Simple tasks showing reasoning breakdown in state-of-the-art LLMs

Given many people don’t have an inner monologue and function just fine, it’s more likely inner monologue is a product of the reasoning process and not it’s mechanism.

but we really want it to be internal and automatic.

Why? We have specialized semi-isolated lobes of our brain. Some "external to the LLM weights" guiding system just needs to be transparent to us. It doesn't need to be internal to the LLM/weights.

because what controls the internal thought process is itself not necessarily language-based

Most of the interesting AI things can't write a sentence (weather simulation, protein folding, inverse kinematics, driving cars, etc...). Language isn't a requirement, only some sort of meaningful mapping to the latent space of each is. I think one could claim that "meaningful latent space mapping" is the language of neural nets, including those in the brain.

For reasoning of any complexity, some sort of internal monologue and iteration is needed.

From a formal perspective you're entirely correct. Transformers with chain-of-thought are strictly more powerful than transformers without it, and can efficiently solve classes of problems that would otherwise require exponentially increasing model depth: https://arxiv.org/abs/2310.07923

Yeah, I don't know people ever thought that the current state of LLMs is enough to actually solve problems; it's trained to be very good at predicting the next word, but any logic solving capability is only emergent due to questions + answers in the training data, when it has to improv then it either makes a randomly correct or similar enough guess, or it falls over.

If these things were trained to actually solve logic problems then it may be different, the training data would have to be logic problems. They've started at the top (language) rather than at the bottom/middle (logic, thought, reasoning) which should be done first - and then the model can be trained/additional models to get it to put thoughts into words.

If you really think about what an LLM is you would think there is no way that leads to general purpose AI

It is an autoregressive sequence predictor/generator. Explain to me how humans are fundamentally different

LLMs are the language center of an AI. They in no way purport to be: long term memory, critical thinking engines, calculators.

If we want those things we can build them. Building them into the language center would be absurd and weird.

I honestly thought about this recently when I was trying to see the limits of Claude Opus. Some of the problems I gave it, what if instead of telling it to solve the problem I asked it to write the script and then give me the command and inputs needed to properly run it to get the answer I needed. That way instead of relying on the LLM to do properly analysis of the numbers it just needs to understand enough to write the logic.

It is an interesting prospect but I feel like it has some limitations. For math problems like this one, yeah it should be simple to write a script to do it. But it does first have to understand the core thing here that Alice would be one of the sisters of the brother to write the script accordingly.

But I would think this would not scale well when dealing with far more complex issues, particularly ones that may not just be simple math logic. If the request was to write a scientific paper for example, it might have to make several scripts and call them multiple times, and that assumes that it understands what it is doing enough to properly make the script.

Right, and do you verify the result?

You have to know what the answer is supposed to be before you can write a test case.

Given that Alice has 13 brothers and 31 sisters, we can update the Prolog program with this information. We need to adjust the fact about Alice's siblings and then use the rule to calculate the number of sisters her brothers have.

Here is the updated program:

```prolog % Facts siblings(alice, 13, 31). % Alice has 13 brothers and 31 sisters

% Rules % Alice's brothers have M + 1 sisters, since Alice is also their sister. sisters_of_brother(TotalSisters) :- siblings(alice, _, M), TotalSisters is M + 1.

% Initialization goal :- initialization(main).

% Predicate to find and print the total number of sisters Alice's brothers have main :- sisters_of_brother(TotalSisters), writeln(TotalSisters). ```

In this program, the fact `siblings(alice, 13, 31)` represents that Alice has 13 brothers and 31 sisters. The rule `sisters_of_brother(TotalSisters)` calculates the total number of sisters that Alice's brothers have by adding 1 to the number of sisters Alice has (since Alice is also their sister).

When we run this program, it will print the total number of sisters Alice's brothers have:

1. Alice has 31 sisters. 2. Each of Alice's brothers will also consider Alice as a sister. 3. Therefore, each brother has 31 + 1 = 32 sisters.

The output of the program will be:

``` 32 ```

Thus, Alice's brothers have 32 sisters.

If that kind of thing worked, we'd have been doing it long before LLM chatbots.

I’m curious how this would work considering knowledge can be fuzzy.

Like if I’m out camping and I sit on a log or a rock those things are not what people usually think of as chairs but they can serve as chairs in that situation.

I just tried that with ChatGPT 3.5 (4o stopped responding after I asked the initial question and it produced the wrong answer). Here's the Prolog it generated:

    % Define the number of brothers and sisters
    brothers(4).
    sisters(1).

    % Predicate to calculate the number of sisters a brother has
    brother_sisters(NumberOfSisters) :-
        sisters(NumberOfSisters).

    % Query to find out how many sisters a brother has
    ?- brother_sisters(Sisters).

asking it to execute this type of program in order to come up with a solution

I may be showing my ignorance about this tech here, but I believe the LLM doesn't even try to solve a problem; they try to generate a discourse that could pass as a solution or answer to the problem; that's more or less what the abstract states if I understand it correctly. But in no way does it try to apply some sort of mechanical reasoning like inference engines do.

To me the solution to this is to associate LLM with mechanical computations, that is an inference engine or an equation solver, rather than recombining the millions of solutions for similar problems it has seen in its training set. I believe I remember reading about teams attempting this approach. I can imagine for instance that if the LLM is in some way able to ask questions and use the answer, maybe it could just generate a prompt for an equation solver and include the result in its answmer.

I took one of the problems that gpt4o got wrong, and asked gpt4o what tools it could use. It suggested and wrote prolog for me that (with one pass back to get things defined in the right order) which correctly worked out the answer.

Your post is complete hype, all about people saying things instead of showing things that've actually been done.

For me, 2024 was the LLM exposed as basically pure hype year.

There is no expert of any field I follow online where they're posting up results from AI tooling for any other reason than to show how awful it is. I consider myself an expert in software, and LLMs specifically have only caused me great pain.

Even the one situation where you describe someone describing the ability to work in an absolute vacuum sounds like a huge negative to me. The recent push for DEI policies were even ostensibly about the importance of people of diverse backgrounds and viewpoints working together.

The most important thing you're missing a perspective of scale on is the step you describe as "quality check it". On things I don't know, and have attempted to enlist an LLMs help on, in every case I have had to go back and just actually learn how something works, after time wasted struggling with subtle wrongness in the output.

At least I have the background expertise to do that, however, I have seen a Jr dev's mind get literally rotted by too much time in pure LLM land. Besides the cost of rewriting their code, the company was now the proud owner of a young dev with a mind filled with nonsense.

How do you even weigh the cost of fixing a corrupted human mind?

AI doesn't need to be ready tomorrow, but its capabilities are already really good.

A majority don’t deny that it’s good. The problem is that so many think it is actually reasoning, believing the answers can be trusted.

We are also don't know what ChatGPT 5 will bring, because they say this will do more reasoning than before...

This paper very clearly demonstrates these LLMs are not reasoning in a fundamental way. Token prediction and reasoning are two different tasks. They may be related, but they are not the same. "Just wait for GPT 5, it will be amazing!" is part of the hype.

Please do not assume an LLM is correct in skill or knowledge unless you already know the answer or can verify by other means.

There is no feedback. You cannot create new knowledge out of thin air.

Seriously? You think individuals are incapable of reasoning without training first?

In "any sense" of the word? Surely anyone who adjusts their behavior when they get undesired or unexpected results is reasoning and thinking. And since most activities are mediated by thought of some kind, most people are reasoning and thinking otherwise they would never recover from even simple mistakes, like walking east when they need to go north.

Saying they're "not thinking in any sense of the word" because they can't solve predicate logic problems from a college textbook is a rather odd claim. Surely those things arise from reasoning and thinking, rather than the other way around.

Logical reasoning is a higher order skill that often requires formal training. It's not a natural ability for human beings.

I've read your comments here, and while I understand your point I think you have it backwards. The only reason we formed societies is because we evolved an innate a theory of mind to reason about how others might be thinking and feeling. That's reasoning. We have a natural ability to do limited arithmetic, otherwise we wouldn't be able to hunt, gather or track enough to endure winters, or keep track of our sheep or children for that matter. That's reasoning.

Reasoning is a natural ability for human beings, but we also carry a lot of evolutionary impulses that add a lot of noise to the decision process, eg. observation->judgment->[set of possible decisions], judgment has "reason" as one path that adds to the set of possible decisions, but there remain other paths we inherited from our evolutionary roots. Education is training that suppresses poorly calibrated judgment paths that lead to frequent mistakes in the decision set, but reasoning remains, thus education improves the signal to noise ratio of our decision making.

So I 100% disagree that an individual cannot separate cause and effect without training. They will just be worse at it than someone who is trained to filter out those impulses that lead us to jump to conclusions, eg. they will produce more noise / a larger set of possibilities than reason would allow.

Well, I believe LLMs are general purpose AI. And, for the record, so does Peter Norvig. He coauthored the most popular textbook on Artificial Intelligence, so this belief can be considered a somewhat mainstream perspective. His article, “Artificial General Intelligence Is Already Here“ [1] is really worth reading.

https://www.noemamag.com/artificial-general-intelligence-is-...

I don't think anybody who actually uses LLMs thinks they are general purpose AI.

I would like to believe that but I have had too many conversations with people who basically think it already is. Including in one situation of a fellow engineer.

It feels like more and more "we" are in a bubble of actually having some knowledge of how this works, what the actual limitations are, and what it just is not. While there is in fact criticism of it out there, particularly around AI "art". It doesn't seem to be focused on the area we are talking about.

The vast majority of people use a model built by someone else through a high-level abstraction and then make broad claims about what the technology will be capable of.

Then you have the people who are leveraging the technology to train models from scratch and that population is far more apt to believe that large models can be general purpose AI (by some definition). Sure there are other things at play like money, publicity, reputation, and a desire to do something important. But there is also the very clear trend line where transformers are able to model pretty much any sequence of tokens where there is pretraining scale data available. Whisper was not a surprise. GPT-4v was not a surprise. Sora was not a surprise.

They might not think that but they sure as hell are trying to sell that idea.

they're pretty general-purpose; you can ask the same model for recipe suggestions, a fanfic, or verilog. like, they're far, far more general-purpose than any humans i know

it's true that they're not very reliable, but they seem to be not very reliable across many different domains. and they don't seem to be particularly less reliable than the average human, so i think possibly your standards for 'general purpose ai' are set high enough that you would declare humans to be unintelligent (or perhaps not 'general-purpose') if you applied them consistently

you can certainly find particular domains where humans can still do things llms can't, but i haven't seen a persuasive account of why those domains are the more important ones, and of course the converse is also true

I don't think anybody who actually uses LLMs thinks they are general purpose AI.

They are being sold as such. Most people don't know anything about the topic and will buy that marketing. The entire concept of these models is that you can put a whole bunch of data in and eventually some kind of magic will happen and you get AGI out. They would not see the kind of investment that they do if all that was being promised was "really good predictive text". In fact some philosophers argue that sentience is just really good predictive text to try and make the point that these models are AGI.

Yeah, I think these chatbots are just too sure of themselves. They only really do "system 1 thinking" and only do "system 2 thinking" if you prompt them to. If I ask gpt-4o the riddle in this paper and tell it to assume its reasoning contains possible logical inconsistencies and to come up with reasons why that might be then it does correctly identify the problems with its initial answer and arrives at the correct one.

Here is my prompt:

I have a riddle for you. Please reason about possible assumptions you can make, and paths to find the answer to the question first. Remember this is a riddle so explore lateral thinking possibilities. Then run through some examples using concrete values. And only after doing that attempt to answer the question by reasoning step by step.

The riddle is "Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?"

After you answer the riddle please review your answer assuming that you have made a logical inconsistency in each step and explain what that inconsistency is. Even if you think there is none do your best to confabulate a reason why it could be logically inconsistent.

Finally after you have done this re-examine your answer in light of these possible inconsistencies and give what you could consider a second best answer.

In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work.

The last part of that is the problem and why a paper like this is critical.

These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc.

People are being told it can do far more than it really is.

Such is the strength of corporate tech propaganda that a whole mass of people will instead insist that we have never worn clothes either.

This is the line of reasoning I find most dispiriting. I still believe tech people cling to this line of reasoning because it helps them justify replacing people in jobs with LLMs.

LLMs are idiots. They can't reason properly and only parrot stuff

https://chatgpt.com/share/dcb4ff4e-e8a2-463b-86ec-9caf10b6e6...

Sometimes they get the answer right to something really complex because it fits a pattern, but sometimes they answer with something really really stupid.

Even if this were all true, it points to a fundamental risk of using LLM's for important tasks, which is that it is not at all clear to a user that this prompt would cause a problem. The LLM doesn't say "I'm sorry Dave, I just can't do that", it just complies with it and gets the wrong answer.

You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.

New option needed besides concise. Think quietly on your own, then answer concisely.

Sometimes I think I'd prefer it to "think" before answering anyhow. The immediate thinking out loud text can be irritating for some irrational reason.

I often want chatgpt to answer concisely and tell it that.

If it really needs to do this 'thinking out loud', could it do that under the hood and not in the final output on my screen? Its first pass could use as many words as it wants to compute the answer, but once the answer is computed please go back and make it short.

Not to take away from your point that maybe the prompt is the problem in these reasoning questions.

I think you're wrong about that — I just tried prompting ChatGPT 4o to show all its working before giving an answer.

It was still incorrect, but when asked to show its working it formatted the answer prettily.

Yes this is a common thing I see people who think LLMs are idiots do.

If you're among technologists discussing LLMs academically, as we are, that's a reasonable approach. However, I see a lot of people fail to distinguish that from LLM-powerd products sold to the general public as intelligent bots that can understand your plain english and output answers.

People use their existing mental models when interacting with something. If you have 3 different interfaces with a widget to trigger the same exact function, but one look like a music play button, one looks like a gas pedal, and one looks like mechanical pinball plunger, we interact with those things differently because we know how those things work. In this context, chatbots are designed to engage people's existing mental model for chatting with a person via text. The further you stray from people's expectations of human chat, the further you are from people's expectations, for better, or worse.

If you're selling someone a product claiming it understands plain language questions and gives plain language answers, then not getting the right answer to that question makes it idiotic. The subtleties aren't within most users' grasp, and the "FYI: this thing might be full of shit" disclaimer isn't helpful if you don't know enough about what you're asking to administer a proper smell test.

Your statements are obviously not wrong, but I see people saying these things like its reasonable for non-technical end users to reason about those subtleties. Considering how those things are marketed, I really don't think it is.

_because that's the only way it can compute anything_

I'm fairly certain we'll soon realize that what's happening here is that the markov chain being run over latent space needs a certain amount of "warmup" before it starts sampling from the optimal region. HMC samplers for Bayesian methods have this same property.

The terms "reasoning", "computing" or "thinking" for this stage should be considered metaphors rather than explanations for what's happening, which is really waiting for a random walk to start sampling from the typical-set.

As it says in the paper, they already did the research with less limiting prompts. All models were tested with another prompt that gave only the question and no further instruction, and a third prompt that asked the model to consider its answer carefully.

The correct response rate chart doesn't even use the results from the concise prompt.

The more an LLM talks the smarter it gets

I have a blog post in coming on this topic, but yes, this is right.

My method is to first get the LLM to answer the question, and THEN feed the answer back the LLM to extract the answer using constraints + grammar/logit bias/regex to parse the answer. Previously, I constrained to a single true/false token, which worked, but fails on complex queries.

So I split the decision making into a "justification" portion[0], and a "parsing" portion. I found that even crafting the prompt matters here, if you start with or end with, "It's very important to the response includes 'The answer is:'", then the model will lead with that response or only reply with that response. So I put it in the middle of the prompt, and end with with a request to justify the response. As a result, most models will reason their way to the answer, and then end with 'The answer is:'.

https://github.com/ShelbyJenkins/llm_client/blob/e3c4a860dda...

Many people, especially on this site, really want LLMs to be everything the hype train says and more. Some have literally staked their future on it so they get defensive when people bring up that maybe LLMs aren’t a replacement for human cognition.

The number of times I’ve heard “but did you try model X” or “humans hallucinate too” or “but LLMs don’t get sleep or get sick” is hilarious.

Well, why does it need a name? It just seems like a "correct argument". If somebody is claiming that AI is "not all that" because it can make stupid mistakes, surely it's relevant that humans also make stupid mistakes, so AI making them does not necessarily pose a hindrance to it becoming a human-tier reasoner.

Its not just that the LLM fails, its the kind of failures it makes that illustrate how far what the LLM does is from human thinking. Like the problem someone posted where you have a boat and a goat and you want to cross the river, and the LLM's solution is cross the river with the goat, come back with an empty boat, and then cross again with an empty boat. A human might miss the insight that Alice's brother has one more sister than Alice, namely Alice herself. A human would never think the number of sisters her brother has depends on the number of brothers she has. Not if it has any clue at all what "sister" and "brother" mean.

There's also the phenomenon of papers showing ChatGPT getting it wrong, then people posting anecdotal examples of it getting it right when they try it, but that one already has a couple of names: WORKSFORME or Unable to Reproduce

Hey folks, ever considered the possibility that unreproduceability is not a good thing?

You could call it the "Compose a symphony fallacy", after the scene in I, Robot (2004) where Spooner asks Sonny if he can compose a symphony, to which Sonny replies "Can you?"

To be quite honest, I assume they made the restriction so that the models would fail.

This sort of paper is becoming a genre.

When I added a " to the end of the prompt by accident I got a wrong answer.

From the paper:

To account for the response variations due to various prompt forms, we created 3 distinct prompt types asking for the solution to the AIW problem: STANDARD, THINKING, and RESTRICTED. The STANDARD prompt type asks to solve the posed problem and output the final answer in the format as described above. This does not put any specific requirements on model behavior. The THINKING prompt type extends STANDARD with the request to think carefully and double check the solution for any mistakes

Alice has N Brothers, and she has M sisters. How many sisters do Alice’s brothers have?

I have not gotten the correct answer to the question as phrased above in one go from Gpt4o yet! (and today was not the first day i tried.)

Phrase it as shown above and you'll likely need 5 or more interactions to get it to generate the correct output. With Gemini i could not get it below 8 without feeling like i was cheating. fwiw.

The "only output the number" prompt was one of three they used. The others were one with no instructions beyond the question, and one that explicitly told the model to show its reasoning and check for errors.

We all are, but only in meat-space. We continue to improve ourselves much faster than evolution ever could.

But what we are talking about here is the singularity in tech-space.

Do you sleep, get sick, require more than compute hardware/storage/networking/electricity?

Page 42 of the paper :)

One thing that strikes me is that the model first tries using "inclusive language" in one answer - and literally states so, using this specific term - but seems to interpret it in a more mathematical sense (like set inclusion). Then seamlessly switches to the expected DEI spiel in the next paragraph.

For one thing, it makes me suspect that something with the words "inclusive language" was automatically added to the prompt. But more interesting is how it responds to this demand in two different ways, illustrating a "thought process" that is very much unlike that of a human with normal verbal reasoning ability.

I am not a psychologist, but remember reading that schizophrenic people sometimes confuse different meanings of words in a similar way, jumping from one meaning to another without noticing.

> Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?"

The right answer depends on how Alice identifies I guess? :)

Given that the wording of the question specifically identifies Alice as "she", rather than using a gender-neutral pronoun or no pronoun at all, I think inferring that she identifies as female is reasonable.

> My initial response was based on the assumption that all prompts used that script prompt quoted in the parent.

You, and another 20 or so commenters here. We should really re-examine the guideline about asking people to RTFA.

No offense meant- good on you for correcting your error.

Thinking out loud also only gets you so far, if the expectation is a certain type of response it can't always "think out loud". In reality that just proves it isn't really reasoning here and is more likely just self referencing.

Try to calculate it without writing anything down, or thinking any numbers or words in your head.

You can't draw a 1:1 analogue between an AI and the human experience, but remember that we have an internal stream of consciousness. Maybe the outputs of the LLM are more similar to the stream of consciousness in our heads rather than the words we say? After all, Humans also do lots of self referencing.

That being said, I tried this personally allowing it to think out loud and it told me she has 212 sisters. Using your exact prompt.

Fair enough, but worst case it can often solve it correctly with the correct reasoning. GPT3.5 can't solve it correctly with correct reasoning, so we are at least appearing to be on a path where AI's can start to solve this question, albeit potentially not fully reliably.

ChatGPT 4o seems to occasionally get this correct, but still thinks it's a puzzle and will often get this wrong...

https://chatgpt.com/share/b8901f4c-7313-4461-acad-9e561c1e6e...

Still seems like it fares slightly better than some of the other models.

My experience with GPT-4o with the same prompt was pretty solid. With no special system prompt and your same user prompt, I got:

This scenario is intriguing and does seem to carry the potential for a tricky question. In the simplest terms, the presence of a goat doesn't complicate the actual act of crossing the river. If there are no additional constraints or rules involved (like the need to also transport something else that requires careful consideration), you can simply:

1. Place the goat in the boat. 2. Row across the river.

However, if we are to assume there's more depth or a twist to the question, we might consider an outside-the-box answer. Here’s an alternative possibility if there is indeed some trick:

1. Get into the boat. 2. Leave the goat on the shore (if it’s safe) and row across the river.

There might be other interpretations depending on additional constraints that could be implicitly part of the question. If the goal was simply to get yourself across, leaving behind the goat and traversing the river is the straightforward solution.

There are always "bean counters" added somewhere in the process. There are many places where the person lists their number of publications and that is all most people will ever see.

I guess this is the reason:

> BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions.

Regardless, I'm not citing a paper with a whole page of author names unless I'm allowed to shorten it significantly in the bibliography section (e.g. "Srivastava and 450 others").

The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt.

It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data.

For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend).

In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.

an entire paper written on one prompt - is this the state of research these days

Years ago I attended an AI meetup where the organizer would discuss the differences between reductionist and wholistic research.

This is reductionist research.

In this case, reducing the paper to one prompt is to allow for objective comparison of models.

Otherwise, without a reductionist approach, how does one perform such an experiment and ensure that they are comparing apples to apples?

First result:

Me > Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have?

GPT4o (web version) > Alice has 50 sisters. Since all of Alice's brothers and sisters share the same set of siblings, each of Alice's 40 brothers also has 50 sisters. Therefore, Alice's brother has 50 sisters.

---

Step by step:

Me > Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have? Think about it step by step.

GPT4o > To determine how many sisters Alice's brother has, follow these steps:

    Identify the total number of siblings: Alice has 40 brothers and 50 sisters, plus Alice herself. Thus, the total number of siblings is:
    40 (brothers)+50 (sisters)+1 (Alice)=91 siblings
    40 (brothers)+50 (sisters)+1 (Alice)=91 siblings

    Focus on Alice's brother: Alice's brother is one of the 40 brothers.

    Determine the number of sisters for Alice's brother: Each brother has the same number of sisters. Since the brothers do not count themselves as sisters, they only count the 50 sisters, excluding themselves and other brothers.

So, Alice's brother has 50 sisters.

---

Thinking step by step somehow gave an even more nonsensical answer, I'm actually very surprised it didn't work when prompted to do it step by step.

I remember when ChatGPT 4 posted a set of standardized trst benchmarks, including AMC 12 math test, where ChatGPT scored higher than guessing randomly, but lower than leaving the test blank (due to the penalty for wrong answers).

I wonder the same thing. If any academic reading this wants a paper idea:

1. Examine papers and other claims that an LLM gets something wrong that a human would have gotten wrong. How many of those claims have any citations about how many humans actually get it wrong? How many of those citations use the general population instead of the population of people who would be uniquely well-suited to answering the question correctly (i.e. people who signed up for the GRE are more likely to get GRE questions right than the general population).

2. For claims that are totally missing citations on human performance, run some tests with humans from the general population (or as close as you can get), and see how the LLMs compare.

The question the paper claims to address isn't whether current LLMs are AGI, the question is whether LLMs can reason. And solving a problem that seems to require reasoning some of the time is still evidence of reasoning - especially if the question is what we can expect this technology to do in the future.

ChatGPT is really great at handling typos.

I understand that, but the article we are discussing points out that LLMs are so good on many tasks, and so good at passing tests, that many people will be tricked into blindly "taking their word for granted" -- even people who should know better: our brain is a lazy machine, and if something works almost always it starts to assume it works always.

I mean, you can ask an LLM to count letters in thousand of words, and pretty much always it will come with the correct answer! So far I don't know of any word other than "банан" that breaks this function.