> I think it's a succinct word that does the job at explaining what the goal here is.

Sure. If the goal is intelligence then LLMs fail. LLMs do not currently have the same intelligence as humans.

If a human being in front of me were to answer my question like an LLM does, I would think they are an overly confident parrot.

Not saying LLMs are bad, they are an incredible tool. Just not intelligence. Words matter.

Who said anything about matching human intelligence though? If that's the reference point, then nothing in the field of AI has ever had the 'right' to be called that. Computer vision, ML-based optimization of anything or ranking/recommendation systems are all considered to be within AI, despite none of them being remotely similar to 'human intelligence'.

This is the main point of my post - I feel like people retroactively try to see AI as being some kind of an endorsement term, or having to do anything regarding humans - or that 'intelligence' is in itself an endorsement and something so extremely good that only humans can be bestowed with it. In reality, these comparisons only appeared after the boom of generative AI and would've been seen as ludicrous by any AI researchers prior to it.

I think it's more that as the term is widely adopted via something like LLMs they convey different meaning to users of the tools branded by it. Since users and their perspective of "artificial intelligence" and its meaning have no relation to the original term from 50-70 years ago.

Yes, I agree that the rise of gAI has had a major impact on this, but the fact that the meaning of AI in CS and AI in sci-fi are different was also very apparent before all this happened. Marketing and finance people have been trying to associate the hard-math, grounded-in-reality AI with "it's magical, it's smart like a human!" many times before, this time they just were very successful at it.

Yes indeed, I think you are completely right on that

I don't mean to sound corny. LLMs just don't really use or apply information in a way that I think should be considered intelligent. It just repeats its training data. I don't just repeat my training data (even if it was an influence on me)

The idea that LLMs just repeat their training data is just wrong. It’s easy to test them and prove this is not the case. In some situations they may do that, typically when they don’t have much data on some topic. But in many other cases, it’s easy to verify that they are able to synthesize new output that is not simply a repetition of their training data.

Software development is a great example, which also illustrates the ability of LLMs to reason (whether you want to call it e.g. “simulating reasoning” doesn’t matter - the results are what counts.) They can design new programs, write new code, debug code they’ve never seen before, and explain code they’ve never seen before. None of that would be possible if they were simply repeating their training data.

If you've implemented a sampler before, the "repeating the training data" is technically the logits array that you do the sampling on. Good samplers and sometimes even the most basic samplers can produce acceptable output but in the end the output is still technically just a bunch of training data predictions averaged together... or something roughly like that. The fact that I don't consider them intelligent doesn't mean I don't find them useful, I just prefer to use them in a way that acknowledges their shortcomings.

First, to be clear, I'm not arguing that you should consider LLMs intelligent. I was responding more narrowly to the claim that an LLM "just repeats its training data."

On a trivial level, it's obviously true that every token in an LLM's output must have existed in the training data. But that's as far as your observation goes.

The point is that LLMs can produce novel sequences of tokens that exhibit the functional equivalent of "understanding" of the input and the expected output. Further, however they achieve it, functionally their output compares well to output that has been produced by a reasoning process.

None of this would be expected if they simply repeated "a bunch of training data predictions averaged together... or something roughly like that." For example, if that were all that was happening, you couldn't reasonably expect them to respond to a prompt with decent, working new code that's fit for purpose. They would produce code that looks plausible, but that doesn't compile, or run, or do what was intended.

One reason your model of the process fails to capture what's happening is because it's not taking into account the effects of latent space embeddings, and the resulting relationships between token representations. This is a major part of what enables an LLM to generalize and produce "correct" output, taking meaning into account, beyond simply repeating its training data.

As for intelligence - again the question comes down to functional equivalence. If we use traditional ways of measuring intelligence, like IQ tests, then LLMs beat the average human. Of course, that's not as significant as might naively be imagined, but it hints at the problem: how do you define intelligence, and on what basis are you claiming an LLM doesn't have it? Ultimately, it's a question of definitions, and I suspect it'd actually be quite difficult to give a rigorous (non-handwavy) definition of intelligence that an LLM can't satisfy. This may partly be an indictment of our understanding of intelligence.

> how do you define intelligence, and on what basis are you claiming an LLM doesn't have it? Ultimately, it's a question of definitions, and I suspect it'd actually be quite difficult to give a rigorous (non-handwavy) definition of intelligence that an LLM can't satisfy. This may partly be an indictment of our understanding of intelligence.

In my opinion there's nothing wrong with the traditional definition which is "the ability to acquire and apply knowledge and skills". But if you want to reach a minimum of "hand-waviness" then it's additionally required to define 'acquire', 'apply', and 'skills'. My personal definition is that acquiring knowledge requires building some sort of internal semantic model of it, though the occurrence of which there is actually evidence of in LLMs (see "abliteration"), so one out of three so far. But it falls apart at 'apply'. How do we even define applying? Well, I do not define it as what LLMs do, which is to predict the next token of the data.

I, personally, apply my knowledge by recognizing where it may be applicable, bringing it to thought, and then using that in the construction of ideas or strategies that I can act on. There's a degree of separation here between thought and action that doesn't currently seem to exist in LLMs; some creators are trying to simulate it by having an LLM for thoughts and another LLM for actions, or by enabling the thoughts to call tools that perform actions, or by having the LLM think before acting as in DeepSeek R1, but that isn't quite it.

An LLM still doesn't understand, say, spatial reasoning when it is helping me write something like a battle in a story. I have spatial reasoning because I can literally see what is happening while I write. I can see, and feel, and hear, and everything. Maybe that's just my dissociative disorder, but I will continue to await the day where LLMs might be able to do stuff like that. Until they can have that essentially happening "in their head", reason about it, and write using that, I won't believe that LLMs can "apply" much of anything just yet. (Other than machine learning I guess.)

> If we use traditional ways of measuring intelligence, like IQ tests, then LLMs beat the average human.

I think the whole notion of IQ is flawed because of neurodivergence. To put things in vaguely ableist-sounding terms (I don't mean it that way, but it will always sound that way), LLMs right now feel too neurotypical (pattern-based) and I would like to see future LLM developments that allow models leaning closer to autistic (logic-based).

The reason why I said it sounded that way to me is that the 'intelligence' part in AI was never meant to imply some kind of superhuman ability. I highlighted the age of the field to point out precisely that judging any AI systems as humans would've been seen as outlandish and laughable for most of the field's existence. If computer vision is AI, then surely text generators can also be. It doesn't have to be a glamorous term, it's just the technical field.

Besides, if LLMs only recycled training data with no changes, they'd just be really bad search engines. Generative AI was created initially to improve training, not for human consumption - the fact that it did improve training shows that the result is greater than the sum of its parts. And since nowadays they're good enough to pass for conversation, you can even observe that on your own by asking a question that doesn't appear anywhere on the training dataset - if there's enough coverage on that topic otherwise, I've seen them give very reasonable answers.

> Besides, if LLMs only recycled training data with no changes, they'd just be really bad search engines.

I will admit that in my experience, LLMs tend to be really good at tip-of-my-tongue type stuff. There are certain very particular types of queries that LLMs seem to greatly excel at, and they're mostly where the words I want aren't in the words that I am using. I can just spam vibes into the prompt and have an LLM give me words/phrases that exactly fit what I am looking for, even if I couldn't recall any of the proper terms that would allow good results to turn up from a search engine.