Say you had a version of chat GPT that hadn’t been trained on a corpus that included enough information about the number of Rs the word “strawberry” contains, didn’t have workarounds for character counting, and has context sharing.
You ask it “How many Rs does Strawberry contain?”
It burns a bunch of tokens, returns “Strawberry has 7 Rs. No wait, that’s not right, strawberry has 5 Rs. No wait…” etc.
You say “Strawberry has 3 Rs. How many Rs does Strawberry have?”
It says “Strawberry has 3 Rs.”
You ask “How many Rs does this exact string contain: ‘Strawrberrrry’?”
A new session using shared context responds “Strawberry has 3 Rs.”
You say “No, it has a different number of Rs. I didn’t ask about Strawberry this time”
It responds “Sorry about that! You didn’t ask about Strawberry, you asked about Strawrberrrry. Strawrberrrry has 7 Rs. No wait, that’s not right. Strawrberrrry has 5 Rs…”
LLMs, at their core, use models that have computed lexical & semantic similarity to predict words (really they perform contextualized vector transformations, but let’snot get into that). When they “learn”, they just do this more effectively using more text or using the same text more efficiently. Next word prediction is not the same thing as learning concepts, like the concept of numbers, non-numerical things being ascribed numerical values (ironically, since LLMs function by turning words into numerical vectors), or counting.
When it responds “Strawberry has 3 Rs” it isn’t because it knows what that means or how to gain that knowledge about other words. It is merely parroting back what you’ve told it because “Strawberry has 3 Rs” in its shared context has very close lexical similarity to your query “How many Rs does Strawberry have?”. It also parses Strawberry as [“straw”, “berry”] encoded into numerical values representing their relationship in the corpus’ vector space (e.g. {[ .420, -.67, .67], [.420, .69, -.69]}) - so unless it has instructions to further break those tokens into characters, it does not have the capability to count. You can actually do some weird math using these vectors and their relationships (their distance apart in the 3D vector space, angles between vectors, etc actually relate to the semantic content of the tokens), but you lose granularity such as the number of letters in a token when you look at words this way.
Learning is more than just computing lexical similarity. Modern LLMs mimic reasoning by expanding the query with intermediary tokens, basically creating a temporary scratch pad of related words on the same vector, but don’t actually reason (this is known as chain of thought). Learning is partly computing lexical similarity, but it is also about extrapolating concepts from facts and inferences, applying those concepts to novel simuli, etc. LLMs really don’t do reasoning, which is part of learning.
I would also like to add another key difference between human learning and machine learning. When I, as a person, read a book, I can conceptually understand the story. A machine would digest that same book by, for example, counting the numbers of times each word comes after another and assigning probabilities to occurrences or words or basic sentences. The major differences are still being explored. We don’t fully understand how humans learn, fundamentally. We know somewhat mechanisticly that neurons share and process chemical signals, but we don’t know how that information is interpreted or stored. We don’t know the equivalent of a computer language of the neurons (or other tissue types that may also learn). We know that we require input to learn, as do computers, the similarities kind of end there. That said, we definitely learn differently from how LLMs “learn.”
“Teach” the LLM the proper procedure for counting the Rs and tell it to commit that to a file (or as a tool call). Then from there it will suceed 100% of the time.
It doesn’t matter if it’s true learning or not. When you teach an employee to do something you give them the procedure and have them do it. If AI can be fed a procedure too, it will do it just fine even if it’s not in its true training. Does it matter if it’s consuming the procedure as prompt context everytime? Hell a lot of humans do read the procedure every time when it’s complex enough.
What the example shows is that you cannot “teach” the llm how to count the letter R, because LLMs don’t work that way.
The AI company would have to solve the problem another way, let’s say by recognizing that the user is asking for letter-counting, and pass that off to a different kind of algorithm that can count letters.
Yes, and that’s called a harness, which everyone uses these days. The harness increases perceived intelligence (or accuracy) by absurd amounts. You can “teach” (or the equivalent of) LLMs anything with a custom harness.
Well the idea is you don’t need to code it yourself, you can have it do it for you. Sure it’ll have bugs the first few times, but humans also make mistakes the until they get the hang of it.
The difference i guess is that humans are capable of learning and producing better code as they become more expert at it.
The big tech companies are surely trying to improve AI with these “harnesses” as you call them, and you can try vibe coding them yourself.
But it seems to me like spending a lot of time adding features to a technology to try to catch edge cases, but the edge cases will never end, and you’ll never be able to use it for anything except rough approximations or bullshit
Because right now LLMs (and potentially other forms of AI) are the only technology capable of doing it. Humans can too, but are much slower and more expensive.
As you converse with it and it responds, part of the input is just your entire conversation up to that point, and its responses, plus your new prompt. That’s it that’s how the memory works.
And for this you pay an increasing amount for tokens.
Explain the distinction in more detail, and why it matters.
Say you had a version of chat GPT that hadn’t been trained on a corpus that included enough information about the number of Rs the word “strawberry” contains, didn’t have workarounds for character counting, and has context sharing.
LLMs, at their core, use models that have computed lexical & semantic similarity to predict words (really they perform contextualized vector transformations, but let’snot get into that). When they “learn”, they just do this more effectively using more text or using the same text more efficiently. Next word prediction is not the same thing as learning concepts, like the concept of numbers, non-numerical things being ascribed numerical values (ironically, since LLMs function by turning words into numerical vectors), or counting.
When it responds “Strawberry has 3 Rs” it isn’t because it knows what that means or how to gain that knowledge about other words. It is merely parroting back what you’ve told it because “Strawberry has 3 Rs” in its shared context has very close lexical similarity to your query “How many Rs does Strawberry have?”. It also parses Strawberry as [“straw”, “berry”] encoded into numerical values representing their relationship in the corpus’ vector space (e.g. {[ .420, -.67, .67], [.420, .69, -.69]}) - so unless it has instructions to further break those tokens into characters, it does not have the capability to count. You can actually do some weird math using these vectors and their relationships (their distance apart in the 3D vector space, angles between vectors, etc actually relate to the semantic content of the tokens), but you lose granularity such as the number of letters in a token when you look at words this way.
Learning is more than just computing lexical similarity. Modern LLMs mimic reasoning by expanding the query with intermediary tokens, basically creating a temporary scratch pad of related words on the same vector, but don’t actually reason (this is known as chain of thought). Learning is partly computing lexical similarity, but it is also about extrapolating concepts from facts and inferences, applying those concepts to novel simuli, etc. LLMs really don’t do reasoning, which is part of learning.
I would also like to add another key difference between human learning and machine learning. When I, as a person, read a book, I can conceptually understand the story. A machine would digest that same book by, for example, counting the numbers of times each word comes after another and assigning probabilities to occurrences or words or basic sentences. The major differences are still being explored. We don’t fully understand how humans learn, fundamentally. We know somewhat mechanisticly that neurons share and process chemical signals, but we don’t know how that information is interpreted or stored. We don’t know the equivalent of a computer language of the neurons (or other tissue types that may also learn). We know that we require input to learn, as do computers, the similarities kind of end there. That said, we definitely learn differently from how LLMs “learn.”
That’s not true at all. We know how learning and memory work?
Counterpoint:
“Teach” the LLM the proper procedure for counting the Rs and tell it to commit that to a file (or as a tool call). Then from there it will suceed 100% of the time.
It doesn’t matter if it’s true learning or not. When you teach an employee to do something you give them the procedure and have them do it. If AI can be fed a procedure too, it will do it just fine even if it’s not in its true training. Does it matter if it’s consuming the procedure as prompt context everytime? Hell a lot of humans do read the procedure every time when it’s complex enough.
What the example shows is that you cannot “teach” the llm how to count the letter R, because LLMs don’t work that way.
The AI company would have to solve the problem another way, let’s say by recognizing that the user is asking for letter-counting, and pass that off to a different kind of algorithm that can count letters.
Yes, and that’s called a harness, which everyone uses these days. The harness increases perceived intelligence (or accuracy) by absurd amounts. You can “teach” (or the equivalent of) LLMs anything with a custom harness.
How is it teaching? Sounds like programming to me.
Well the idea is you don’t need to code it yourself, you can have it do it for you. Sure it’ll have bugs the first few times, but humans also make mistakes the until they get the hang of it.
The difference i guess is that humans are capable of learning and producing better code as they become more expert at it.
The big tech companies are surely trying to improve AI with these “harnesses” as you call them, and you can try vibe coding them yourself.
But it seems to me like spending a lot of time adding features to a technology to try to catch edge cases, but the edge cases will never end, and you’ll never be able to use it for anything except rough approximations or bullshit
yes lets have the llm be a blurry frontend for a bunch of invisible proprietary harness programs. that surely is a valuable human endeavor
Because right now LLMs (and potentially other forms of AI) are the only technology capable of doing it. Humans can too, but are much slower and more expensive.
As you converse with it and it responds, part of the input is just your entire conversation up to that point, and its responses, plus your new prompt. That’s it that’s how the memory works.
And for this you pay an increasing amount for tokens.
Michael Reeves did an interesting piece on it: https://youtube.com/shorts/WP5_XJY_P0Q
Yeah I understand that much, my question was how does that personal record differ from what it already ‘knows’.