Large language models

LLMs are powerful because they're different from humans. They're tireless, don't need to eat, don't need to sleep, are much faster than humans in writing and comprehension, but they can't count the number of R's in "strawberry."

People tend to hate AI because they see AI as a threat to their livelihood, and to be fair, this is a real concern. But this problem says less about AI and more about our economic system.

LLMs first encode the input text

A token is a coded representation of a sequence of characters; often, these are partial or whole words, or even symbols.

LLMs take inputs and convert them into tokens, and then further convert them into high-dimensional vectors. The LLM then works on those vectors directly.

User --input text--> Tokenizer --tokens--> Embedding lookup --vectors--> Rest of the LLM

In other words, they forget what the exact input was as soon as they've read it, and they convert that input into these vectors.

This is exactly why LLMs don't know how many R's are in the word "strawberry," and other similar errors. (See Why LLMs can't count letters.)

LLMs represent their ideas in higher-dimensional space

An LLM's output token is a random walk through n-dimensional space, the space of all possible continuations, shaped by training.