Tokens: why the model reads "strawberry" differently from you

Note 01 said the model goes round a loop, one word at a time, and admitted in the small print that "word" was a simplification. This note is the small print. It matters more than it sounds, because it explains a whole family of odd behaviour with one fact.

A model never sees letters. Before your text reaches it, the text is cut into pieces called tokens, and each piece is replaced by a number. The model reads the numbers. "Strawberry" is not ten letters to a model. It is three pieces: st, raw, berry, and then three numbers. The letter r is not in any of them as a letter. It is inside the pieces, the way the letter r is inside the sound of the word when you say it aloud.

That is why, for a couple of years, "how many r's in strawberry?" was a famous way to embarrass a chat tool. The model was being asked to count something it had never been shown. It could only answer from what it had learned about the piece "berry" in training, and "berry" as a piece is more often about fruit than about spelling. Newer models get it right, and the "one layer deeper" section says how, because the reason is not that they can now see letters.

How the pieces are chosen

Nobody sits down and decides that "strawberry" is st + raw + berry. The pieces are found by counting. Take a very large amount of text, start with single characters, and repeatedly merge the pair that appears together most often into a new piece: t and h become th, th and e become the, and so on, tens of thousands of times. Stop when you have a list of around a hundred thousand pieces. That list is the model's vocabulary, fixed before training starts, and it is a mirror of the text it was counted from. Common English words are one piece each. Anything rarer gets chopped.

You can see the mirror in the practical results. The same word in a different case is a different set of pieces. A city name is chopped. A long South Indian name is chopped into seven. A rupee amount with Indian comma grouping becomes six pieces where a plain number would be two or three.

Why this one fact explains so much

Spelling and counting. Anything that needs the letters, such as counting them, reversing a word, or finding words that start with a given letter, is being asked of something that was never shown the letters. It answers from memory of what the pieces usually contain, which is good for common words and poor for rare ones.

Arithmetic. Look at the last row of the drawing. 1234567 is not seven digits to the model; it is 123, 456, 7. Long multiplication done on pieces of three digits that do not line up with place value is hard, and the model was never taught column arithmetic; it learned patterns of what digits tend to follow which. Note 06 is about this.

Cost and room. Every tool that charges for AI charges per token, and every model's working memory (the context window, Section 3) is measured in tokens. So the number of pieces a text becomes is the number that decides what you pay and how much fits. English, the most common language in the training text, is the cheapest. A name like Thiruvananthapuram costs seven pieces where "London" costs one.

Languages. This is the one I most wanted to see. The same sentence in Hindi, in Devanagari script and then in the romanised Hinglish many of us actually type, came out as nine pieces and eleven. The romanised version cost more than the script, which surprised me: the tokeniser had clearly seen enough Devanagari to give common Hindi words their own pieces, but "varsh" and "mahine" written in Latin letters were chopped as if they were misspelt English. Note 32 takes this properly.

The practical, and what it showed

Practical 02 has two halves. First, ask a chat tool to count the r's, plainly and then with the word spelled out one letter per line. Second, put a handful of words into a public tokeniser and write down the pieces. Fifteen minutes, nothing to install.

My run: the counting question came back correct, "3", every time, both plainly and spelled out. Two years ago that was not the case, and the change is worth understanding (below). The tokeniser half produced the two drawings above; the full table is on the practical page.

it never saw the r's. it remembered them.

One layer deeper *

The method has a name. Merging the most frequent pair, over and over, is byte-pair encoding, borrowed from a 1994 compression technique and applied to language models from about 2016. Modern tokenisers work on bytes rather than characters, so any text in any script can be represented, at worst one byte per piece; that is why unusual scripts and emoji cost so many pieces. The list is typically 100,000 to 200,000 pieces long.

Why models now count the r's. Three things changed, none of which is "the model can see letters". The question became famous, so the answer appears many times in newer training text. Models are now trained to work step by step before answering, and spelling a word out letter by letter turns it into pieces the model can count, which is exactly what the second prompt in the practical does by hand. And some chat tools quietly run a small program for questions that look like counting. A fair test today is a rare word the internet has not argued about; try counting the letters in "Thiruvananthapuram".

The number is not the meaning. The token ID is just a position in the list; 19772 does not mean "berry". What the model learns about each piece is stored separately, as a long list of numbers per piece (an embedding), and that is where "berry usually has two r's" would have to live, if it lives anywhere. Note 14 is about embeddings.

Each model has its own tokeniser. The pieces in this note come from one public tokeniser (OpenAI's). Another company's model would cut the same words differently, so token counts, and therefore prices and context limits, are not comparable across models without checking. The shape of the results is the same everywhere: English cheap, everything else chopped.

The Hindi result is worth a careful look. That Devanagari cost fewer pieces than romanised Hindi is one tokeniser's behaviour on one sentence, not a law. It does suggest the training text contained a fair amount of Hindi in script and rather less transliterated Hinglish. Whether that holds for Tamil, Gujarati or Bengali is an open question for Note 32; if you run the practical in your language, the practical page has a row for it.

Sources

Sennrich, Haddow and Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016) for byte-pair encoding in language models; Gage, "A New Algorithm for Data Compression" (1994) for the original. Karpathy, "Let's build the GPT Tokenizer" (2024) for the plain-language walk-through. Token splits are my own, taken from OpenAI's public tokeniser page on 8 September 2026; counting runs are my own in a consumer chat tool the same day. Terms used: token, model, context window.