The route
What's one of the first things you learn in school? You learn how to read, and AIs aren't much different here. They have to read your text, words, characters. But how does that actually work?
An AI will take your words & phrases and separate them into "bitesize" chunks for it to compute. Let's start with an example
Hello. Welcome to strawberrrrry city!
If we head on over to https://platform.openai.com/tokenizer1 we can see what an AI model will do to separate them
You'll notice that message is 10 tokens. Each color represents an individual token that the AI model will "digest". Note that punctuations are single tokens and something interesting is happening in the word "strawberrrrry". It's not split into syllables although you might think that's how it worked.
The actual principle is tokens are this:
Frequently occurring character sequences in the training data, found by an algorithm, that repeatedly merges the most common adjacent pairs. Pure statistics without any prior knowledge of syllables, prefixes, or meaning.
Common things get one token no matter how long. Amsterdam and Florida are common words so they're single tokens. Hamsterdam and Floridia aren't, so the model has to piece them together from fragments.
Now here's where I lied earlier. Remember when I said "They have to read your text, words, characters"? Well the model doesn't actually read characters, at all. By the time the model gets involved your list of characters, words, phrases, is actually just a bunch of numbers.
Each chunk has a fixed ID. The model never actually receives the word "Amsterdam". "Amsterdam" is actually entry #167754 and THAT is what the model receives. Keep in mind different models have different IDs for tokens; each model family builds its own dictionary.
Let's head on over to the next lesson to learn about context windows!
B - What's a context window?
Or if you want to understand tokenizers in more detail we can go to
A2 WIP