A token is the basic unit of text or code that a language model (e.g., Large Language Models) reads, processes, and generates. In other words, tokens are the fundamental elements that models use to break down the input and create the output.
These units can vary based on how the model provider defines them, but in general, they are a few characters long and could represent:
- A whole word (e.g., “daydream”),
- A part of a word (e.g., “day”),
- A number (e.g., 3),
- Or other linguistic components such as punctuation or spaces.
A model (such an embedding model) tokenises the input based on its tokeniser algorithm, which converts the input into tokens. Similarly, the model’s output is presented as a stream of tokens, which are then decoded back into human-readable text.
Why tokens instead of characters?
Since tokens represent meaningful units, like whole words or parts of words, models can better capture language structure than by processing raw characters. As a consequence, tokens enable models to understand context and grammar more effectively.
On another front, models handle fewer units when processing tokens rather than characters. Thus, tokens provide faster computation.
In contrast, character-level processing would require manipulating a much larger input sequence, hindering the model’s ability to learn relationships and context. Tokens allow models to focus on linguistic meaning, making them more accurate and efficient in generating responses.
Tokens aren’t necessarily text
Tokens represent multimodal data, including text, images, audio, video, and other formats.
A priori, anything that we can represent as a sequence of tokens could be modelled with transformers. For example, DNA includes four bases in its strands — adenine (A), thymine (T), cytosine (C), and guanine (G). Thus, in theory, we could tokenise and model the DNA using transformers to capture patterns, predict gene expression, and so on. Therefore, transformers have immense potential in various fields, utilising either structured or unstructured sequences.
At the time of writing this post, I am only aware of a few models that can handle multimodal inputs (e.g., text combined with images or audio), and none support multimodal output.
As AI continues to evolve, I hope that multimodality becomes a commonplace approach. In this case, models would be able to process and generate a broader range of media, significantly expanding the scope of what tokens can represent and how models can interact with diverse types of data.
« Back to Glossary Index