Glossary › Tokenization
Tokenization
Converting text into tokens - the atomic units a language model processes. Common approaches: word tokenization (splits on spaces, loses morphology), character tokenization (maximally granular), subword tokenization (BPE, WordPiece - the standard in modern LLMs). Subword tokenization handles unknown words by splitting them into known subword units.
Where Tokenization shows up
All matching postsTokenization: The Part of LLMs Nobody Explains Well
LLMs do not see words or characters - they see tokens. Understanding tokenization explains why LLMs fail at some surprisingly easy tasks and how to work around it.
#llm#transformers#nlp
Transformers and Modern NLP Fundamentals
Bridge into LLM systems with practical understanding of tokenization, attention, and fine-tuning.
#transformers#nlp#deep-learning#attention