What Are Tokens and Tokenization?

A token is one entry in a model’s fixed vocabulary, typically a subword fragment rather than a whole word. Tokenization is the step that maps your text onto those entries and then onto integers. Models never see characters; they see token ids, and everything you pay for is counted in them.

How is text actually split?

By a learned merge table, not by whitespace. The dominant algorithms all build a vocabulary from a training corpus and then apply it deterministically.

Algorithm	How the vocabulary is built	Seen in
BPE	Repeatedly merge the most frequent adjacent pair	GPT-2, RoBERTa
WordPiece	Merge the pair that most increases corpus likelihood	BERT, Electra
Unigram / SentencePiece	Prune a large candidate set down by loss	T5, ALBERT, XLNet

Sennrich et al. introduced byte-pair encoding to translation so rare and unseen words could be represented as sequences of known subwords rather than falling back to an unknown token. The tokenizer summary sets out how the three differ.

Why does the same sentence cost different amounts?

Because the vocabulary was fitted to a corpus, and text that resembles that corpus compresses better. Common English words are often single tokens; a rare identifier, a chemical name, or a language under-represented in training gets shredded into many fragments. Same meaning, more tokens, more of your context window consumed.

Should you retrain the tokenizer for your domain?

Almost never, because the vocabulary is welded to the embedding matrix, so changing it invalidates the pretrained weights that make a small model worth starting from. Do you need to fine-tune the tokenizer works through the exceptions.

It’s also why student and teacher can differ freely in architecture but not interchangeably in vocabulary, a point picked up in does the student need the same architecture as the teacher.