Skip to content
explainroo
AI models explained

What Is a Token in AI?

Updated 2 min read

With explainroo Text-to-Video Models vs. Code-Rendered Video for Explainers
On this page
  1. How text becomes tokens
  2. Why token counts affect limits and costs
  3. Tokens are the units a model predicts

When you ask a language model to summarize a document or answer a question, it splits the text into small units called tokens. A token may be a whole word or part of one. Tokens also include punctuation, spaces, and pieces of code. The model’s tokenizer determines how the text is split.

Token counts help determine how much text a model can process at once, the cost of an API request, and how much text the model produces. Because a token is not the same as a word, a word count gives only a rough estimate.

How text becomes tokens

A language model’s tokenizer splits text into tokens and matches them to entries in the model’s vocabulary. A common word might make up one token, while a less common word might split into smaller pieces. Tokens can also include punctuation and spaces. The model processes these tokens rather than whole words.

Many tokenizers use a method related to Byte-Pair Encoding (BPE). BPE combines text pieces that appear often into larger units. Less common text stays representable as smaller pieces. This lets the vocabulary cover familiar words efficiently and still handle unfamiliar ones. Tokenization in language models explains how these choices shape the units a model learns to predict.

Providers use different tokenizers, so the same sentence can have different token counts across models. Language also affects counts, so an estimate for English may not fit text in another language. For an accurate count, use the tokenizer or counting tool for the model you plan to use. OpenAI, for example, provides token-counting tools.

Why token counts affect limits and costs

A model’s context window sets the maximum number of tokens it can handle in a request. The count can include instructions and conversation history sent to the model. It can also include the answer the model generates. If the total exceeds the limit, the request may fail or the input may need to be shortened.

For English, a rough estimate is one token per four characters, or about three-quarters of a word. These figures are approximate rules of thumb. The OpenAI token guide gives examples and explains how the text and tokenizer affect the count.

Many AI APIs charge separately for tokens sent to the model and tokens it generates. Rates vary by provider and model, and also by token category. Check the service’s current prices before estimating a bill. Some providers charge a different rate for cached input, which is text reused from earlier requests.

To estimate usage, count the prompt and expected response with the model’s tokenizer. Some reasoning models also report tokens used for internal reasoning. Internal reasoning tokens count toward usage, though they do not appear in the visible answer.

Tokens are the units a model predicts

During training, a language model learns to predict the next token in a sequence. When it generates a response, it produces tokens one at a time, and those tokens appear as text. Since a token can be part of a word, the number of tokens does not translate directly into a word count.

Shortening a prompt can help keep it within the context limit and reduce usage costs. Remove repeated instructions or irrelevant passages, then check the new token count. For important requests, use the target model’s tokenizer instead of converting from a word count.

Free and open source

Let your AI agent make the video

explainroo lets a coding agent like Claude Code or Codex make narrated explainer videos and product demos. The voice, the word timing and the rendering run on your own computer, with no API key and no cost per video. Optional outside services use your own key.

Copy this into your AI agent

Make me a short explainer video about [your topic]. Use explainroo for it: run npx --yes explainroo docs and follow what it prints.

Getting started Example videos GitHub

Made with explainroo: How explainroo makes a video

More on ai models explained

All guides on ai models explained