What is a token in AI? Word count, limits, pricing

What is a token?

People read text as letters and words. An AI first splits text into small pieces, gives each piece a number, and then does its math on those numbers. Each piece is a token, and the program that does the splitting is called a tokenizer.

Anthropic, the company behind Claude, describes a token in its glossary as the smallest unit of a language model. It can correspond to a word, part of a word, a character, or even a byte. Common words usually become one token. Rare or unfamiliar words get split into several pieces.

So "one token equals one word" is wrong. Use it only as a rough feel. The same glossary says one token is roughly 3.5 English characters, and adds that this varies by model and language.

Does non-English text use more tokens?

Often, yes, as a general tendency. AI models see a lot of English in training, and tokenizers tend to keep common English words whole. Text in many other languages, especially those with different writing systems, is often split into smaller pieces.

How big the gap is depends on the company and the model version. Published research and developer tests report anything from a bit more than English to several times more, and newer tokenizers usually narrow the gap. So no single number, like "always double," is reliable.

The same company can change this too. When a new model ships with a different tokenizer, the same text can come out as more or fewer tokens. A token count only applies to the model it was measured on.

To see how many tokens your own text uses, use the token counter from the AI company. Different tokenizers give different results, so use the one from the company that makes the model you plan to use.

• How text is split

  English

  Common words often stay as one token

  Many other languages

  Often split into smaller pieces

• Same meaning, same sentence

  English

  Usually fewer tokens

  Many other languages

  Often more tokens

• Size of the gap

  English

  Varies by model and tokenizer

  Many other languages

  Varies by model and tokenizer

Chat length limits and tokens

An AI can only look at a limited amount of text at once. That limit is the context window, measured in tokens. Anthropic's documentation describes it as everything the model can refer to when it writes a reply, a kind of working memory. It includes what you typed, what the AI wrote, and any document you pasted in.

As a conversation grows, earlier questions and answers keep piling up. The documentation says each message and reply accumulates in the context window as the conversation goes on. Chat apps may also trim older parts. That is why, in a very long chat, the AI can lose track of something said early on.

Window size varies by model. According to Anthropic's documentation (checked October 2026), many of its latest models have 1 million tokens, and some others have 200,000. A bigger number is not automatically better. The same documentation notes that accuracy and recall can drop as the content gets longer.

Usage limits on the service you use, such as messages per few hours, also relate to chat length. Long chats and big files contain more tokens, so they tend to use up your allowance faster. The exact rules differ by service and plan and change often, so check each service's help pages.

Pricing is counted in tokens

A monthly subscription (ChatGPT Plus, Claude Pro and so on) does not make you count tokens. You use it within a usage limit. The API, which developers use to build AI into apps, charges by the number of tokens. Input (what you send) and output (what the AI writes) usually have different prices, and output costs more.

For models that "think" before answering, that thinking is counted too. Anthropic's documentation says thinking tokens are billed as output tokens. So a short visible answer can still cost more than you expect.

This means the same task can cost different amounts in different languages, and a long conversation gets more expensive because the earlier messages are sent again each time.

How to use fewer tokens

• When a chat gets long and the AI starts missing earlier details, start a new chat. Paste a short summary of what matters as the first message.
• Do not attach long documents or photos that have nothing to do with your question. Paste only the part you need.
• Ask for a format instead of "in detail." "In 5 lines or fewer" cuts the output tokens.
• Write clear instructions the first time instead of asking for the same text to be redone several times.
• Use a smaller, faster model for simple jobs. The lightweight models in the model picker are usually cheaper.

Summarize only what we have decided so far, in 10 lines or fewer. I will paste it into a new chat to continue. Split it into: decisions made, open questions, and things I said I do not want.

Related terms

Tokenizer

The program that splits text into tokens. Each AI company and model does it differently, so the same text can have different token counts.

Context window

How much text the AI can consider at once. It is counted in tokens.

Input / output tokens

The text you send and the text the AI writes. In the API they have different prices.

FAQ