shapesinfinity.
Learn AI · Beginner · 3 min

Tokens and Tokenization

Did you know AI reads words piece by piece, not whole?

How AI Breaks Text Into Tokens
Your sentenceText splitterWord piecesNumber each pieceToken numbersContext limit

AI converts your words into numbered pieces called tokens, then reads them one by one.

goes inthe AI workscomes outwatch out

In simple words

Tokens are tiny word pieces. AI breaks words into pieces, like cutting a cake into small bites.

The real definition

Tokenization breaks text into tokens (small units: words, punctuation, or subwords). AI processes tokens one by one.

Like… Breaking a Letter

Tokenization is like a postal worker breaking a long letter into pages. Each page is a token. The worker reads page one, then page two.

But: Unlike letters, tokens can split words in the middle for efficiency.

You see it every day

on your phone

Phone chat reply

You type 'Hello friend' on your phone. AI tokenizes it into 'Hello,' 'friend' and sends the reply.

out in the world

Translation service

A translation tool breaks your sentence into tokens, translates each, and reassembles it.

at work

Customer support

A work chatbot tokenizes your question, understands each token, and generates a helpful response.

Step by step

  1. 1

    You write text

    You enter a sentence or question into the AI system.

  2. 2

    Tokenizer splits it

    Special software cuts your text into small, meaningful pieces.

  3. 3

    Assign token numbers

    Each piece gets a unique number the AI understands.

  4. 4

    AI reads tokens

    The AI model processes numbers one at a time.

  5. 5

    AI generates output

    The AI creates new tokens and converts them back to words.

Remember

Tokens are small pieces of text, not whole words.Tokenization lets AI understand language in manageable units.Each token gets a number the AI recognizes.More tokens cost more time and money to process.Token limits constrain how much context AI can use.

Memory trick

Think: LEGO blocks. Words are big; tokens are tiny blocks. AI builds understanding block by block.

Words

token TOH-kun
One small piece of text (word, part, or symbol).
tokenization toh-kun-eye-ZAY-shun
The process of breaking text into small, numbered pieces.
tokenizer TOH-kun-eye-zer
The tool or software that splits text into tokens.
model MAH-dul
The trained AI program that understands and generates text.
context KON-text
The recent conversation or text the AI remembers and uses.
input IN-put
Text or data you give to the AI to process.
output OWT-put
Text or data the AI creates and returns to you.

Check yourself

quick check

What does tokenization mean?

Hint

Think about cutting something into smaller parts.

quick check

You ask an AI chatbot: 'What is AI?' Why does AI use tokens?

Hint

How does AI read and understand text?

think

If a long story needs many tokens, why might this be a problem?

Show a good answer

More tokens take longer to process and cost more money. AI also has a token limit, so very long texts may not fit.

Tell a friend

“AI reads your words as tiny numbered pieces called tokens, one by one.”

People also ask

What is a token in AI?

A token is one small piece of text—a word, part of a word, or punctuation mark. AI reads tokens one at a time to understand language.

Why do AI systems use tokenization?

Tokenization breaks language into pieces small enough for AI to process quickly. It makes understanding and generating text more efficient.

Does tokenization affect how much text AI can remember?

Yes. AI has a token limit (context window). Longer texts use more tokens, so there is less room for new conversation.