> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ai21.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenization

> Tokenization is the process of converting text into numerical tokens that language models can understand. Learn how to use AI21's tokenizer for Jamba models with practical examples.

## What is Tokenization?

Tokenization is both the first and final step in language model processing. Since machine learning models can only work with numerical data, text must be converted into numbers that models can understand and manipulate.

The tokenization process breaks down text into smaller units called tokens, which can represent:

* **Words or subwords**: "hello" → `[15496]`
* **Characters**: "AI" → `[32, 73]`
* **Byte-level representations**: For handling any Unicode text across all languages

Each token is assigned a unique numerical ID from the model's vocabulary.

## When is Tokenization Used?

Tokenization serves as both the entry point and exit point of text processing in language models. Since models can only work with numerical data, text must be converted into tokens with corresponding numerical indices from the tokenizer's vocabulary.

In a standard language model workflow:

1. **Encoding Phase**: We first convert input text into tokens using a tokenizer. Each token receives a unique index number that the model can process.

2. **Model Processing**: The tokenized input flows through the model architecture:
   * **Embedding layer**: Transforms tokens into dense vector representations that capture semantic relationships
   * **Transformer blocks**: Process these vectors to understand context, relationships, and generate meaningful responses

3. **Decoding Phase**: Finally, we convert the model's output tokens back into readable text by mapping token indices back to their corresponding words or subwords using the tokenizer's vocabulary.

This encode → process → decode cycle ensures seamless conversion between human language and machine-readable formats, enabling effective communication with language models.

## AI21's Tokenizer

We provides a [AI21-Tokenizer](https://github.com/AI21Labs/ai21-tokenizer) specifically engineered for Jamba models.

### Key Features

* **Jamba Mini and Large support**
* **Async/sync operations**:
* **Production-ready**: Enterprise-grade reliability

## Installation

### Prerequisites

<Note>
  To use tokenizers for Jamba, you'll need access to the relevant model's HuggingFace repository.
</Note>

### Install the Tokenizer

```bash theme={"system"}
pip install ai21-tokenizer
```

## Model-Specific Tokenizers

Choose the appropriate tokenizer for your Jamba model:

<Tabs>
  <Tab title="Jamba Mini">
    ```python theme={"system"}
    from ai21_tokenizer import Tokenizer, PreTrainedTokenizers

    tokenizer = Tokenizer.get_tokenizer(PreTrainedTokenizers.JAMBA_MINI_TOKENIZER)

    text = "Jamba Mini model says hello"
    encoded = tokenizer.encode(text)
    print(f"Jamba Mini encoded: {encoded}")
    ```
  </Tab>

  <Tab title="Jamba Large">
    ```python theme={"system"}
    from ai21_tokenizer import Tokenizer, PreTrainedTokenizers

    tokenizer = Tokenizer.get_tokenizer(PreTrainedTokenizers.JAMBA_LARGE_TOKENIZER)

    text = "Jamba Large model says hello"
    encoded = tokenizer.encode(text)
    print(f"Jamba Large encoded: {encoded}")
    ```
  </Tab>
</Tabs>

## Basic Usage

<Steps>
  <Step title="Encode Text to Tokens">
    ```python theme={"system"}
    from ai21_tokenizer import Tokenizer

    # Create tokenizer (defaults to Jamba Mini)
    tokenizer = Tokenizer.get_tokenizer()

    # Convert text to token IDs
    text = "Hello, world!"
    encoded = tokenizer.encode(text)
    print(f"Encoded: {encoded}")
    # Output: Encoded: [15496, 11, 1917, 0]
    ```
  </Step>

  <Step title="Decode Tokens to Text">
    ```python theme={"system"}
    from ai21_tokenizer import Tokenizer

    tokenizer = Tokenizer.get_tokenizer()

    # Convert token IDs back to text
    decoded = tokenizer.decode(encoded)
    print(f"Decoded: {decoded}")
    # Output: Decoded: Hello, world!
    ```
  </Step>
</Steps>

## Asynchronous Usage

For high-performance/server applications, use the async tokenizer:

```python theme={"system"}
import asyncio
from ai21_tokenizer import Tokenizer

async def main():
    tokenizer = await Tokenizer.get_async_tokenizer()
    
    text = "Async tokenization for async operations!"
    encoded = await tokenizer.encode(text)
    decoded = await tokenizer.decode(encoded)
    
    print(f"Original: {text}")
    print(f"Encoded: {encoded}")
    print(f"Decoded: {decoded}")

asyncio.run(main())
```

## Practical Use Cases

* **Cost estimation**: Calculate API usage costs based on token consumption
* **Prompt optimization**: Ensure prompts fit within model context limits

For more advanced usage examples, visit the [AI21 tokenizer examples](https://github.com/AI21Labs/ai21-tokenizer/tree/main/examples) folder.