We are pleased to announce the launch of our new
ai21-tokenizer client library. This library allows for local tokenization, eliminating the need to rely on the
/tokenizer API endpoint. Users can now efficiently perform tokenization directly within their local environment, providing a secure method to determine the number of tokens for a given prompt or answer.
- Local Tokenization: Conduct tokenization directly on your local environment without reliance on the
- Efficiency: Streamline the tokenization process for improved workflow efficiency.
- Security: Ensure a secure environment for tokenizing prompts or answers.