New Client Tokenizer is now available! 🚀

We are pleased to announce the launch of our new ai21-tokenizer client library. This library allows for local tokenization, eliminating the need to rely on the /tokenizer API endpoint. Users can now efficiently perform tokenization directly within their local environment, providing a secure method to determine the number of tokens for a given prompt or answer.

Key Advantages:

  • Local Tokenization: Conduct tokenization directly on your local environment without reliance on the /tokenizer API endpoint.
  • Efficiency: Streamline the tokenization process for improved workflow efficiency.
  • Security: Ensure a secure environment for tokenizing prompts or answers.

For detailed information and implementation, please refer to our GitHub repository: ai21-tokenizer or access the package on PyPI: ai21-tokenizer on PyPI.