New Client Tokenizer is now available! 🚀
5 months ago by Or Fattal
We are pleased to announce the launch of our new ai21-tokenizer client library. This library allows for local tokenization, eliminating the need to rely on the /tokenizer API endpoint. Users can now efficiently perform tokenization directly within their local environment, providing a secure method to determine the number of tokens for a given prompt or answer.
Key Advantages:
- Local Tokenization: Conduct tokenization directly on your local environment without reliance on the
/tokenizerAPI endpoint. - Efficiency: Streamline the tokenization process for improved workflow efficiency.
- Security: Ensure a secure environment for tokenizing prompts or answers.
For detailed information and implementation, please refer to our GitHub repository: ai21-tokenizer or access the package on PyPI: ai21-tokenizer on PyPI.