Tokenize text for Llama, Gemini, GPT-4, DeepSeek, Mistral and many others; in the web, on the client and any platform. Kitoken can load and convert many existing tokenizer formats. Every supported ...

A SQLite FTS5 extension that provides International Components for Unicode (ICU) based tokenization for full-text search, with excellent support for Japanese, Chinese, Korean, and other ...