GitHub

SQLite ICU Tokenizer Extension

A SQLite FTS5 extension that provides International Components for Unicode (ICU) based tokenization for full-text search, with excellent support for Japanese, Chinese, Korean, and other ...
GitHub

FTS5 ICU Tokenizer for SQLite

This project provides custom FTS5 tokenizers for SQLite that use the International Components for Unicode (ICU) library to provide robust word segmentation for various languages. The project supports ...
IEEE

Problematic Tokens: Tokenizer Bias in Large Language Models

Abstract: Recent advancements in large language models (LLMs), such as GPT-4 and GPT-4o, have shown exceptional performance, especially in languages with abundant resources like English, thanks to ...