Tokenizer Collection

◆

◆

◆

◆

📦Normal

★★★★★

“Implement a ChatGPT-like LLM in PyTorch from scratch, step by step”

◆

◆

◆

◆

☠️Poison

★★★

“A small library for converting tokenized PHP source code into XML (and potentially other formats)”

◆

◆

◆

◆

💎Aqua

★★

“Parser Building Toolkit for JavaScript”

◆

◆

◆

◆

💎Aqua

★★

“Online playground for OpenAPI tokenizers”

◆

◆

◆

◆

🔮Psychic

★★

“Persian NLP Toolkit”

◆

◆

◆

◆

🔮Psychic

★★

“Solves basic Russian NLP tasks, API for lower level Natasha projects”

◆

◆

◆

◆

⚡Electric

★★

“A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint — parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.”

◆

◆

◆

◆

🔮Psychic

★★

“한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.”

◆

◆

◆

◆

🌊Water

★★

“Self-contained Japanese Morphological Analyzer written in pure Go”

◆

◆

◆

◆

⚡Electric

★★

“Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.”

◆

◆

◆

◆

🔥Fire

★★

“支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin”

◆

◆

◆

◆

💎Aqua

★★

“The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.”

◆

◆

◆

◆

🔮Psychic

★★

“An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation”

◆

◆

◆

◆

⚡Thunder

★★

“数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。”

◆

◆

◆

◆

🔥Fire

★★

“🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.”

◆

◆

◆

◆

🔮Psychic

★★

“Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).”

◆

◆

◆

◆

🔥Fire

★★

“Open Korean Text Processor - An Open-source Korean Text Processor”

◆

◆

◆

◆

⚙️Steel

★★

“A multilingual morphological analysis library.”