kitoken
v0.11.0 GrowingFast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Quick Verdict
- !Pre-1.0: API may have breaking changes
- !Heavy dependency tree (24 direct deps)
Security
Deep Insights
1.6K downloads in the last 30 days (54/day), up 95% from the previous period.
Despite being 1+ years old, kitoken hasn't reached 1.0 yet. Expect potential API changes between versions.
24 direct dependencies. Consider the impact on compile times and supply chain complexity.
Notable dependents include miktik, rust-genai, rlx-voxtral-tts. When high-quality crates choose kitoken, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on kitoken
Version Adoption
Release Timeline
Feature Flags
default =["std", "serialization", "normalization", "convert", "regex-perf", "multiversion"]