kitoken
v0.10.1 GrowingFast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Quick Verdict
- ✕Not updated for 1+ year
- !Pre-1.0: API may have breaking changes
- !Heavy dependency tree (23 direct deps)
Security
Deep Insights
7.4K downloads in the last 30 days (247/day), up 30% from the previous period.
Despite being 1+ years old, kitoken hasn't reached 1.0 yet. Expect potential API changes between versions.
23 direct dependencies. Consider the impact on compile times and supply chain complexity.
At 43KB, kitoken is lightweight. Small crate size correlates with focused, well-scoped functionality.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on kitoken
Version Adoption
Release Timeline
Feature Flags
default =["std", "serialization", "normalization", "convert", "regex-perf", "multiversion"]
