tokenizers
v0.23.1 GrowingProvides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Quick Verdict
- ✓Actively maintained (updated 28d ago)
- !Pre-1.0: API may have breaking changes
- ✓Massive adoption (5.5K crates depend on it)
- !Heavy dependency tree (33 direct deps)
- ✓Permissive license (Apache-2.0)
Security
Deep Insights
2.6M downloads in the last 30 days (85.1K/day), up 38% from the previous period.
5.5K crates depend on tokenizers. Strong ecosystem adoption means battle-tested code and long-term stability.
Despite being 6+ years old, tokenizers hasn't reached 1.0 yet. Expect potential API changes between versions.
33 direct dependencies. Consider the impact on compile times and supply chain complexity.
Notable dependents include candle-core, text-splitter, fastembed, xgrammar, toktrie_hf_tokenizers. When high-quality crates choose tokenizers, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on tokenizers
Version Adoption
Release Timeline
Feature Flags
default =["progressbar", "onig", "esaxx_fast"]
