tokenizers
v0.23.2 GrowingFastest implementation of the most used tokenizers in AI.
Quick Verdict
- βActively maintained (updated 18d ago)
- !Pre-1.0: API may have breaking changes
- βMassive adoption (12.6K crates depend on it)
- !Heavy dependency tree (33 direct deps)
- βPermissive license (Apache-2.0)
Security
Deep Insights
6.9M downloads in the last 30 days (228.8K/day), up 37% from the previous period.
12.6K crates depend on tokenizers β it's part of the Rust ecosystem's core infrastructure. Removing it from your dependency tree would be extremely difficult.
Despite being 7+ years old, tokenizers hasn't reached 1.0 yet. Expect potential API changes between versions.
33 direct dependencies. Consider the impact on compile times and supply chain complexity.
Notable dependents include candle-core, fastembed, text-splitter, llm-tokenizer, fastokens. When high-quality crates choose tokenizers, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on tokenizers
Version Adoption
Release Timeline
Feature Flags
default =["progressbar", "onig", "esaxx_fast"]