rust_tokenizers
v8.1.1 GrowingHigh performance tokenizers for Rust
Quick Verdict
- ✕Not updated for 2+ years
- ✓Stable API (8.x for 6+ years)
- ✓Trusted by 117 crates
- ✓Permissive license (Apache-2.0)
Security
Deep Insights
8.2K downloads in the last 30 days (272/day), up 15% from the previous period.
117 crates depend on rust_tokenizers. Reasonable ecosystem adoption, though not yet a core dependency.
The API has been stable (1.x) for over 6 years with 34 releases. This level of maturity means you can depend on it without worrying about breaking changes.
Notable dependents include rust-bert, llm-llama, sbert, pocket-cli, rust_hero. When high-quality crates choose rust_tokenizers, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on rust_tokenizers
Version Adoption
Release Timeline
Feature Flags
default =["default-tls"]
