rustio rustio.net
65

tokenizers

v0.23.2 Growing

Fastest implementation of the most used tokenizers in AI.

Apache-2.0 Edition 2018
#simd#tokenizer#wordpiece#huggingface#bpe

Quick Verdict

  • βœ“Actively maintained (updated 18d ago)
  • !Pre-1.0: API may have breaking changes
  • βœ“Massive adoption (12.6K crates depend on it)
  • !Heavy dependency tree (33 direct deps)
  • βœ“Permissive license (Apache-2.0)

Security

Checking security advisories...
Downloads
38.1M
Dependents
12.6K
Releases
42
Size
216KB

Deep Insights

πŸ“ˆ
Strong growth momentum

6.9M downloads in the last 30 days (228.8K/day), up 37% from the previous period.

πŸ›οΈ
Foundational crate

12.6K crates depend on tokenizers β€” it's part of the Rust ecosystem's core infrastructure. Removing it from your dependency tree would be extremely difficult.

πŸ”¬
Pre-1.0 for over a year

Despite being 7+ years old, tokenizers hasn't reached 1.0 yet. Expect potential API changes between versions.

πŸ“¦
Heavy dependency tree

33 direct dependencies. Consider the impact on compile times and supply chain complexity.

🌟
Used by top crates

Notable dependents include candle-core, fastembed, text-splitter, llm-tokenizer, fastokens. When high-quality crates choose tokenizers, it's a strong quality signal.

Health Breakdown

Maintenance 17/25

Recency, release consistency, active ratio

Quality 12/25

Yanked ratio, deps, size, maturity, features

Community 16/20

Reverse deps, ownership, ecosystem

Popularity 8/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads Β· last 90 days
45K/day avg-100%
0100K200K300K8/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v0.22.2
45%
v0.21.4
34%
v0.23.1
8%
v0.22.1
7%
v0.21.2
6%

Release Timeline

10 releasessince 2025
J
F
M
A
M
J
J
A
S
O
N
D
2025
7
2026
3
Less
More

Feature Flags

default =["progressbar", "onig", "esaxx_fast"]

httpesaxx_fast*rustls-tlsprogressbar*unstable_wasmparity-aware-bpe

README

Loading README...

Maintainers

Dependencies
33
direct dependencies
Dependents
12.6K
crates depend on tokenizers

Similar Crates