rustio rustio.net
49

rust_tokenizers

v8.1.1 Growing

High performance tokenizers for Rust

Apache-2.0 Edition 2018
#tokenizer#nlp#machine-learning

Quick Verdict

  • โœ•Not updated for 3+ years
  • โœ“Stable API (8.x for 6+ years)
  • โœ“Trusted by 117 crates
  • โœ“Permissive license (Apache-2.0)

Security

Checking security advisories...
Downloads
452.3K
Dependents
117
Releases
34
Size
134KB

Deep Insights

๐Ÿ“ˆ
Strong growth momentum

33.3K downloads in the last 30 days (1.1K/day), up 85% from the previous period.

๐Ÿ”—
Moderate adoption

117 crates depend on rust_tokenizers. Reasonable ecosystem adoption, though not yet a core dependency.

โœ…
Stable for 6+ years

The API has been stable (1.x) for over 6 years with 34 releases. This level of maturity means you can depend on it without worrying about breaking changes.

๐ŸŒŸ
Used by top crates

Notable dependents include rust-bert, semchunk-rs, llm-llama, sbert, luminal. When high-quality crates choose rust_tokenizers, it's a strong quality signal.

Health Breakdown

Maintenance 6/25

Recency, release consistency, active ratio

Quality 18/25

Yanked ratio, deps, size, maturity, features

Community 9/20

Reverse deps, ownership, ecosystem

Popularity 7/15

Downloads, momentum, growth trend

Documentation 9/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
104/day avg-100%
01K2K3K8/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v8.1.1
57%
v7.0.2
27%
v8.1.0
11%
v7.0.0
3%
v6.2.4
2%

Release Timeline

10 releasessince 2021
J
F
M
A
M
J
J
A
S
O
N
D
2021
6
2022
1
2023
3
2024
2025
2026
Less
More

Feature Flags

default =["default-tls"]

rustls-tlsdefault-tls*proto-compile

README

Loading README...

Maintainers

Dependencies
17
direct dependencies
Dependents
117
crates depend on rust_tokenizers

Similar Crates