rustio rustio.net
54

tokie

v0.1.4 Experimental

Blazingly fast tokenizer - 50x faster tokenization, 10x smaller model files, 100% accurate drop-in replacement for HuggingFace

MIT OR Apache-2.0 Edition 2024
ScienceText processing #tokenizer#nlp#wordpiece#bpe#transformers

Quick Verdict

  • โœ“Actively maintained (updated 77d ago)
  • !Pre-1.0: API may have breaking changes
  • !Heavy dependency tree (21 direct deps)
  • โœ“Permissive license (MIT OR Apache-2.0)

Security

Checking security advisories...
Downloads
1.9K
Dependents
36
Releases
17
Size
185KB

Deep Insights

๐Ÿ“‰
Download decline

345 downloads in the last 30 days, down 42% from the previous period. May indicate migration to alternatives.

๐Ÿ“ฆ
Heavy dependency tree

21 direct dependencies. Consider the impact on compile times and supply chain complexity.

๐ŸŒŸ
Used by top crates

Notable dependents include winx-code-agent, maolan-generate, mandex. When high-quality crates choose tokie, it's a strong quality signal.

Health Breakdown

Maintenance 15/25

Recency, release consistency, active ratio

Quality 12/25

Yanked ratio, deps, size, maturity, features

Community 10/20

Reverse deps, ownership, ecosystem

Popularity 4/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
0/day avg
08/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v0.1.4
63%
v0.0.10
27%
v0.0.8
7%
v0.0.9
2%
v0.0.10-alpha.1
1%

Release Timeline

10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
Less
More

Feature Flags

hfbuild

README

Loading README...

Maintainers

Dependencies
21
direct dependencies
Dependents
36
crates depend on tokie

Similar Crates