rustio rustio.net
46

tok3niz3r-trainer

v0.2.0 Experimental

Trainer for byte-level BPE tokenizers: learns a vocabulary and merge table from a text corpus. The training half of tok3niz3r.

MIT Edition 2024 MSRV 1.85
AlgorithmsScienceText processing #tokenizer#nlp#tokenization#training#bpe

Quick Verdict

  • !Pre-1.0: API may have breaking changes
  • ✓Permissive license (MIT)

Security

Checking security advisories...
Downloads
94
Dependents
2
Releases
2
Size
52KB

Deep Insights

📊
Steady growth

13 downloads in the last 30 days (0/day), up 8% from the previous period.

Health Breakdown

Maintenance 11/25

Recency, release consistency, active ratio

Quality 14/25

Yanked ratio, deps, size, maturity, features

Community 7/20

Reverse deps, ownership, ecosystem

Popularity 2/15

Downloads, momentum, growth trend

Documentation 12/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days
0/day avg
08/239/109/2810/1611/311/20

Top Dependents

Most downloaded crates that depend on tok3niz3r-trainer

Version Adoption

v0.2.0
79%
v0.1.0
21%

Release Timeline

2 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
2
Less
More

Feature Flags

default =["hf-compat"]

hf-compat*

README

Loading README...

Maintainers

Dependencies
8
direct dependencies
Dependents
2
crates depend on tok3niz3r-trainer

Similar Crates