rustio rustio.net
45

ferric-tokenizer

v0.3.0 Experimental

Ferric tokenizer — a pure-Rust byte-level BPE (GPT-2/tiktoken family): lossless encode/decode, loads vocab.json + merges.txt. The prompt→ids→text glue for end-to-end LLM use.

Apache-2.0 Edition 2021
ScienceText processing #ai#tokenizer#nlp#bpe#llm

Quick Verdict

  • ✓Actively maintained (updated 56d ago)
  • !Pre-1.0: API may have breaking changes
  • ✓Tiny footprint (9KB, 1 deps)
  • ✓Permissive license (Apache-2.0)

Security

Checking security advisories...
Downloads
100
Dependents
3
Releases
2
Size
9KB

Deep Insights

📊
Stable downloads

40 downloads in the last 30 days (1/day). Volume is roughly flat compared to the previous period.

👤
Prolific maintainer

The primary maintainer publishes 66 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.

🪶
Minimal dependencies

Only 1 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.

📐
Compact crate

At 8KB, ferric-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.

🌟
Used by top crates

Notable dependents include ferric-web, ferric-serve, ferric-ffi. When high-quality crates choose ferric-tokenizer, it's a strong quality signal.

Health Breakdown

Maintenance 11/25

Recency, release consistency, active ratio

Quality 14/25

Yanked ratio, deps, size, maturity, features

Community 8/20

Reverse deps, ownership, ecosystem

Popularity 2/15

Downloads, momentum, growth trend

Documentation 10/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days
0/day avg
08/229/99/2710/1511/211/19

Top Dependents

Most downloaded crates that depend on ferric-tokenizer

Version Adoption

v0.3.0
79%
v0.1.0
21%

Release Timeline

2 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
2
Less
More

README

Loading README...

Maintainers

Dependencies
1
direct dependencies
Dependents
3
crates depend on ferric-tokenizer

Similar Crates