ferric-tokenizer
v0.3.0 ExperimentalFerric tokenizer — a pure-Rust byte-level BPE (GPT-2/tiktoken family): lossless encode/decode, loads vocab.json + merges.txt. The prompt→ids→text glue for end-to-end LLM use.
Quick Verdict
- ✓Actively maintained (updated 56d ago)
- !Pre-1.0: API may have breaking changes
- ✓Tiny footprint (9KB, 1 deps)
- ✓Permissive license (Apache-2.0)
Security
Deep Insights
40 downloads in the last 30 days (1/day). Volume is roughly flat compared to the previous period.
The primary maintainer publishes 66 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.
Only 1 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.
At 8KB, ferric-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.
Notable dependents include ferric-web, ferric-serve, ferric-ffi. When high-quality crates choose ferric-tokenizer, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on ferric-tokenizer