tokenizer-lib
v1.6.0 GrowingTokenization utilities for building parsers in Rust
Quick Verdict
- ✕Not updated for 2+ years
- ✓Stable API (1.x for 5+ years)
- ✓Tiny footprint (7KB, 1 deps)
- ✓Permissive license (MIT)
Security
Deep Insights
533 downloads in the last 30 days (18/day), up 30% from the previous period.
The primary maintainer publishes 52 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.
The API has been stable (1.x) for over 5 years with 15 releases. This level of maturity means you can depend on it without worrying about breaking changes.
Only 1 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.
At 7KB, tokenizer-lib is lightweight. Small crate size correlates with focused, well-scoped functionality.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on tokenizer-lib
Version Adoption
Release Timeline
Feature Flags
default =["parallel", "buffered", "generator"]
