english-tokenizer
v0.2.0 ExperimentalRegex-driven tokenizer that splits a sentence and type-tags each token (word, number, emoji, url, email, hashtag, mention, time, currency, and more).
Quick Verdict
- !Pre-1.0: API may have breaking changes
- โTiny footprint (27KB, 3 deps)
- โPermissive license (MIT)
Security
Deep Insights
4 downloads in the last 30 days (0/day). Volume is roughly flat compared to the previous period.
The primary maintainer publishes 113 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.
Only 3 direct dependencies. Lean dependency tree means faster builds and lower supply chain risk.
At 26KB, english-tokenizer is lightweight. Small crate size correlates with focused, well-scoped functionality.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata