Decision Workspace
verbora-tokenizers vs panko vs english-tokenizer
Side-by-side comparison of Rust crates
50
verbora-tokenizers
experimentalv0.2.0
UAX #29 word, segment and sentence tokenizers for Rust
45
panko
experimentalv0.2.0
A small, zero-copy text tokenizer that crumbles strings into Words, Symbols, and Newlines.
43
english-tokenizer
experimentalv0.2.0
Regex-driven tokenizer that splits a sentence and type-tags each token (word, number, emoji, url, email, hashtag, mention, time, currency, and more).
Core Metrics
| verbora-tokenizers | panko | english-tokenizer | |
|---|---|---|---|
| Health Score | 50 | 45 | 43 |
| Total Downloads | 292 | 112 | 52 |
| 30d Downloads | 0 | 0 | 0 |
| Dependents | 16 | 0 | 0 |
| Releases | 2 | 2 | 2 |
| Last Updated | 50d ago | 120d ago | 94d ago |
| Age | 1m | 4m | 3m |
Health Breakdown
verbora-tokenizers
Maintenance
11
Quality
15
Community
9
Popularity
3
Documentation
12
panko
Maintenance
11
Quality
13
Community
6
Popularity
2
Documentation
13
english-tokenizer
Maintenance
12
Quality
14
Community
5
Popularity
2
Documentation
10
Technical Details
| verbora-tokenizers | panko | english-tokenizer | |
|---|---|---|---|
| Version | 0.2.0 | 0.2.0 | 0.2.0 |
| Stable (≥1.0) | ✗ No | ✗ No | ✗ No |
| License | MIT | MIT | MIT |
| Dependencies | 4 | 4 | 3 |
| Crate Size | 57KB | 13KB | 27KB |
| Features | 2 | 0 | 0 |
| Yanked % | 0.0% | 0.0% | 0.0% |
| Edition | 2024 | 2021 | 2021 |
| MSRV | 1.85 | — | — |
| Owners | 1 | 1 | 1 |
Links
Quick Verdict
- •verbora-tokenizers leads with a health score of 50/100, but none of the options score above 80.