rustio rustio.net

tokenizer vs text-splitter vs unicode-segmentation

Side-by-side comparison of Rust crates

Core Metrics

tokenizertext-splitterunicode-segmentation
Health Score396463
Total Downloads4.1K2.4M597.6M
30d Downloads000
Dependents01.0K22.3K
Releases26527
Last Updated2323d ago16d ago130d ago
Age6y 4m3y 5m11y 6m

Health Breakdown

tokenizer
Maintenance
3
Quality
17
Community
6
Popularity
4
Documentation
9
text-splitter
Maintenance
18
Quality
13
Community
13
Popularity
8
Documentation
12
unicode-segmentation
Maintenance
11
Quality
17
Community
16
Popularity
8
Documentation
11

Technical Details

tokenizertext-splitterunicode-segmentation
Version0.1.20.33.01.13.3
Stable (≥1.0)✗ No✗ No✓ Yes
LicenseBSD-3-ClauseMITMIT OR Apache-2.0
Dependencies2203
Crate Size17KB62KB112KB
Features341
Yanked %0.0%1.5%7.4%
Edition201820212018
MSRV—1.90.01.85.0
Owners115

Quick Verdict

  • •text-splitter leads with a health score of 64/100, but none of the options score above 80.
  • •unicode-segmentation has the most downloads (597.6M), suggesting wider adoption.
  • •unicode-segmentation is depended on by 22.3K crates — strongest ecosystem trust.
  • •⚠ tokenizer has not been updated in over a year.
  • •tokenizer, text-splitter are pre-1.0 — API may change.