rustio rustio.net

rust_tokenizers vs sentencepiece-model vs rusty-tokenizer

Side-by-side comparison of Rust crates

Core Metrics

rust_tokenizerssentencepiece-modelrusty-tokenizer
Health Score494645
Total Downloads452.3K747.5K9
30d Downloads000
Dependents11790
Releases3451
Last Updated1105d ago731d ago1d ago
Age6y 7m2y 10m1d

Health Breakdown

rust_tokenizers
Maintenance
6
Quality
18
Community
9
Popularity
7
Documentation
9
sentencepiece-model
Maintenance
4
Quality
16
Community
9
Popularity
7
Documentation
10
rusty-tokenizer
Maintenance
13
Quality
15
Community
6
Popularity
1
Documentation
10

Technical Details

rust_tokenizerssentencepiece-modelrusty-tokenizer
Version8.1.10.1.40.1.0
Stable (≥1.0)✓ Yes✗ No✗ No
LicenseApache-2.0BSD-2-ClauseMIT
Dependencies1740
Crate Size134KB8KB6KB
Features420
Yanked %0.0%0.0%0.0%
Edition201820212024
MSRV———
Owners111

Quick Verdict

  • •rust_tokenizers leads with a health score of 49/100, but none of the options score above 80.
  • •sentencepiece-model has the most downloads (747.5K), suggesting wider adoption.
  • •rust_tokenizers is depended on by 117 crates — strongest ecosystem trust.
  • •⚠ rust_tokenizers, sentencepiece-model have not been updated in over a year.
  • •sentencepiece-model, rusty-tokenizer are pre-1.0 — API may change.