rustio rustio.net

simd-tokenizer vs gproxy-tokenize vs llm-tokenizer

Side-by-side comparison of Rust crates

Core Metrics

simd-tokenizergproxy-tokenizellm-tokenizer
Health Score515065
Total Downloads1.2K8092.0M
30d Downloads000
Dependents0044
Releases34115
Last Updated28d ago36d ago1d ago
Age1m2m9m

Health Breakdown

simd-tokenizer
Maintenance
12
Quality
15
Community
5
Popularity
4
Documentation
15
gproxy-tokenize
Maintenance
15
Quality
16
Community
6
Popularity
3
Documentation
10
llm-tokenizer
Maintenance
22
Quality
14
Community
11
Popularity
8
Documentation
10

Technical Details

simd-tokenizergproxy-tokenizellm-tokenizer
Version0.1.22.9.101.9.0
Stable (≥1.0)✗ No✓ Yes✓ Yes
LicenseApache-2.0MITApache-2.0
Dependencies51124
Crate Size35KB1.9MB242KB
Features350
Yanked %0.0%0.0%0.0%
Edition202120242021
MSRV1.85——
Owners111

Quick Verdict

  • •llm-tokenizer leads with a health score of 65/100, but none of the options score above 80.
  • •simd-tokenizer is pre-1.0 — API may change.