tiktoken
v4.1.2 StableA high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
Quick Verdict
- โActively maintained (updated 57d ago)
- โStable API (4.x for 3+ years)
- โTrusted by 245 crates
- โPermissive license (MIT OR Apache-2.0)
Security
Deep Insights
11.9K downloads in the last 30 days (398/day). Volume is roughly flat compared to the previous period.
245 crates depend on tiktoken. Reasonable ecosystem adoption, though not yet a core dependency.
The primary maintainer publishes 226 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.
The API has been stable (1.x) for over 3 years with 26 releases. This level of maturity means you can depend on it without worrying about breaking changes.
5.3MB is relatively large. May impact download and compile times.
27% of releases have been yanked. Check version history before pinning.
Notable dependents include vtcode-commons, instructors, deeplossless, chunkedrs, opf-mlx. When high-quality crates choose tiktoken, it's a strong quality signal.
Health Breakdown
Recency, release consistency, active ratio
Yanked ratio, deps, size, maturity, features
Reverse deps, ownership, ecosystem
Downloads, momentum, growth trend
Docs, repo, license, metadata
Download Trend
Top Dependents
Most downloaded crates that depend on tiktoken
Version Adoption
Release Timeline
Feature Flags
default =["vocabs-all"]