rustio rustio.net
61

tiktoken

v4.1.2 Stable

A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer

MIT OR Apache-2.0 Edition 2024
AlgorithmsText processing #tokenizer#openai#gpt#bpe#tiktoken

Quick Verdict

  • โœ“Actively maintained (updated 57d ago)
  • โœ“Stable API (4.x for 3+ years)
  • โœ“Trusted by 245 crates
  • โœ“Permissive license (MIT OR Apache-2.0)

Security

Checking security advisories...
Downloads
57.8K
Dependents
245
Releases
26
Size
5.3MB

Deep Insights

๐Ÿ“Š
Stable downloads

11.9K downloads in the last 30 days (398/day). Volume is roughly flat compared to the previous period.

๐Ÿ”—
Moderate adoption

245 crates depend on tiktoken. Reasonable ecosystem adoption, though not yet a core dependency.

๐Ÿ‘ค
Prolific maintainer

The primary maintainer publishes 226 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.

โœ…
Stable for 3+ years

The API has been stable (1.x) for over 3 years with 26 releases. This level of maturity means you can depend on it without worrying about breaking changes.

๐Ÿ“
Large crate size

5.3MB is relatively large. May impact download and compile times.

โš ๏ธ
Notable yanked versions

27% of releases have been yanked. Check version history before pinning.

๐ŸŒŸ
Used by top crates

Notable dependents include vtcode-commons, instructors, deeplossless, chunkedrs, opf-mlx. When high-quality crates choose tiktoken, it's a strong quality signal.

Health Breakdown

Maintenance 16/25

Recency, release consistency, active ratio

Quality 13/25

Yanked ratio, deps, size, maturity, features

Community 13/20

Reverse deps, ownership, ecosystem

Popularity 6/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
56/day avg-100%
02004006008008/229/99/2710/1511/211/19

Top Dependents

Version Adoption

v4.1.2
84%
v3.8.3
11%
v3.8.2
4%
v3.7.1
1%
v4.1.1
0%

Release Timeline

10 releasessince 2026
J
F
M
A
M
J
J
A
S
O
N
D
2026
10
Less
More

Feature Flags

default =["vocabs-all"]

parallelvocab-glm4vocab-glm5vocab-metavocab-qwenvocabs-all*vocab-qwen2vocab-zhipuvocab-llama3vocab-openaivocab-kimi_k2vocab-minimaxvocab-mistralvocab-deepseekvocab-moonshotvocab-p50k_basevocab-r50k_basevocab-minimax_m2vocab-mistral_v3vocab-o200k_basevocab-cl100k_basevocab-deepseek_v3

README

Loading README...

Maintainers

Dependencies
9
direct dependencies
Dependents
245
crates depend on tiktoken

Similar Crates