rustio rustio.net
65

tiktoken-rs

v0.12.1 Growing

Library for encoding and decoding with the tiktoken library in Rust

MIT Edition 2024 MSRV 1.85.0
#ai#openai#gpt#bpe

Quick Verdict

  • โœ“Actively maintained (updated 15d ago)
  • !Pre-1.0: API may have breaking changes
  • โœ“Massive adoption (8.7K crates depend on it)
  • โœ“Permissive license (MIT)

Security

Checking security advisories...
Downloads
20.2M
Dependents
8.7K
Releases
36
Size
3.8MB

Deep Insights

๐Ÿ“ˆ
Strong growth momentum

4.1M downloads in the last 30 days (135.2K/day), up 62% from the previous period.

๐Ÿ”—
Widely adopted

8.7K crates depend on tiktoken-rs. Strong ecosystem adoption means battle-tested code and long-term stability.

โš ๏ธ
Single maintainer risk

tiktoken-rs has only 1 owner despite 8.7K dependents. This is a bus-factor concern โ€” consider the implications for long-term support.

๐Ÿ”ฌ
Pre-1.0 for over a year

Despite being 3+ years old, tiktoken-rs hasn't reached 1.0 yet. Expect potential API changes between versions.

๐Ÿ“
Large crate size

3.8MB is relatively large. May impact download and compile times.

๐ŸŒŸ
Used by top crates

Notable dependents include text-splitter, spider, llm-tokenizer, dynamo-tokenizers, toon-format. When high-quality crates choose tiktoken-rs, it's a strong quality signal.

Health Breakdown

Maintenance 18/25

Recency, release consistency, active ratio

Quality 12/25

Yanked ratio, deps, size, maturity, features

Community 12/20

Reverse deps, ownership, ecosystem

Popularity 8/15

Downloads, momentum, growth trend

Documentation 15/15

Docs, repo, license, metadata

Download Trend

Daily downloads ยท last 90 days
28K/day avg-100%
050K100K150K200K8/239/109/2810/1611/311/20

Top Dependents

Version Adoption

v0.7.0
30%
v0.9.1
23%
v0.12.0
20%
v0.11.0
14%
v0.6.0
12%

Release Timeline

10 releasessince 2023
J
F
M
A
M
J
J
A
S
O
N
D
2023
1
2024
2
2025
3
2026
4
Less
More

Feature Flags

dhat-heapasync-openai

README

Loading README...

Maintainers

Dependencies
8
direct dependencies
Dependents
8.7K
crates depend on tiktoken-rs

Similar Crates