rullama-datasets

v0.12.0 Experimental

Training data pipelines for rullama: JSONL I/O, tokenization, dedup, format conversion (Alpaca / ChatML / OpenAI / ShareGPT / Together), and quality stats. Shared by cloud (`rullama-finetune`) and local (`rullama-lora`, in the rullama workspace) fine-tuning.

MIT OR Apache-2.0 Edition 2024 MSRV 1.91

Artificial intelligence #tokenizer#dataset#jsonl#fine-tuning#llm-training

GitHub ↗ Docs ↗ crates.io ↗ Compare with…

Quick Verdict

✓Actively maintained (updated 12d ago)
!Pre-1.0: API may have breaking changes
✓Permissive license (MIT OR Apache-2.0)

Security

Checking security advisories...

Downloads

Dependents

Releases

Size

34KB

Deep Insights

📊

Download activity

28 downloads in the last 30 days (1/day avg).

👤

Prolific maintainer

The primary maintainer publishes 101 crates. This suggests deep Rust expertise and long-term commitment to the ecosystem.

📐

Compact crate

At 33KB, rullama-datasets is lightweight. Small crate size correlates with focused, well-scoped functionality.

Health Breakdown

Maintenance 12/25

Recency, release consistency, active ratio

Quality 15/25

Yanked ratio, deps, size, maturity, features

Community 6/20

Reverse deps, ownership, ecosystem

Popularity 2/15

Downloads, momentum, growth trend

Documentation 15/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days

0/day avg

Top Dependents

Most downloaded crates that depend on rullama-datasets

Version Adoption

v0.12.0

100%

Release Timeline

1 releasessince 2026

2026

Less

Feature Flags

datasets-fulldatasets-dedupdatasets-tiktokendatasets-hf-tokenizer

README

Loading README...

Maintainers

Josh Guyette

101 crates

Dependencies

direct dependencies