spider

v2.48.8 Stable

A web crawler and scraper, building blocks for data curation workloads.

MIT Edition 2021

Command line utilities Web programming #spider#crawler#scraper

GitHub ↗ Docs ↗ crates.io ↗ Compare with…

Quick Verdict

✓Actively maintained (updated 2d ago)
✓Stable API (2.x for 8+ years)
✓Massive adoption (5.0K crates depend on it)
!Heavy dependency tree (80 direct deps)
✓Permissive license (MIT)

Security

Checking security advisories...

Downloads

2.0M

Dependents

5.0K

Releases

1952

Size

455KB

Deep Insights

📈

Strong growth momentum

16.2K downloads in the last 30 days (540/day), up 118% from the previous period.

🔗

Widely adopted

5.0K crates depend on spider. Strong ecosystem adoption means battle-tested code and long-term stability.

✅

Stable for 8+ years

The API has been stable (1.x) for over 8 years with 1952 releases. This level of maturity means you can depend on it without worrying about breaking changes.

📦

Heavy dependency tree

80 direct dependencies. Consider the impact on compile times and supply chain complexity.

🌟

Used by top crates

Notable dependents include spider_cli, spider_worker, spider_transformations, website_crawler, accessibility-rs. When high-quality crates choose spider, it's a strong quality signal.

Health Breakdown

Maintenance 19/25

Recency, release consistency, active ratio

Quality 18/25

Yanked ratio, deps, size, maturity, features

Community 17/20

Reverse deps, ownership, ecosystem

Popularity 8/15

Downloads, momentum, growth trend

Documentation 13/15

Docs, repo, license, metadata

Download Trend

Daily downloads · last 90 days

349/day avg+102%

Top Dependents

Most downloaded crates that depend on spider

spider_transformations

swiftide-integrations

Version Adoption

v1.99.37

28%

v2.37.159

20%

v1.99.5

18%

v1.1.2

17%

v2.27.8

17%

Release Timeline

1952 releases57 yankedsince 2018

2018

2019

2020

2021

2022

2023

329

2024

659

2025

626

2026

241

Less

More|

Yanked

Feature Flags

default =["basic", "io_uring", "tcp_fastopen", "splice", "numa", "zero_copy"]

fscmdcrondiskglobnuma*simdsynctimewarcwreqagentbasic*bloomcachehedgeregexserdeservosmartsocksspoofchromecowboygeminiopenaisearchsplice*__basicadblockbalancecontrolcookiesheaderssitemaptracingagent_fsdisk_awsencodingfirewallio_uring*llm_jsonbasic_tlscache_memdns_cachewebdriverzero_copy*agent_fullchrome_cpuetag_cachelightpandarate_limitchrome_simdflexbuffersinline-moreremote_addrsearch_bingagent_chromeagent_openaiagent_skillscache_geminicache_openaih2_multiplexreal_browserreqwest_jsonrobots_cachesearch_bravespider_cloudtcp_fastopen*tokio_io_stdua_generatorauto_throttlecache_requestchrome_headeddecentralizedsearch_serpersearch_tavilychrome_stealthfull_resourceswebdriver_edgeagent_skills_s3agent_webdriverdisk_native_tlsopenai_slim_fitchrome_interceptrequest_coalescewebdriver_chromewebdriver_headedagent_search_bingchrome_screenshotchrome_store_pageextra_informationparallel_backendspriority_frontierreqwest_multipartwebdriver_firefoxwebdriver_stealthagent_search_bravereqwest_native_tlsreqwest_rustls_tlsrquest_hickory_dnsagent_search_serperagent_search_tavilycache_chrome_hybridchrome_headless_newchrome_remote_cachereqwest_hickory_dnsadaptive_concurrencychrome_serde_stackerwebdriver_screenshotchrome_tls_connectiondecentralized_headersparallel_backends_fullcache_chrome_hybrid_memchrome_remote_cache_memreqwest_native_tls_alpnchrome_remote_cache_diskpage_error_status_detailsreqwest_native_tls_vendoredstring_interner_bucket_backendstring_interner_buffer_backendstring_interner_string_backendreqwest_native_tls_manual_rootsreqwest_native_tls_native_rootsreqwest_native_tls_webpki_roots