fetchdir

Every URL becomes a directory.

A fast, modular Rust library for downloading and extracting remote resources.

---

✨ What is fetchdir?

fetchdir turns remote resources into local directories.

Give it a URL. Get a folder.

https://example.com/project.tar.gz
                |
                v
          ┌───────────┐
          │ fetchdir  │
          └───────────┘
                |
                v
        ./local/project/

It automatically downloads, detects, and extracts supported formats.

---

🚀 Features

• ⚡ Fast streaming downloads
• 🦀 Native Rust implementation
• 🔄 Sync and async APIs
• 🧩 Feature-based architecture
• 📦 Archive extraction
• 🌱 Git repository support
• 🗂 URL → directory abstraction

---

📦 Supported formats

Feature	Format
zip	ZIP archives
tar	TAR archives
tar-gz	gzip compressed TAR
tar-xz	xz compressed TAR
tar-bz2	bzip2 compressed TAR
tar-zstd	zstd compressed TAR
git	Git repositories

---

Installation

Add fetchdir:

[dependencies]
fetchdir = "0.1"

Enable the fetchers you need:

[dependencies.fetchdir]
version = "0.1"
features = [
    "sync",
    "zip",
    "tar",
    "tar-gz",
    "git",
]

For async support:

[dependencies.fetchdir]
version = "0.1"
features = [
    "async",
    "zip",
    "tar",
    "tar-gz",
    "git",
]

---

Usage

Sync

use std::path::Path;

fn main() -> anyhow::Result<()> {
    fetchdir::fetch_dir(
        "https://example.com/archive.tar.gz",
        Path::new("./output"),
    )?;

    Ok(())
}

---

Async

use std::path::Path;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    fetchdir::async_fetch_dir(
        "https://example.com/archive.zip",
        Path::new("./output"),
    )
    .await?;

    Ok(())
}

---

How it works

fetchdir follows a simple pipeline:

                 URL
                  |
                  v
          ┌─────────────┐
          │  Downloader │
          └─────────────┘
                  |
                  v
          ┌─────────────┐
          │ Temp Storage│
          └─────────────┘
                  |
                  v
          ┌─────────────┐
          │ Extractor   │
          └─────────────┘
                  |
                  v
             Directory

Each format has its own isolated fetcher.

Adding a new format does not require changing the core API.

---

Feature system

fetchdir is fully modular.

Example:

features = [
    "async",
    "zip",
    "git",
]

Only enabled fetchers are compiled.

No unnecessary dependencies. No unused code.

---

Git support

Clone repositories directly:

fetchdir::fetch_dir(
    "https://github.com/example/project.git",
    "./project".as_ref(),
)?;

Git fetching uses native Rust bindings.

No external git command required.

---

Design goals

fetchdir is built around:

• simplicity
• performance
• portability
• extensibility

The goal:

Any downloadable resource should be usable like a local directory.

---

Roadmap

Possible future additions:

• Initial features
• HTTP directory indexes
• GitHub release assets
• S3 support
• checksum verification
• progress callbacks
• parallel extraction
• caching

---

License

Licensed under either:

• MIT License
• Apache License 2.0

at your option.

---

Made with 🦀 Rust