Skip to content

Add examples of warc2zipped output #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lfoppiano merged 3 commits into from
Sep 14, 2026
Merged

Add examples of warc2zipped output #24

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3ae16b8
docs: Update README with new WARC examples and repackage naming conve…
lfoppiano Sep 8, 2026
eaa8fa4
docs: Add sidecar examples
lfoppiano Sep 8, 2026
4a77ace
Merge branch 'main' into feat/add-zipped-examples
lfoppiano Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,10 @@ This testing release of the software supports 2 output formats: flat and sidecar
Flat puts the metadata into a small number of large files, and sidecar instead
creates a lot of metadata files, one for each payload.

To see real output before installing anything, download one of the ready-made zips
listed under [WARC examples for testing](#warc-examples-for-testing): each example WARC
sits next to its `--format flat` conversion, its `--format sidecar` conversion and its `--metadata-only` conversion.

### Flat format (`--format flat`, default)

Counter-named payload files with global CSV/JSONL metadata. Optimized for bulk analysis.
Expand Down Expand Up @@ -424,7 +428,11 @@ Three caveats:

We prepared some smaller (~1GBytes or less) and interesting WARC files for testing: US Federal government websites, homepages, etc.
These files are in a [Huggingface bucket](https://huggingface.co/buckets/commoncrawl/warc2zip-examples) and the `warc2zip` commands
below read directly from that bucket. These examples are `--format flat` ... you can also try `--format sidecar`
below read directly from that bucket.
Next to each WARC the bucket also holds the zips `warc2zip` made from it: one per output format
(`--format flat` and `--format sidecar`) and a `--metadata-only` zip (every CSV, no payload files),
so you can look at the output without running anything.
The commands below use `--format flat` ... you can also try `--format sidecar`

Details:

Expand All @@ -434,6 +442,7 @@ Details:
```
warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz?download=true' --format flat
```
- or download the zip we made: [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.zip?download=true) (13 MBytes), sidecar format: [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.sidecar.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.sidecar.zip?download=true) (14 MBytes), metadata only: [500_RECORDS-REPACKAGE-CC-MAIN-2026-30.metadata-only.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/500_RECORDS-REPACKAGE-CC-MAIN-2026-30.metadata-only.zip?download=true) (0.8 MBytes)
- here is the warcinfo

```
Expand All @@ -453,6 +462,7 @@ Details:
```
warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz?download=true' --format flat --limit 1000
```
- or download the zip we made, which is the full conversion without the limit: [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.zip?download=true) (1 GByte), sidecar format: [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.sidecar.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.sidecar.zip?download=true) (1.1 GBytes), metadata only: [HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.metadata-only.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.metadata-only.zip?download=true) (35 MBytes)
- here is the warcinfo
```
software: pypi_cdx_toolkit/0.9.40.dev89+g53a7ef76c
Expand All @@ -467,6 +477,7 @@ warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/H
```
warc2zip 'https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz?download=true' --format flat --limit 1000
```
- or download the zip we made, which is the full conversion without the limit: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.zip?download=true) (434 MBytes), sidecar format: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.sidecar.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.sidecar.zip?download=true) (494 MBytes), metadata only: [IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.metdata-only.zip](https://huggingface.co/buckets/commoncrawl/warc2zip-examples/resolve/IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.metdata-only.zip?download=true) (36 MBytes)
- here is the warcinfo
```
software: pypi_cdx_toolkit/0.9.40.dev91+ga04800ea0
Expand Down Expand Up @@ -507,11 +518,11 @@ the whole files (also available as the manual "Long tests" workflow under Action

**Prefixes**: https://eotarchive.s3.amazonaws.com/ or `s3://eotarchive/`:

#### Heretrix/IA style warcs from EOT 2024
#### Heretrix/IA style WARCs from EOT 2024

- crawl-data/EOT-2024/segments/IA-000/warc/EOT24PRE-20240926172119-crawl804_EOT24PRE-20240926172119-00000.warc.gz

#### Nutch/CCF style warcs from EOT 2024
#### Nutch/CCF style WARCs from EOT 2024

- crawl-data/EOT-2024/segments/CC-000/warc/EOT-2024-REPACKAGE-CC-MAIN-2024-42-GOV-000000-001.warc.gz

Expand All @@ -521,11 +532,11 @@ the whole files (also available as the manual "Long tests" workflow under Action
- crawl-data/EOT-2024/segments/WR-000/warc/EOT24WR-0015_20250114215650265-8c53efcc-e2d-0_eot-http-energy-gov-eere-office-energy-efficiency-renewable-energy-manual-20250114215335-8c53efcc-e2d-screenshots-20250114215649547.warc.gz
- crawl-data/EOT-2024/segments/WR-000/warc/EOT24WR-0015_20250114215650265-8c53efcc-e2d-0_eot-http-energy-gov-eere-office-energy-efficiency-renewable-energy-manual-20250114215335-8c53efcc-e2d-text-20250114215649747.warc.gz

#### ArchiveTeam style megawarcs, EOT 2024 (warning: 10 gigabytes)
#### ArchiveTeam style megaWARCs, EOT 2024 (warning: 10 gigabytes)

- crawl-data/EOT-2024/segments/AT-000/warc/archiveteam_usgovernment_20250131232111_96ad506d_usgovernment_20250131232111_96ad506d.1738361595.megawarc.warc.gz

#### Heretrix-style arcs from EOT 2004 (arc is the predecessor to warc)
#### Heretrix-style ARCs from EOT 2004 (arc is the predecessor to warc)

- crawl-data/EOT-2004/segments/NARA-000/warc/NARA-PEOT-2004-20041014205819-00000-crawling009-c_NARA-PEOT-2004-20041014205819-00000-crawling009.archive.org.arc.gz

Expand All @@ -542,5 +553,5 @@ the whole files (also available as the manual "Long tests" workflow under Action
## TODO

- CC-NEWS - old, pre-upgrade, post-upgrade
- ArchiveTeam warcs (not megawarcs)
- ArchiveTeam WARCs (not megaWARCs)
- ArchiveIt old and new, for various flavors
Loading