Skip to content

Fix FIXME and avoid override zips with the same original name - #18

Merged
lfoppiano merged 10 commits into
mainfrom
fix/replace-fixme
Sep 14, 2026
Merged

lfoppiano merged 10 commits into
mainfrom
fix/replace-fixme

Conversation

@lfoppiano

@lfoppiano lfoppiano commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator
  • Fixed the filenames convention as {SCOPE}-REPACKAGED-{ORIGINAL_NAME}.warc.gz
  • Fixed issue with overriding zip files (reported in confusing names #13
  • Updated instructions for massaging the manifest.csv + download
    • Added a new command warc2zip manifest.csv --fetch

@lfoppiano lfoppiano linked an issue Sep 1, 2026 that may be closed by this pull request
5 tasks
…fied, a hex suffix (like the directories) is appended, if --limit is specified an _partial is appended as well, like for directories
@lfoppiano lfoppiano linked an issue Sep 2, 2026 that may be closed by this pull request
@lfoppiano lfoppiano changed the title Fix FIXME Fix FIXME and avoid override zips with the same original name Sep 4, 2026
@lfoppiano

lfoppiano commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

The hex appended will match the directory.

$ warc2zip s3://commoncrawl/crawl-data/CC-MAIN-2026-34/segments/1786091384908.68/crawldiagnostics/CC-MAIN-20260807101845-20260807131845-00000.warc.gz
Reading WARC: 27.0MB [00:04, 5.54MB/s]
Warning: 150 orphan group(s) without a response record, skipped
Created CC-MAIN-20260807101845-20260807131845-00000_44ca.zip: 3639 responses, 3789 requests, 3789 metadata records

$ warc2zip s3://commoncrawl/crawl-data/CC-MAIN-2026-34/segments/1786091384908.68/robotstxt/CC-MAIN-20260807101845-20260807131845-00000.warc.gz      
Reading WARC: 1.49MB [00:00, 1.61MB/s]
Created CC-MAIN-20260807101845-20260807131845-00000_2a58.zip: 833 responses, 833 requests, 0 metadata records

The directory name will be slighly different because it contains also the timestamp, so it will partially match the original zip file:

$ zipinfo -1 CC-MAIN-20260807101845-20260807131845-00000_2a58.zip | cut -d/ -f1 | sort -u 
CC-MAIN-20260807101845-20260807131845-00000_20260904T074146_2a58
$ zipinfo -1 CC-MAIN-20260807101845-20260807131845-00000_44ca.zip | cut -d/ -f1 | sort -u  
CC-MAIN-20260807101845-20260807131845-00000_20260904T074137_44ca

@lfoppiano lfoppiano mentioned this pull request Sep 4, 2026
@lfoppiano
lfoppiano marked this pull request as ready for review September 4, 2026 16:43
@lfoppiano
lfoppiano requested a review from wumpus September 4, 2026 16:44
@lfoppiano
lfoppiano merged commit 359a742 into main Sep 14, 2026
4 checks passed
@lfoppiano
lfoppiano deleted the fix/replace-fixme branch September 14, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FIXME all the FIXME in README.md confusing names

2 participants