Modernise repository to open-source standards

[siligam]

Summary

Brings the repository up to open-source standards, fixes several structural issues identified in an architecture review, and adds two analysis improvements for real-world HPC pool comparisons.

Changes

Open-source standards

• pyproject.toml — replaces legacy setup.py (PEP 517/518 compliant, resolves pip deprecation warning)
• Metadata fixes: license MIT → GPL-3.0, Development Status classifier, url GitLab → GitHub
• CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, CHANGELOG.md
• .github/ISSUE_TEMPLATE/ — bug report and feature request templates
• .github/PULL_REQUEST_TEMPLATE.md — PR template
• .github/workflows/ci.yml — GitHub Actions CI running tests on Python 3.9–3.12
• .gitignore — expanded with standard Python, editor, and OS entries

Test suite (59 tests)

• tests/test_checksums.py — split, ignore_re, Results, hasher, stats, scanner, get_files
• tests/test_analyse.py — read_csv, compare, compare_compact, merge, directory_map, TF-IDF folder mapping, suspicious pair detection
• tests/test_cli.py — all four CLI commands via CliRunner
• tests/test_imohash_false_negatives.py — documents the known imohash sampling limitation (see below)

Documentation

• Added "How it works" section to README with the file association classification table (identical / modified / renamed / unique) and an explanation of the folder mapping algorithm and --threshold

Architecture fixes

• Deleted ptool/utils.py — dead code; referenced a non-existent ptool_config.yaml and imported yaml (undeclared dependency)
• Removed duplicate cli() from checksums.py — the registered entry point is ptool.cli:cli; the standalone command was never reachable
• Fixed prepare_rsync double-open bug — replaced open(outfile, "w") with outfile.write() on the Click-provided handle; fixed hardcoded "sync_cmd.sh" in the success message
• Clarified sanitise() docstring — documents the AWI shared-filesystem rationale for the awi.de special case
• Fixed DataFrame metadata attributes — replaced monkey-patching (df.site = ...) with a _SnapshotFrame subclass using pandas _metadata + _constructor so site and filename survive merges, filters, and copies
• Fixed docstring escape sequences in split() — \, → \\, (Python 3.12 DeprecationWarning)
• Replaced groupby.apply in merge() — eliminated FutureWarning by rewriting the max-association algorithm using groupby.size() + idxmax()

Analysis improvements

TF-IDF weighted folder association

• Folder mapping previously used raw file-match counts, which favoured directories full of generic sequential filenames (e.g. my_list00001.out) over directories sharing a single rare, specific file
• Now weights each filename match by its IDF score (log(n_dirs / n_dirs_containing_fname)), so rare filenames anchor directory mappings and ubiquitous ones are downweighted
• Verified with a dedicated test: a directory sharing one specific file + one generic correctly maps to its true counterpart rather than a directory with three generic matches

Spurious folder mapping detection (_suspicious_pairs)

• When a folder pair shares filenames but every matched file differs in content (0 identical, 100% modified rate), the mapping is likely spurious — e.g. MPI partition files generated for different core counts at different sites
• _suspicious_pairs() detects these pairs and summary() marks them with [!] in the directory map table with an explanatory note
• Three tests cover: all-modified pairs flagged, mixed pairs (some identical) not flagged, unique-only directories not flagged

Old CSV format compatibility

• read_csv() now handles CSVs with the legacy md5 column name (renamed to checksum on load) for backward compatibility with older snapshots

imohash false-negative awareness tests

• imohash samples only 3×16 KB windows (start, middle, end) for files ≥ 128 KB, so two files that differ only in the unsampled region produce the same digest — a known limitation
• Three deterministic tests document this: (1) the false-negative exists and is reproducible for large files, (2) small files are fully read and immune, (3) different file sizes never collide because size is encoded in the digest

Test plan

• 59 tests pass with zero warnings (conda run -n ptool python -m pytest tests/ -v)
• CI workflow triggers on push/PR and all matrix jobs pass
• pip install -e ".[dev]" works without deprecation warnings
• Issue templates appear correctly on GitHub when creating a new issue

🤖 Generated with Claude Code