diff --git a/AGENTS.md b/AGENTS.md index 397c07634..a3eca825d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -181,13 +181,14 @@ Key principles: | File | Purpose | |------------------------------------|------------------------------------| -| `src/vip/cli/` | CLI entry point, one module per command: `version.py`, `verify.py` (including `--basic` to skip `@slow`-tagged scenarios), `cleanup.py` (Connect content + orphaned Workbench sessions via `--workbench-url`), `install.py` (install, uninstall), `auth.py`, `report.py`, `status.py`, `scaffold.py`; `app.py` builds the parser and holds `main()` and the `--version` flag; `_common.py` holds helpers shared by several commands; `__init__.py` re-exports the names callers import from `vip.cli`, but a selftest must patch a helper on the submodule that calls it | +| `src/vip/cli/` | CLI entry point, one module per command: `version.py`, `verify.py` (including `--basic` to skip `@slow`-tagged scenarios), `cleanup.py` (Connect content + orphaned Workbench sessions via `--workbench-url`), `install.py` (install, uninstall), `auth.py`, `report.py`, `bundle.py`, `status.py`, `scaffold.py`; `app.py` builds the parser and holds `main()` and the `--version` flag; `_common.py` holds helpers shared by several commands; `__init__.py` re-exports the names callers import from `vip.cli`, but a selftest must patch a helper on the submodule that calls it | +| `src/vip/bundle/` | `vip bundle` internals: `summary.py` (allowlisted config summary and the secret list, with a selftest that fails on an unclassified config field), `archive.py` (allowlisted report files, encoded-form secret scan, atomic zip write), and `stamp.py` (the `_output/.vip-render.json` stamp that `vip report` writes, so that a bundle holds only rendered files made from its own `results.json`) | | `src/vip/config.py` | TOML config loader and dataclasses (includes `[proxy]` → `ProxyConfig`) | | `src/vip/proxy.py` | Single source of truth for outbound-proxy resolution. `ProxyConfig` + `build_proxy_map` (mirrors httpx's `get_environment_proxies`, incl. NO_PROXY formatting), `build_mounts` (per-scheme `HTTPTransport` mounts that keep `verify`), `proxy_for_url` (httpx-identical most-specific-pattern selection, used by non-httpx probes), `playwright_proxy` (renders a Playwright `launch(proxy=)` dict). Every HTTP egress path routes through this so VIP never diverges from httpx's own env-proxy behavior — see "Outbound proxy support" below | | `src/vip/auth/` | Interactive and headless browser authentication for OIDC providers, split into `browser.py` (Chromium launch, `InteractiveAuthSession`), `cache.py`, `flows.py` (`start_interactive_auth`, `start_headless_auth`), `sso.py`, `workbench.py`, `scheme.py` (`resolve_url_scheme`) and `apikey.py` (Connect API-key minting); `__init__.py` re-exports the names callers import from `vip.auth`, but a selftest must patch a helper on the submodule that calls it; `authenticated_page` opens a headless page from a cached auth session for `vip cleanup --workbench-url`; `auth_cache_path()` is the single source of truth for the `.vip-auth-cache.json` location (both `plugin/auth.py` and `cli/cleanup.py` must use it), and `_load_cached_auth` probes Workbench before trusting a cached session; `refresh_auth_cache_from_storage_state` writes a live context's cookies back over a cache whose session has been invalidated (atomic, 0600, existing caches only); both cache writers drop Entra's long-lived `ESTSAUTHPERSISTENT` cookie (`_CACHE_DROPPED_COOKIES`) because Entra sets it even when "Stay signed in?" is answered No | | `src/vip/idp.py` | IdP login form strategies for headless auth (Keycloak, Okta, Snowflake, Entra ID) | | `src/vip/attest.py` | The two skip helpers (`not_applicable`, `unproven`) that record whether a skipped check was out of scope or simply never verified | -| `src/vip/errors.py` | The `VipError` exception hierarchy for user-facing CLI failures (`ConfigError`, `AuthError`, `ProductUnreachableError`, `InstallError`, `ReportError`, `AuthConfigError`, `AuthTimeoutError`), each carrying an `exit_code`; raise one instead of calling `sys.exit` from command code — see "Error handling" below | +| `src/vip/errors.py` | The `VipError` exception hierarchy for user-facing CLI failures (`ConfigError`, `AuthError`, `ProductUnreachableError`, `InstallError`, `ReportError`, `BundleError`, `AuthConfigError`, `AuthTimeoutError`), each carrying an `exit_code`; raise one instead of calling `sys.exit` from command code — see "Error handling" below | | `src/vip/plugin/` | pytest plugin, split into `options.py` (`pytest_addoption`), `configure.py` (`pytest_configure`: markers, including `slow` used by `verify --basic`, and warning filters), `auth.py` (the `--interactive-auth`/`--headless-auth` login), `selection.py` (deselection and version auto-skip), `results.py` (JSON report output), `terminal.py` (concise display, heartbeat) and `state.py` (module-level mutable state; read and write it as `state.`); `__init__.py` re-exports every hook, since pytest registers only the package module, plus the names callers import from `vip.plugin`, but a selftest must patch a helper on the submodule that calls it; `pytest_configure` also registers `vip.fixtures` as its own named plugin (`"vip-fixtures"`) so core fixtures resolve regardless of directory ancestry — see that module's docstring | | `src/vip/fixtures.py` | VIP's core pytest fixtures and shared BDD "Given" steps (`vip_config`, `connect_client`, `browser_context_args`, etc.), registered by `vip.plugin.pytest_configure` rather than defined in a `conftest.py` — pytest scopes `conftest.py` fixtures by directory ancestry, which would hide them from extension directories | | `src/vip/version.py` | `ProductVersion` parsing/comparison for `min_version` gating; `MINIMUM_SUPPORTED_POSIT_TEAM` support floor (powers `vip version`) | diff --git a/README.md b/README.md index 9e8eb5885..2ed80877a 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,7 @@ content from Connect. | `vip status` | Quick health check for each configured product | | `vip cleanup` | Delete VIP `_vip_test` content from Connect | | `vip report` | Render the HTML report from test results (requires [Quarto CLI](https://quarto.org/docs/download/)) | +| `vip bundle` | Write a zip of the report output to share. It leaves out vip.toml and the auth cache, and checks each file for known credentials. | | `vip auth` | Authentication tools (e.g. mint Connect API keys) | | `vip version` | Print the vip version and the minimum supported Posit Team version | | `vip --version` | Print the installed vip version | diff --git a/selftests/test_bundle_archive.py b/selftests/test_bundle_archive.py new file mode 100644 index 000000000..3ce53f23f --- /dev/null +++ b/selftests/test_bundle_archive.py @@ -0,0 +1,536 @@ +"""Tests for the ``vip bundle`` archive builder and secret scan.""" + +from __future__ import annotations + +import json +import os +import shutil +import zipfile +from pathlib import Path + +import pytest + +from vip.bundle.archive import ( + collect_members, + find_secret_hits, + short_secret_labels, + write_zip, +) +from vip.bundle.stamp import STAMP_NAME, record_rendered, results_digest +from vip.errors import BundleError + +_SIDECARS = ("failures.json", "junit.xml", "results.sarif") +_RENDERED = ("index.html", "details.html", "vip-report.pdf") +_RESULTS = json.dumps({"deployment_name": "Acme", "results": []}) + + +def _write(path: Path, text: str, mtime: float) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + os.utime(path, (mtime, mtime)) + + +def _seed( + report: Path, + sidecars=_SIDECARS, + rendered=_RENDERED, + results: str = _RESULTS, + results_mtime: float = 1_000_000.0, +) -> None: + """Write a report directory as ``vip verify`` and ``vip report`` leave it. + + Each sidecar is one second newer than results.json. The render stamp + names each rendered file and the digest of results.json. + """ + _write(report / "results.json", results, results_mtime) + for name in sidecars: + _write(report / name, f"content of {name}", results_mtime + 1) + for name in rendered: + _write(report / "_output" / name, f"content of {name}", results_mtime + 2) + if rendered: + record_rendered(report / "_output", results_digest(results.encode()), rendered) + + +def _all_members() -> set[str]: + return ( + {"report/results.json"} + | {f"report/{n}" for n in _SIDECARS} + | {f"report/_output/{n}" for n in _RENDERED} + ) + + +def test_collects_only_allowlisted_files(tmp_path): + report = tmp_path / "report" + _seed(report) + for name in ("notes.txt", "index.qmd", "_output/search.json", "_output/site_libs/b.css"): + _write(report / name, "x", 2_000_000.0) + got = collect_members(report / "results.json", report) + assert set(got.members) == _all_members() + assert got.missing == [] + assert got.left_out == [] + assert got.warnings == [] + assert got.sources["report/failures.json"] == report / "failures.json" + + +def test_optional_files_absent_is_not_an_error(tmp_path): + report = tmp_path / "report" + _seed(report, sidecars=(), rendered=()) + got = collect_members(report / "results.json", report) + assert set(got.members) == {"report/results.json"} + assert "report/failures.json" in got.missing + assert "report/_output/vip-report.pdf" in got.missing + assert any("Run 'vip report'" in w for w in got.warnings) + + +def test_missing_results_raises(tmp_path): + with pytest.raises(BundleError, match=r"results\.json"): + collect_members(tmp_path / "report/results.json", tmp_path / "report") + + +@pytest.mark.parametrize( + "name", + ["vip.toml", ".vip-auth-cache.json", ".vip-auth-cache.meta.json", ".vip-install.json"], +) +def test_results_cannot_name_a_credential_file(tmp_path, name): + path = tmp_path / name + path.write_text('{"results": []}', encoding="utf-8") + with pytest.raises(BundleError, match="is not a results file"): + collect_members(path, tmp_path / "report") + + +@pytest.mark.parametrize("text", ["api_key = 'x'", "[]", '{"no_results": 1}']) +def test_results_must_be_a_vip_results_file(tmp_path, text): + path = tmp_path / "results.json" + path.write_text(text, encoding="utf-8") + with pytest.raises(BundleError, match="is not a VIP results file"): + collect_members(path, tmp_path / "report") + + +def test_render_from_other_results_is_left_out(tmp_path): + # The user ran vip verify again but not vip report. + report = tmp_path / "report" + _seed(report) + _write(report / "results.json", json.dumps({"results": [{"new": 1}]}), 3_000_000.0) + for name in _SIDECARS: + os.utime(report / name, (3_000_001.0, 3_000_001.0)) + got = collect_members(report / "results.json", report) + assert not any(name.startswith("report/_output/") for name in got.members) + assert {item["name"] for item in got.left_out} == {f"report/_output/{n}" for n in _RENDERED} + assert all(item["reason"].startswith("was not made from") for item in got.left_out) + assert sum("Run 'vip report' again" in w for w in got.warnings) == 3 + + +def test_copied_results_with_an_old_time_does_not_let_in_an_old_render(tmp_path): + # vip report --results copies with shutil.copy2, which keeps the old file + # time. If the PDF step then fails, the PDF of the earlier run stays and + # is newer than results.json. Only the stamp shows that it is foreign. + report = tmp_path / "report" + _seed(report, results_mtime=2_000_000.0) + ci = tmp_path / "ci-results.json" + _write(ci, json.dumps({"results": [{"ci": 1}]}), 1_500_000.0) + shutil.copy2(ci, report / "results.json") + record_rendered(report / "_output", results_digest(ci.read_bytes()), ("index.html",)) + got = collect_members(report / "results.json", report) + assert "report/_output/index.html" in got.members + assert "report/_output/vip-report.pdf" not in got.members + assert "report/_output/details.html" not in got.members + + +def test_results_outside_report_dir_keeps_render_with_the_same_content(tmp_path): + report = tmp_path / "report" + _seed(report) + other = tmp_path / "out" / "run1.json" + _write(other, _RESULTS, 1_000_000.0) + got = collect_members(other, report) + assert {f"report/_output/{n}" for n in _RENDERED} <= set(got.members) + + +def test_results_outside_report_dir_leaves_out_other_render(tmp_path): + report = tmp_path / "report" + _seed(report) + other = tmp_path / "other" / "results.json" + _write(other, json.dumps({"results": [{"other": 1}]}), 2_000_000.0) + got = collect_members(other, report) + assert set(got.members) == {"report/results.json"} + assert any(f"was not made from {other}" in w for w in got.warnings) + + +def test_render_without_a_stamp_is_left_out(tmp_path): + # A render from a vip version that writes no stamp, or a render that failed. + report = tmp_path / "report" + _seed(report) + (report / "_output" / STAMP_NAME).unlink() + got = collect_members(report / "results.json", report) + assert not any(name.startswith("report/_output/") for name in got.members) + + +def test_stale_sidecar_is_left_out(tmp_path): + # failures.json from an earlier run that failed. The new run passed, so + # vip verify did not write a new failures.json. + report = tmp_path / "report" + _seed(report) + os.utime(report / "failures.json", (999_000.0, 999_000.0)) + got = collect_members(report / "results.json", report) + assert "report/failures.json" not in got.members + assert {"name": "report/failures.json", "reason": "is older than results.json"} in ( + got.left_out + ) + assert any(str(report / "failures.json") in w for w in got.warnings) + + +@pytest.mark.parametrize("name", ["index.html", "vip-report.pdf"]) +def test_rendered_file_that_changed_after_the_stamp_is_left_out(tmp_path, name): + # A copy of an older file over the stamped file keeps the name. + report = tmp_path / "report" + _seed(report) + _write(report / "_output" / name, "content of an older render", 1_000_002.0) + got = collect_members(report / "results.json", report) + assert f"report/_output/{name}" not in got.members + reason = "changed after 'vip report' made it" + assert {"name": f"report/_output/{name}", "reason": reason} in got.left_out + + +@pytest.mark.parametrize("absent", ["index.html", "details.html"]) +def test_pdf_without_both_html_pages_is_left_out(tmp_path, absent): + # The scan cannot read the text in the PDF. The HTML pages hold the same text. + report = tmp_path / "report" + _seed(report) + (report / "_output" / absent).unlink() + got = collect_members(report / "results.json", report) + assert "report/_output/vip-report.pdf" not in got.members + assert "report/_output/vip-report.pdf" not in got.sources + reason = "cannot be checked without the HTML pages of the same render" + assert {"name": "report/_output/vip-report.pdf", "reason": reason} in got.left_out + + +def _seed_run_pair(report: Path, results_run: str, failures_run: str) -> None: + """Write results.json and failures.json with a ``generated_at`` value each.""" + results = json.dumps({"generated_at": results_run, "results": []}) + _seed(report, sidecars=(), rendered=(), results=results) + failures = json.dumps({"generated_at": failures_run, "failures": []}) + _write(report / "failures.json", failures, 1_000_001.0) + + +def test_failures_json_of_the_same_run_stays_when_results_is_newer(tmp_path): + # An edit of results.json gives it a newer file time than failures.json. + report = tmp_path / "report" + _seed_run_pair(report, "2026-11-12T14:00:00+00:00", "2026-11-12T14:00:00+00:00") + os.utime(report / "results.json", (1_000_500.0, 1_000_500.0)) + got = collect_members(report / "results.json", report) + assert "report/failures.json" in got.members + assert got.left_out == [] + + +def _sidecar_text(name: str, run: str) -> str: + """Return a junit.xml or a results.sarif that holds *run* in the form that vip writes.""" + if name == "junit.xml": + return ( + '' + f'' + ) + if name == "failures.json": + return json.dumps({"generated_at": run, "failures": []}) + return json.dumps({"version": "2.1.0", "runs": [{"properties": {"generated_at": run}}]}) + + +@pytest.mark.parametrize("name", ["junit.xml", "results.sarif"]) +def test_sidecar_with_the_run_time_stays_when_results_is_newer(tmp_path, name): + # An edit of results.json gives it a newer file time than the sidecar. + report = tmp_path / "report" + run = "2026-11-12T14:00:00+00:00" + results = json.dumps({"generated_at": run, "results": []}) + _seed(report, sidecars=(), rendered=(), results=results, results_mtime=1_000_500.0) + _write(report / name, _sidecar_text(name, run), 1_000_001.0) + got = collect_members(report / "results.json", report) + assert f"report/{name}" in got.members + assert got.left_out == [] + + +@pytest.mark.parametrize("name", ["junit.xml", "results.sarif"]) +@pytest.mark.parametrize("sidecar_mtime", [1_000_001.0, 1_000_900.0]) +def test_sidecar_with_another_run_time_is_left_out(tmp_path, name, sidecar_mtime): + # The file time does not matter: a copy can make an old sidecar newer. + report = tmp_path / "report" + results = json.dumps({"generated_at": "2026-11-12T14:00:00+00:00", "results": []}) + _seed(report, sidecars=(), rendered=(), results=results, results_mtime=1_000_500.0) + _write(report / name, _sidecar_text(name, "2026-11-12T13:00:00+00:00"), sidecar_mtime) + got = collect_members(report / "results.json", report) + reason = "is from a different run than results.json" + assert got.left_out == [{"name": f"report/{name}", "reason": reason}] + + +@pytest.mark.parametrize("name", _SIDECARS) +@pytest.mark.parametrize("sidecar_mtime", [1_000_001.0, 1_000_900.0]) +@pytest.mark.parametrize("text", ['{"runs": [{"note": "2026-11-12T14:00:00+00:00"}]}', "{not json"]) +def test_sidecar_with_no_run_time_is_left_out_when_results_has_one( + tmp_path, name, sidecar_mtime, text +): + # If results.json has a run time, a sidecar with none is not from this + # run, even if it is newer, for example a later pytest --junitxml. The + # text of the value in some other place of the file does not count. + report = tmp_path / "report" + results = json.dumps({"generated_at": "2026-11-12T14:00:00+00:00", "results": []}) + _seed(report, sidecars=(), rendered=(), results=results, results_mtime=1_000_500.0) + _write(report / name, text, sidecar_mtime) + got = collect_members(report / "results.json", report) + reason = "has no run time, so it is not from the run of results.json" + assert got.left_out == [{"name": f"report/{name}", "reason": reason}] + + +@pytest.mark.parametrize("name", _SIDECARS) +@pytest.mark.parametrize(("sidecar_mtime", "included"), [(1_000_001.0, True), (999_999.0, False)]) +def test_with_no_run_time_in_results_the_file_time_decides(tmp_path, name, sidecar_mtime, included): + report = tmp_path / "report" + _seed(report, sidecars=(), rendered=()) + _write(report / name, _sidecar_text(name, "2026-11-12T14:00:00+00:00"), sidecar_mtime) + got = collect_members(report / "results.json", report) + assert (f"report/{name}" in got.members) is included + + +def test_failures_json_of_another_run_is_left_out_when_it_is_newer(tmp_path): + report = tmp_path / "report" + _seed_run_pair(report, "2026-11-12T14:00:00+00:00", "2026-11-12T15:00:00+00:00") + got = collect_members(report / "results.json", report) + assert "report/failures.json" not in got.members + assert got.left_out == [ + {"name": "report/failures.json", "reason": "is from a different run than results.json"} + ] + + +@pytest.mark.skipif(os.name == "nt", reason="symlinks need extra rights on Windows") +def test_symlinks_are_left_out(tmp_path): + report = tmp_path / "report" + _seed(report, sidecars=(), rendered=()) + secret = tmp_path / "vip.toml" + secret.write_text("api_key = 'x'", encoding="utf-8") + (report / "failures.json").symlink_to(secret) + (report / "_output").mkdir() + (report / "_output" / "index.html").symlink_to(secret) + got = collect_members(report / "results.json", report) + assert set(got.members) == {"report/results.json"} + assert sum("is a link" in w for w in got.warnings) == 2 + + +@pytest.mark.skipif(os.name == "nt", reason="symlinks need extra rights on Windows") +def test_links_to_files_in_the_same_directory_are_left_out(tmp_path): + report = tmp_path / "report" + _seed(report, sidecars=(), rendered=("index.html",)) + _write(report / "other.json", "x", 1_000_001.0) + (report / "failures.json").symlink_to(report / "other.json") + stamp = report / "_output" / STAMP_NAME + stamp.rename(report / "_output" / "real-stamp.json") + stamp.symlink_to(report / "_output" / "real-stamp.json") + got = collect_members(report / "results.json", report) + assert set(got.members) == {"report/results.json"} + + +@pytest.mark.skipif(os.name == "nt", reason="symlinks need extra rights on Windows") +def test_symlinked_output_dir_is_left_out(tmp_path): + report = tmp_path / "report" + _seed(report, rendered=()) + elsewhere = tmp_path / "elsewhere" + _seed(elsewhere) + (report / "_output").symlink_to(elsewhere / "_output") + got = collect_members(report / "results.json", report) + assert not any(name.startswith("report/_output/") for name in got.members) + + +@pytest.mark.skipif(os.name == "nt", reason="symlinks need extra rights on Windows") +def test_symlinked_results_raises(tmp_path): + report = tmp_path / "report" + report.mkdir() + real = tmp_path / "real.json" + real.write_text(_RESULTS, encoding="utf-8") + (report / "results.json").symlink_to(real) + with pytest.raises(BundleError, match="is a link"): + collect_members(report / "results.json", report) + + +# Shorter than 16 characters, so only the full value is scanned. Each case +# below then passes only through the encoded form that it names. +SHORT = 'pa"ss&wörd-01' +SPACED = "pass word-0123" +BACKSLASH = "ab\\cd-Secret9" +WRAPPED = "Wrap-Me-1234" +# At least 16 characters, so the scan also looks for the first and the last 8. +LONG = "K3y-Start-0123456789-a1b2c3d4e5" +QUOTED = 'p"ss\\w0rd-ABCDEFGHIJ0123' +PADDED = "QU1DREVGR0hJSktMTU5PUA==" + + +@pytest.mark.parametrize( + ("secret", "stored"), + [ + (SHORT, SHORT.encode()), + (SHORT, b'pa\\"ss&w\\u00f6rd-01'), # json.dumps with ensure_ascii + (SHORT, 'pa\\"ss&wörd-01'.encode()), # json.dumps without ensure_ascii + (SHORT, "pa"ss&wörd-01".encode()), # html.escape + (SHORT, 'pa"ss&wörd-01'.encode()), # html.escape with quote=False (XML text) + (SHORT, b"pa%22ss%26w%C3%B6rd-01"), # percent-encoded with quote + (SHORT, b"pa%22ss%26w%c3%b6rd-01"), # percent-encoded with lowercase hex + (SPACED, b"pass+word-0123"), # percent-encoded with quote_plus + (BACKSLASH, b"'ab\\\\cd-Secret9'"), # repr in error text + (BACKSLASH, json.dumps(repr(BACKSLASH)).encode()), # repr, then json.dumps + (WRAPPED, b"Wrap\n-Me-1234"), # wrapped across lines + (WRAPPED, b"Wrap\\n-Me-1234"), # wrapped, then json.dumps + (LONG, LONG[:10].encode()), # cut at a length limit + (LONG, b"..." + LONG[-12:].encode()), # cut in the middle by pytest saferepr + (QUOTED, json.dumps(QUOTED[:10])[1:-1].encode()), # cut, then json.dumps + (PADDED, PADDED[:12].encode()), # a value with "=" still gets a partial form + ], +) +def test_find_secret_hits_sees_encoded_forms(secret, stored): + members = {"report/a": b"x " + stored + b" y", "report/b": b"clean"} + assert find_secret_hits(members, [("auth.password", secret)]) == [("report/a", "auth.password")] + + +@pytest.mark.parametrize( + ("secret", "stored"), + [ + ("pass word-0123", b"pass%20word-0123"), # quote gives %20, quote_plus gives + + ("Pa\\nss-w0rd9", b"Pa\\nss-w0rd9"), # the value holds a backslash and an n + ], +) +def test_find_secret_hits_sees_a_space_and_a_backslash(secret, stored): + assert find_secret_hits({"report/a": b"x " + stored + b" y"}, [("k", secret)]) == [ + ("report/a", "k") + ] + + +def test_find_secret_hits_finds_nothing_in_clean_members(): + members = {"report/a": b"nothing here", "report/b": b"clean"} + assert find_secret_hits(members, [("connect.api_key", "ABC123")]) == [] + + +@pytest.mark.parametrize( + ("conn", "text"), + [ + ("Host=db01;Port=5432;Password=Xy9-secret-value", b"could not reach Host=db01:5432"), + ("pg15://dbuser:DB-PASSWORD-01@db:5432/app", b"engine pg15://..."), + ], +) +def test_partial_rule_skips_connection_strings(conn, text): + # The start of a whole connection string or URL is a key or a scheme, which + # is in normal error text. Its credential parts are separate secrets. + assert find_secret_hits({"report/a": text}, [("db", conn)]) == [] + + +def test_partial_rule_skips_the_start_of_a_jwt(): + # Every JWT starts with eyJhbGci, which has no digit, so only the end of a + # JWT is a partial form. + jwt = "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ2aXAifQ.c2lnbmF0dXJlLX9hbH1l" + other = b"token eyJhbGciOiJSUzI1NiJ9.other" + assert find_secret_hits({"report/a": other}, [("package_manager.token", jwt)]) == [] + tail = jwt[-10:].encode() + assert find_secret_hits({"report/a": tail}, [("package_manager.token", jwt)]) == [ + ("report/a", "package_manager.token") + ] + + +def test_partial_rule_skips_a_part_with_no_digit_or_no_letter(): + # A password that ends in a word must not match every "password_set" key. + secrets = [("auth.password", "Company-Secret-password"), ("x", "abcdefgh-20261105")] + members = {"config-summary.json": b'"password_set": true, "date": "20261105"'} + assert find_secret_hits(members, secrets) == [] + + +def test_short_secrets_are_not_scanned_but_are_reported(): + secrets = [("auth.password", "abc"), ("connect.api_key", "ABC123")] + assert find_secret_hits({"report/a": b"abc"}, secrets) == [] + assert short_secret_labels(secrets) == ["auth.password"] + + +def test_scan_threshold_boundaries(): + # A value of MIN_SECRET_LENGTH (4) is scanned. A value of + # PARTIAL_MIN_SECRET_LENGTH (16) also gets its first and last + # PARTIAL_LENGTH (8) characters as forms. A value of 15 does not. + assert find_secret_hits({"a": b"x ab1c y"}, [("k", "ab1c")]) == [("a", "k")] + at, below = "Abcdefg1-xyz-23Q", "Abcdefg1-xyz-23" + assert find_secret_hits({"a": b"x Abcdefg1 y"}, [("k", at)]) == [("a", "k")] + assert find_secret_hits({"a": b"x Abcdefg y"}, [("k", at)]) == [] + assert find_secret_hits({"a": b"x Abcdefg1 y"}, [("k", below)]) == [] + + +def test_scan_reads_a_value_with_bytes_that_are_not_utf8(): + # Python keeps such environment bytes as surrogates. + assert find_secret_hits({"a": b"x pw\xff1234 y"}, [("k", "pw\udcff1234")]) == [("a", "k")] + assert find_secret_hits({"a": b"x"}, [("k", "pw\ud8001234")]) == [] + + +_PAGE = ( + b"" + b'' + b'

metadata:K3yText01

' +) + + +@pytest.mark.parametrize("secret", ["K3yFont1", "K3yScript1", "K3yImage1"]) +def test_scan_skips_data_uris_scripts_and_styles_in_html(secret): + assert find_secret_hits({"report/_output/index.html": _PAGE}, [("k", secret)]) == [] + assert find_secret_hits({"report/results.json": _PAGE}, [("k", secret)]) == [ + ("report/results.json", "k") + ] + + +def test_scan_reads_the_text_of_an_html_page(): + hits = find_secret_hits({"report/_output/index.html": _PAGE}, [("k", "K3yText01")]) + assert hits == [("report/_output/index.html", "k")] + + +@pytest.mark.parametrize( + "data", + [ + _RESULTS.encode("utf-16"), + b"\xef\xbb\xbf" + _RESULTS.encode(), + b'{"results": [], "x": "\xff"}', + ], +) +def test_results_that_is_not_utf8_raises(tmp_path, data): + report = tmp_path / "report" + report.mkdir() + (report / "results.json").write_bytes(data) + with pytest.raises(BundleError, match="not UTF-8"): + collect_members(report / "results.json", report) + + +@pytest.mark.parametrize("name", [*_SIDECARS, "_output/index.html"]) +def test_sidecar_or_page_that_is_not_utf8_raises(tmp_path, name): + report = tmp_path / "report" + _seed(report) + (report / name).write_bytes(f"content of {name}".encode("utf-16")) + os.utime(report / name, (1_000_001.0, 1_000_001.0)) + if name.startswith("_output/"): + record_rendered(report / "_output", results_digest(_RESULTS.encode()), _RENDERED) + with pytest.raises(BundleError, match="not UTF-8"): + collect_members(report / "results.json", report) + + +def test_deeply_nested_results_raises_bundle_error(tmp_path): + report = tmp_path / "report" + _seed(report, sidecars=(), rendered=(), results="[" * 100_000 + "]" * 100_000) + with pytest.raises(BundleError, match="is not a VIP results file"): + collect_members(report / "results.json", report) + + +def test_write_zip_writes_every_member(tmp_path): + out = tmp_path / "b.zip" + write_zip(out, {"y": b"2", "x": b"1"}) + with zipfile.ZipFile(out) as zf: + assert zf.namelist() == ["x", "y"] + assert zf.read("y") == b"2" + + +def test_write_zip_leaves_nothing_on_error(tmp_path, monkeypatch): + real_writestr = zipfile.ZipFile.writestr + calls: list[str] = [] + + def failing_writestr(self, name, data, *args, **kwargs): + calls.append(name) + if len(calls) == 2: + raise OSError("disk full") + return real_writestr(self, name, data, *args, **kwargs) + + monkeypatch.setattr(zipfile.ZipFile, "writestr", failing_writestr) + with pytest.raises(OSError, match="disk full"): + write_zip(tmp_path / "b.zip", {"x": b"1", "y": b"2"}) + assert list(tmp_path.iterdir()) == [] diff --git a/selftests/test_bundle_summary.py b/selftests/test_bundle_summary.py new file mode 100644 index 000000000..5e58ea0ad --- /dev/null +++ b/selftests/test_bundle_summary.py @@ -0,0 +1,667 @@ +"""Tests for the ``vip bundle`` configuration summary and secret list.""" + +from __future__ import annotations + +import dataclasses +import json +import types +import typing +from typing import Any + +import pytest + +from vip.bundle.summary import ( + EXCLUDED_FIELDS, + SECRET_ENV_VARS, + SECRET_FIELDS, + SUMMARIZED_FIELDS, + auth_cache_secrets, + build_config_summary, + secret_values, + strip_url_userinfo, +) +from vip.config import ( + AuthConfig, + ConnectConfig, + DataSourceEntry, + GitTestConfig, + PackageManagerConfig, + VIPConfig, + WorkbenchConfig, +) +from vip.errors import BundleError +from vip.proxy import ProxyConfig + +# Every VIP_* variable that VIPConfig or the bundle reads. A developer's own +# shell must not change what these tests see. +_VIP_SECRET_ENV = ( + "VIP_CONFIG", + "VIP_CONNECT_API_KEY", + "VIP_WORKBENCH_API_KEY", + "VIP_GIT_TOKEN", + "VIP_PACKAGE_MANAGER_TOKEN", + "VIP_PM_TOKEN", + "VIP_TEST_USERNAME", + "VIP_TEST_PASSWORD", + "VIP_TEST_TOTP_SECRET", +) + +# One distinct value per credential source, so a test can name the one that leaked. +SECRETS = { + "connect.api_key": "SECRET-connect-api-key", + "workbench.api_key": "SECRET-workbench-api-key", + "workbench.git_test.token": "SECRET-git-token", + "clone_url user": "SECRET-git-user-token", + "clone_url query": "SECRET-git-query-token", + "package_manager.token": "SECRET-pm-token", + "auth.password": "SECRET-auth-password", + "connection_string password": "SECRET-db-password", + "connect.url password": "SECRET-connect-url-password", + "proxy.url password": "SECRET-proxy-password", +} + + +@pytest.fixture(autouse=True) +def _no_vip_env(monkeypatch): + for var in _VIP_SECRET_ENV: + monkeypatch.delenv(var, raising=False) + + +@pytest.fixture +def full_config() -> VIPConfig: + return VIPConfig( + deployment_name="Acme Prod", + connect=ConnectConfig( + url=f"https://cuser:{SECRETS['connect.url password']}@connect.example.com", + api_key=SECRETS["connect.api_key"], + ), + workbench=WorkbenchConfig( + url="https://workbench.example.com", + api_key=SECRETS["workbench.api_key"], + git_test=GitTestConfig( + clone_url=( + f"https://{SECRETS['clone_url user']}@github.com/org/repo.git" + f"?access_token={SECRETS['clone_url query']}" + ), + token=SECRETS["workbench.git_test.token"], + ), + ), + package_manager=PackageManagerConfig( + url="https://pm.example.com", token=SECRETS["package_manager.token"] + ), + auth=AuthConfig(username="alice", password=SECRETS["auth.password"]), + data_sources=[ + DataSourceEntry( + name="pg", + type="postgres", + connection_string=( + f"postgresql://dbuser:{SECRETS['connection_string password']}@db:5432/app" + ), + ) + ], + proxy=ProxyConfig(url=f"http://puser:{SECRETS['proxy.url password']}@proxy:8080"), + ) + + +def _walk_config_fields(cls: Any, prefix: str = "") -> set[str]: + """Return the dotted path of every leaf field of a config dataclass. + + This helper lives in the test, not in the module, so that the test does + not trust the code that it checks. config.py uses postponed annotations, + so the field types come from typing.get_type_hints. + """ + hints = typing.get_type_hints(cls) + paths: set[str] = set() + for f in dataclasses.fields(cls): + hint = hints[f.name] + args = typing.get_args(hint) + origin = typing.get_origin(hint) + path = prefix + f.name + if dataclasses.is_dataclass(hint): + paths |= _walk_config_fields(hint, path + ".") + elif origin is list and args and dataclasses.is_dataclass(args[0]): + paths |= _walk_config_fields(args[0], path + "[].") + elif origin in (typing.Union, types.UnionType) and any( + dataclasses.is_dataclass(a) for a in args + ): + inner = next(a for a in args if dataclasses.is_dataclass(a)) + paths |= _walk_config_fields(inner, path + ".") + else: + paths.add(path) + return paths + + +# Summary keys whose value is a plain dict, not a nested section. +_DICT_VALUED_KEYS = {"node_pool_profiles", "profile_cpu_limit", "profile_memory_limit_gib"} + + +def _summary_paths(node: dict, prefix: str = "") -> set[str]: + """Return the dotted path of every leaf key in a summary, without ``*_set`` keys.""" + paths: set[str] = set() + for key, value in node.items(): + path = prefix + key + if key.endswith("_set"): + continue + if isinstance(value, dict) and key not in _DICT_VALUED_KEYS: + paths |= _summary_paths(value, path + ".") + elif isinstance(value, list) and value and isinstance(value[0], dict): + paths |= _summary_paths(value[0], path + "[].") + else: + paths.add(path) + return paths + + +def test_every_config_field_is_classified(): + assert _walk_config_fields(VIPConfig) == SUMMARIZED_FIELDS | EXCLUDED_FIELDS + assert not (SUMMARIZED_FIELDS & EXCLUDED_FIELDS) + assert SECRET_FIELDS <= EXCLUDED_FIELDS + + +def test_summary_has_exactly_the_summarized_fields(full_config): + assert _summary_paths(build_config_summary(full_config)) == SUMMARIZED_FIELDS + + +def _config_classes(cls: Any, prefix: str = "") -> list[tuple[Any, str]]: + """Return ``(class, dotted prefix)`` for *cls* and every config dataclass under it.""" + found = [(cls, prefix)] + hints = typing.get_type_hints(cls) + for f in dataclasses.fields(cls): + hint = hints[f.name] + args = typing.get_args(hint) + inner = next((a for a in (hint, *args) if dataclasses.is_dataclass(a)), None) + if inner is not None: + suffix = "[]." if typing.get_origin(hint) is list else "." + found += _config_classes(inner, prefix + f.name + suffix) + return found + + +# Field values that __post_init__ needs before the other fields keep their value. +_REQUIRED_VALUES = {GitTestConfig: {"auth_method": "https-token"}} + + +_STR_TYPES = (str, str | None) +_LIST_TYPES = (list[str], list[str] | None) +_DICT_TYPES = (dict[str, str], dict[str, str] | None) + + +def _sentinel_values(cls: type, prefix: str) -> dict[str, tuple[object, str]]: + """Return ``{field: (value, sentinel)}`` for each init field that can hold a str. + + A str field gets the sentinel. A list field gets a list that holds it, + and a dict field gets a dict with it as the value. + """ + hints = typing.get_type_hints(cls) + found: dict[str, tuple[object, str]] = {} + for f in dataclasses.fields(cls): + sentinel = f"SENTINEL-{prefix}{f.name}" + hint = hints[f.name] + if not f.init: + continue + if hint in _STR_TYPES: + found[f.name] = (sentinel, sentinel) + elif hint in _LIST_TYPES: + found[f.name] = ([sentinel], sentinel) + elif hint in _DICT_TYPES: + found[f.name] = ({"key": sentinel}, sentinel) + return found + + +def test_sentinel_values_cover_str_list_and_dict_fields(): + @dataclasses.dataclass + class Sample: + name: str = "" + token: str | None = None + hosts: list[str] = dataclasses.field(default_factory=list) + keys: list[str] | None = None + headers: dict[str, str] = dataclasses.field(default_factory=dict) + count: int = 0 + limits: dict[str, float] = dataclasses.field(default_factory=dict) + + assert _sentinel_values(Sample, "s.") == { + "name": ("SENTINEL-s.name", "SENTINEL-s.name"), + "token": ("SENTINEL-s.token", "SENTINEL-s.token"), + "hosts": (["SENTINEL-s.hosts"], "SENTINEL-s.hosts"), + "keys": (["SENTINEL-s.keys"], "SENTINEL-s.keys"), + "headers": ({"key": "SENTINEL-s.headers"}, "SENTINEL-s.headers"), + } + + +def test_every_repr_masked_field_is_a_secret(): + # A config class that hides a field in __repr__ treats it as a secret. + # Every field that can hold a str, in a list, or in a dict gets a distinct + # sentinel, so a new hidden field is found. + classes = _config_classes(VIPConfig) + known = {ConnectConfig, WorkbenchConfig, GitTestConfig, PackageManagerConfig, AuthConfig} + assert known | {DataSourceEntry} <= {cls for cls, _ in classes} + hidden: set[str] = set() + for cls, prefix in classes: + required = _REQUIRED_VALUES.get(cls, {}) + sentinels = { + name: pair + for name, pair in _sentinel_values(cls, prefix).items() + if name not in required + } + text = repr(cls(**{name: value for name, (value, _) in sentinels.items()}, **required)) + hidden |= {prefix + name for name, (_, s) in sentinels.items() if s not in text} + assert hidden == SECRET_FIELDS + + +def test_fields_left_out_of_the_summary_that_are_not_secrets_are_pinned(): + # The scan looks only for SECRET_FIELDS. A new field here is left out of + # the summary but is not scanned, so it must be a reviewed choice. + assert { + "auth.username", + "connect.url_scheme_inferred", + "workbench.url_scheme_inferred", + "package_manager.url_scheme_inferred", + } == EXCLUDED_FIELDS - SECRET_FIELDS + + +_SET_ONE_SECRET = { + "connect.api_key": lambda v: VIPConfig(connect=ConnectConfig(api_key=v)), + "workbench.api_key": lambda v: VIPConfig(workbench=WorkbenchConfig(api_key=v)), + "workbench.git_test.token": lambda v: VIPConfig( + workbench=WorkbenchConfig(git_test=GitTestConfig(token=v)) + ), + "package_manager.token": lambda v: VIPConfig(package_manager=PackageManagerConfig(token=v)), + "auth.password": lambda v: VIPConfig(auth=AuthConfig(password=v)), + "data_sources[].connection_string": lambda v: VIPConfig( + data_sources=[DataSourceEntry(name="db", connection_string=v)] + ), +} + + +@pytest.mark.parametrize("path", sorted(SECRET_FIELDS)) +def test_each_secret_field_is_in_secret_values(path): + # A new entry in SECRET_FIELDS needs a builder here, and secret_values must return it. + config = _SET_ONE_SECRET[path]("SENTINEL-VALUE-0123") + assert "SENTINEL-VALUE-0123" in {value for _, value in secret_values(config)} + + +def test_every_secret_field_has_a_builder(): + assert set(_SET_ONE_SECRET) == SECRET_FIELDS + + +def test_summary_has_no_secret_values(full_config): + text = json.dumps(build_config_summary(full_config)) + for value in SECRETS.values(): + assert value not in text + assert "cuser" not in text + assert "puser" not in text + assert "dbuser" not in text + assert "alice" not in text + + +def test_summary_reports_secret_presence(full_config): + s = build_config_summary(full_config) + assert s["connect"]["api_key_set"] is True + assert s["workbench"]["api_key_set"] is True + assert s["package_manager"]["token_set"] is True + assert s["auth"]["password_set"] is True + assert s["workbench"]["git_test"]["token_set"] is True + assert s["data_sources"] == [{"name": "pg", "type": "postgres", "connection_string_set": True}] + assert "username" not in s["auth"] + + +def test_summary_keeps_clone_url_path(full_config): + s = build_config_summary(full_config) + assert s["workbench"]["git_test"]["clone_url"] == "https://github.com/org/repo.git" + assert s["proxy"]["url"] == "http://proxy:8080" + assert s["connect"]["url"] == "https://connect.example.com" + + +def test_summary_without_git_test(): + assert build_config_summary(VIPConfig())["workbench"]["git_test"] is None + + +@pytest.mark.parametrize( + ("url", "expected"), + [ + ( + "https://x-access-token:SECRET@github.com/org/repo.git", + "https://github.com/org/repo.git", + ), + ("https://TOKEN@github.com/org/repo.git", "https://github.com/org/repo.git"), + ("https://github.com/org/repo.git", "https://github.com/org/repo.git"), + ("https://github.com/org/repo.git?access_token=T#frag", "https://github.com/org/repo.git"), + ("http://u:p@[::1]:8080/x", "http://[::1]:8080/x"), + ("x-access-token:SECRET@github.com/org/repo.git", "github.com/org/repo.git"), + ("git@github.com:org/repo.git", "git@github.com:org/repo.git"), + ("", ""), + ], +) +def test_strip_url_userinfo(url, expected): + assert strip_url_userinfo(url) == expected + + +def test_secret_values_cover_every_source(full_config, tmp_path): + meta = tmp_path / ".vip-auth-cache.meta.json" + meta.write_text(json.dumps({"api_key": "MINTED-KEY-0001"}), encoding="utf-8") + state = tmp_path / ".vip-auth-cache.json" + state.write_text( + json.dumps( + { + "cookies": [ + {"name": "session", "value": "COOKIE-VALUE-0123456789"}, + {"name": "flag", "value": "1"}, + ], + "origins": [ + { + "origin": "https://connect.example.com", + "localStorage": [{"name": "token", "value": "LOCAL-STORAGE-0123456789"}], + } + ], + } + ), + encoding="utf-8", + ) + cache = auth_cache_secrets(state) + auth_cache_secrets(meta) + values = {value for _, value in secret_values(full_config, cache)} + assert set(SECRETS.values()) <= values + assert {"MINTED-KEY-0001", "COOKIE-VALUE-0123456789", "LOCAL-STORAGE-0123456789"} <= values + assert full_config.data_sources[0].connection_string in values + assert "1" not in values + assert "" not in values + # A user name next to a password is not a credential, except in the Git clone URL. + assert not {"cuser", "puser", "dbuser"} & values + + +def test_auth_cache_secrets_skip_login_names(tmp_path): + # Okta keeps the login name in the "ln" cookie. Test output often prints + # that name, and a login name is not a credential. + state = tmp_path / ".vip-auth-cache.json" + state.write_text( + json.dumps( + { + "cookies": [ + {"name": "ln", "value": "test.user@example.com"}, + {"name": "ln2", "value": "test.user%40example.com"}, + {"name": "sid", "value": "a1b2c3@example-0123456789"}, + ], + "origins": [ + { + "origin": "https://login.example.com", + "localStorage": [{"name": "user", "value": "test.user@example.com"}], + } + ], + } + ), + encoding="utf-8", + ) + assert auth_cache_secrets(state) == [("auth cache cookie sid", "a1b2c3@example-0123456789")] + + +@pytest.mark.parametrize( + "clone_url", + [ + "https://GIT-TOKEN-0123456789:x-oauth-basic@github.com/org/repo.git", + "https://GIT-TOKEN-0123456789:@github.com/org/repo.git", + "x-access-token:GIT-TOKEN-0123456789@github.com/org/repo.git", + ], +) +def test_secret_values_find_the_clone_url_token_in_each_form(clone_url): + config = VIPConfig(workbench=WorkbenchConfig(git_test=GitTestConfig(clone_url=clone_url))) + assert "GIT-TOKEN-0123456789" in {value for _, value in secret_values(config)} + assert "GIT-TOKEN" not in json.dumps(build_config_summary(config)) + + +def test_secret_values_labels_name_the_source(full_config): + labels = {value: label for label, value in secret_values(full_config)} + assert labels[SECRETS["auth.password"]] == "auth.password" + assert labels[SECRETS["clone_url user"]] == "workbench.git_test.clone_url user" + assert labels[SECRETS["clone_url query"]] == "workbench.git_test.clone_url query access_token" + + +def test_clone_url_user_and_password_have_their_own_labels(): + # --allow-match for a false match on a public user name must not also + # let the token through. + url = "https://x-access-token:GIT-TOKEN-0123456789@github.com/org/repo.git" + config = VIPConfig(workbench=WorkbenchConfig(git_test=GitTestConfig(clone_url=url))) + labels = {value: label for label, value in secret_values(config)} + assert labels["x-access-token"] == "workbench.git_test.clone_url user" + assert labels["GIT-TOKEN-0123456789"] == "workbench.git_test.clone_url password" + + +def test_secret_values_read_env_only_secrets(monkeypatch): + monkeypatch.setenv("VIP_GIT_TOKEN", "ENV-GIT-TOKEN") + monkeypatch.setenv("VIP_TEST_TOTP_SECRET", "ENV-TOTP-SEED") + monkeypatch.setenv("HTTPS_PROXY", "http://u:ENV-PROXY-PASS@proxy:3128") + monkeypatch.setenv("VIP_CONNECT_API_KEY", "ENV-CONNECT-KEY") + values = {value for _, value in secret_values(VIPConfig())} + assert {"ENV-GIT-TOKEN", "ENV-TOTP-SEED", "ENV-PROXY-PASS", "ENV-CONNECT-KEY"} <= values + + +def test_secret_values_split_key_value_connection_string(): + conn = ( + "Driver=Snowflake;Server=db;Uid=sa;Pwd=ODBC-PASS;token=SNOW-TOKEN-01;" + "AccountKey=AZURE-KEY-01==;Password={pa;ss-0123};" + ) + config = VIPConfig(data_sources=[DataSourceEntry(name="db", connection_string=conn)]) + labels = {value: label for label, value in secret_values(config)} + assert labels[conn] == "data_sources.db.connection_string" + assert labels["ODBC-PASS"] == "data_sources.db.connection_string pwd" + assert labels["SNOW-TOKEN-01"] == "data_sources.db.connection_string token" + assert labels["AZURE-KEY-01=="] == "data_sources.db.connection_string accountkey" + assert labels["pa;ss-0123"] == "data_sources.db.connection_string password" + assert "sa" not in labels + + +def test_auth_cache_secrets_skip_a_missing_file(tmp_path): + assert auth_cache_secrets(tmp_path / "absent.json") == [] + + +_GOOD_CACHE = json.dumps({"cookies": [{"name": "sid", "value": "COOKIE-VALUE-0123456789"}]}) + + +@pytest.mark.parametrize( + "data", + [ + b"not json", + _GOOD_CACHE[:-3].encode(), # cut before the closing brackets + b"\xef\xbb\xbf" + _GOOD_CACHE.encode(), # UTF-8 BOM + _GOOD_CACHE.encode("utf-16"), + b"[]", + b'{"cookies": 5}', + b'{"cookies": ["sid"]}', + b'{"origins": {"localStorage": []}}', + b'{"origins": [{"localStorage": 5}]}', + ], +) +def test_auth_cache_file_that_cannot_be_used_raises(tmp_path, data): + path = tmp_path / ".vip-auth-cache.json" + path.write_bytes(data) + with pytest.raises(BundleError, match="could not read the auth cache file"): + auth_cache_secrets(path) + + +def test_auth_cache_directory_raises(tmp_path): + with pytest.raises(BundleError, match="could not read the auth cache file"): + auth_cache_secrets(tmp_path) + + +def test_auth_cache_labels_name_the_domain_and_the_origin(tmp_path): + state = tmp_path / ".vip-auth-cache.json" + cookies = [ + {"name": "sid", "domain": "a.example.com", "value": "COOKIE-A-0123456789"}, + {"name": "sid", "domain": "b.example.com", "value": "COOKIE-B-0123456789"}, + ] + origins = [ + { + "origin": "https://a.example.com", + "localStorage": [{"name": "t", "value": "LOCAL-A-0123456789"}], + } + ] + state.write_text(json.dumps({"cookies": cookies, "origins": origins}), encoding="utf-8") + assert auth_cache_secrets(state) == [ + ("auth cache cookie a.example.com sid", "COOKIE-A-0123456789"), + ("auth cache cookie b.example.com sid", "COOKIE-B-0123456789"), + ("auth cache localStorage https://a.example.com t", "LOCAL-A-0123456789"), + ] + + +@pytest.mark.parametrize( + "value", + [ + "user:PASSWORD-0123456789@host.example.com", + "https://user:PASSWORD-0123@host.example.com", + "token=ABCDEFGH01234567&email=a@example.com", + "TOKEN-0123456789|test.user@example.com", + 'prefix "test.user@example.com" TOKEN-0123456789', + "a@b@example.com-0123456789", + "tok@en-0123456789@example.com", + "no-at-sign-0123456789.example.com", + '{"login":"test.user@example.com","t":"abc"}', + "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ2aXAifQ.signature", + ], +) +def test_auth_cache_value_that_is_more_than_an_email_is_a_secret(tmp_path, value): + state = tmp_path / ".vip-auth-cache.json" + state.write_text(json.dumps({"cookies": [{"name": "sid", "value": value}]}), encoding="utf-8") + assert auth_cache_secrets(state) == [("auth cache cookie sid", value)] + + +def test_auth_cache_value_floor_is_exactly_16_characters(tmp_path): + at, below = "a1b2c3d4e5f6g7h8", "a1b2c3d4e5f6g7h" + state = tmp_path / ".vip-auth-cache.json" + cookies = [{"name": "at", "value": at}, {"name": "below", "value": below}] + state.write_text(json.dumps({"cookies": cookies}), encoding="utf-8") + assert auth_cache_secrets(state) == [("auth cache cookie at", at)] + + +def test_email_shaped_password_and_minted_key_are_secrets(tmp_path): + # The login name rule is only for cookies and localStorage values. + meta = tmp_path / ".vip-auth-cache.meta.json" + meta.write_text(json.dumps({"api_key": "svc.account@example.com"}), encoding="utf-8") + config = VIPConfig(auth=AuthConfig(password="svc.user@example.com")) + values = {value for _, value in secret_values(config, auth_cache_secrets(meta))} + assert {"svc.user@example.com", "svc.account@example.com"} <= values + + +def test_each_product_url_credential_is_a_secret_and_is_not_in_the_summary(): + config = VIPConfig( + connect=ConnectConfig(url="https://u:URL-PASS-connect1@connect.example.com"), + workbench=WorkbenchConfig(url="https://u:URL-PASS-workbench1@wb.example.com"), + package_manager=PackageManagerConfig(url="https://u:URL-PASS-pm1@pm.example.com"), + ) + values = {value for _, value in secret_values(config)} + assert {"URL-PASS-connect1", "URL-PASS-workbench1", "URL-PASS-pm1"} <= values + assert "URL-PASS" not in json.dumps(build_config_summary(config)) + + +def test_url_credential_is_a_secret_raw_and_percent_decoded(): + config = VIPConfig(connect=ConnectConfig(url="https://u:p%40ss-0123@connect.example.com")) + assert {"p%40ss-0123", "p@ss-0123"} <= {value for _, value in secret_values(config)} + + +_PROXY_VARS = ("HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "http_proxy", "https_proxy", "all_proxy") + + +@pytest.mark.parametrize("var", _PROXY_VARS) +def test_each_proxy_variable_is_read(monkeypatch, var): + for other in _PROXY_VARS: + monkeypatch.delenv(other, raising=False) + monkeypatch.setenv(var, "http://u:ENV-PROXY-PASS@proxy:3128") + assert "ENV-PROXY-PASS" in {value for _, value in secret_values(VIPConfig())} + + +@pytest.mark.parametrize( + ("url", "expected"), + [ + # urlsplit raises ValueError for this host. It is not a plain host. + ("http://u:SECRET@[::1/x?token=T", "http:///x"), + # A Git remote with no scheme. + ( + "x-access-token:SECRET@github.com/org/repo.git?access_token=T#f", + "github.com/org/repo.git", + ), + ], +) +def test_strip_url_userinfo_handles_a_bad_host_and_a_bare_remote(url, expected): + assert strip_url_userinfo(url) == expected + + +@pytest.mark.parametrize( + ("url", "secret", "summary"), + [ + ( + "https://admin:Abc123#Secret9@connect.example.com", + "Abc123#Secret9", + "https://connect.example.com", + ), + ( + "https://admin:Abc?123x@connect.example.com/p", + "Abc?123x", + "https://connect.example.com/p", + ), + ( + "https://admin:Ab/c123x@connect.example.com/p", + "Ab/c123x", + "https://connect.example.com/p", + ), + ("https://Tok#en-0123@connect.example.com", "Tok#en-0123", "https://connect.example.com"), + ], +) +def test_password_with_a_raw_delimiter_is_a_secret_and_is_not_in_the_summary(url, secret, summary): + config = VIPConfig(connect=ConnectConfig(url=url)) + assert secret in {value for _, value in secret_values(config)} + assert strip_url_userinfo(url) == summary + + +def test_url_with_an_at_sign_in_the_path_keeps_the_real_password_too(): + # The two parsers differ when the URL holds two "@". The scan looks for both values. + url = "https://u:REAL-PASS-0123@connect.example.com/a@b" + values = {value for _, value in secret_values(VIPConfig(connect=ConnectConfig(url=url)))} + assert "REAL-PASS-0123" in values + + +@pytest.mark.parametrize( + ("url", "summary"), + [ + ("https://admin:Abc123", "https://"), + ("https://connect.example.com:8443/x", "https://connect.example.com:8443/x"), + ], +) +def test_summary_redacts_a_host_that_is_not_a_plain_host(url, summary): + assert strip_url_userinfo(url) == summary + + +def test_each_part_of_a_url_has_its_own_label(): + conn = "postgresql://appuser@db.example.com:5432/app?sslmode=require&password=PG-PASS-0123" + config = VIPConfig( + data_sources=[DataSourceEntry(name="pg", connection_string=conn)], + connect=ConnectConfig(url="https://u:URL-PASS-0123@c.example.com/?token=URL-TOKEN-0123"), + ) + labels = {value: label for label, value in secret_values(config)} + assert labels["appuser"] == "data_sources.pg.connection_string user" + assert labels["PG-PASS-0123"] == "data_sources.pg.connection_string query password" + assert labels["URL-PASS-0123"] == "connect.url password" + assert labels["URL-TOKEN-0123"] == "connect.url query token" + + +_INLINE_FOR_ENV = { + "VIP_CONNECT_API_KEY": lambda: VIPConfig(connect=ConnectConfig(api_key="INLINE-0123")), + "VIP_WORKBENCH_API_KEY": lambda: VIPConfig(workbench=WorkbenchConfig(api_key="INLINE-0123")), + "VIP_PACKAGE_MANAGER_TOKEN": lambda: VIPConfig( + package_manager=PackageManagerConfig(token="INLINE-0123") + ), + "VIP_PM_TOKEN": lambda: VIPConfig(package_manager=PackageManagerConfig(token="INLINE-0123")), + "VIP_TEST_PASSWORD": lambda: VIPConfig(auth=AuthConfig(password="INLINE-0123")), + "VIP_GIT_TOKEN": lambda: VIPConfig( + workbench=WorkbenchConfig(git_test=GitTestConfig(token="INLINE-0123")) + ), + "VIP_TEST_TOTP_SECRET": VIPConfig, +} + + +@pytest.mark.parametrize("var", SECRET_ENV_VARS) +def test_each_env_variable_is_read_next_to_an_inline_value(monkeypatch, var): + # The configuration keeps the inline value, so only SECRET_ENV_VARS + # gives the scan the value of the variable. + monkeypatch.setenv(var, f"ENV-{var}-0123") + config = _INLINE_FOR_ENV[var]() + assert f"ENV-{var}-0123" not in repr(dataclasses.asdict(config)) + assert f"ENV-{var}-0123" in {value for _, value in secret_values(config)} + + +def test_every_env_variable_has_an_inline_builder(): + assert set(_INLINE_FOR_ENV) == set(SECRET_ENV_VARS) diff --git a/selftests/test_cli_bundle.py b/selftests/test_cli_bundle.py new file mode 100644 index 000000000..39ae4b144 --- /dev/null +++ b/selftests/test_cli_bundle.py @@ -0,0 +1,737 @@ +"""Tests for the ``vip bundle`` subcommand (run_bundle in vip.cli).""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +import zipfile +from pathlib import Path + +import pytest + +import vip.cli.bundle as bundle_module +from vip.bundle.stamp import record_rendered, results_digest +from vip.cli import main, run_bundle +from vip.config import VIPConfig +from vip.errors import BundleError + +_VIP_SECRET_ENV = ( + "VIP_CONFIG", + "VIP_CONNECT_API_KEY", + "VIP_WORKBENCH_API_KEY", + "VIP_GIT_TOKEN", + "VIP_PACKAGE_MANAGER_TOKEN", + "VIP_PM_TOKEN", + "VIP_TEST_USERNAME", + "VIP_TEST_PASSWORD", + "VIP_TEST_TOTP_SECRET", +) + +CONNECT_KEY = "SECRET-connect-api-key" +MINTED_KEY = "MINTED-KEY-0001" +SESSION_COOKIE = "COOKIE-SESSION-0123456789" + +VIP_TOML = f"""\ +[general] +deployment_name = "Acme Prod" + +[connect] +url = "https://connect.example.com" +api_key = "{CONNECT_KEY}" + +[workbench] +url = "https://workbench.example.com" +api_key = "SECRET-workbench-api-key" + +[workbench.git_test] +clone_url = "https://SECRET-git-user-token@github.com/org/repo.git" +token = "SECRET-git-token" + +[package_manager] +url = "https://pm.example.com" +token = "SECRET-pm-token" + +[auth] +username = "alice" +password = "SECRET-auth-password" + +[data_sources.pg] +type = "postgres" +connection_string = "postgresql://dbuser:SECRET-db-password@db:5432/app" + +[proxy] +url = "http://puser:SECRET-proxy-password@proxy:8080" +""" + +ALL_SEEDED_SECRETS = ( + CONNECT_KEY, + "SECRET-workbench-api-key", + "SECRET-git-user-token", + "SECRET-git-token", + "SECRET-pm-token", + "SECRET-auth-password", + "SECRET-db-password", + "SECRET-proxy-password", + MINTED_KEY, + SESSION_COOKIE, +) + +RESULTS_MTIME = 1_000_000.0 + + +def _args(**overrides) -> argparse.Namespace: + defaults: dict[str, object] = { + "results": None, + "config": None, + "no_config": False, + "output": None, + "force": False, + "allow_match": [], + } + defaults.update(overrides) + return argparse.Namespace(**defaults) + + +def _write(path: Path, text: str, mtime: float) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + os.utime(path, (mtime, mtime)) + + +def _write_report( + root: Path, results_extra: str = "", failures_extra: str = "", deployment: str = "Acme Prod" +) -> None: + """Write report files as vip verify and vip report leave them. + + failures.json is newer than results.json, and the render stamp ties + index.html to results.json. + """ + report = root / "report" + results = { + "deployment_name": deployment, + "products": {"connect": {"url": "https://connect.example.com"}}, + "results": [], + "note": results_extra, + } + _write(report / "results.json", json.dumps(results), RESULTS_MTIME) + failures = {"failures": [{"error_summary": f"HTTP 401 {failures_extra}"}]} + _write(report / "failures.json", json.dumps(failures), RESULTS_MTIME + 1) + _write(report / "_output" / "index.html", "report", RESULTS_MTIME + 2) + digest = results_digest((report / "results.json").read_bytes()) + record_rendered(report / "_output", digest, ("index.html",)) + _write(report / "notes.txt", "not allowlisted", RESULTS_MTIME + 2) + + +@pytest.fixture(autouse=True) +def _isolated_env(monkeypatch, tmp_path): + for var in _VIP_SECRET_ENV: + monkeypatch.delenv(var, raising=False) + # A later auth_cache_path() reads XDG_CACHE_HOME. Keep it out of the real home. + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "xdg")) + + +@pytest.fixture +def root(tmp_path, monkeypatch) -> Path: + """An invocation directory as ``vip verify`` and ``vip report`` leave it.""" + (tmp_path / "vip.toml").write_text(VIP_TOML, encoding="utf-8") + (tmp_path / ".vip-auth-cache.json").write_text( + json.dumps({"cookies": [{"name": "session", "value": SESSION_COOKIE}]}), + encoding="utf-8", + ) + (tmp_path / ".vip-auth-cache.meta.json").write_text( + json.dumps({"api_key": MINTED_KEY}), encoding="utf-8" + ) + (tmp_path / ".vip-install.json").write_text("{}", encoding="utf-8") + _write_report(tmp_path) + monkeypatch.chdir(tmp_path) + return tmp_path + + +def _names(zip_path: Path) -> set[str]: + with zipfile.ZipFile(zip_path) as zf: + return set(zf.namelist()) + + +def _leftovers(directory: Path) -> list[str]: + return sorted(p.name for p in directory.iterdir() if p.suffix in {".zip", ".tmp"}) + + +def test_bundle_holds_only_allowlisted_files_and_no_secret(root): + run_bundle(_args(output="b.zip")) + with zipfile.ZipFile(root / "b.zip") as zf: + names = set(zf.namelist()) + data = b"".join(zf.read(n) for n in names) + assert names == { + "report/results.json", + "report/failures.json", + "report/_output/index.html", + "config-summary.json", + "version.txt", + "manifest.json", + } + for secret in ALL_SEEDED_SECRETS: + assert secret.encode() not in data + assert b"[connect]" not in data + + +def test_manifest_lists_members_and_missing_files(root): + run_bundle(_args(output="b.zip")) + with zipfile.ZipFile(root / "b.zip") as zf: + manifest = json.loads(zf.read("manifest.json")) + assert manifest["members"] == sorted(_names(root / "b.zip")) + assert "report/_output/vip-report.pdf" in manifest["missing"] + assert manifest["left_out"] == [] + assert manifest["allowed_matches"] == [] + assert manifest["config_summary"] is True + assert manifest["secret_scan"] == "config" + + +def test_manifest_records_how_many_secrets_the_scan_knew(root, capsys): + run_bundle(_args(output="b.zip")) + with zipfile.ZipFile(root / "b.zip") as zf: + manifest = json.loads(zf.read("manifest.json")) + # vip.toml holds nine credential values. The cache holds a cookie and a minted key. + assert manifest["secret_scan_sources"] == { + "configuration_and_environment": 9, + "results_product_urls": 0, + "auth_cache": 2, + } + out = capsys.readouterr().out + assert ( + "9 from the configuration and the environment, 0 from the product URLs in " + "results.json, 2 from the auth cache" + ) in out + + +def test_scan_with_no_known_secret_warns(root, capsys): + (root / "vip.toml").unlink() + (root / ".vip-auth-cache.json").unlink() + (root / ".vip-auth-cache.meta.json").unlink() + run_bundle(_args(output="b.zip", no_config=True)) + with zipfile.ZipFile(root / "b.zip") as zf: + manifest = json.loads(zf.read("manifest.json")) + assert manifest["secret_scan_sources"] == { + "configuration_and_environment": 0, + "results_product_urls": 0, + "auth_cache": 0, + } + assert any("knew no credential" in w for w in manifest["warnings"]) + assert "Warning: the secret scan knew no credential" in capsys.readouterr().err + + +def test_scan_with_no_auth_cache_does_not_warn_when_the_config_has_secrets(root, capsys): + (root / ".vip-auth-cache.json").unlink() + (root / ".vip-auth-cache.meta.json").unlink() + run_bundle(_args(output="b.zip")) + assert "knew no credential" not in capsys.readouterr().err + + +def test_secret_in_failures_json_blocks_bundle(root, capsys): + _write_report(root, failures_extra=f"key={CONNECT_KEY}") + with pytest.raises(BundleError) as exc: + run_bundle(_args(output="b.zip")) + message = str(exc.value) + # The error names the file on disk, relative to the working directory. + assert f"\n {Path('report') / 'failures.json'}: connect.api_key\n" in message + assert "Edit each file to remove the value" in message + assert "--allow-match LABEL" in message + assert CONNECT_KEY not in message + captured = capsys.readouterr() + assert CONNECT_KEY not in captured.out + captured.err + assert _leftovers(root) == [] + + +@pytest.mark.parametrize( + "member", + [ + "junit.xml", + "results.sarif", + "_output/index.html", + "_output/details.html", + "_output/vip-report.pdf", + ], +) +def test_secret_in_each_report_file_blocks_bundle(root, member): + report = root / "report" + rendered = ("index.html", "details.html", "vip-report.pdf") + for name in rendered: + _write(report / "_output" / name, "a clean file", RESULTS_MTIME + 5) + _write(report / member, "x SECRET-pm-token y", RESULTS_MTIME + 5) + digest = results_digest((report / "results.json").read_bytes()) + record_rendered(report / "_output", digest, rendered) + with pytest.raises(BundleError, match=r"package_manager\.token") as exc: + run_bundle(_args(output="b.zip")) + assert Path(member).name in str(exc.value) + assert not (root / "b.zip").exists() + + +def test_session_cookie_in_root_cache_blocks_bundle(root): + _write_report(root, results_extra=SESSION_COOKIE) + with pytest.raises(BundleError, match="auth cache cookie session"): + run_bundle(_args(output="b.zip")) + + +def test_session_cookie_in_auth_cache_path_blocks_bundle(root, tmp_path, monkeypatch): + user_cache = tmp_path / "user-cache" / ".vip-auth-cache.json" + user_cache.parent.mkdir() + user_cache.write_text( + json.dumps({"cookies": [{"name": "sid", "value": "USER-CACHE-COOKIE-0123"}]}), + encoding="utf-8", + ) + monkeypatch.setattr(bundle_module, "auth_cache_path", lambda: user_cache) + _write_report(root, results_extra="USER-CACHE-COOKIE-0123") + with pytest.raises(BundleError, match="auth cache cookie sid"): + run_bundle(_args(output="b.zip")) + + +def test_secret_in_a_made_member_names_it(root): + # A weak password that is also a setting that is not secret. + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace('password = "SECRET-auth-password"', 'password = "postgres"'), + encoding="utf-8", + ) + with pytest.raises(BundleError) as exc: + run_bundle(_args(output="b.zip")) + assert "config-summary.json (made by vip bundle): auth.password" in str(exc.value) + assert "Change the credential" in str(exc.value) + + +def test_allow_match_writes_the_bundle_and_records_it(root, capsys): + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace('password = "SECRET-auth-password"', 'password = "postgres"'), + encoding="utf-8", + ) + run_bundle(_args(output="b.zip", allow_match=["auth.password"])) + with zipfile.ZipFile(root / "b.zip") as zf: + manifest = json.loads(zf.read("manifest.json")) + assert manifest["allowed_matches"] == [ + {"member": "config-summary.json", "label": "auth.password"} + ] + assert "because --allow-match auth.password is set" in capsys.readouterr().err + + +def test_allow_match_does_not_cover_other_labels(root): + _write_report(root, failures_extra=CONNECT_KEY) + with pytest.raises(BundleError, match=r"connect\.api_key"): + run_bundle(_args(output="b.zip", allow_match=["auth.password"])) + + +def test_allow_match_with_an_unknown_label_fails(root): + with pytest.raises(BundleError, match=r"names no known credential: connect\.apikey"): + run_bundle(_args(output="b.zip", allow_match=["connect.apikey"])) + + +def test_manifest_is_scanned_too(root, monkeypatch): + # Only the manifest holds the value: the path of --results is in a warning there. + run_dir = root / "Run-Dir-0123" + _write(run_dir / "results.json", '{"deployment_name": "Acme Prod", "results": []}', 5.0) + monkeypatch.setenv("VIP_TEST_TOTP_SECRET", "Run-Dir-0123") + with pytest.raises(BundleError, match=r"manifest\.json \(made by vip bundle\)"): + run_bundle(_args(output="b.zip", results=str(run_dir / "results.json"))) + assert _leftovers(root) == [] + + +def _refuse_with_a_secret_hit(root, monkeypatch): + _write_report(root, results_extra=CONNECT_KEY) + + +def _refuse_with_a_manifest_hit(root, monkeypatch): + run_dir = root / "Run-Dir-0123" + _write(run_dir / "results.json", '{"deployment_name": "Acme Prod", "results": []}', 5.0) + monkeypatch.setenv("VIP_TEST_TOTP_SECRET", "Run-Dir-0123") + return {"results": str(run_dir / "results.json")} + + +def _refuse_with_an_unknown_label(root, monkeypatch): + return {"allow_match": ["connect.apikey"]} + + +def _refuse_with_a_bad_auth_cache(root, monkeypatch): + (root / ".vip-auth-cache.json").write_text('{"cookies": [', encoding="utf-8") + + +def _refuse_with_a_config_mismatch(root, monkeypatch): + _write_report(root, deployment="Other") + + +@pytest.mark.parametrize( + "refuse", + [ + _refuse_with_a_secret_hit, + _refuse_with_a_manifest_hit, + _refuse_with_an_unknown_label, + _refuse_with_a_bad_auth_cache, + _refuse_with_a_config_mismatch, + ], +) +def test_each_refusal_leaves_no_zip_and_no_temp_file(root, monkeypatch, refuse): + overrides = refuse(root, monkeypatch) or {} + with pytest.raises(BundleError): + run_bundle(_args(output="b.zip", **overrides)) + assert _leftovers(root) == [] + assert _leftovers(root / "report") == [] + + +def test_minted_key_in_root_cache_blocks_bundle(root, tmp_path, monkeypatch): + # auth_cache_path() points elsewhere, so only the file in the root holds the key. + other = tmp_path / "other-cache" / ".vip-auth-cache.json" + monkeypatch.setattr(bundle_module, "auth_cache_path", lambda: other) + _write_report(root, results_extra=MINTED_KEY) + with pytest.raises(BundleError, match="auth cache api_key"): + run_bundle(_args(output="b.zip")) + + +def test_root_cache_is_read_before_auth_cache_path(root, monkeypatch): + # The cache files in the root must be read before auth_cache_path() runs, + # because an auth_cache_path() with a side effect can remove them. + def migrating_auth_cache_path(): + (root / ".vip-auth-cache.meta.json").unlink() + return root / "moved" / ".vip-auth-cache.json" + + monkeypatch.setattr(bundle_module, "auth_cache_path", migrating_auth_cache_path) + _write_report(root, results_extra=MINTED_KEY) + with pytest.raises(BundleError, match="auth cache api_key"): + run_bundle(_args(output="b.zip")) + + +def test_results_path_sets_the_root(root, tmp_path, monkeypatch): + # Run from another directory: vip.toml and the auth cache come from the + # directory that holds report/. + elsewhere = tmp_path / "elsewhere" + elsewhere.mkdir() + monkeypatch.chdir(elsewhere) + _write_report(root, results_extra=MINTED_KEY) + with pytest.raises(BundleError, match="auth cache api_key"): + run_bundle(_args(output="b.zip", results=str(root / "report" / "results.json"))) + + +def test_config_for_another_deployment_fails(root): + _write_report(root, deployment="Acme Staging") + with pytest.raises(BundleError, match=r"does not match results\.json"): + run_bundle(_args(output="b.zip")) + + +def test_config_with_another_product_url_fails(root): + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace("https://connect.example.com", "https://connect.staging.example.com"), + encoding="utf-8", + ) + with pytest.raises(BundleError, match="the connect URL is"): + run_bundle(_args(output="b.zip")) + + +def test_config_with_another_scheme_and_path_passes(root): + # A browser login can change the scheme and the path of the URL that the + # run records. The host is the same, so the file is the correct one. + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace("https://connect.example.com", "http://connect.example.com/connect/"), + encoding="utf-8", + ) + _write_report(root) + run_bundle(_args(output="b.zip")) + assert (root / "b.zip").is_file() + + +def test_config_with_another_port_fails(root): + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace("https://connect.example.com", "https://connect.example.com:8443"), + encoding="utf-8", + ) + _write_report(root) + with pytest.raises(BundleError, match="the connect URL is"): + run_bundle(_args(output="b.zip")) + + +def test_config_with_no_url_for_a_product_of_the_run_fails(root): + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace("https://connect.example.com", ""), encoding="utf-8" + ) + _write_report(root) + with pytest.raises(BundleError, match="the connect URL is not set in the configuration"): + run_bundle(_args(output="b.zip")) + + +def test_config_with_a_url_for_a_product_that_the_run_did_not_have_fails(tmp_path): + config = VIPConfig() + config.workbench.url = "https://workbench.example.com" + results = json.dumps({"products": {"workbench": {"url": ""}, "connect": {"url": ""}}}) + with pytest.raises(BundleError, match=r"not set in results\.json"): + bundle_module._check_config_matches(config, tmp_path / "vip.toml", results.encode()) + + +def test_inline_connection_string_blocks_bundle_when_the_env_variable_replaces_it( + root, monkeypatch +): + # load_config gives the value of the variable, not the inline string. + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace("[data_sources.pg]", '[data_sources.pg]\nconnection_string_env = "PG_CONN"'), + encoding="utf-8", + ) + monkeypatch.setenv("PG_CONN", "postgresql://other:OTHER-db-password-9@db:5432/app") + _write_report(root, results_extra="SECRET-db-password") + with pytest.raises(BundleError, match=r"data_sources\.pg\.connection_string"): + run_bundle(_args(output="b.zip")) + + +def test_inline_git_token_blocks_bundle_when_the_clone_is_anonymous(root): + # With auth_method "none", the loaded configuration has no token. + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace("[workbench.git_test]", '[workbench.git_test]\nauth_method = "none"'), + encoding="utf-8", + ) + _write_report(root, results_extra="SECRET-git-token") + with pytest.raises(BundleError, match=r"workbench\.git_test\.token"): + run_bundle(_args(output="b.zip")) + + +def test_env_credential_blocks_bundle_when_the_config_has_an_inline_value(root, monkeypatch): + # The loaded configuration keeps the inline key, so the scan must read + # the variable directly. + monkeypatch.setenv("VIP_CONNECT_API_KEY", "ENV-ONLY-CONNECT-KEY-77") + _write_report(root, results_extra="ENV-ONLY-CONNECT-KEY-77") + with pytest.raises(BundleError, match="VIP_CONNECT_API_KEY"): + run_bundle(_args(output="b.zip")) + + +def test_key_in_auth_cache_path_blocks_bundle(root, tmp_path, monkeypatch): + # The auth cache can live outside the invocation directory. The bundle + # must read the file at auth_cache_path() too, not only the file in the root. + user_cache = tmp_path / "user-cache" / ".vip-auth-cache.json" + user_cache.parent.mkdir() + user_cache.with_suffix(".meta.json").write_text( + json.dumps({"api_key": "USER-CACHE-KEY-01"}), encoding="utf-8" + ) + monkeypatch.setattr(bundle_module, "auth_cache_path", lambda: user_cache) + _write_report(root, results_extra="USER-CACHE-KEY-01") + with pytest.raises(BundleError, match="auth cache api_key"): + run_bundle(_args(output="b.zip")) + + +def test_results_path_sets_the_rendered_files_dir(root, tmp_path, monkeypatch): + elsewhere = tmp_path / "elsewhere" + elsewhere.mkdir() + monkeypatch.chdir(elsewhere) + run_bundle(_args(output="b.zip", results=str(root / "report" / "results.json"))) + assert "report/_output/index.html" in _names(elsewhere / "b.zip") + + +def test_runs_from_inside_report_dir(root, monkeypatch): + monkeypatch.chdir(root / "report") + run_bundle(_args()) + (zip_path,) = root.glob("vip-bundle-*.zip") + assert "report/results.json" in _names(zip_path) + assert "config-summary.json" in _names(zip_path) + assert not list((root / "report").glob("*.zip")) + + +def test_inside_report_dir_still_scans_config_and_cache(root, monkeypatch): + _write_report(root, results_extra=MINTED_KEY, failures_extra=CONNECT_KEY) + monkeypatch.chdir(root / "report") + with pytest.raises(BundleError) as exc: + run_bundle(_args()) + assert "connect.api_key" in str(exc.value) + assert "auth cache api_key" in str(exc.value) + + +def test_directory_with_no_run_names_the_missing_results_first(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + with pytest.raises(BundleError, match="no results found") as exc: + run_bundle(_args()) + assert "vip.toml" not in str(exc.value) + + +def test_summary_says_what_the_auth_cache_of_the_run_holds(root): + run_bundle(_args(output="b.zip")) + with zipfile.ZipFile(root / "b.zip") as zf: + summary = json.loads(zf.read("config-summary.json")) + assert summary["auth_cache"] == {"session": True, "minted_connect_api_key": True} + assert "vip bundle" in summary["credential_flags"] + + +def test_summary_says_when_the_run_left_no_auth_cache(root): + (root / ".vip-auth-cache.json").unlink() + (root / ".vip-auth-cache.meta.json").unlink() + run_bundle(_args(output="b.zip")) + with zipfile.ZipFile(root / "b.zip") as zf: + summary = json.loads(zf.read("config-summary.json")) + assert summary["auth_cache"] == {"session": False, "minted_connect_api_key": False} + + +def test_missing_config_fails_without_flag(root): + (root / "vip.toml").unlink() + with pytest.raises(BundleError, match="--no-config"): + run_bundle(_args(output="b.zip")) + assert _leftovers(root) == [] + + +def test_explicit_missing_config_fails(root): + with pytest.raises(BundleError, match=r"no configuration file found at other\.toml"): + run_bundle(_args(output="b.zip", config="other.toml")) + + +def test_bad_config_fails_cleanly(root): + (root / "vip.toml").write_text("[connect\n", encoding="utf-8") + with pytest.raises(BundleError, match="could not read"): + run_bundle(_args(output="b.zip")) + + +def test_no_config_flag_bundles_and_warns(root, capsys): + (root / "vip.toml").unlink() + run_bundle(_args(output="b.zip", no_config=True)) + with zipfile.ZipFile(root / "b.zip") as zf: + manifest = json.loads(zf.read("manifest.json")) + names = set(zf.namelist()) + assert manifest["config_summary"] is False + assert manifest["secret_scan"] == "no-config" + assert "config-summary.json" not in names + assert "Warning: --no-config is set" in capsys.readouterr().err + + +def test_no_config_scan_still_uses_env_key(root, monkeypatch): + (root / "vip.toml").unlink() + monkeypatch.setenv("VIP_CONNECT_API_KEY", "ENV-KEY-1234") + _write_report(root, results_extra="ENV-KEY-1234") + with pytest.raises(BundleError, match=r"connect\.api_key"): + run_bundle(_args(output="b.zip", no_config=True)) + + +def test_stale_render_is_left_out_and_warns(root, capsys): + # vip verify ran again, but vip report did not. + results = root / "report" / "results.json" + text = results.read_text(encoding="utf-8").replace('"results": []', '"results": [{"n": 1}]') + results.write_text(text, encoding="utf-8") + os.utime(root / "report" / "failures.json") + run_bundle(_args(output="b.zip")) + assert "report/_output/index.html" not in _names(root / "b.zip") + err = capsys.readouterr().err + assert "was not made from" in err + assert "Run 'vip report' again" in err + + +def test_short_secret_warns(root, capsys): + toml = (root / "vip.toml").read_text(encoding="utf-8") + (root / "vip.toml").write_text( + toml.replace('password = "SECRET-auth-password"', 'password = "abc"'), + encoding="utf-8", + ) + run_bundle(_args(output="b.zip")) + assert "the value of auth.password is shorter than 4 characters" in capsys.readouterr().err + + +def test_existing_output_needs_force(root): + (root / "b.zip").write_bytes(b"old") + with pytest.raises(BundleError, match="--force"): + run_bundle(_args(output="b.zip")) + run_bundle(_args(output="b.zip", force=True)) + assert "manifest.json" in _names(root / "b.zip") + + +def test_default_output_name(root, capsys): + run_bundle(_args()) + (zip_path,) = root.glob("vip-bundle-*.zip") + assert re.fullmatch(r"vip-bundle-acme-prod-\d{8}T\d{6}Z\.zip", zip_path.name) + assert "Bundle written:" in capsys.readouterr().out + + +def test_unwritable_output_dir_fails_cleanly(root): + with pytest.raises(BundleError, match="could not write"): + run_bundle(_args(output="no-such-dir/b.zip")) + + +def test_bundle_subcommand_is_registered(monkeypatch, capsys): + monkeypatch.setattr(sys, "argv", ["vip", "bundle", "--help"]) + with pytest.raises(SystemExit) as exc: + main() + assert exc.value.code == 0 + out = capsys.readouterr().out + assert "--output" in out + assert "--no-config" in out + assert "--allow-match" in out + + +def test_bundle_error_exits_with_code_one(tmp_path, monkeypatch, capsys): + monkeypatch.chdir(tmp_path) + monkeypatch.setattr(sys, "argv", ["vip", "bundle", "--no-config"]) + with pytest.raises(SystemExit) as exc: + main() + assert exc.value.code == 1 + assert "Error: no results found" in capsys.readouterr().err + + +def test_main_writes_a_bundle_from_real_arguments(root, monkeypatch, capsys): + # The parser, not a hand-built Namespace, sets each option. A change to an + # option's action or dest fails here. + _write_report(root, results_extra="SECRET-auth-password") + argv = ["vip", "bundle", "--output", "b.zip", "--force", "--allow-match", "auth.password"] + (root / "b.zip").write_bytes(b"old") + monkeypatch.setattr(sys, "argv", argv) + main() + with zipfile.ZipFile(root / "b.zip") as zf: + manifest = json.loads(zf.read("manifest.json")) + assert manifest["allowed_matches"] == [ + {"member": "report/results.json", "label": "auth.password"} + ] + assert "Bundle written" in capsys.readouterr().out + + +def test_main_repeats_allow_match(root, monkeypatch): + _write_report(root, results_extra="SECRET-auth-password SECRET-pm-token") + argv = ["vip", "bundle", "--output", "b.zip"] + argv += ["--allow-match", "auth.password", "--allow-match", "package_manager.token"] + monkeypatch.setattr(sys, "argv", argv) + main() + assert (root / "b.zip").is_file() + + +def test_vip_config_selects_the_configuration_file(root, tmp_path, monkeypatch): + elsewhere = tmp_path / "conf" / "site.toml" + elsewhere.parent.mkdir() + (root / "vip.toml").rename(elsewhere) + monkeypatch.setenv("VIP_CONFIG", str(elsewhere)) + _write_report(root, results_extra="SECRET-pm-token") + with pytest.raises(BundleError, match=r"package_manager\.token"): + run_bundle(_args(output="b.zip")) + + +_RUN_URL = "https://admin:RUN-URL-PASS-0123@connect.example.com" + + +def _write_report_with_run_url(root: Path) -> None: + report = root / "report" + results = { + "deployment_name": "Acme Prod", + "products": {"connect": {"url": _RUN_URL}}, + "results": [], + } + _write(report / "results.json", json.dumps(results), RESULTS_MTIME) + + +def test_no_config_scan_uses_the_product_url_credentials_in_results(root): + (root / "vip.toml").unlink() + _write_report_with_run_url(root) + with pytest.raises(BundleError, match=r"results\.json connect\.url password"): + run_bundle(_args(output="b.zip", no_config=True)) + + +def test_config_scan_uses_the_product_url_credentials_in_results(root): + # vip.toml has the same host with no userinfo, so it passes the match check. + _write_report_with_run_url(root) + with pytest.raises(BundleError, match=r"results\.json connect\.url password"): + run_bundle(_args(output="b.zip")) + + +def test_bad_auth_cache_stops_the_bundle(root): + (root / ".vip-auth-cache.meta.json").write_text('{"api_key": "x"', encoding="utf-8") + with pytest.raises(BundleError, match="could not read the auth cache file"): + run_bundle(_args(output="b.zip")) diff --git a/selftests/test_cli_report.py b/selftests/test_cli_report.py index 7dad44313..ef0f97c08 100644 --- a/selftests/test_cli_report.py +++ b/selftests/test_cli_report.py @@ -9,6 +9,7 @@ from __future__ import annotations import argparse +import hashlib import os import subprocess import sys @@ -375,6 +376,116 @@ def _failing_html(cmd, cwd=None, **kwargs): assert rendered == ["index.qmd"], "must stop at the first failed page" +class TestRenderStamp: + """vip report ties each rendered file to the results.json it was made from.""" + + @staticmethod + def _report_dir(tmp_path, monkeypatch, results: str = '{"results": []}'): + monkeypatch.chdir(tmp_path) + report_dir = tmp_path / "report" + report_dir.mkdir(exist_ok=True) + (report_dir / "results.json").write_text(results, encoding="utf-8") + return report_dir + + @staticmethod + def _age_outputs(report_dir): + """Give each rendered file an old file time, as files of an earlier render have.""" + for path in (report_dir / "_output").iterdir(): + os.utime(path, (1_000_000.0, 1_000_000.0)) + + def test_stamp_names_every_rendered_file(self, tmp_path, monkeypatch): + from vip import cli + from vip.bundle.stamp import read_render_stamp, results_digest + + report_dir = self._report_dir(tmp_path, monkeypatch) + monkeypatch.setattr("vip.cli.report.subprocess.run", _fake_quarto(create_output=True)) + cli.run_report(_make_args()) + digest = results_digest((report_dir / "results.json").read_bytes()) + stamp = read_render_stamp(report_dir / "_output") + assert stamp is not None + assert stamp[0] == digest + assert stamp[1] == { + "index.html": hashlib.sha256(b"report").hexdigest(), + "details.html": hashlib.sha256(b"report").hexdigest(), + "vip-report.pdf": hashlib.sha256(b"%PDF-fake").hexdigest(), + } + + def test_failed_pdf_is_not_in_the_new_stamp(self, tmp_path, monkeypatch): + # The PDF of the earlier run stays on disk, but the stamp leaves it out. + from vip import cli + from vip.bundle.stamp import read_render_stamp + + report_dir = self._report_dir(tmp_path, monkeypatch) + monkeypatch.setattr("vip.cli.report.subprocess.run", _fake_quarto(create_output=True)) + cli.run_report(_make_args()) + self._age_outputs(report_dir) + self._report_dir(tmp_path, monkeypatch, results='{"results": [{"n": 1}]}') + monkeypatch.setattr( + "vip.cli.report.subprocess.run", _fake_quarto(create_output=True, pdf_returncode=1) + ) + cli.run_report(_make_args()) + stamp = read_render_stamp(report_dir / "_output") + assert stamp is not None + assert set(stamp[1]) == {"index.html", "details.html"} + assert (report_dir / "_output" / "vip-report.pdf").is_file() + + def test_file_that_the_render_did_not_write_gets_no_stamp(self, tmp_path, monkeypatch, capsys): + # The second render exits with 0 and writes nothing, so each file on + # disk is from the first results.json. + from vip import cli + from vip.bundle.stamp import read_render_stamp + + report_dir = self._report_dir(tmp_path, monkeypatch) + monkeypatch.setattr("vip.cli.report.subprocess.run", _fake_quarto(create_output=True)) + cli.run_report(_make_args()) + self._age_outputs(report_dir) + self._report_dir(tmp_path, monkeypatch, results='{"results": [{"n": 1}]}') + monkeypatch.setattr("vip.cli.report.subprocess.run", _fake_quarto(create_output=False)) + cli.run_report(_make_args()) + stamp = read_render_stamp(report_dir / "_output") + assert stamp is not None + assert stamp[1] == {} + assert "did not write" in capsys.readouterr().err + + def test_failed_html_clears_the_stamp(self, tmp_path, monkeypatch): + from vip import cli + from vip.bundle.stamp import read_render_stamp + + report_dir = self._report_dir(tmp_path, monkeypatch) + monkeypatch.setattr("vip.cli.report.subprocess.run", _fake_quarto(create_output=True)) + cli.run_report(_make_args()) + monkeypatch.setattr( + "vip.cli.report.subprocess.run", + lambda cmd, cwd=None, **kwargs: types.SimpleNamespace(returncode=3), + ) + with pytest.raises(SystemExit): + cli.run_report(_make_args()) + assert read_render_stamp(report_dir / "_output") is None + + def test_stamp_that_cannot_be_cleared_is_a_report_error(self, tmp_path, monkeypatch): + from vip import cli + from vip.errors import ReportError + + self._report_dir(tmp_path, monkeypatch) + + def fail(output_dir): + raise PermissionError("read-only file system") + + monkeypatch.setattr("vip.cli.report.clear_render_stamp", fail) + with pytest.raises(ReportError, match="render stamp"): + cli.run_report(_make_args()) + + def test_results_that_cannot_be_read_is_a_report_error(self, tmp_path, monkeypatch): + from vip import cli + from vip.errors import ReportError + + report_dir = self._report_dir(tmp_path, monkeypatch) + (report_dir / "results.json").unlink() + (report_dir / "results.json").mkdir() + with pytest.raises(ReportError, match="cannot read"): + cli.run_report(_make_args()) + + class TestSupportFileResolution: """troubleshooting.toml and feature files resolve from checkout or install.""" diff --git a/selftests/test_errors.py b/selftests/test_errors.py index 72b32b8f9..8c8665d39 100644 --- a/selftests/test_errors.py +++ b/selftests/test_errors.py @@ -13,6 +13,7 @@ AuthConfigError, AuthError, AuthTimeoutError, + BundleError, ConfigError, InstallError, ProductUnreachableError, @@ -33,7 +34,14 @@ def test_message_is_the_str_representation(self): @pytest.mark.parametrize( "cls", - [ConfigError, AuthError, ProductUnreachableError, InstallError, ReportError], + [ + ConfigError, + AuthError, + ProductUnreachableError, + InstallError, + ReportError, + BundleError, + ], ) def test_subclasses_are_vip_errors(self, cls): assert issubclass(cls, VipError) diff --git a/selftests/test_reporting.py b/selftests/test_reporting.py index 6c73d3e49..83a1e2fa5 100644 --- a/selftests/test_reporting.py +++ b/selftests/test_reporting.py @@ -640,6 +640,21 @@ def test_failure_fallback_when_no_error_details(self, tmp_path): assert failure is not None assert failure.get("message") == "test failed" + def test_generated_at_is_a_suite_property(self, tmp_path): + out = tmp_path / "junit.xml" + write_junit_xml(self._sample(), out) + suite = ET.parse(out).getroot().find("testsuite") + prop = suite.find("properties/property[@name='generated_at']") + assert prop.get("value") == "2026-07-21T12:00:00+00:00" + # The properties element must come before the test cases. + assert suite[0].tag == "properties" + assert b"2026-07-21T12:00:00+00:00" in out.read_bytes() + + def test_no_generated_at_gives_no_properties(self, tmp_path): + out = tmp_path / "junit.xml" + write_junit_xml(ReportData(), out) + assert ET.parse(out).getroot().find("testsuite/properties") is None + def test_empty_report_data_is_well_formed(self, tmp_path): out = tmp_path / "junit.xml" write_junit_xml(ReportData(), out) @@ -703,6 +718,18 @@ def test_na_version_wording_is_fallback_without_skip_reason(self, tmp_path): class TestWriteSarif: + def test_generated_at_is_a_run_property(self, tmp_path): + out = tmp_path / "results.sarif" + write_sarif(ReportData(generated_at="2026-07-21T12:00:00+00:00"), out) + run = json.loads(out.read_text(encoding="utf-8"))["runs"][0] + assert run["properties"] == {"generated_at": "2026-07-21T12:00:00+00:00"} + assert b"2026-07-21T12:00:00+00:00" in out.read_bytes() + + def test_no_generated_at_gives_no_run_properties(self, tmp_path): + out = tmp_path / "results.sarif" + write_sarif(ReportData(), out) + assert "properties" not in json.loads(out.read_text(encoding="utf-8"))["runs"][0] + def _sample(self) -> ReportData: return ReportData( results=[ diff --git a/src/vip/bundle/__init__.py b/src/vip/bundle/__init__.py new file mode 100644 index 000000000..3eaa65130 --- /dev/null +++ b/src/vip/bundle/__init__.py @@ -0,0 +1 @@ +"""Build a ``vip bundle`` zip: report files and a configuration summary, with no credentials.""" diff --git a/src/vip/bundle/archive.py b/src/vip/bundle/archive.py new file mode 100644 index 000000000..463b0cbc1 --- /dev/null +++ b/src/vip/bundle/archive.py @@ -0,0 +1,379 @@ +"""Collect the allowlisted report files, scan them for secrets, and write the zip.""" + +from __future__ import annotations + +import html +import json +import re +import tempfile +import zipfile +from collections.abc import Sequence +from dataclasses import dataclass, field +from pathlib import Path +from urllib.parse import quote, quote_plus + +from vip.bundle.stamp import file_digest, read_render_stamp, results_digest +from vip.errors import BundleError + +# Read from the directory of results.json. results.json must exist. +REPORT_SIDECAR_FILES = ("results.json", "failures.json", "junit.xml", "results.sarif") +# Read from /_output/. _quarto.yml sets embed-resources, so each +# page is one self-contained file and no site_libs/ directory is needed. +RENDERED_FILES = ("index.html", "details.html", "vip-report.pdf") +# The secret scan cannot read the compressed text in the PDF. The HTML pages +# hold the same text, so the PDF is in the bundle only together with them. +PDF_FILE = "vip-report.pdf" +# Files that hold credentials. --results must never name one of them. +FORBIDDEN_NAMES = frozenset( + {"vip.toml", ".vip-auth-cache.json", ".vip-auth-cache.meta.json", ".vip-install.json"} +) + +# A value shorter than this matches almost any file, so the scan skips it. +MIN_SECRET_LENGTH = 4 +# Report text can hold only a part of a secret. failures.json and the error +# parser cut long text, which keeps the start. pytest shortens a long repr in +# the middle, which keeps the start and the end. So for each secret that is +# at least PARTIAL_MIN_SECRET_LENGTH long, the scan also looks for its first +# and its last PARTIAL_LENGTH characters, in each encoded form. A part with +# no digit or no letter, such as "password" or "20261105", matches normal +# report text, so the scan skips it. Random keys almost always have both. The +# start of every JWT, "eyJhbGci", has no digit, so the scan uses only the end +# of a JWT. +PARTIAL_LENGTH = 8 +PARTIAL_MIN_SECRET_LENGTH = 16 + +# A rendered page is about 2 MB, most of it fonts and images in data: URIs and +# the bodies of