Skip to content

feat(cli): add vip bundle to build a shareable report archive - #824

Merged
timtalbot merged 11 commits into
mainfrom
feat/vip-bundle
Nov 17, 2026
Merged

timtalbot merged 11 commits into
mainfrom
feat/vip-bundle

Conversation

@timtalbot

@timtalbot timtalbot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #794.

Summary

vip bundle writes a zip of VIP report output that you can send to another person. It builds the zip from an allowlist (a fixed list of permitted files), not from a copy of the directory. Before it writes the zip, it looks in each file for each credential that it knows. If it finds one, it stops.

vip verify --config vip.toml
vip report
vip bundle

What the zip holds

  • report/results.json.
  • failures.json, junit.xml, and results.sarif. Each one must be from the same run as results.json.
  • report/_output/index.html, details.html, and vip-report.pdf. Each one must be a file that vip report made from this results.json, with no change after the render. The PDF goes in only together with both HTML pages, because the scan cannot read the text in a PDF.
  • config-summary.json: the products, the URLs, the auth provider, and other fields with no credentials.
  • version.txt: the output of vip version.
  • manifest.json: the members, the files that were left out with the reason, the warnings, and the scan counts.

The zip never holds vip.toml, the auth cache files, or .vip-install.json.

The secret scan

The scan gets its values from three sources: vip.toml, the environment variables, and the auth cache (session cookies, localStorage values, and a minted Connect API key). It looks for each value in each member, in these forms: raw, JSON-escaped, HTML-escaped, XML text, percent-encoded, and Python repr. It also looks for the first and the last 8 characters of a long value, because report text can cut a value.

  • If the scan finds a value, the command stops and writes no zip. The error names the file and the label of the credential. It never prints the value.
  • If the value is not a real secret, --allow-match LABEL lets the command continue for that label. The manifest records it.
  • The manifest and the last output line give the number of values that the scan knew from each source. If the scan knew no value, the command prints a warning.
  • An auth cache value that is only an email address is a login name, so the scan does not treat it as a secret.

The scan cannot find a credential that it does not know. Examples are a value that an extension test printed, or a value that was set only in the shell that ran vip verify. The User Guide says this and tells the user to look at the zip before they send it.

Differences from the issue

  1. No vip status output. vip status sends a request with live credentials to each product, so it needs network access. It also shows the state at bundle time, not at the time of the run. results.json already has the product URLs, the vip version, the platform, and the Python version. The product version in results.json is the value from vip.toml, not the version that the server reports, and it is often empty. So this branch does not deliver the "platform details vip status already collects" item of the issue. Please accept this, or ask for a follow-up issue that adds the data.
  2. No copy of vip.toml, redacted or raw. config-summary.json is built from a list of named fields. A new configuration field stays out of the summary until a developer adds it to the list.
  3. The secret field list is in src/vip/bundle/summary.py, not in one constant in config.py. config.py did not change. If a configuration field has no class (summary, secret, or excluded), a selftest fails. If __repr__ hides a field that is not in the secret list, a second selftest fails.
  4. Old files are left out. A rendered page or PDF must come from the results.json in the bundle. If it does not, the command leaves it out. A failures.json, junit.xml, or results.sarif from a different run is left out. The manifest gives the reason for each one.
  5. With no configuration file, the command stops. Without vip.toml, the scan misses each credential that only that file holds. --no-config writes the bundle with a warning, for a run that used vip verify --connect-url and no file. If vip.toml does not match results.json, the command also stops. It compares the deployment name, and the host and the port of each product URL.

Changes outside the new command

  • vip report writes _output/.vip-render.json. The file holds the SHA-256 of the results.json that was rendered and the SHA-256 of each rendered file. vip report deletes it before each render, and then adds only the files that the render wrote.
  • vip verify writes the generated_at value of the run into junit.xml (a suite property) and results.sarif (a run property). vip bundle uses the value to pair these files with results.json.
  • vip report gives a clear error for a results.json that it cannot read.
  • New error class BundleError, with exit code 1.

What the recipient gets, and does not get

The bundle holds the login email of the test user (test output prints it), the user name in local paths, and the product host names. It has no server version for a product unless vip.toml sets one. In config-summary.json, each *_set value describes the configuration file and the environment of the vip bundle command. The auth_cache values show two facts about the run: a saved browser session, and a minted Connect API key.

Interaction with #793

#793 moves the auth cache out of the invocation directory. This branch reads the cache files in the invocation directory first, and then the files at auth_cache_path(). A trial merge is clean in both orders, and the selftests pass on the merged tree. After #793 merges, the AGENTS.md row for auth_cache_path() and the README section of that PR need a small edit.

One limit comes with #793. If a run directory moves to a different machine, the cache does not move with it. The scan then knows no cache value. The scan counts in the manifest and in the output show this.

Tests

  • just check and just typecheck pass. The selftests have 2809 passed and 3 skipped.
  • selftests/test_cli_bundle.py builds a bundle from a directory that holds vip.toml with a value in each credential field, both auth cache files, and .vip-install.json. It makes sure that the member list is the allowlist and that no seeded value is in the bytes of the zip.
  • A set of tests puts a credential into each report file in turn. The files are results.json, failures.json, junit.xml, results.sarif, each HTML page, and the PDF. Each one stops the bundle.
  • Hand mutation tests: 28 small code changes in the scan, the allowlist, the stamp, and the summary. Each one makes a selftest fail.
  • A real run against a deployment with Okta and --interactive-auth: the zip has 8 members. An independent script found none of the 15 auth cache values in it, in nine encoded forms and in the inflated PDF streams.

Known limits

  • The scan does not look for a token that is inside a JSON localStorage value. It looks for the whole value.
  • The scan does not look for base64 forms.
  • A junit.xml or results.sarif from an older vip has no generated_at value. For those files the command uses the file time: a file that is older than results.json is left out.
  • The manifest holds the absolute local path of a file that was left out.

Read rendered files from the report directory of --results. Give the user
and password of the clone URL their own labels, so that --allow-match for
one does not cover the other. A stamp that cannot be cleared is a
ReportError. The repr drift test now covers str | None fields.
Okta keeps the login name in the ln cookie, and test output often prints
it, so every bundle after an Okta login stopped on a false match.
The manifest and the output now give the number of known credentials by
source, and the command warns if the scan knew none. failures.json is
matched to results.json by generated_at, so an edit of results.json does
not drop it. The email exemption no longer accepts a URL with
credentials or a query string. vip report gives a ReportError for a
results.json that it cannot read.
Add tests for the email rule shape, the 16-character cache floor, URL
credentials for each product, the proxy variables, a secret in each
report file, the session cookie from both cache places, links in the
report directory, and two more encoded forms.
… summary

vip verify now writes generated_at to junit.xml and results.sarif, and
the bundle keeps a file that holds the value of results.json. The
configuration summary says what the auth cache of the run holds and
what the *_set flags describe. A directory with no run gives the error
for the missing results.json first.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 18:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Credential collection and configuration-matching gaps can allow secrets into archives, while artifact validation can include reports from a different run.

5 open issues
What changed in this PR

Adds vip bundle to package VIP deployment reports for sharing, addressing the archive workflow in #794 while excluding configuration and auth-cache files.

Changes:

  • Adds an allowlisted archive builder, credential scanning, configuration summary, and manifest.
  • Records render provenance and run timestamps to help identify matching report artifacts.
  • Adds selftests and documents the command, safeguards, and scan limits.
File Description
website/​src/​pages/​guide.astro Documents bundling, credential scanning, and artifact selection.
src/​vip/​reporting.py Adds run timestamps to JUnit and SARIF output.
src/​vip/​errors.py Defines BundleError.
src/​vip/​cli/​report.py Records render provenance and handles unreadable results.
src/​vip/​cli/​bundle.py Implements bundle orchestration and validation.
src/​vip/​cli/​app.py Registers the command and its options.
src/​vip/​cli/​__init__.py Exports run_bundle.
src/​vip/​bundle/​summary.py Builds configuration summaries and collects known credentials.
src/​vip/​bundle/​stamp.py Reads and writes render-provenance stamps.
src/​vip/​bundle/​archive.py Collects artifacts, scans credentials, and writes archives atomically.
src/​vip/​bundle/​__init__.py Introduces the bundle package.
selftests/​test_reporting.py Tests JUnit and SARIF run properties.
selftests/​test_errors.py Checks BundleError inheritance.
selftests/​test_cli_report.py Tests render stamps and report read failures.
selftests/​test_cli_bundle.py Tests command behavior and archive safeguards.
selftests/​test_bundle_summary.py Tests field classification and credential collection.
selftests/​test_bundle_archive.py Tests artifact selection, scanning, and atomic writes.
README.md Adds the command to the CLI reference.
AGENTS.md Documents bundle modules and error handling.

Review effort: Balanced

Comment thread src/vip/bundle/summary.py Resolved
Comment thread src/vip/cli/bundle.py Resolved
Comment thread src/vip/cli/bundle.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/cli/bundle.py Resolved
…ecars by run

The scan reads each credential variable and each inline connection
string directly, because the loaded configuration holds only one of an
inline value and its variable. The configuration match compares the
host and the port of each product URL, and stops if only one side has
a URL. A junit.xml or results.sarif with a different generated_at is
left out at any file time.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/cli/bundle.py Resolved
Comment thread src/vip/cli/report.py Resolved
The render stamp holds the SHA-256 of each rendered file, and vip
report stamps only a file that the render wrote. vip bundle leaves out
a rendered file that changed after the stamp, and includes the PDF
only together with both HTML pages, because the scan cannot read PDF
text. The scan also reads workbench.git_test.token from the text of
vip.toml, because the loaded configuration drops it for an anonymous
clone.
@timtalbot
timtalbot marked this pull request as ready for review November 16, 2026 15:30

@ian-flores ian-flores left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, Tim. The design holds up: the allowlist, the scan in memory before the write, the atomic zip, and the render stamp with a digest for each file. The selftest that #794 asked for is there and checks a literal member list and the inflated bytes.

I am requesting changes because I found five cases where the command exits 0, prints "Bundle written", and the zip holds a credential that the scan could have known. Each one is an inline comment with a short reproduction. All five have small fixes.

On the differences from the issue:

  • No vip status output: accepted. Your reasons are right, and a bundle command should not need network access.
  • No copy of vip.toml: accepted.
  • Secret field list in summary.py with selftests, not one constant in config.py: accepted, with one gap to close. _sentinel_values fills only str and str | None fields, so a secret field of type list or dict that __repr__ hides is not seen as hidden. Also, a new field that a developer puts in the non-secret part of EXCLUDED_FIELDS and does not hide in __repr__ passes both tests and the scan never looks for it. Please make the tests cover both.
  • Old files are left out, and no configuration file stops the command: accepted.

Tests to add in this branch:

  • One test that runs main() with real arguments (vip bundle --output ... --force --allow-match X) through to a written zip. Every current test builds the Namespace by hand, so a change to an option's action or dest leaves the suite green.
  • The scan thresholds at their boundary: a 4-character secret is scanned, a 16-character secret gets the partial forms, and an exact 8-character part is found. Today MIN_SECRET_LENGTH = 5 and PARTIAL_MIN_SECRET_LENGTH = 24 both pass the suite.
  • Each entry of SECRET_ENV_VARS next to a different inline value. Removing VIP_PM_TOKEN, VIP_TEST_PASSWORD, or VIP_WORKBENCH_API_KEY from the list passes the suite.
  • On each refusal path, no zip and no temp file is left. test_manifest_is_scanned_too checks only the exception.
  • VIP_CONFIG selects the configuration file. No test sets it.

Docs:

  • The guide lists a configuration summary in the zip with no condition, but --no-config leaves it out. The only example in the guide is vip bundle --no-config.
  • The guide says the command stops when there is no vip.toml. It also stops when the deployment name, or the host or port of a product URL, differs from results.json.
  • For --allow-match, say that a label with spaces needs quotes, that the label must be one from the error, and that the value then stays in the zip.
  • The comment above SECRET_ENV_VARS in summary.py does not parse as one sentence.

Comment thread src/vip/bundle/summary.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/cli/bundle.py Resolved
Comment thread src/vip/bundle/summary.py Resolved
Comment thread src/vip/bundle/summary.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
Comment thread src/vip/bundle/archive.py Resolved
A damaged auth cache file or a report file that is not UTF-8 stops the
bundle. The scan reads the product URL credentials in results.json, finds
a URL password with a raw "#", "?", or "/", gives each URL part and each
cookie domain its own label, and skips data: URIs, scripts, and styles in
HTML pages. If results.json has a run time, a sidecar with none is left
out. Tests cover the scan limits, each credential variable, main(),
VIP_CONFIG, and cleanup after a refusal.
@timtalbot

Copy link
Copy Markdown
Contributor Author

The items in the review body are fixed in a3d946f. The inline threads have their own replies.

Secret field tests

  • _sentinel_values now also gives a sentinel to list[str] and dict[str, str] fields, optional or not. A list or dict field that __repr__ hides is now found as a secret (test_sentinel_values_cover_str_list_and_dict_fields, test_every_repr_masked_field_is_a_secret).
  • The part of EXCLUDED_FIELDS that is not secret is now a literal set in the test. A new field there needs a test edit that a reviewer sees (test_fields_left_out_of_the_summary_that_are_not_secrets_are_pinned).

Tests

  • main() with real arguments: test_main_writes_a_bundle_from_real_arguments runs vip bundle --output b.zip --force --allow-match auth.password through to a written zip. test_main_repeats_allow_match passes --allow-match two times.
  • Scan limits: test_scan_threshold_boundaries checks a 4-character value, a 16-character value with its partial forms, a 15-character value with none, and an exact 8-character part.
  • Credential variables: test_each_env_variable_is_read_next_to_an_inline_value runs once for each entry of SECRET_ENV_VARS. It sets the variable before the configuration loads, with a different inline value. If a new entry has no case, test_every_env_variable_has_an_inline_builder fails.
  • Refusals: test_each_refusal_leaves_no_zip_and_no_temp_file covers a scan hit, a manifest hit, an unknown --allow-match label, a bad auth cache file, and a configuration mismatch. test_manifest_is_scanned_too now also checks for left-over files.
  • VIP_CONFIG: test_vip_config_selects_the_configuration_file moves vip.toml and points VIP_CONFIG at it.

Before the fix, the suite passed with MIN_SECRET_LENGTH = 5 and with PARTIAL_MIN_SECRET_LENGTH = 24. It also passed with no VIP_PM_TOKEN, VIP_TEST_PASSWORD, VIP_WORKBENCH_API_KEY, or VIP_PACKAGE_MANAGER_TOKEN in SECRET_ENV_VARS. Now each of these changes fails at least one test. PARTIAL_MIN_SECRET_LENGTH = 15, PARTIAL_LENGTH = 9, and action="store" for --allow-match also fail.

Docs

  • The guide says that the zip holds the configuration summary only without --no-config.
  • The guide says that the command also stops on a different deployment name, or a different host or port of a product URL. It also says that the command stops on an auth cache file that it cannot read, or a report file that is not UTF-8.
  • A new paragraph on --allow-match says that the label must be one from the error. It also says that a label with spaces needs quotes, and that the value then stays in the zip. The --allow-match help text and the scan error say the same.
  • The comment above SECRET_ENV_VARS is rewritten as separate sentences.

@ian-flores ian-flores left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Tim. I checked a3d946f against my review, and everything is fixed.

  • I ran my repro for each of the five cases against the real CLI. Each one now stops with a BundleError and leaves no zip behind.
  • I also checked the smaller items: the lowercase %2f, a deeply nested results.json, a non-UTF-8 env value, and sidecars with no run time.
  • Setting MIN_SECRET_LENGTH = 5 or PARTIAL_MIN_SECRET_LENGTH = 24 now fails a test.
  • The docs now say what I asked for.

I am fine with vip verify still writing the URL userinfo. vip bundle refuses until the user removes it, so nothing leaks.

Approving.

@timtalbot
timtalbot merged commit 54db20c into main Nov 17, 2026
55 checks passed
@timtalbot
timtalbot deleted the feat/vip-bundle branch November 17, 2026 14:47
@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.2
Preview removed because the pull request was closed.
2026-11-17 14:48 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] vip bundle: build a shareable archive from an allowlist of report artifacts

3 participants