Repository navigation
feat(cli): add vip bundle to build a shareable report archive - #824
Conversation
Read rendered files from the report directory of --results. Give the user and password of the clone URL their own labels, so that --allow-match for one does not cover the other. A stamp that cannot be cleared is a ReportError. The repr drift test now covers str | None fields.
Okta keeps the login name in the ln cookie, and test output often prints it, so every bundle after an Okta login stopped on a false match.
The manifest and the output now give the number of known credentials by source, and the command warns if the scan knew none. failures.json is matched to results.json by generated_at, so an edit of results.json does not drop it. The email exemption no longer accepts a URL with credentials or a query string. vip report gives a ReportError for a results.json that it cannot read.
Add tests for the email rule shape, the 16-character cache floor, URL credentials for each product, the proxy variables, a secret in each report file, the session cookie from both cache places, links in the report directory, and two more encoded forms.
… summary vip verify now writes generated_at to junit.xml and results.sarif, and the bundle keeps a file that holds the value of results.json. The configuration summary says what the auth cache of the run holds and what the *_set flags describe. A directory with no run gives the error for the missing results.json first.
There was a problem hiding this comment.
🟡 Changes recommended
Credential collection and configuration-matching gaps can allow secrets into archives, while artifact validation can include reports from a different run.
5 open issues
Scan all credential environment variables, not just unresolved ones · New Treat missing product URLs as configuration mismatches · New Scan inline connection-string secrets alongside environment credentials · New Reject artifact run-ID mismatches regardless of modification time · New Compare original configured URLs, not resolved reporting URLs · New
What changed in this PR
Adds vip bundle to package VIP deployment reports for sharing, addressing the archive workflow in #794 while excluding configuration and auth-cache files.
Changes:
- Adds an allowlisted archive builder, credential scanning, configuration summary, and manifest.
- Records render provenance and run timestamps to help identify matching report artifacts.
- Adds selftests and documents the command, safeguards, and scan limits.
| File | Description |
|---|---|
website/src/pages/guide.astro |
Documents bundling, credential scanning, and artifact selection. |
src/vip/reporting.py |
Adds run timestamps to JUnit and SARIF output. |
src/vip/errors.py |
Defines BundleError. |
src/vip/cli/report.py |
Records render provenance and handles unreadable results. |
src/vip/cli/bundle.py |
Implements bundle orchestration and validation. |
src/vip/cli/app.py |
Registers the command and its options. |
src/vip/cli/__init__.py |
Exports run_bundle. |
src/vip/bundle/summary.py |
Builds configuration summaries and collects known credentials. |
src/vip/bundle/stamp.py |
Reads and writes render-provenance stamps. |
src/vip/bundle/archive.py |
Collects artifacts, scans credentials, and writes archives atomically. |
src/vip/bundle/__init__.py |
Introduces the bundle package. |
selftests/test_reporting.py |
Tests JUnit and SARIF run properties. |
selftests/test_errors.py |
Checks BundleError inheritance. |
selftests/test_cli_report.py |
Tests render stamps and report read failures. |
selftests/test_cli_bundle.py |
Tests command behavior and archive safeguards. |
selftests/test_bundle_summary.py |
Tests field classification and credential collection. |
selftests/test_bundle_archive.py |
Tests artifact selection, scanning, and atomic writes. |
README.md |
Adds the command to the CLI reference. |
AGENTS.md |
Documents bundle modules and error handling. |
Review effort: Balanced
…ecars by run The scan reads each credential variable and each inline connection string directly, because the loaded configuration holds only one of an inline value and its variable. The configuration match compares the host and the port of each product URL, and stops if only one side has a URL. A junit.xml or results.sarif with a different generated_at is left out at any file time.
There was a problem hiding this comment.
🟡 Changes recommended
Missing artifact-content verification and incomplete credential scanning can allow stale or replaced reports containing known secrets into bundles.
4 open issues
5 resolved since last review
Scan inline connection-string secrets alongside environment credentials Treat missing product URLs as configuration mismatches Scan all credential environment variables, not just unresolved ones Compare original configured URLs, not resolved reporting URLs Reject artifact run-ID mismatches regardless of modification time
Review effort: Balanced
The render stamp holds the SHA-256 of each rendered file, and vip report stamps only a file that the render wrote. vip bundle leaves out a rendered file that changed after the stamp, and includes the PDF only together with both HTML pages, because the scan cannot read PDF text. The scan also reads workbench.git_test.token from the text of vip.toml, because the loaded configuration drops it for an anonymous clone.
ian-flores
left a comment
There was a problem hiding this comment.
Thanks for this, Tim. The design holds up: the allowlist, the scan in memory before the write, the atomic zip, and the render stamp with a digest for each file. The selftest that #794 asked for is there and checks a literal member list and the inflated bytes.
I am requesting changes because I found five cases where the command exits 0, prints "Bundle written", and the zip holds a credential that the scan could have known. Each one is an inline comment with a short reproduction. All five have small fixes.
On the differences from the issue:
- No
vip statusoutput: accepted. Your reasons are right, and a bundle command should not need network access. - No copy of
vip.toml: accepted. - Secret field list in
summary.pywith selftests, not one constant inconfig.py: accepted, with one gap to close._sentinel_valuesfills onlystrandstr | Nonefields, so a secret field of type list or dict that__repr__hides is not seen as hidden. Also, a new field that a developer puts in the non-secret part ofEXCLUDED_FIELDSand does not hide in__repr__passes both tests and the scan never looks for it. Please make the tests cover both. - Old files are left out, and no configuration file stops the command: accepted.
Tests to add in this branch:
- One test that runs
main()with real arguments (vip bundle --output ... --force --allow-match X) through to a written zip. Every current test builds theNamespaceby hand, so a change to an option'sactionordestleaves the suite green. - The scan thresholds at their boundary: a 4-character secret is scanned, a 16-character secret gets the partial forms, and an exact 8-character part is found. Today
MIN_SECRET_LENGTH = 5andPARTIAL_MIN_SECRET_LENGTH = 24both pass the suite. - Each entry of
SECRET_ENV_VARSnext to a different inline value. RemovingVIP_PM_TOKEN,VIP_TEST_PASSWORD, orVIP_WORKBENCH_API_KEYfrom the list passes the suite. - On each refusal path, no zip and no temp file is left.
test_manifest_is_scanned_toochecks only the exception. VIP_CONFIGselects the configuration file. No test sets it.
Docs:
- The guide lists a configuration summary in the zip with no condition, but
--no-configleaves it out. The only example in the guide isvip bundle --no-config. - The guide says the command stops when there is no
vip.toml. It also stops when the deployment name, or the host or port of a product URL, differs fromresults.json. - For
--allow-match, say that a label with spaces needs quotes, that the label must be one from the error, and that the value then stays in the zip. - The comment above
SECRET_ENV_VARSinsummary.pydoes not parse as one sentence.
A damaged auth cache file or a report file that is not UTF-8 stops the bundle. The scan reads the product URL credentials in results.json, finds a URL password with a raw "#", "?", or "/", gives each URL part and each cookie domain its own label, and skips data: URIs, scripts, and styles in HTML pages. If results.json has a run time, a sidecar with none is left out. Tests cover the scan limits, each credential variable, main(), VIP_CONFIG, and cleanup after a refusal.
|
The items in the review body are fixed in a3d946f. The inline threads have their own replies. Secret field tests
Tests
Before the fix, the suite passed with Docs
|
ian-flores
left a comment
There was a problem hiding this comment.
Thanks, Tim. I checked a3d946f against my review, and everything is fixed.
- I ran my repro for each of the five cases against the real CLI. Each one now stops with a
BundleErrorand leaves no zip behind. - I also checked the smaller items: the lowercase
%2f, a deeply nestedresults.json, a non-UTF-8 env value, and sidecars with no run time. - Setting
MIN_SECRET_LENGTH = 5orPARTIAL_MIN_SECRET_LENGTH = 24now fails a test. - The docs now say what I asked for.
I am fine with vip verify still writing the URL userinfo. vip bundle refuses until the user removes it, so nothing leaks.
Approving.
|


Closes #794.
Summary
vip bundlewrites a zip of VIP report output that you can send to another person. It builds the zip from an allowlist (a fixed list of permitted files), not from a copy of the directory. Before it writes the zip, it looks in each file for each credential that it knows. If it finds one, it stops.What the zip holds
report/results.json.failures.json,junit.xml, andresults.sarif. Each one must be from the same run asresults.json.report/_output/index.html,details.html, andvip-report.pdf. Each one must be a file thatvip reportmade from thisresults.json, with no change after the render. The PDF goes in only together with both HTML pages, because the scan cannot read the text in a PDF.config-summary.json: the products, the URLs, the auth provider, and other fields with no credentials.version.txt: the output ofvip version.manifest.json: the members, the files that were left out with the reason, the warnings, and the scan counts.The zip never holds
vip.toml, the auth cache files, or.vip-install.json.The secret scan
The scan gets its values from three sources:
vip.toml, the environment variables, and the auth cache (session cookies, localStorage values, and a minted Connect API key). It looks for each value in each member, in these forms: raw, JSON-escaped, HTML-escaped, XML text, percent-encoded, and Python repr. It also looks for the first and the last 8 characters of a long value, because report text can cut a value.--allow-match LABELlets the command continue for that label. The manifest records it.The scan cannot find a credential that it does not know. Examples are a value that an extension test printed, or a value that was set only in the shell that ran
vip verify. The User Guide says this and tells the user to look at the zip before they send it.Differences from the issue
vip statusoutput.vip statussends a request with live credentials to each product, so it needs network access. It also shows the state at bundle time, not at the time of the run.results.jsonalready has the product URLs, the vip version, the platform, and the Python version. The product version inresults.jsonis the value fromvip.toml, not the version that the server reports, and it is often empty. So this branch does not deliver the "platform detailsvip statusalready collects" item of the issue. Please accept this, or ask for a follow-up issue that adds the data.vip.toml, redacted or raw.config-summary.jsonis built from a list of named fields. A new configuration field stays out of the summary until a developer adds it to the list.src/vip/bundle/summary.py, not in one constant inconfig.py.config.pydid not change. If a configuration field has no class (summary, secret, or excluded), a selftest fails. If__repr__hides a field that is not in the secret list, a second selftest fails.results.jsonin the bundle. If it does not, the command leaves it out. Afailures.json,junit.xml, orresults.sariffrom a different run is left out. The manifest gives the reason for each one.vip.toml, the scan misses each credential that only that file holds.--no-configwrites the bundle with a warning, for a run that usedvip verify --connect-urland no file. Ifvip.tomldoes not matchresults.json, the command also stops. It compares the deployment name, and the host and the port of each product URL.Changes outside the new command
vip reportwrites_output/.vip-render.json. The file holds the SHA-256 of theresults.jsonthat was rendered and the SHA-256 of each rendered file.vip reportdeletes it before each render, and then adds only the files that the render wrote.vip verifywrites thegenerated_atvalue of the run intojunit.xml(a suite property) andresults.sarif(a run property).vip bundleuses the value to pair these files withresults.json.vip reportgives a clear error for aresults.jsonthat it cannot read.BundleError, with exit code 1.What the recipient gets, and does not get
The bundle holds the login email of the test user (test output prints it), the user name in local paths, and the product host names. It has no server version for a product unless
vip.tomlsets one. Inconfig-summary.json, each*_setvalue describes the configuration file and the environment of thevip bundlecommand. Theauth_cachevalues show two facts about the run: a saved browser session, and a minted Connect API key.Interaction with #793
#793 moves the auth cache out of the invocation directory. This branch reads the cache files in the invocation directory first, and then the files at
auth_cache_path(). A trial merge is clean in both orders, and the selftests pass on the merged tree. After #793 merges, the AGENTS.md row forauth_cache_path()and the README section of that PR need a small edit.One limit comes with #793. If a run directory moves to a different machine, the cache does not move with it. The scan then knows no cache value. The scan counts in the manifest and in the output show this.
Tests
just checkandjust typecheckpass. The selftests have 2809 passed and 3 skipped.selftests/test_cli_bundle.pybuilds a bundle from a directory that holdsvip.tomlwith a value in each credential field, both auth cache files, and.vip-install.json. It makes sure that the member list is the allowlist and that no seeded value is in the bytes of the zip.results.json,failures.json,junit.xml,results.sarif, each HTML page, and the PDF. Each one stops the bundle.--interactive-auth: the zip has 8 members. An independent script found none of the 15 auth cache values in it, in nine encoded forms and in the inflated PDF streams.Known limits
junit.xmlorresults.sariffrom an older vip has nogenerated_atvalue. For those files the command uses the file time: a file that is older thanresults.jsonis left out.