Add datacontract test --dry-run

[OGsiji]

Closes #1510.

datacontract test datacontract.yaml --dry-run

⚪ no checks were executed. Planned 33 checks. Took 0.16 seconds.

Every check is already registered as a stub before execution begins, so the plan is that list with a result of skipped, carrying the assertion each check would have made. The existing output formats render it, so the machine-readable view comes for free:

{ "name": "Check that field order_total has no missing values",
  "implementation": "missing_count(order_total) = 0",
  "result": "skipped" }

Since nothing connects, a plan works where a run cannot. On a contract with a Postgres server and no credentials set:

$ datacontract test contract.yaml
missing_env_DATACONTRACT_POSTGRES_USERNAME: Required configuration ... is not set.   (exit 1)

$ datacontract test contract.yaml --dry-run
⚪ no checks were executed. Planned 10 checks.                                        (exit 0)

On the two questions from the issue

Azure blob. Excluded, as you suggested. Those checks read the file listing to decide which checks exist at all, so there is nothing to plan without reading. A dry run appends a warning check saying the plan is incomplete rather than quietly omitting them.

JSON Schema. Included, since it looked like the case where an empty plan would be most misleading. The schema is still built and compiled and only the file read is skipped, so a contract that could never validate still fails in a dry run — which seemed closer to the point of the feature than skipping it wholesale.

skipped for the checks works as you said. The run-level result needed a little more care, though, and it is the one thing I would like a second opinion on.

The one judgement call

An all-skipped run resolves to unknown, which the writer treats as invalid and exits 1. So a plan initially failed the build, which defeats the CI use case.

My first attempt was to make calculate_result report skipped when every check is skipped. That broke test_metadata_only_all_skipped_remains_unknown, which says plainly that you decided otherwise for --metadata-only, so I backed it out.

Instead the run records dryRun: true and a dry run resolves to skipped on that basis — a plan asserts nothing, so it can neither pass nor fail. --metadata-only still resolves to unknown exactly as before, and its test is untouched.

The field is also what lets anything reading the JSON tell a plan from a run that happened to skip everything, which matters for the case in the issue: without it, "all checks skipped" is ambiguous.

Happy to change how that is modelled if you would rather it worked differently — it is additive to the Run schema, which I did not want to do silently.

Notes

datacontract ci does not get the flag; it seemed odd for a command whose job is to run. Easy to add if you want it.

Full suite green (2103 passed). Nine tests added, the main one asserting the plan lists exactly the checks a real run against the same contract executes, so the plan cannot drift from reality without failing.

[OGsiji]

Pushed two fixes after testing the paths I had not exercised.

An incomplete plan was invisible. The blob-schema warning was being added correctly, but a dry run resolved to skipped unconditionally, so the summary read ⚪ no checks were executed while the plan was quietly missing checks — the opposite of the requirement to make incompleteness obvious. A dry run now reports the most severe result its own checks carry:

🟠 data contract has warnings. Found the following warnings:
1) Check that blob files match the contract: Dry run is incomplete:
   file-metadata checks for blob schemas are not planned.

Still exits 0. Two tests cover it.

The JSON Schema condition was duplicated. Deciding whether those checks apply was written out twice, once in the execution path and once in the dry run. That is the one place a plan could silently drift from what actually runs, which would defeat the point of the feature. Both now call the same predicate, so the execution path lost a copy rather than gaining one.

Also verified along the way, none of which needed changes: the plan matches a real run under --checks and --dimension filters (now asserted), Kafka and no-server contracts behave as they do without the flag, and --output-format junit reports the planned checks as skipped. The credentials test now clears the environment variables it assumes are absent, so it cannot block on a connection where a developer happens to have them set.

On the 3.13 job: it was cancelled at the 20 minute limit rather than failing an assertion, with orphan pytest/uv processes. I ran the whole suite on 3.13 locally — 2104 passed, and the only two failures were in test_download_datacontract_file.py, which pass on their own. Wall clock was 58 minutes at 5% CPU, so the time went on network stalls in those download tests, not on work. Nothing in this change makes a network call; it removes one, since a dry run no longer fetches the API response. Happy to be told otherwise if you have seen that job wedge differently.

[OGsiji]

@jochenchrist

[jschoedl]

Thank you for the PR!

[jschoedl]

That's not entirely true: A contract can reference semantics from an external source, in which case those are fetched and need to be accessible. (Docs: https://docs.datacontract.com/semantics)

[OGsiji]

Thanks — all four applied.

The filter ordering was the real one. You were right that it has to come after --metadata-only, and the effect was bigger than I expected: on the local-json fixture the plan listed 40 checks where a real --metadata-only run executes 19. A plan narrowed by only some of the filters is exactly the failure this feature is supposed to prevent, and I had built the guard for it on the JSON Schema side and then walked past it here.

Two tests now hold the plan and the run to the same filters, and the plan keeps each reason distinct, so --metadata-only skips read as their own thing rather than as dry-run skips.

The help text contradicted the warning I had added myself — fixed to mention warning.

The semantics point I had not considered at all. The docs now say a dry run is offline for the server rather than offline in general, and link to the semantics page, since the checks cannot be worked out until those are resolved.

Wording suggestion taken as written.

Also rebased onto main; the CHANGELOG entry moved to Unreleased now that 1.1.2 has shipped. Full suite green at 2198.

On #1531 — thanks for adding the dimensions, that was a real miss. I checked the new checks worked but not that they survived the filters that already existed, which is the same mistake as the ordering here. Noted.