lockfile: key params by def_path to match how they are loaded - #11097
lockfile: key params by def_path to match how they are loaded#11097KR-Ravindra wants to merge 1 commit into
Conversation
The params section of dvc.lock was keyed through Output.dumpd(), which rewrites in-repo paths relative to the stage wdir, while deps and outs in the same lockfile are keyed by def_path verbatim and StageLoader looks everything up by def_path. A params file written as `./my_params.yaml` in dvc.yaml was therefore stored as `my_params.yaml`, never found on load, and reported as `new` by `dvc status` (and re-run by `dvc repro`) forever. Use def_path for the params key too, so the serializer and the loader agree. Closes treeverse#9518
|
|
|
Closing: opened against the wrong repository; this change is for iterative/dvc. |
|
Reopening: iterative/dvc now redirects to this repository, so this is the correct target. Issue #9518 lives here as well. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11097 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.30%
==========================================
Files 504 505 +1
Lines 39795 41158 +1363
Branches 3141 3263 +122
==========================================
+ Hits 36087 37448 +1361
- Misses 3042 3071 +29
+ Partials 666 639 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Round 1 self-review. Checked against CI: the red jobs are the same ones failing on Marking ready. |
Problem
When a params file is written in
dvc.yamlwith a./prefix (e.g.params: [./my_params.yaml]),dvc statusreports the whole file asnewforever, anddvc reprore-runs the stage on every invocation, even though nothing changed:Reproduced on current
main(3.67.2.dev) with the script from the issue.Root cause
dvc.lockentries fordepsandoutsare keyed verbatim bydef_path(dvc/stage/serialize.py:159,_dumpd), but theparamssection is keyed throughOutput.dumpd()(dvc/stage/serialize.py:104-105,_serialize_params_values), which rewrites in-repo paths asrelpath(fs_path, stage.wdir)(dvc/output.py:843-846). The lock therefore ends up withparams: {my_params.yaml: ...}while theParamsDependencykeepsdef_path == "./my_params.yaml".On load,
StageLoader.fill_from_locklooks every item up bydef_path(dvc/stage/loader.py:63-65):get_in(lock_data, ["params", "./my_params.yaml"])returnsNone,fill_valuesis a no-op,hash_info.valuestaysNone, andworkspace_statusreports the whole file asnew.Fix
Key the lockfile
paramssection byparam_dep.def_path, the same waydeps/outsare already keyed in the same lockfile, so the serializer and the loader agree. One line in_serialize_params_values; the now-unused module constantPARAM_PATHis removed.Lock files previously written with the normalized key are rewritten on the next
dvc repro/dvc commit(which affected repos already trigger on every run today); after that the stage is reported clean.How tested
Added three tests that fail on
mainand pass with the fix:tests/unit/stage/test_serialize_pipeline_lock.py::test_lock_params_keeps_def_pathtests/unit/stage/test_loader_pipeline_file.py::test_load_stage_with_params_def_path_roundtrip(serialize to lock, load back, values filled)tests/func/test_status.py::test_params_file_with_dot_slash_path(stage add+reproduce, thenstatus == {})Before:
After:
pytest tests/unit/stage tests/unit/dependency tests/func/test_status.py tests/func/params tests/func/test_lockfile.py tests/func/test_stage_load.py tests/func/test_run.py tests/func/test_stage.pypasses.ruff check,ruff format --checkandmypyare clean on the changed files.End-to-end with the script from the issue:
dvc.locknow containsparams: {./my_params.yaml: {foo: 1}},dvc statusprints "Data and pipelines are up to date.", a seconddvc reproskips the stage, and editingfoois still reported asmodified.Links
I have followed the Contributing to DVC checklist.
Documentation: no changes to the docs are needed for this fix.
This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.