Skip to content

lockfile: key params by def_path to match how they are loaded - #11097

Open
KR-Ravindra wants to merge 1 commit into
treeverse:mainfrom
KR-Ravindra:fix/lockfile-params-def-path
Open

lockfile: key params by def_path to match how they are loaded#11097
KR-Ravindra wants to merge 1 commit into
treeverse:mainfrom
KR-Ravindra:fix/lockfile-params-def-path

Conversation

@KR-Ravindra

@KR-Ravindra KR-Ravindra commented Sep 9, 2026

Copy link
Copy Markdown

Problem

When a params file is written in dvc.yaml with a ./ prefix (e.g. params: [./my_params.yaml]), dvc status reports the whole file as new forever, and dvc repro re-runs the stage on every invocation, even though nothing changed:

$ dvc status
test:
	changed deps:
		new:                my_params.yaml

Reproduced on current main (3.67.2.dev) with the script from the issue.

Root cause

dvc.lock entries for deps and outs are keyed verbatim by def_path (dvc/stage/serialize.py:159, _dumpd), but the params section is keyed through Output.dumpd() (dvc/stage/serialize.py:104-105, _serialize_params_values), which rewrites in-repo paths as relpath(fs_path, stage.wdir) (dvc/output.py:843-846). The lock therefore ends up with params: {my_params.yaml: ...} while the ParamsDependency keeps def_path == "./my_params.yaml".

On load, StageLoader.fill_from_lock looks every item up by def_path (dvc/stage/loader.py:63-65): get_in(lock_data, ["params", "./my_params.yaml"]) returns None, fill_values is a no-op, hash_info.value stays None, and workspace_status reports the whole file as new.

Fix

Key the lockfile params section by param_dep.def_path, the same way deps/outs are already keyed in the same lockfile, so the serializer and the loader agree. One line in _serialize_params_values; the now-unused module constant PARAM_PATH is removed.

Lock files previously written with the normalized key are rewritten on the next dvc repro/dvc commit (which affected repos already trigger on every run today); after that the stage is reported clean.

How tested

Added three tests that fail on main and pass with the fix:

  • tests/unit/stage/test_serialize_pipeline_lock.py::test_lock_params_keeps_def_path
  • tests/unit/stage/test_loader_pipeline_file.py::test_load_stage_with_params_def_path_roundtrip (serialize to lock, load back, values filled)
  • tests/func/test_status.py::test_params_file_with_dot_slash_path (stage add + reproduce, then status == {})

Before:

FAILED tests/unit/stage/test_serialize_pipeline_lock.py::test_lock_params_keeps_def_path
E     {'params': OrderedDict({'my_params.yaml': OrderedDict({'foo': 'foo'})})} != {'params': {'./my_params.yaml': OrderedDict({'foo': 'foo'})}}
FAILED tests/unit/stage/test_loader_pipeline_file.py::test_load_stage_with_params_def_path_roundtrip
E       name: None != 'params'
E       value: None != {'foo': 1}
FAILED tests/func/test_status.py::test_params_file_with_dot_slash_path
E   AssertionError: assert {'my_params.yaml': {'foo': 1}} == {'./my_params...': {'foo': 1}}
3 failed

After:

3 passed in 0.36s

pytest tests/unit/stage tests/unit/dependency tests/func/test_status.py tests/func/params tests/func/test_lockfile.py tests/func/test_stage_load.py tests/func/test_run.py tests/func/test_stage.py passes. ruff check, ruff format --check and mypy are clean on the changed files.

End-to-end with the script from the issue: dvc.lock now contains params: {./my_params.yaml: {foo: 1}}, dvc status prints "Data and pipelines are up to date.", a second dvc repro skips the stage, and editing foo is still reported as modified.

Links


  • I have followed the Contributing to DVC checklist.

  • Documentation: no changes to the docs are needed for this fix.

This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.

The params section of dvc.lock was keyed through Output.dumpd(), which
rewrites in-repo paths relative to the stage wdir, while deps and outs
in the same lockfile are keyed by def_path verbatim and StageLoader
looks everything up by def_path. A params file written as
`./my_params.yaml` in dvc.yaml was therefore stored as
`my_params.yaml`, never found on load, and reported as `new` by
`dvc status` (and re-run by `dvc repro`) forever.

Use def_path for the params key too, so the serializer and the loader
agree.

Closes treeverse#9518
@github-project-automation github-project-automation Bot moved this to Backlog in DVC Sep 9, 2026
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@KR-Ravindra

Copy link
Copy Markdown
Author

Closing: opened against the wrong repository; this change is for iterative/dvc.

@KR-Ravindra KR-Ravindra closed this Sep 9, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to Done in DVC Sep 9, 2026
@KR-Ravindra KR-Ravindra reopened this Sep 9, 2026
@KR-Ravindra

Copy link
Copy Markdown
Author

Reopening: iterative/dvc now redirects to this repository, so this is the correct target. Issue #9518 lives here as well.

@github-project-automation github-project-automation Bot moved this from Done to Backlog in DVC Sep 9, 2026
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.98%. Comparing base (2431ec6) to head (5f73986).
⚠️ Report is 213 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #11097      +/-   ##
==========================================
+ Coverage   90.68%   90.98%   +0.30%     
==========================================
  Files         504      505       +1     
  Lines       39795    41158    +1363     
  Branches     3141     3263     +122     
==========================================
+ Hits        36087    37448    +1361     
- Misses       3042     3071      +29     
+ Partials      666      639      -27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@KR-Ravindra

Copy link
Copy Markdown
Author

Round 1 self-review.

Checked against main (56e5982): _dumpd keys deps/outs by def_path (dvc/stage/serialize.py:159), while _serialize_params_values keyed params through Output.dumpd(), which rewrites in-repo paths as relpath(fs_path, wdir) (dvc/output.py:843-846); fill_from_lock then looks params up by def_path (dvc/stage/loader.py:63-65), so ./my_params.yaml never matched its lock entry. Keying by def_path makes the writer agree with the reader and with how the rest of the lockfile is already keyed. The three new tests fail on main and pass here.

CI: the red jobs are the same ones failing on main's latest scheduled run (34299452210): mypy in the lint job and tests/func/test_repo_index.py::test_ignored_dir_unignored_pattern on Windows. Nothing in this diff touches either.

Marking ready.

@KR-Ravindra
KR-Ravindra marked this pull request as ready for review September 9, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dvc status: shows that params have changed when they haven't

2 participants