Skip to content

Normalize raw session metadata before ingest in the nightly cron - #354

Merged
lecriste merged 1 commit into
mainfrom
leo_add-sync-days-to-cron
Aug 14, 2026
Merged

lecriste merged 1 commit into
mainfrom
leo_add-sync-days-to-cron

Conversation

@lecriste

Copy link
Copy Markdown
Collaborator

What

Add a step to the nightly local cron that reconciles the per-session index metadata stored in the raw acquisition files (.npy) with the database before
populate_rig ingests them.

  • docker/cron_common.sh: new cron helper that runs the repo's existing raw-metadata resync entrypoint in the client container, using the same setup as the scenario runner (vr4mice_check_env_file + BASE_INSTALL + source .env).
  • cron_script.sh: invoke it after compose up and before vr4mice_run_cron_scenario local.

The resync action itself already exists in the repo; this PR only invokes it from the cron.

Why

Some session tables (exp.Session / SessionScoreSheet) use a per-session index that is baked into the raw files as part of their primary key. A duplicated value colliding with an earlier session makes the new session silently skipped on ingest — it never lands in the database, with no error. The resync recomputes the
correct values from the database, removing the collision at the source before ingest.

Behavior / notes

  • Runs in the client container against /data/data, after compose up (containers up) and before populate_rig moves files to processed/.
  • Fatal on failure: cron_script.sh uses set -euo pipefail, so if the resync (or its env check) fails, the run aborts and the raw files are left untouched in data/data for the next run — rather than ingesting inconsistent metadata.
  • Idempotent: the resync rewrites a file only when the stored value differs from the computed one, so already-correct files are a no-op.
  • Local job only: added to cron_script.sh and not cron_script_aws.sh.
  • No crontab change — the step lives inside the script.

Testing

Ran the resync entrypoint on the production server: it corrected freshly-acquired files whose stored index was inconsistent, and was a verified no-op on already-correct files.
The new cron helper is a thin wrapper that runs exactly that entrypoint with the standard cron environment.

@lecriste lecriste self-assigned this Aug 14, 2026
@lecriste
lecriste merged commit e0eb2a6 into main Aug 14, 2026
6 checks passed
@lecriste
lecriste deleted the leo_add-sync-days-to-cron branch August 14, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant