Skip to content

✨ Add intermediate rewards with paper semantics - #799

Draft
flowerthrower wants to merge 10 commits into
v3/configurable-pass-timeoutsfrom
v3/670-intermediate-rewards
Draft

flowerthrower wants to merge 10 commits into
v3/configurable-pass-timeoutsfrom
v3/670-intermediate-rewards

Conversation

@flowerthrower

@flowerthrower flowerthrower commented Aug 27, 2026

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Description

Adds configurable intermediate rewards while preserving the existing PredictorEnv lifecycle and matching the paper prototype's reward behavior.

The public controls are intermediate_reward=True, reward_scale=1.0, and no_effect_penalty=-0.001. Setting intermediate_reward=False retains terminal-only rewards.

Comparable expected-fidelity and estimated-success-probability steps use exact or approximate figure-of-merit deltas. Terminal ESP uses ASAP scheduling and stops charging a qubit after its final measurement or reset. Critical-depth actions use exact non-terminal deltas, while estimated Hellinger distance remains neutral until termination. Structural progress and structural no-effect steps remain neutral; optimization no-effect and failed actions receive no_effect_penalty, and failures truncate the episode.

This is position 8 of the stack. It depends on #798 and is followed by #800. No new package dependencies are introduced.

Fixes #670

Checklist

  • The pull request only contains commits that are focused and relevant to this change.
  • I have added appropriate tests that cover the new/changed functionality.
  • I have updated the documentation to reflect these changes.
  • I have added entries to the changelog for any noteworthy additions, changes, fixes, or removals.
  • I have added migration instructions to the upgrade guide (if needed).
  • The changes follow the project's style guidelines and introduce no new warnings.
  • The changes are fully tested and pass the CI checks.
  • I have reviewed my own code changes.

If PR contains AI-assisted content:

  • Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope, as required by our AI Usage Guidelines.
  • Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖 (titles are exempt).
  • I have disclosed AI assistance in the PR description.
  • I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

@flowerthrower flowerthrower added enhancement New feature or request major Part of a major release labels Aug 27, 2026
@flowerthrower flowerthrower self-assigned this Aug 27, 2026
@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch from 9446b53 to f8f444f Compare August 27, 2026 14:46
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch 2 times, most recently from a25d10f to 21e573a Compare August 27, 2026 15:02
@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch 2 times, most recently from 0302862 to d3609ee Compare August 27, 2026 15:07
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch from 21e573a to 1fbe20f Compare August 27, 2026 15:07
@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 38.37838% with 114 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/mqt/predictor/rl/approx_reward.py 13.8% 81 Missing ⚠️
src/mqt/predictor/rl/predictorenv.py 28.2% 33 Missing ⚠️

📢 Thoughts on this report? Let us know!

@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch from d3609ee to db2cee4 Compare August 28, 2026 07:33
flowerthrower added a commit that referenced this pull request Aug 28, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch from 1fbe20f to 3905e40 Compare August 28, 2026 07:33
@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch from db2cee4 to 7606d81 Compare August 28, 2026 07:52
flowerthrower added a commit that referenced this pull request Aug 28, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch 2 times, most recently from 462ea5c to 894d842 Compare August 28, 2026 08:13
@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch from 5e9d1d7 to 837159b Compare August 28, 2026 08:30
flowerthrower added a commit that referenced this pull request Aug 28, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch from 894d842 to 04224ef Compare August 28, 2026 08:30
@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch from 837159b to dd6fd15 Compare August 28, 2026 09:57
flowerthrower added a commit that referenced this pull request Aug 28, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch 3 times, most recently from 27372b1 to 8c2a30f Compare August 28, 2026 10:18
@flowerthrower
flowerthrower force-pushed the v3/configurable-pass-timeouts branch from e802a3f to 3943b1b Compare August 28, 2026 10:18
flowerthrower and others added 2 commits August 28, 2026 12:29
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
flowerthrower and others added 8 commits August 28, 2026 12:29
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch from 8c2a30f to 623b4ab Compare August 28, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request major Part of a major release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant