Skip to content

✨ Add intermediate rewards with paper semantics - #782

Closed
flowerthrower wants to merge 41 commits into
v3/670-intermediate-rewardsfrom
v3/paper-reward-parity
Closed

flowerthrower wants to merge 41 commits into
v3/670-intermediate-rewardsfrom
v3/paper-reward-parity

Conversation

@flowerthrower

@flowerthrower flowerthrower commented Aug 27, 2026

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Description

Combines the intermediate-reward feature and the paper reward semantics in one top-of-stack PR while retaining the existing PredictorEnv lifecycle.

The public reward controls are intermediate_reward=True, reward_scale=1.0, and no_effect_penalty=-0.001. Setting intermediate_reward=False retains terminal-only rewards.

  • comparable non-terminal expected-fidelity and ESP steps use exact or approximate figure-of-merit deltas;
  • approximate expected fidelity uses translated target-basis gate counts and averaged calibration errors, including measurement;
  • approximate ESP additionally applies the prototype's duration, parallelism, liveness, and coherence idle penalty;
  • terminal ESP uses ASAP scheduling and stops charging a qubit after its final measurement or reset;
  • critical-depth actions use exact non-terminal deltas, while estimated Hellinger distance remains neutral until termination;
  • structural compilation progress and structural no-effect steps remain neutral;
  • optimization no-effect and failed actions receive no_effect_penalty, with failures truncating the episode.

This is position 11 and the top of the stack. It depends directly on the GNN PR #788 and absorbs the former standalone intermediate-reward PR #760. No action, GNN, dependency, or experiment changes are included.

Validation

  • repository lint;
  • direct bit-exact comparisons with add-individual-bqskit-passes for approximate expected fidelity, approximate ESP, calibration maps, and terminal ESP;
  • focused reward and timeout checks (21 passed, 25 deselected).

Fixes #670

Checklist

  • The pull request only contains focused changes required for intermediate rewards and paper reward parity.
  • Existing focused tests and release metadata cover the changed public behavior.
  • I reviewed the complete base-to-head diff.

If PR contains AI-assisted content:

  • Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope.
  • Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖.
  • I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 38.88889% with 99 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/mqt/predictor/rl/approx_reward.py 13.8% 81 Missing ⚠️
src/mqt/predictor/rl/predictorenv.py 21.7% 18 Missing ⚠️

📢 Thoughts on this report? Let us know!

@flowerthrower
flowerthrower force-pushed the v3/670-intermediate-rewards branch from 245f8c3 to 28fe747 Compare August 27, 2026 11:08
@flowerthrower
flowerthrower force-pushed the v3/paper-reward-parity branch from 0ef3cca to 7c65897 Compare August 27, 2026 11:08
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
flowerthrower added a commit that referenced this pull request Aug 27, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
@flowerthrower
flowerthrower force-pushed the v3/paper-reward-parity branch from 7c65897 to 5d711c5 Compare August 27, 2026 13:09
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/paper-reward-parity branch from 5d711c5 to 6dd26c4 Compare August 27, 2026 13:18
flowerthrower added a commit that referenced this pull request Aug 27, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
flowerthrower and others added 13 commits August 27, 2026 15:19
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Co-authored-by: Antonio Tudisco <anto.tu98@hotmail.it>

Assisted-by: GPT-5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5 via Codex

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5 via Codex

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
flowerthrower and others added 19 commits August 27, 2026 15:20
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Co-authored-by: Antonio Tudisco <anto.tu98@hotmail.it>

Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
@flowerthrower
flowerthrower force-pushed the v3/paper-reward-parity branch from 6dd26c4 to c785d7a Compare August 27, 2026 13:21
@flowerthrower flowerthrower changed the title 🐛 Match the paper reward semantics ✨ Add intermediate rewards with paper semantics Aug 27, 2026
@flowerthrower

Copy link
Copy Markdown
Member Author

🤖 AI text below 🤖

Superseded by #792 after combining intermediate rewards and paper reward semantics at the top of the reordered stack.

flowerthrower added a commit that referenced this pull request Aug 27, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
flowerthrower added a commit that referenced this pull request Aug 27, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ add step-wise intermediate rewards

1 participant