Skip to content

✨ Add intermediate rewards with paper semantics - #792

Closed
flowerthrower wants to merge 12 commits into
v3/669-gnn-rlfrom
v3/paper-reward-parity
Closed

flowerthrower wants to merge 12 commits into
v3/669-gnn-rlfrom
v3/paper-reward-parity

Conversation

@flowerthrower

@flowerthrower flowerthrower commented Aug 27, 2026

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Description

Combines the intermediate-reward feature and the paper reward semantics in one top-of-stack PR while retaining the existing PredictorEnv lifecycle.

The public reward controls are intermediate_reward=True, reward_scale=1.0, and no_effect_penalty=-0.001. Setting intermediate_reward=False retains terminal-only rewards.

  • comparable non-terminal expected-fidelity and ESP steps use exact or approximate figure-of-merit deltas;
  • approximate expected fidelity uses translated target-basis gate counts and averaged calibration errors, including measurement;
  • approximate ESP additionally applies the prototype's duration, parallelism, liveness, and coherence idle penalty;
  • terminal ESP uses ASAP scheduling and stops charging a qubit after its final measurement or reset;
  • critical-depth actions use exact non-terminal deltas, while estimated Hellinger distance remains neutral until termination;
  • structural compilation progress and structural no-effect steps remain neutral;
  • optimization no-effect and failed actions receive no_effect_penalty, with failures truncating the episode.

This is position 11 and the top of the stack. It depends directly on the GNN PR #788 and absorbs the former standalone intermediate-reward PR #760. No action, GNN, dependency, or experiment changes are included.

Validation

  • repository lint;
  • direct bit-exact comparisons with add-individual-bqskit-passes for approximate expected fidelity, approximate ESP, calibration maps, and terminal ESP;
  • focused reward and timeout checks (21 passed, 25 deselected).

Fixes #670

Part of #664

Checklist

  • The pull request only contains focused changes required for intermediate rewards and paper reward parity.
  • Existing focused tests and release metadata cover the changed public behavior.
  • I reviewed the complete base-to-head diff.

If PR contains AI-assisted content:

  • Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope.
  • Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖.
  • I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

flowerthrower added a commit that referenced this pull request Aug 27, 2026
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/paper-reward-parity branch from c785d7a to c25766d Compare August 27, 2026 13:26
flowerthrower and others added 12 commits August 27, 2026 15:28
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Assisted-by: GPT-5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Signed-off-by: flowerthrower <flowerthrower@users.noreply.github.com>
Assisted-by: GPT 5.6 via Codex
@flowerthrower
flowerthrower force-pushed the v3/paper-reward-parity branch from c25766d to 4721ec8 Compare August 27, 2026 13:28
@flowerthrower flowerthrower added enhancement New feature or request major Part of a major release labels Aug 27, 2026
@flowerthrower flowerthrower self-assigned this Aug 27, 2026
@flowerthrower

Copy link
Copy Markdown
Member Author

🤖 AI text below 🤖\n\nSuperseded by the rebuilt combined-reward layer at position 8.

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 38.37838% with 114 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/mqt/predictor/rl/approx_reward.py 13.8% 81 Missing ⚠️
src/mqt/predictor/rl/predictorenv.py 28.2% 33 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request major Part of a major release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant