Skip to content

[feat] Add phase-aware modular training performance metrics - #1826

Open
Jack47 wants to merge 1 commit into
hao-ai-lab:mainfrom
Jack47:codex/training-performance-metrics
Open

[feat] Add phase-aware modular training performance metrics#1826
Jack47 wants to merge 1 commit into
hao-ai-lab:mainfrom
Jack47:codex/training-performance-metrics

Conversation

@Jack47

@Jack47 Jack47 commented Sep 7, 2026

Copy link
Copy Markdown

Summary

  • add per-step optimizer/sample throughput and per-role Transformer call accounting to the modular trainer
  • estimate Wan Transformer-core TFLOP/s and MFU using actual BDir/VSA, block-causal AR, streaming KV-cache, and DMD multi-role/multi-rollout geometry
  • expose an optional peak BF16 override, document metric semantics and limitations, and keep all performance hooks configurable

Why

The existing standalone MFU helper assumes one dense Wan forward. That does not represent causal chunking, sparse attention, DMD teacher/critic calls, or repeated student rollouts. Counting each role at the Transformer boundary makes the metric follow the algorithm actually executed.

Validation

  • 40 targeted CPU tests passed, including existing Trainer managed-optimization coverage
  • pre-commit passed: YAPF, Ruff, codespell, PyMarkdown, mypy, filename and suggestion checks

GPU validation was not run locally; FLOP geometry is covered with CPU unit tests.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome to FastVideo! Thanks for your first pull request.

How our CI works:

PRs run a three-tier CI system:

  1. Pre-commit — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
  2. Fastcheck — six core GPU lanes run automatically via Buildkite (~10-15 min).
  3. Merge gate — a reviewer adds ready; changed paths select only the relevant integration, training, golden, or SSIM coverage.

Before your PR is reviewed:

  • pre-commit run --all-files passes locally
  • You've added or updated tests for your changes
  • The PR description explains what and why

If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and merge-gate results appear in the Checks section below.

Useful links:

@mergify mergify Bot added type: feat New feature or capability scope: training Training pipeline, methods, configs scope: infra CI, tests, Docker, build scope: docs Documentation labels Sep 7, 2026
@mergify

mergify Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

Protection Waiting on
🔴 PR merge requirements 👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • check-success=full-suite-passed
This rule is failing.
  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • check-success=full-suite-passed
  • check-success~=pre-commit
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope: docs Documentation scope: infra CI, tests, Docker, build scope: training Training pipeline, methods, configs type: feat New feature or capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant