feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline by vividf · Pull Request #115 · tier4/AWML

vividf · 2025-10-08T15:13:51Z

AWML Unified Deployment Framework

Introduced a unified deployment and evaluation framework for AWML, significantly improving consistency, scalability, and maintainability across projects

PR feat(deployment):prototype deployment framework #137: Core deployment framework (ONNX Runtime + TensorRT support)
PR feat(deployment): centerpoint deployment integration #181: CenterPoint integration into the unified pipeline

Key Contributions

1. Unified Deployment Pipeline

Standardized model deployment flow: PyTorch → ONNX → TensorRT
Eliminated duplicated and project-specific deployment scripts
Enabled config-driven, reusable pipelines across multiple models
Supported seamless backend switching (PyTorch / ONNX / TensorRT)

2. Cross-Backend Verification

Implemented automatic output comparison across backends
Ensured numerical consistency between PyTorch, ONNX Runtime, and TensorRT
Prevented silent accuracy regressions during deployment
Improved reliability for production-level inference

3. Integrated Evaluation Framework

Built unified evaluation pipeline based on T4MetricV2
Supports:
- mAP / mAPH
- Per-class metrics
- Threshold-based evaluation
- Enables consistent benchmarking across different backends and models

How to run

python -m deployment.cli.main centerpoint deployment/projects/centerpoint/config/deploy_config.py   projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_8xb16_121m_j6gen2_base_amp_t4metric_v2.py   --rot-y-axis-reference

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

KSeangTan

Only review the first part, please address the comments accordingly first.

KSeangTan · 2026-04-21T06:35:51Z

+    Args:
+        log_file_path: Absolute or resolved path to the log file.
+    """
+    path = os.path.abspath(os.path.expanduser(log_file_path))


Use pathlib instead

Thanks, addressed in cf9157a

KSeangTan · 2026-04-21T06:41:34Z

+        """Require a non-empty checkpoint path that exists as a regular file."""
+        if not isinstance(checkpoint_path, str):
+            raise TypeError(f"checkpoint_path must be a string, got {type(checkpoint_path).__name__}.")
+        path = os.path.expanduser(checkpoint_path.strip())


Use pathlib instead

KSeangTan · 2026-04-21T06:44:05Z

+
+    def _needs_cuda_device(self) -> bool:
+        """Determine if current deployment config requires a CUDA device."""
+        if self.export_config.should_export_tensorrt:


I believe this function is unnecessary complicated, for example, we only need to check if tensorrt is enabled since we will use only tensorrt with cuda right?

Thanks, renamed _needs_cuda_device → _uses_tensorrt to make the intent clear. Kept the three branches: TensorRT (hence CUDA) can be pulled in without an export step — evaluation can run a pre-built .engine, and a verification scenario can set ref/test_backend = tensorrt — so should_export_tensorrt alone would miss those and fail mid-run instead of at config validation.

KSeangTan · 2026-04-21T06:45:41Z

+        if not output_names:
+            output_names = ("output",)
+        settings_dict = {
+            "opset_version": onnx_config.opset_version,


self._onnx_export_config.opset_config

self._onnx_config is the shared OnnxConfig section (global ONNX knobs), not the per-component ONNXExportConfig — the latter is assembled per component in get_onnx_settings by combining OnnxConfig with that component's I/O. The field is opset_version, so I kept these names as-is for that distinction

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

…se.py Signed-off-by: vividf <yihsiang.fang@tier4.jp>

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf changed the title ~~feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline~~ WIP: feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline Nov 14, 2025

vividf force-pushed the feat/new_deployment_and_evaluation_pipeline branch from 68d1700 to 7ac5b8b Compare January 29, 2026 08:08

vividf and others added 10 commits March 11, 2026 13:24

chore: init

c77cf92

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add deployment framework intro

117dead

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

feat(deployment):prototype deployment framework (#137)

41a3e9a

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

feat(deployment): restructure deployment framework (#160)

d80db0a

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add missing pipeline directory and docstring

a1edc32

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

feat(deployment): refactor config and clean code (#180)

a362b27

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: refactor base config

d99296c

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up code: device spec, remove unused fucntion .etc

e23c210

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: refactor export compenent

dd46052

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix more Device spec

2b28f60

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf force-pushed the feat/new_deployment_and_evaluation_pipeline branch from 5256306 to 2b28f60 Compare March 11, 2026 04:27

vividf and others added 14 commits March 25, 2026 20:49

chore: enable only trt evaluation

85f0d64

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix verification

e7439a6

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix main

ca6f730

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix base evaluator

d150f5e

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up base

23df89b

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add base logger

36d7568

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: update docs

5bd5b76

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add more readme

0eed1f1

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: base add -> None for __init__

d706ff7

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: remove task profile

9dfb6a1

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix matrix report base

a75d739

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean metrics

efa71d1

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: update project registry base

0c7c57c

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

feat(deployment): centerpoint deployment integration (#181)

ddc5106

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

vividf self-assigned this Apr 14, 2026

vividf added 2 commits April 16, 2026 16:16

chore: improve docs

d0b491d

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up docs

5080b40

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf added 4 commits April 16, 2026 20:56

chore: clean up code

a46f933

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix verification

b7b6954

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix verification 2

f873e85

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up __init__.py

9329b11

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf marked this pull request as ready for review April 20, 2026 12:41

vividf requested a review from KSeangTan as a code owner April 20, 2026 12:41

vividf changed the title ~~WIP: feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline~~ feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline Apr 20, 2026

chor: clean code

1d4c27f

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf force-pushed the feat/new_deployment_and_evaluation_pipeline branch from 8756669 to 1d4c27f Compare April 20, 2026 13:05

KSeangTan requested changes Apr 21, 2026

View reviewed changes

vividf added 12 commits May 19, 2026 20:35

chore:pathlib

cf9157a

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix sequence

7256b29

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix message to dict-like mapping

e242301

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix normalize, validate cuda device, parse deploy log path, ba…

f2e5a94

…se.py Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean code

b70171e

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up private member in configs base

736a31b

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up more code

159eab3

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix record time, clean funcs

e85d3cf

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: update readme and clean more code

ccb1482

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean more code

63e4e0e

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean more code

3cf2438

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: seperate responsiblility of verifier and evaluator

2f38e92

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline#115

feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline#115
vividf wants to merge 43 commits into
mainfrom
feat/new_deployment_and_evaluation_pipeline

vividf commented Oct 8, 2025 •

edited

Loading

Uh oh!

KSeangTan left a comment

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

vividf Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

vividf Jun 26, 2026

Uh oh!

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

vividf Jun 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

vividf commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AWML Unified Deployment Framework

Key Contributions

1. Unified Deployment Pipeline

2. Cross-Backend Verification

3. Integrated Evaluation Framework

How to run

Uh oh!

KSeangTan left a comment

Choose a reason for hiding this comment

Uh oh!

KSeangTan Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

vividf Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KSeangTan Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

KSeangTan Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

vividf Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KSeangTan Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

vividf Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vividf commented Oct 8, 2025 •

edited

Loading