fix(training): exclude unavailable metrics from checkpoint ranking

.github/workflows/test-checkpoint-selection.yml

@@ -0,0 +1,65 @@
+name: Validate checkpoint selection
+
+on:
+  pull_request:
+    paths:
+      - "funasr/train_utils/**"
+      - "tests/test_checkpoint_metric_*.py"
+      - "tests/test_trainer_ds_unvalidated_ckpt.py"
+      - "tests/test_train_ds_distributed_config.py"
+      - "tests/test_sensevoice_continual_finetuning_docs.py"
+      - "examples/industrial_data_pretraining/sense_voice/**"
+      - ".github/workflows/test-checkpoint-selection.yml"
+  push:
+    branches: [main]
+    paths:
+      - "funasr/train_utils/**"
+      - "tests/test_checkpoint_metric_*.py"
+      - "tests/test_trainer_ds_unvalidated_ckpt.py"
+      - "tests/test_train_ds_distributed_config.py"
+      - "tests/test_sensevoice_continual_finetuning_docs.py"
+      - "examples/industrial_data_pretraining/sense_voice/**"
+      - ".github/workflows/test-checkpoint-selection.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  checkpoint-selection:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    env:
+      OMP_NUM_THREADS: "1"
+      MKL_NUM_THREADS: "1"
+      PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
+      TORCH_FORCE_WEIGHTS_ONLY_LOAD: "1"
+      HF_HUB_OFFLINE: "1"
+      GLOO_SOCKET_IFNAME: lo
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install CPU test dependencies
+        run: |
+          python -m pip install torch==2.10.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cpu
+          python -m pip install "numpy<2" kaldiio librosa rapidfuzz six pytest omegaconf hydra-core pyyaml tqdm
+      - name: Check recipe syntax
+        run: bash -n examples/industrial_data_pretraining/sense_voice/finetune.sh
+      - name: Test validation, checkpoint lifecycle and two-rank collectives
+        run: |
+          python -m pytest -q \
+            tests/test_checkpoint_metric_presence.py \
+            tests/test_checkpoint_metric_distributed.py \
+            tests/test_trainer_ds_unvalidated_ckpt.py \
+            tests/test_train_ds_distributed_config.py \
+            tests/test_sensevoice_continual_finetuning_docs.py \
+            --junitxml=checkpoint-selection.xml
+      - name: Save test report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: checkpoint-selection-report
+          path: checkpoint-selection.xml
+          if-no-files-found: error

examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.md

@@ -24,16 +24,19 @@ relative regression = (candidate CER - baseline CER) / baseline CER
 The threshold is a product decision, not a FunASR guarantee. Select a checkpoint
 only when it meets every retention threshold and the new-domain target.
 
-A real starting point from [issue #3388](https://github.com/modelscope/FunASR/issues/3388)
-is:
-
-| Validation set | Utterances | Baseline CER |
-| --- | ---: | ---: |
-| Mandarin FLEURS test | 300 | 8.9% |
-| Cantonese ASCEND dialogue | 314 | 9.1% |
-| Teochew held-out, about 20 speakers in the source corpus | 300 | 93.3% |
-
-Keep the Teochew held-out speakers out of the 8.45-hour training set.
+References must be independently audited for transcription policy and provenance.
+Check whether new-domain labels are verbatim transcripts or written-language
+paraphrases, and whether test references were generated by the same process as
+training labels. Speaker separation alone does not make those references reliable.
+An unaudited new-domain CER must not select checkpoints or tune replay ratios.
+
+The reporter in [issue #3388](https://github.com/modelscope/FunASR/issues/3388#issuecomment-5578763142)
+later withdrew the earlier new-domain evaluation and rebuilt an independent test
+set. The old baseline table is therefore not a valid selection reference. Recompute
+the frozen baseline and all candidates with the same reviewed references and CER
+normalization. For Mandarin and Cantonese, also separate read and spontaneous
+speech when both matter in production. Keep held-out speakers out of the 8.45-hour
+training example below; its replay budgets are illustrative, not validated optima.
 
 ## 2. Build a replay manifest
 
@@ -72,15 +75,29 @@ pool totals disagree.
 
 ## 3. Use a supported language token
 
-SenseVoiceSmall currently has a closed language-id set: `<|zh|>`, `<|en|>`,
-`<|yue|>`, `<|ja|>`, and `<|ko|>`. For Teochew transcripts written as
-standard Chinese, use `<|zh|>` for the first experiment.
+The current model's named inference language mapping includes `zh`, `en`, `yue`,
+`ja`, and `ko` (plus automatic/no-speech handling). This is not the complete
+tokenizer vocabulary. Sharing `<|zh|>` with a new domain is an experiment, not an
+isolation mechanism: it can interfere with Mandarin and must pass retention gates.
+
+The #3388 reporter found a useful `<|minnan|>` comparison in their setup. A token
+existing in a vocabulary does not mean it is unused, freely repurposable, or
+accepted by the stock inference language mapping. Pin the tokenizer/model revision
+and inspect the token id, training target and query embedding path, and inference
+mapping before adopting a language-tag change. Do not generalize that experiment
+into a guarantee for another language or checkpoint.
 
 Do not add `<|teochew|>` to the JSONL and assume it becomes a new language id.
 The current tokenizer does not encode it as one supported special token. A real
 custom language id requires coordinated tokenizer, vocabulary, language
 embedding, checkpoint migration, export, and inference changes.
 
+Check downstream language validation and every allowlist as well. Test an actual
+result through export, service parsing and the final client: a valid model result
+must not be silently discarded because its language tag is unfamiliar. Preserve
+raw model output while diagnosing an empty response; do not simply disable all
+validation or relabel the result to pass a closed list.
+
 ## 4. Train in two stages
 
 Copy `finetune.sh` to an experiment directory, point it at the mixed training
@@ -129,6 +146,31 @@ parameters trainable because no parameter name matches it.
 
 ## 5. Select the checkpoint by constraints
 
+The maintained `finetune.sh` explicitly uses:
+
+```bash
+++train_conf.avg_keep_nbest_models_type=loss
+```
+
+SenseVoice returns ASR CTC loss plus a rich-tag objective and `acc_rich`, not the
+trainer's ASR `acc`. `acc_rich` is not CER or ASR accuracy. A missing metric must not
+be treated as a real zero; genuine `acc=0` remains a valid score for models that
+actually report it. The trainers exclude an unavailable/nonfinite selected metric
+from best selection, keep-N ranking and metric-based averaging, with an explicit
+warning. This can leave unranked checkpoints on disk; correct the criterion/data
+instead of assuming the keep-N limit applies to them.
+
+Use a new output directory when changing from accuracy to loss ranking. Resuming
+with a different criterion is rejected; old zero-valued histories cannot reliably
+be identified as missing after the fact, and pruned weights cannot be recovered.
+Earlier runs affected by this defect need independent reevaluation of retained
+checkpoints, not just an upgrade. Revalidation cannot overwrite the current best
+under the same checkpoint name if its selected metric becomes unavailable.
+
+`model.pt.best` now means best **total validation loss** for this recipe, not the
+lowest CER or best retained-language behavior. Weight averaging is also only a
+candidate: reevaluate the averaged model under all the same gates.
+
 Evaluate every candidate checkpoint on Mandarin, Cantonese, and Teochew
 separately.
 
@@ -156,6 +198,8 @@ Verify the trainable parameter list before relying on any adapter experiment.
 - Baseline CER/WER and acceptable relative regression are recorded per language.
 - Pseudo-labels are filtered and manually sampled for audit.
 - Replay is sampled to an explicit per-epoch budget with a Cantonese floor.
-- Teochew uses the supported `<|zh|>` path unless the model is deliberately forked.
+- References and normalization are independently audited before any CER comparison.
+- Language-token mapping and downstream allowlists pass an end-to-end result check.
+- The ranking metric is explicit; changed criteria use a new output directory.
 - Freeze logs and trainable parameters match the intended stage.
 - Every saved checkpoint is evaluated on all retained and new domains.

examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.md

@@ -20,15 +20,15 @@ replay 数据之间必须说话人不重叠。不要用一个混合 CER 选择 c
 阈值应由业务要求决定，不是 FunASR 的保证值。候选 checkpoint 必须同时满足所有旧语种
 阈值和新域目标。
 
-[issue #3388](https://github.com/modelscope/FunASR/issues/3388) 提供了一个真实起点：
+参考文本的标注规范和来源必须经过独立核验：新域标签是逐字转写还是书面语意译？测试
+参考是否与训练标签来自同一套生成流程？仅做到说话人不重叠，并不能证明标注可靠。
+未经核验的新域 CER 不能用来选 checkpoint，也不能用来调整 replay 比例。
 
-| 验证集 | 句数 | 基线 CER |
-| --- | ---: | ---: |
-| 普通话 FLEURS test | 300 | 8.9% |
-| 粤语 ASCEND 口语对话 | 314 | 9.1% |
-| 潮汕话 held-out（源语料约 20 名说话人） | 300 | 93.3% |
-
-潮汕话 held-out 中的说话人不能出现在 8.45 小时训练数据中。
+[issue #3388 的后续反馈](https://github.com/modelscope/FunASR/issues/3388#issuecomment-5578763142)
+已撤回早期新域评测，改用另建的独立测试集。因此旧基线表不能继续作为选点依据。
+应使用相同、经过人工复核的参考文本和 CER 归一化规则，重新评测原模型及所有候选。
+线上同时包含朗读和自发口语时，普通话、粤语也应分别检查这两类数据。下文 8.45 小时
+训练示例仍须排除 held-out 说话人；其 replay 小时数是预算演示，不是已验证的最优值。
 
 ## 2. 构建 replay 清单
 
@@ -61,13 +61,23 @@ replay 数据之间必须说话人不重叠。不要用一个混合 CER 选择 c
 
 ## 3. 使用现有支持的语言 token
 
-SenseVoiceSmall 当前语言 id 是闭集：`<|zh|>`、`<|en|>`、`<|yue|>`、
-`<|ja|>` 和 `<|ko|>`。潮汕话转写目标为标准中文时，第一轮实验使用 `<|zh|>`。
+当前模型具名推理语言映射包含 `zh`、`en`、`yue`、`ja`、`ko`，以及自动判断和无语音
+处理；这不是 tokenizer 完整词表。让新域共享 `<|zh|>` 只是实验方案，不是隔离机制，
+可能干扰普通话，必须通过旧域保持评测。
+
+#3388 报告者在自己的设置中获得了有价值的 `<|minnan|>` 对照结果，但“词表中存在”
+不等于“空闲、可任意复用”，也不代表原版推理语言映射支持它。采用语言 tag 改动前，
+固定 tokenizer/模型版本，核对 token id、训练目标和查询 embedding 路径以及推理映射。
+不能把该个例写成其他语种或 checkpoint 的保证。
 
 不要直接在 JSONL 中添加 `<|teochew|>` 并假设它会成为新语言 id。当前 tokenizer
 不会把它编码成一个受支持的特殊 token。真正新增自定义语言 id，需要同步修改 tokenizer、
 词表、language embedding、旧权重迁移、导出和推理代码。
 
+还要检查下游语言校验与每一处白名单，用真实结果走通导出、服务端解析和最终客户端。
+不能因为语言 tag 不在旧枚举中，就静默丢弃有效的模型输出。排查空响应时保留原始模型
+结果，不要简单关闭全部校验或改写语言标签来绕过白名单。
+
 ## 4. 分两阶段训练
 
 把 `finetune.sh` 复制到实验目录，指向混合训练 manifest 和固定验证集，然后把默认的
@@ -110,6 +120,26 @@ Setting encoder...requires_grad = False
 
 ## 5. 按约束选择 checkpoint
 
+维护中的 `finetune.sh` 已明确指定：
+
+```bash
+++train_conf.avg_keep_nbest_models_type=loss
+```
+
+SenseVoice 返回 ASR CTC loss 与 rich-tag 目标，以及 `acc_rich`，并不返回训练器所选的
+ASR `acc`。`acc_rich` 不是 CER 或 ASR 准确率。缺失指标不能伪装成真实零值；对确实
+返回准确率的模型，`acc=0` 仍是有效值。所选指标缺失、非有限值或只覆盖部分 batch/rank
+时，训练器会明确告警，并将该点排除在 best、keep-N 排名和按指标平均之外。未参与排名
+的 checkpoint 可能继续留在磁盘上，应修正指标或数据，不要假设 keep-N 会删除它们。
+
+从 accuracy 改为 loss 排名时，必须使用新的输出目录；恢复时混用两套指标会被拒绝。
+历史上的零值无法可靠追溯为“缺失”，已裁剪的权重也无法恢复。受影响的旧训练需要对
+仍保留的 checkpoint 独立复测，而不是只升级软件。同名重验若导致当前 best 的所选指标
+失效，也会被拒绝，避免覆盖现有 best。
+
+这个配方的 `model.pt.best` 表示**总验证 loss 最优**，不是 CER 最低或旧域保持最好。
+权重平均结果也只是候选模型，仍须重新通过全部评测门槛。
+
 每个候选 checkpoint 都要分别评估普通话、粤语和潮汕话。
 
 1. 丢弃任何一个旧语种超过退化阈值的 checkpoint。
@@ -133,6 +163,8 @@ SenseVoice adapter；任何 adapter 实验都必须先核对实际可训练参
 - 已记录每个语种的基线 CER/WER 和可接受相对退化。
 - 已过滤伪标签，并按语种和说话人做人工抽检。
 - replay 使用明确的每轮预算，并设置粤语最低占比。
-- 未 fork 模型时，潮汕话使用当前支持的 `<|zh|>` 路径。
+- 参考文本和归一化规则已经独立核验，再做 CER 对比。
+- 语言 token 映射和下游白名单已经完成真实结果的端到端检查。
+- 已明确排名指标；更换指标时使用新的输出目录。
 - 冻结日志和可训练参数与当前阶段一致。
 - 所有保存的 checkpoint 都分别评估旧语种和新域。

examples/industrial_data_pretraining/sense_voice/finetune.sh

@@ -63,6 +63,7 @@ ${train_tool} \
 ++train_conf.save_checkpoint_interval=2000 \
 ++train_conf.keep_nbest_models=20 \
 ++train_conf.avg_nbest_model=10 \
+++train_conf.avg_keep_nbest_models_type=loss \
 ++train_conf.use_deepspeed=false \
 ++train_conf.deepspeed_config=${deepspeed_config} \
 ++optim_conf.lr=0.0002 \

funasr/train_utils/average_nbest_models.py

@@ -14,6 +14,7 @@
 import re
 from collections import OrderedDict
 from functools import cmp_to_key
+from funasr.train_utils.checkpoint_metrics import finite_scalar
 
 
 def _get_checkpoint_paths(output_dir: str, last_n: int = 5, use_deepspeed=False, **kwargs):
@@ -31,7 +32,19 @@ def _get_checkpoint_paths(output_dir: str, last_n: int = 5, use_deepspeed=False,
             )
         avg_keep_nbest_models_type = checkpoint["avg_keep_nbest_models_type"]
         val_step_or_epoch = checkpoint[f"val_{avg_keep_nbest_models_type}_step_or_epoch"]
-        sorted_items = sorted(val_step_or_epoch.items(), key=lambda x: x[1], reverse=True)
+        eligible = []
+        for key, value in val_step_or_epoch.items():
+            value = finite_scalar(value)
+            ckpt = os.path.join(output_dir, key)
+            if use_deepspeed:
+                ckpt = os.path.join(ckpt, "mp_rank_00_model_states.pt")
+            if value is not None and os.path.isfile(ckpt):
+                eligible.append((key, value))
+        if not eligible:
+            logging.warning("No checkpoints with an available finite %s metric for averaging.",
+                            avg_keep_nbest_models_type)
+            return []
+        sorted_items = sorted(eligible, key=lambda x: x[1], reverse=True)
         sorted_items = (
             sorted_items[:last_n] if avg_keep_nbest_models_type == "acc" else sorted_items[-last_n:]
         )
@@ -43,8 +56,8 @@ def _get_checkpoint_paths(output_dir: str, last_n: int = 5, use_deepspeed=False,
                 ckpt = os.path.join(output_dir, key, "mp_rank_00_model_states.pt")
             checkpoint_paths.append(ckpt)
 
-    except:
-        print(f"{checkpoint} does not exist, avg the lastet checkpoint.")
+    except (FileNotFoundError, KeyError):
+        logging.warning("Checkpoint ranking metadata unavailable; averaging latest checkpoints.")
         # List all files in the output directory
         files = os.listdir(output_dir)
         # Filter out checkpoint files and extract epoch numbers