Check Meta validation during indexing

[sujik18]

✅ PR Checklist

• Target branch is dev

📌 Note: PRs must be raised against dev. Do not commit directly to main.

✅ Testing & CI

• Have tested the changes in my local environment, else have properly conveyed in the PR description
• The change includes a GitHub Action to test the script(if it is possible to be added).
• No existing GitHub Actions are failing because of this change.

📚 Documentation

• README or help docs are updated for new features or changes.
• CLI help messages are meaningful and complete.

📁 File Hygiene & Output Handling

• No unintended files (e.g., logs, cache, temp files, pycache, output folders) are committed.

🛡️ Safety & Security

• No secrets or credentials are committed.
• Paths, shell commands, and environment handling are safe and portable.

🙌 Contribution Hygiene

• PR title and description are concise and clearly state the purpose of the change.
• Related issues (if any) are properly referenced using Fixes # or Closes #.
• All reviewer feedback has been addressed.

Moved Meta validation checking from process_config_file function as it is only called if a meta file is modified after index json file is created.

Helps in checking Syntax errors in meta file of scripts, related to issue #228.

Example Log when there is error in meta file:

[2026-05-17 06:17:04,235 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/build-mlperf-inference-server-nvidia/meta.yaml: docker: unknown key 'mlc_repo_flags1'
[2026-05-17 06:17:04,235 index.py       : 276 DEBUG] - Meta validation passed for /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/build-mlperf-inference-server-nvidia/meta.yaml
[2026-05-17 06:17:04,237 main.py        : 134 ERROR] - Error during 'script' action: Script run execution failed in /home/sujith/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py.
Error : ParserError: while parsing a block mapping
  in "/home/sujith/MLC/repos/sujik18@LLM-Evaluation-Framework/script/parse-gate-question/meta.yaml", line 1, column 1
expected <block end>, but found '<scalar>'
  in "/home/sujith/MLC/repos/sujik18@LLM-Evaluation-Framework/script/parse-gate-question/meta.yaml", line 25, column 2
[2026-05-17 06:17:04,238 main.py        : 148 ERROR] -   at /home/sujith/mlc/lib/python3.12/site-packages/mlc/script_action.py:287 in call_script_module_function
[2026-05-17 06:17:04,238 main.py        : 178 ERROR] - Failed script: llm-evaluation
[2026-05-17 06:17:04,239 main.py        : 179 ERROR] - To rerun just the failed part: mlcr llm-evaluation
[2026-05-17 06:17:04,239 main.py        : 188 ERROR] - Please file an issue at https://github.com/mlcommons/mlperf-automations/issues with the full console log.

All Meta validation warnings for the scripts in automation repo:

(mlc) sujith@ideapadG3:~$ time mlcr llm-evaluation -v --env.EXAM_NAME=2026
[2026-05-17 06:36:58,313 index.py       :  35 DEBUG] - Repos path for Index: /home/sujith/MLC/repos
[2026-05-17 06:36:58,384 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-ml-model-stable-diffusion/meta.yaml: Unknown top-level key 'test'
[2026-05-17 06:36:58,473 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-qualcomm/meta.yaml: deps[14]: unknown dep key 'update_tags_from_env_with_prefix1'
[2026-05-17 06:36:58,528 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-aocc/meta.yaml: Unknown top-level key 'post_deps_off'
[2026-05-17 06:36:58,595 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-src/meta.yaml: Unknown top-level key 'print_env_at_the_end_disabled'
[2026-05-17 06:36:58,638 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-automotive/meta.yaml: docker: unknown key 'mlc_repos_off'
[2026-05-17 06:36:58,696 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-dataset-cnndm/meta.yaml: variations.validation,llama3: unknown variation key 'default-variations'
[2026-05-17 06:36:59,499 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-dummy/meta.yaml: deps[7]: unknown dep key 'updats_tags_from_env_with_prefix'
[2026-05-17 06:36:59,605 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-profiler-uprof/meta.yaml: Unknown top-level key 'post_deps_off'
[2026-05-17 06:37:00,109 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/calibrate-model-for.qaic/meta.yaml: variations.bert_: unknown variation key 'default-variations'
[2026-05-17 06:37:00,546 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-oneapi/meta.yaml: Unknown top-level key 'post_deps_off'
[2026-05-17 06:37:00,739 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/meta.yaml: variations.run-harness: unknown variation key 'alis'
[2026-05-17 06:37:00,740 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/meta.yaml: variations.default_variations: unknown variation key 'batch-size'
[2026-05-17 06:37:00,740 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/app-mlperf-inference-nvidia/meta.yaml: variation 'default_variations' looks like a variation property key used as a variation name
[2026-05-17 06:37:01,033 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-endpoints-src/meta.yaml: Unknown top-level key 'print_env_at_the_end_disabled'
[2026-05-17 06:37:01,237 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-automotive-app/meta.yaml: variations.dashboard: unknown variation key 'default_gui'
[2026-05-17 06:37:01,237 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-automotive-app/meta.yaml: variations.submission: unknown variation key 'default_gui'
[2026-05-17 06:37:01,346 index.py       : 271 DEBUG] - Meta validation warning: /home/sujith/MLC/repos/mlcommons@mlperf-automations/script/build-mlperf-inference-server-nvidia/meta.yaml: docker: unknown key 'mlc_repo_flags1'
[2026-05-17 06:37:01,366 script_utils.py:  88 INFO ] - * mlcr llm-evaluation

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅