Skip to content

fix(strix): preserve NIM fallback evidence #1254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seonghobae wants to merge 3 commits into from
Closed

fix(strix): preserve NIM fallback evidence #1254

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e21c149
fix(strix): preserve NIM fallback evidence
seonghobae Aug 23, 2026
2c6aaef
fix(strix): preserve NIM quality evidence
seonghobae Aug 23, 2026
bcd1318
test(strix): retain trusted smoke compatibility
seonghobae Aug 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions scripts/ci/strix_quick_gate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,8 @@ has_strix_report_failure_signal() {
report_root="$newest_report_root"
fi
while IFS= read -r -d '' report_log; do
if grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning|WARNING|Timeout)([^[:alpha:]]|$)' "$report_log"; then
if grep -Eiv '^[[:space:]]*[^[:alnum:]]*[[:space:]]*MODEL QUALITY WARNING[[:space:]]*[^[:alnum:]]*[[:space:]]*$' "$report_log" |
grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning|WARNING|Timeout)([^[:alpha:]]|$)'; then
Comment on lines +224 to +225

@devin-ai-integration devin-ai-integration Bot Aug 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Report warning-signal check can miss large log artifacts

Under pipefail, grep -Eiq exits on first match and can kill the upstream grep -Eiv with SIGPIPE, so the pipeline fails despite matching. A Fatal/Warn/Timeout signal early in a large report log is then read as absent, and the artifact no longer fails the gate closed.

Suggested change
if grep -Eiv '^[[:space:]]*[^[:alnum:]]*[[:space:]]*MODEL QUALITY WARNING[[:space:]]*[^[:alnum:]]*[[:space:]]*$' "$report_log" |
grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning|WARNING|Timeout)([^[:alpha:]]|$)'; then
if grep -Eiv '^[[:space:]]*[^[:alnum:]]*[[:space:]]*MODEL QUALITY WARNING[[:space:]]*[^[:alnum:]]*[[:space:]]*$' "$report_log" |
grep -Ei '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning|WARNING|Timeout)([^[:alpha:]]|$)' >/dev/null; then
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +224 to +225

@devin-ai-integration devin-ai-integration Bot Aug 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Banner filter only matches the single title line

The exclusion regex matches only a whole line equal to the banner title plus box/padding. If the real NIM MODEL QUALITY WARNING banner spans multiple lines, any body line containing warning/quality still trips the Fatal/Warn detector and fails the scan closed. Safe, but may not fully tolerate clean NIM fallbacks as intended. Worth confirming the actual banner is a single line.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +224 to +225

@devin-ai-integration devin-ai-integration Bot Aug 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Filter regex correctly narrows to the exact heading

The anchored filter only exempts the bare MODEL QUALITY WARNING heading with box/space decoration. A line such as MODEL QUALITY WARNING: detail keeps alnum after WARNING, so it is not filtered and still fails closed. The exemption stays narrow as intended.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

return 0
fi
done < <(find "$report_root" -type f -name '*.log' -print0)
Expand Down Expand Up @@ -3151,7 +3152,8 @@ is_llm_token_limit_error() {
# was interrupted or incomplete. Used as a guard to prevent the
# below-threshold override from silently passing an aborted scan.
has_detected_infrastructure_error() {
if grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)' "$STRIX_LOG"; then
if grep -Eiv '^[[:space:]]*[^[:alnum:]]*[[:space:]]*MODEL QUALITY WARNING[[:space:]]*[^[:alnum:]]*[[:space:]]*$' "$STRIX_LOG" |
grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)'; then
Comment on lines +3155 to +3156

@devin-ai-integration devin-ai-integration Bot Aug 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Infrastructure-error guard can miss signals in large logs

Under the script's pipefail, grep -Eiq exits on the first match and can kill the upstream grep -Eiv with SIGPIPE, so the pipeline fails despite matching. An early warning in a large log then reads as absent, letting an rc==0 scan that should fail closed pass the gate.

Suggested change
if grep -Eiv '^[[:space:]]*[^[:alnum:]]*[[:space:]]*MODEL QUALITY WARNING[[:space:]]*[^[:alnum:]]*[[:space:]]*$' "$STRIX_LOG" |
grep -Eiq '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)'; then
if grep -Eiv '^[[:space:]]*[^[:alnum:]]*[[:space:]]*MODEL QUALITY WARNING[[:space:]]*[^[:alnum:]]*[[:space:]]*$' "$STRIX_LOG" |
grep -Ei '(^|[^[:alpha:]])(Fatal|Denied|Warn|Warning)([^[:alpha:]]|$)' >/dev/null; then
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

return 0
fi

Expand Down
40 changes: 40 additions & 0 deletions scripts/ci/test_strix_quick_gate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3416,6 +3416,23 @@ REPORT
;;
esac
;;
nvidia-nim-quality-warning-fallback-success)
case "${STRIX_LLM:-}" in
nvidia_nim/nvidia/nemotron-3-super-120b-a12b)
echo "Error: litellm.RateLimitError: Nvidia_nimException - Error code: 429"
exit 1
;;
nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5)
echo "│ MODEL QUALITY WARNING"
echo "scan ok with NVIDIA NIM fallback"
exit 0
;;
*)
echo "unexpected model ${STRIX_LLM:-}" >&2
exit 9
;;
esac
;;
vertex-all-notfound)
echo "Error: litellm.NotFoundError: Vertex_aiException - x"
echo '"status": "NOT_FOUND"'
Expand Down Expand Up @@ -5951,6 +5968,18 @@ run_filtered_gate_case_if_requested() {
"vertex_ai/ready-primary" \
"<unset>"
;;
nvidia-nim-quality-warning-fallback-success)
run_gate_case "nvidia-nim-quality-warning-fallback-success" \
"nvidia_nim/nvidia/nemotron-3-super-120b-a12b" \
"nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \
"0" \
"REGEX:Strix quick scan succeeded with fallback model 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5' in [0-9]+s\\." \
"2" \
"nvidia_nim/nvidia/nemotron-3-super-120b-a12b|nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \
"<unset>|<unset>" \
"openai" \
""
;;
pr-rust-workspace-context)
run_gate_case "pr-rust-workspace-context" \
"openai/gpt-4o-mini" \
Expand Down Expand Up @@ -12370,6 +12399,17 @@ run_gate_case "direct-openai-gpt-does-not-require-github-models-api-base" \
"openai" \
""

run_gate_case "nvidia-nim-quality-warning-fallback-success" \
"nvidia_nim/nvidia/nemotron-3-super-120b-a12b" \
"nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \
"0" \
"REGEX:Strix quick scan succeeded with fallback model 'nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5' in [0-9]+s\\." \
"2" \
"nvidia_nim/nvidia/nemotron-3-super-120b-a12b|nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5" \
"<unset>|<unset>" \
"openai" \
""

run_gate_case "github-models-model-prefix-with-api-base-succeeds" \
"openai/gpt-5" \
"" \
Expand Down
Loading