Skip to content

fix(noema): remove fixed LLM response timeout - #1511

Closed
seonghobae wants to merge 1 commit into
mainfrom
fix/noema-unbounded-response
Closed

fix(noema): remove fixed LLM response timeout#1511
seonghobae wants to merge 1 commit into
mainfrom
fix/noema-unbounded-response

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • remove Noema's hard-coded 120-second urllib response timeout
  • keep GitHub Actions concurrency/cancellation as the execution boundary
  • preserve SSRF, redirect refusal, bounded prompt, and substantive verdict validation
  • document the two exact-head ContextualWisdomLab/naruon#1480 timeout reproductions

Root cause

Both Noema jobs 99462790578 and 99472141106 reached a healthy contextual-orchestrator sidecar, then scripts/ci/noema_review_gate.py raised TimeoutError: timed out at opener.open(..., timeout=120). The standing central review contract explicitly allows model reviews to take two hours or more.

Verification

  • RED: two focused tests failed because the runtime still passed 120
  • GREEN: Python 3.12, PYTHONWARNINGS=error, plugin autoload disabled: 60 passed
  • focused root-cause pair: 2 passed
  • Ruff: clean
  • compileall: clean
  • git diff --check: clean
  • independent agent review: PASS, no P1/P2/P3

Safety

timeout=None removes the connection/read socket deadline only. Workflow cancellation and concurrency remain authoritative; URL/loopback validation and redirect refusal are unchanged.


Devin Review

Summary by CodeRabbit

  • 변경 사항
    • Noema 검토 요청의 고정된 120초 응답 제한을 제거했습니다.
    • 장시간 검토는 워크플로의 취소 관리에 따라 완료될 수 있습니다.
    • 기존 URL 검증, 리디렉션 차단, 프롬프트 제한 및 결과 검증은 유지됩니다.
    • 관련 문서와 회귀 테스트를 업데이트했습니다.

@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d329a6c-11bf-4bf4-975f-59c4a8763b3d

📥 Commits

Reviewing files that changed from the base of the PR and between 1cbb6aa and 4fca0b6.

📒 Files selected for processing (5)
  • CHANGELOG.md
  • docs/doctoring/noema-orchestrator-free-zdr.md
  • scripts/ci/noema_review_gate.py
  • tests/test_noema_review_gate.py
  • tests/test_repository_branch_coverage_review_schedulers.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Noema LLM 요청의 고정된 120초 타임아웃을 제거했습니다. timeout=None 계약을 적용하고, 관련 테스트와 오케스트레이터 문서 및 변경 로그를 갱신했습니다.

Changes

Noema 호출 타임아웃 변경

Layer / File(s) Summary
LLM 요청 타임아웃 계약 변경
scripts/ci/noema_review_gate.py
LLM 요청이 timeout=None을 사용하도록 변경되었습니다.
타임아웃 검증 및 문서 갱신
tests/test_noema_review_gate.py, tests/test_repository_branch_coverage_review_schedulers.py, docs/doctoring/noema-orchestrator-free-zdr.md, CHANGELOG.md
테스트가 timeout=None을 검증합니다. 오케스트레이터 문서와 변경 로그가 새 타임아웃 동작을 기록합니다.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: 🔵 Low · up to 4fca0

The change allows legitimate multi-hour model reviews to complete, but a stalled response may keep a review job running until cancellation or platform limits terminate it because no explicit workflow job timeout is configured. The PR is mergeable with owner awareness and follow-up to add a bounded workflow timeout.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. (2 skipped: 2… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 Noema의 고정된 LLM 응답 시간 제한 제거라는 주요 변경 사항을 정확하고 간결하게 설명합니다.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. (2 skipped: 2 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/noema-unbounded-response

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment on lines +656 to +657
# Noema reviews can legitimately take hours; the workflow owns cancellation.
with opener.open(request, timeout=None) as response: # nosec B310

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Cancellation remains externally bounded

timeout=None removes connection and read deadlines. Workflow concurrency still cancels superseded runs, so stalled requests remain bounded by Actions cancellation.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Closing as superseded: this PR, #1415, #1508, and #1509 all independently fix the same contextual-orchestrator#946 call_llm-timeout root cause. This PR's timeout=None approach is the same underlying idea as #1508's (equivalent, un-named-constant) removal and #1509's earlier iteration before it settled on a bounded, monotonic total-budget deadline plus a credential-refresh fix for reviews that outlive the job's GitHub App token.

#1415 already has a bounded fix merged into its branch (CALL_LLM_TIMEOUT_SECONDS=9600, derived backwards from noema-review.yml's own timeout-minutes: 360 job ceiling so the worst case actually fits inside it) and is otherwise the most mature of the four PRs touching this area. #1509's credential-refresh and deadline-watchdog work is independently valuable and slated to be ported into #1415 as a fast-follow. Given that, an unbounded timeout=None doesn't add anything the others don't already cover more precisely, so closing this one to avoid further duplicated review effort. No criticism of the diagnosis — it's correct, just superseded by more complete fixes elsewhere.


Generated by Claude Code

@seonghobae seonghobae closed this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant