test(e2e): stop racing the controller's own drain budget - #57
Open
agenticode wants to merge 1 commit into
Open
agenticode wants to merge 1 commit into
agenticode wants to merge 1 commit into
Conversation
4b waited 60x5s = 5m for a node to disappear. The actuator's WaitNodeEmpty gives the drain NodeDrainTimeout (default 5m) before it gives up, and only the next reconcile retries. The two budgets were identical, so any drain that was not near-instant became a coin flip: the test killed the controller mid-drain and blamed it for "never removed a node". Symptom in CI: ~1 run in 3 failed, always at 4b, always with step 3 delete-node err="context canceled" at the exact moment of teardown. Failing runs sat the full 7m20s; green runs finished the whole script in ~2m10s. Wait 10m instead, which covers a worst-case drain plus one retry, log progress so the wait is not a silent stall, and dump node/pod state on timeout so a real failure says why. Co-authored-by: kording <74226694+kording@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
문제
e2e잡이 최근 100회 중 34회 실패했다. 실패는 항상 같은 지점이다.원인 — 테스트와 컨트롤러의 인내심이 같았다
e2e.sh4b 대기60 x 5s= 300sactuate.Config.NodeDrainTimeout기본값pkg/actuate/actuate.go:71)StepDeleteNode는WaitNodeEmpty로 노드가 빌 때까지 최대 5분을 기다린다.테스트도 정확히 5분을 기다린다. 두 시계가 같으니, 드레인이 즉시 끝나지 않는
순간부터는 누가 먼저 포기하느냐의 동전던지기가 된다. 테스트가 먼저 시작했으니
항상 테스트가 진다 — 컨트롤러를 드레인 도중에 죽여놓고 "노드를 안 지웠다"고 보고한다.
context canceled는 원인이 아니라 정리(cleanup)가 컨트롤러를 죽인 결과다.측정치가 이 해석과 일치한다:
조치
제품 코드는 건드리지 않았다. 이건 테스트 하니스의 예산 충돌이다.