Skip to content

Node re-read loses its diagnostic when kubectl fails without writing stderr #3886

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

scripts/refresh-flux-ghcr-auth.sh guards a failed node re-read against losing its diagnostic, and
the guard does not cover the case its own comment describes. At the node-claim path (currently
around line 2028 on main):

if ! kubectl ... >"${state_file}" 2>"${reread_error_file}"; then
  # ... a failed copy would leave it EMPTY -- and
  # emit_safe_operation_output skips an empty file entirely, which is the very
  # silence this block exists to prevent. ...
  if ! cat "${reread_error_file}" >"${result_file}" 2>/dev/null; then
    echo "node re-read failed; its diagnostic could not be read" >"${result_file}"
  fi

When kubectl get node exits non-zero without writing to stderr, reread_error_file exists and
is zero bytes. cat then succeeds, so the guard never fires — while the > redirection has
already truncated result_file. emit_safe_operation_output opens with
[[ -s "${result_file}" ]] || return 0, so it emits nothing at all.

The operator is then told only ::error::Could not atomically claim and cordon Talos node …, with
no cause — which is exactly the silence the block was written to prevent.

Who this affects

Whoever is diagnosing a failed Talos node drain. The rollout stops, and the one line that would say
why the node claim could not be re-verified is dropped. It degrades precisely when the API server
is unhealthy enough to fail a request without a message, i.e. when the diagnostic is most valuable.

Expected behaviour

An empty diagnostic is treated as a failed copy, so the deterministic fallback line is written and
emitted.

Acceptance criteria

  • A test drives the node re-read to fail with no stderr and asserts the cordon-claim: fallback
    line reaches the output.
  • A companion test asserts a non-empty diagnostic still survives verbatim and is not
    replaced by the fallback (guards against an over-broad fix).
  • Both fail before the change and pass after.

Size

Small — a one-line [[ ! -s "${reread_error_file}" ]] || guard plus the two tests.

Relationship to #3882

#3882 fixes the identical defect in the Flux parent-fence re-read that PR introduces, with
exactly that guard and that test pair. This issue is the pre-existing sibling on the node-claim
path, which is on main and outside that PR's concern, so it is captured separately rather than
widening a fix PR. The fix and its proof pattern can be lifted directly from #3882.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions