bug: Health check state diverges across nginx workers — unhealthy node keeps receiving traffic

[yeganeahmadnejad]

Description

We use a priority-based upstream (nodes[].priority) with active + passive health checks to fail over from a primary node to backup nodes when the primary becomes unhealthy. We're seeing the primary node continue to receive real traffic for minutes after one nginx worker process has already logged that it crossed the unhealthy.http_failures threshold for that node.

This reproduces consistently on 3.13.0, 3.15.0, and 3.18.0 with an identical upstream config — it does not appear to be tied to any specific APISIX version.

Upstream config (sanitized)

upstreams:
  - id: upstream-api-service-priority-failover
    type: roundrobin
    scheme: https
    pass_host: node
    checks:
      active:
        type: https
        timeout: 2
        http_path: /internal-health
        https_verify_certificate: false
        healthy:
          interval: 3
          http_statuses: [200, 404]
          successes: 2
        unhealthy:
          interval: 3
          http_statuses: [500, 502, 503, 504]
          http_failures: 2
          tcp_failures: 2
          timeouts: 2
      passive:
        type: https
        healthy:
          http_statuses: [200, 201]
          successes: 2
        unhealthy:
          http_statuses: [500, 502, 503, 504]
          http_failures: 2
          tcp_failures: 2
          timeouts: 2
    nodes:
      - host: primary.example.internal
        port: 443
        weight: 100
        priority: 0
      - host: fallback-a.example.internal
        port: 443
        weight: 50
        priority: -1
      - host: fallback-b.example.internal
        port: 443
        weight: 50
        priority: -1

Reproduction / evidence

Deployment runs with multiple nginx worker processes per pod, and multiple pod replicas behind a Kubernetes Service. We added $pid to our access log format to correlate real requests with the worker process that handled them, alongside APISIX's own [healthcheck] warn logs (which already include the worker's OS pid in the <pid>#<tid>: prefix).

2026-08-26T14:37:34Z  [warn] 65#65: [lua] healthcheck.lua:1394: log(): [healthcheck]
  (upstream#/upstreams/upstream-api-service-priority-failover)
  unhealthy HTTP increment (2/2) for '10.0.1.100(10.0.1.100:443)', context: ngx.timer

→ worker process 65 in pod api-gateway-abc123def-11111 has just crossed the unhealthy threshold for the primary node.

6.5 minutes later, a real request is served by the same pod, but by a different worker process (pid 70), and is still routed to the same primary node:

{
  "timestamp": "2026-08-26T14:44:04Z",
  "pod_name": "api-gateway-abc123def-11111",
  "pid": "70",
  "upstream_addr": "10.0.1.100:443",
  "status": "201"
}

No further [healthcheck] log lines were emitted for this target on this pod in between — in particular, no "healthy SUCCESS" recovery line, which lua-resty-healthcheck's incr_counter() would emit on any real state transition back to healthy (an unhealthy→healthy transition doesn't short-circuit the way same-state reports do). That suggests worker 65's checker state genuinely stayed "unhealthy" the whole time, and worker 70 simply never converged to it.

Environment

• APISIX 3.13.0, 3.15.0, and 3.18.0 (reproduces on all three) — same upstream config on all.
• Multiple nginx worker processes per instance with auto worker config, multiple Apisix replicas.
• Priority-based upstream nodes (primary + 2 fallback), active (https) + passive checks enabled.

Expected behavior

Once any worker process's active health checker crosses the unhealthy threshold for a node, all worker processes in that instance should stop routing traffic to that node until it's confirmed healthy again — health check state should not diverge silently between workers within the same process group.