lbadm --timeseries reports success (rc=0, populated CSV) for selectors that matched no series

[yakyakyak]

Summary

A --timeseries query for a metric that does not exist (or exists at a different scope) returns rc=0 with a populated CSV, and lbadm prints Success. Saving timeseries to CSV. There is no way to distinguish a matched selector from an unmatched one in either the exit code or the output.

Version: v0.4.0rc1 (container ibaldin/e2sar:0.4.0rc1).
Control plane: udplbd at ejfat-lb.es.net:18008 (ESnet beta).

Detail

Every timeseries CSV includes an /lb/<rid>/epoch/boundary_event column regardless of what was requested. So a query that matched nothing still produces a non-empty body, a zero exit code, and a success message.

On Slurm job 58327682 we queried fifteen metric names. The two that exist returned 72 rows; the thirteen that do not all reported rc=0 rows=6:

2 TS_METRIC drop_packets  rc=0 rows=6      <- does not exist
2 TS_METRIC mbr_tx_pkts   rc=0 rows=6      <- exists, but not at this scope
...  (13 in total, identical)
1 TS_METRIC rx_packets    rc=0 rows=72     <- real data
1 TS_METRIC rx_bytes      rc=0 rows=72     <- real data

That run's entire LB drop accounting was lost. Nothing in the logs indicated a problem — the failure surfaced only when the numbers were compared against an older capture that happened to have used correct names.

Most of those names were ours and wrong, so that part of the root cause was on our side. The bug is that a wrong name is indistinguishable from a right one.

This compounds with the session-scope rule

Note mbr_tx_pkts in the list above. That name is correct — the request failed only because the metric lives at session scope, not LB scope. A caller who does not already know the scoping rule gets Success and an empty result, with nothing to distinguish:

• this metric does not exist
• you asked at the wrong scope
• this counter is genuinely zero

Reporting which selectors matched would make the scoping problem self-diagnosing, and would have saved us a run.

Suggested fix

Either:

1. Fail the query when none of the requested series are present in the response, or
2. Report per-series which selectors matched, e.g.
   Success: 2 of 15 requested series returned data (13 not found: drop_packets, mbr_tx_pkts, ...)

Option 2 is friendlier for wildcard queries, where a partial match is normal and expected.

Callers currently have to test for the metric's own column in the CSV header. Testing for a non-empty body — the obvious check — is a false positive by construction.

Reproduction

lbadm -6 --timeseries --lbpath "/lb/<RID>/no_such_metric" --since <ISO8601> --csv /tmp/x.csv
echo "rc=$?"
# prints "Success. Saving timeseries to CSV.", rc=0, and /tmp/x.csv is non-empty

---

One of three observability issues found in the same investigation; see #198 (omitted WorkerStatus.sessionId) and #199 (timeseries CSV float precision).