lbadm --status/--overview omit WorkerStatus.sessionId, making session-scoped timeseries (mbr_tx_*) unreachable

[yakyakyak]

Summary

lbadm --status and lbadm --overview receive WorkerStatus.sessionId from the control plane and never print it. Because the per-member egress counters (mbr_tx_pkts / mbr_tx_bytes) are addressable only at session scope, and no other route to the session id is currently open, this makes LB egress unmeasurable in practice.

Version: v0.4.0rc1 (container ibaldin/e2sar:0.4.0rc1). Line numbers below are against the v0.4.0rc1 tag; bin/lbadm.cpp, src/e2sarCP.cpp and src/pybind/py_e2sarCP.cpp are unchanged between that tag and v0.4.0-wip HEAD (3626da9), so they apply to both.
Control plane: udplbd at ejfat-lb.es.net:18008 (ESnet beta).

Detail

mbr_tx_pkts / mbr_tx_bytes are the LB's per-member egress counters. udplbd constructs exactly one path for them:

// udplbd2/src/db/timeseries.rs:995
name: format!("/lb/{}/session/{}/mbr_tx_pkts", reservation_id, session_id)

There is no LB-wide rollup, so a query at LB scope returns empty rather than erroring. Reading them requires a session_id.

The CP already sends it — WorkerStatus field 22:

message WorkerStatus {
    ...
    int64 totalPacketsRecv = 18;  // from latest SendState
    repeated SlotRange slots = 19;
    ...
    int64 sessionId = 22; // session ID from database
}

lbadm --status prints eighteen fields from that message and then closes the bracket without ever calling w.sessionid():

// bin/lbadm.cpp:241-248
std::cout << "[ name="s << w.name() << ", controlsignal="s << w.controlsignal() <<
    ...
    ", totalBytesRecv=" << w.totalbytesrecv() << ", totalPacketsRecv=" << w.totalpacketsrecv() <<
    "] "s << std::endl;

The same omission is repeated in the --overview printer at bin/lbadm.cpp:288.

Why the workarounds are also closed

• Wildcard /lb/<rid>/session/* — supported server-side (udplbd2/src/api/handlers/timeseries.rs:138), but the beta deployment currently rejects every wildcard selector with rc=255. Fully-specified per-metric paths are the only accepted form, and those need the id.
• The receiver process — LBManager::registerWorker stores the id (src/e2sarCP.cpp:484, _cpuri.set_sessionId(rep.sessionid())) and EjfatURI::to_string would serialize it as sessionid=, but e2sar_perf never prints the URI after registration and Reassembler exposes no getter for its internal LBManager.
• Python bindings — src/pybind/py_e2sarCP.cpp:50 binds WorkerStatus with only get_name, get_fill_percent, get_control_signal and get_slots_assigned. sessionId is not exposed there either.

Impact

We are characterising the load balancer on NERSC Perlmutter — 4 senders × 8 receivers, 64 MB events, MTU 9000, 50 Gbps aggregate (Slurm job 58327682, CP reservation 576). The packet chain from sender application to receiver application:

stage	packets
sender app sendmsg	209,875,900
LB FPGA rx_packets (ingress)	209,876,000
LB FPGA mbr_tx_pkts (egress)	unobtainable
receiver app	209,857,432

18,468 fragments went missing. The sender→LB leg is clean, so the loss is at or below the LB — but without the egress counter we cannot separate "the LB discarded it" from "the LB→receiver fabric discarded it". It is the only leg in the chain with no independent counter at either end.

Suggested fix

One field in each printer:

", totalPacketsRecv=" << w.totalpacketsrecv() << ", sessionId=" << w.sessionid() <<

plus a matching .def("get_session_id", &WorkerStatus::sessionid) in the pybind binding.

Note for the caller side

Member-scoped counters reset when a session is created, so each session must be read within its own lifetime — a PRE/POST baseline cannot be subtracted across iterations the way LB-scoped counters can. With 8 receivers, and beta accepting only fully-specified paths, a full egress snapshot is 16 lbadm invocations.

Reproduction

Any reservation on ejfat-lb.es.net:18008 with an admin token and at least one registered worker:

lbadm -6 --status --lbid <RID>
# no sessionId anywhere in the output, though field 22 is populated

---

One of three observability issues found in the same investigation; see #199 (timeseries CSV precision) and #200 (silent success on unmatched selectors). Happy to test patches on Perlmutter against beta — we run this configuration regularly and have the full packet chain instrumented on both sides.