Skip to content

Correct clock start, reconnect, write batching and receive CPU - #9

Open
rednayan wants to merge 8 commits into
mainfrom
fix/network-robustness
Open

rednayan wants to merge 8 commits into
mainfrom
fix/network-robustness

Conversation

@rednayan

Copy link
Copy Markdown
Owner

Summary

This branch corrects four network faults and one smaller fault. Each fix has a test. The reference is liblsl at conformance/liblsl/ (e651023c).

  1. Clock offset 0 before the first measurement. An inlet with clock sync now waits up to 5 s (clock::FIRST_OFFSET_TIMEOUT) for the first offset, as liblsl time_correction(5) does. The post-processor never sees a zero offset first.
  2. No reconnect after a lost TCP connection. The reader now connects again to the same outlet. If the outlet moved, the reader finds it again. Then the clock thread goes to the new service port (ClockState::retarget) and the offset resets (ClockState::reset_offset), as in src/time_receiver.cpp:26-33, 212-218.
  3. One socket write per sample. The outlet now writes a pushed chunk with one write, and obeys Max-Chunk-Length (0 = no limit). New Outlet::push_samples. The C API chunk calls use it.
  4. Receive-path CPU. The reader moves its buffer once per read (it was O(n²)) and locks the queue once per read (SampleQueue::push_owned). Measured: 1 MB of 8-channel data decodes in 5 ms (was 58-66 ms). 64-channel: 3.7 ms (was 11 ms).
  5. Multicast TTL. Discovery queries now use multicast_ttl of the configuration, as src/resolve_attempt_udp.cpp:45 does.

Deliberate differences from liblsl

  • After the 5 s wait, liblsl throws timeout_error. labstream gives the samples with the last offset (0 if none). A first pull with clock sync can take longer than its timeout.
  • After a lost connection, labstream first tries the same endpoint directly, then resolves.
  • The timestamp deducer stays after a reconnect to the same outlet (same uid).
  • The watchdog resolve runs on the reader thread.
  • The outlet writes when its queue is empty. This replaces the pushthrough flag. The C API ignores the pushthrough argument.
  • The port walk of the resolver stays. labstream outlets that cannot hold the multicast port, and the machine scope (127.0.0.1), need it.
  • The resolver early exit is unchanged. It already stops at wave boundaries, as liblsl does.

Not in this branch

  • The per-sample Vec<Value> allocation. A fix needs an API change.
  • A half-open connection still waits for the watchdog while the outlet stays visible (same as liblsl). TCP keepalive is the next step.

Tests

  • Gate on this machine: fmt check, clippy -D warnings, doc -D warnings, full workspace tests. All pass.
  • New live tests: tests/recover_live.rs (a proxy cuts the connection; a restarted outlet), tests/clock_live.rs (retarget). The proxy test fails on main after 10 s and passes here in about 0.05 s.
  • Conformance is the same as main: plain, --zero-stamps, --desc, --postproc 7 28/28. --chunk 16 26/28 and --sync 24/28: liblsl itself refuses the failing string cells. selftest, refuse100, pylsl_check 5/5 and examples pass.

Before release (v0.1.3)

  • CI on Linux, macOS, Windows and MSRV 1.75 (1.75 was not run on this machine).
  • Two-machine run with conformance/oracle/labrecorder.sh (reconnect and TTL changes).
  • Release steps of docs/releasing.md.

🤖 Generated with Claude Code

Before, the post-processor got an offset of 0 until the first burst
finished. The first timestamps and the start of the jitter fit therefore
used no offset. The real offset then arrived about 0.6 s later as a step.

Now `LiveSource::correction` waits up to 5 s for the first offset. liblsl
uses the same wait (src/stream_inlet_impl.h:44, src/time_receiver.cpp:57-76).
The wait ends early when no probe thread runs.

If no offset arrives in 5 s, liblsl raises timeout_error. This source
gives the last value instead and does not wait again until a reset. The
samples therefore continue to flow when UDP is blocked.

ClockState also gets `reset_offset` and `retarget` for the recovery path.
A burst that started before a change of target publishes nothing.
Before, a closed TCP connection to an outlet that still ran froze the
stream. `find_again` found the old instance identifier and returned
nothing, and the reader then read the dead socket again.

Now the reader connects again to the same endpoint after the socket
closes. The handshake sends the old instance identifier, so a different
outlet on the same port cannot attach. liblsl does the same
(src/data_receiver.cpp:331-345).

If the stream has a new instance identifier, the reader attaches to it.
The clock offset then resets as in liblsl (src/time_receiver.cpp:212-218),
and the probes go to the new service port (src/time_receiver.cpp:26-33).

If only the watchdog fires and the old stream is still visible, the reader
keeps the open socket, as liblsl does (src/inlet_connection.cpp:186-188).

Two live tests use a loopback proxy and a restarted outlet.
Before, a consumer connection with `Max-Chunk-Length` 0 wrote each sample
with its own socket write. With a chunk length above 0, a partial chunk
waited up to 200 ms for an idle timeout.

liblsl writes when a sample carries the push-through flag, or when the
count reaches the chunk size (src/tcp_server.cpp:782). A chunk length of 0
means no limit (src/tcp_server.cpp:749-753). Each push call sets the flag
on its last sample by default (src/stream_outlet_impl.h:208-209).

Now the samples of one push call go into each consumer queue with one
lock. The writer takes all samples that wait. It writes when the count
reaches the chunk size, or when the queue is empty. An empty queue acts as
the push-through flag, so no sample waits for a later push.

`Outlet::push_samples` sends a block with one timestamp for each sample.
`push_chunk` and the chunk calls of the C API use it. `push_chunk_fast` is
now the same call as `push_chunk`. A blocking outlet writes a chunk with
one write for each consumer.

The C API reads no pushthrough argument, as before. A push with the flag
off therefore does not hold samples back.
Before, the reader removed the bytes of each decoded sample from the
front of its buffer. On a full 64 KB read, that costs time in proportion
to the square of the buffer size. The reader also took the queue lock and
woke the consumer once for each sample.

Now `decode_buffered` decodes every whole sample of a read and moves the
rest of the buffer once. `SampleQueue::push_owned` then moves the samples
into the queue with one lock and no copy.

A measurement on this machine decoded 1 MB of 8-channel float32 samples
in 64 KB reads. The old loop took 58-66 ms. The new loop took 5 ms.

The decoder still allocates one `Vec<Value>` for each sample. The type
`Sample` is public, so this change keeps it.
Before, a query socket always used a TTL of 1. A query of the site scope
or a wider scope therefore did not pass a router.

liblsl sets the TTL of its query socket from the configuration
(src/resolve_attempt_udp.cpp:45). This resolver now does the same, for
IPv4 and IPv6.
Clippy rejects an enum with one large variant and two empty variants.
The box keeps the result small.
If the reset ran first, a burst to the old service port could publish
after the reset. The epoch test now runs against the new target first.

A live test moves the probes from one outlet to a second outlet. The test
fails when the retarget does nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant