From b76316a58baebf3b3c65cbb56ab137e18ffdb010 Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Wed, 15 Jul 2026 12:43:35 +0100 Subject: [PATCH 1/2] Data streaming ADR for period indexing --- .../datastreaming/ADRs/007_periods.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 doc/specific_iocs/datastreaming/ADRs/007_periods.md diff --git a/doc/specific_iocs/datastreaming/ADRs/007_periods.md b/doc/specific_iocs/datastreaming/ADRs/007_periods.md new file mode 100644 index 000000000..a62c0efd9 --- /dev/null +++ b/doc/specific_iocs/datastreaming/ADRs/007_periods.md @@ -0,0 +1,23 @@ +# 7 - Periods + +## Status + +Accepted. + +## Context + +Periods are used to split up runs. This may be because a scientist wants to scan over several points and keep events separated before data analysis, or it may be for other arbitrary reasons. + +In the previous system, periods are 1-indexed to a user. This means that a dashboard can display things like `1/1 periods` if using a single period, or for example `9/12` periods to show a run that is in progress and currently collecting data to period 9. This means that PVs that interface the client and/or `genie_python` need to display as 1-indexed. Conversely, the hardware was 0-indexed, so the ICP had to subtract 1 from the served PV. + +As well as hardware, periods in a NeXus file were previously 0-indexed, which made array slicing more straightforward as most programming languages similarly index starting at 0. + +## Decision + +We will use 0-indexing everywhere (in hardware, udp format, flatbuffers, nexus files) other than the period PVs served by `kafka_dae_control` and `kafka_dae_diagnostics` which will subtract and add one respectively to set and display period numbers. + +There was a consideration to reserve period 0 as an "unknown" period, similar to how traditionally the 0th detector was reserved for malformed data, however this doesn't make sense in the streaming world and an event without a period should just be discarded by the hardware or UDP to Kafka layer. + +## Consequences +- We can use 0-indexed periods as a standard for everything that isn't user facing (and NeXus files which _are_ user facing) +- We need to be careful to make sure that the user is shown 1-indexed periods when viewing and setting periods From 13a493e7a21ff2a755461d45e3c6f8ea0183e609 Mon Sep 17 00:00:00 2001 From: Jack Harper Date: Thu, 16 Jul 2026 12:12:57 +0100 Subject: [PATCH 2/2] review comments --- .../datastreaming/ADRs/007_periods.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/doc/specific_iocs/datastreaming/ADRs/007_periods.md b/doc/specific_iocs/datastreaming/ADRs/007_periods.md index a62c0efd9..595cce5e8 100644 --- a/doc/specific_iocs/datastreaming/ADRs/007_periods.md +++ b/doc/specific_iocs/datastreaming/ADRs/007_periods.md @@ -8,16 +8,20 @@ Accepted. Periods are used to split up runs. This may be because a scientist wants to scan over several points and keep events separated before data analysis, or it may be for other arbitrary reasons. -In the previous system, periods are 1-indexed to a user. This means that a dashboard can display things like `1/1 periods` if using a single period, or for example `9/12` periods to show a run that is in progress and currently collecting data to period 9. This means that PVs that interface the client and/or `genie_python` need to display as 1-indexed. Conversely, the hardware was 0-indexed, so the ICP had to subtract 1 from the served PV. - -As well as hardware, periods in a NeXus file were previously 0-indexed, which made array slicing more straightforward as most programming languages similarly index starting at 0. +In the previous system, periods are 1-indexed to a user. This means that a dashboard can display things like `1/1 periods` if using a single period, or for example `9/12` periods to show a run that is in progress and currently collecting data to period 9. This means that PVs that interface the client and/or `genie_python` need to display as 1-indexed. NeXus files, specifically the `/raw_data_1/framelog/period_log/value` dataset, follow this logic. Conversely, the hardware was 0-indexed, so the ICP had to subtract/add 1 from the served PVs. ## Decision -We will use 0-indexing everywhere (in hardware, udp format, flatbuffers, nexus files) other than the period PVs served by `kafka_dae_control` and `kafka_dae_diagnostics` which will subtract and add one respectively to set and display period numbers. +We will use 0-indexing everywhere internal and 1-indexing for user facing period controls and readbacks: +- The streaming hardware will use zero-indexed periods. +- UDP messages from detectors will use zero-indexed periods. +- Flatbuffers messages, both before and after event_aggregator, will be zero-indexed +- `kafka_dae_control` will use zero-indexed periods everywhere except user-facing PVs (PVs that have no component starting/ending with a `_`), where it will need to do a conversion +- `kafka_dae_diagnostics` will use zero-indexed periods everywhere except user-facing PVs(see above), where it will need to do a conversion +- The filewriter will need to do a conversion internally to remain consistent with old format, such that flatbuffers messages saying "period 0" end up in a frame log called "period 1". There was a consideration to reserve period 0 as an "unknown" period, similar to how traditionally the 0th detector was reserved for malformed data, however this doesn't make sense in the streaming world and an event without a period should just be discarded by the hardware or UDP to Kafka layer. ## Consequences -- We can use 0-indexed periods as a standard for everything that isn't user facing (and NeXus files which _are_ user facing) +- We can use 0-indexed periods as a standard for everything that isn't user facing (and NeXus files which _are_ user facing) which means dealing with arrays is much simpler - We need to be careful to make sure that the user is shown 1-indexed periods when viewing and setting periods