Data streaming ADR for period indexing#227
Conversation
|
|
||
| In the previous system, periods are 1-indexed to a user. This means that a dashboard can display things like `1/1 periods` if using a single period, or for example `9/12` periods to show a run that is in progress and currently collecting data to period 9. This means that PVs that interface the client and/or `genie_python` need to display as 1-indexed. Conversely, the hardware was 0-indexed, so the ICP had to subtract 1 from the served PV. | ||
|
|
||
| As well as hardware, periods in a NeXus file were previously 0-indexed, which made array slicing more straightforward as most programming languages similarly index starting at 0. |
There was a problem hiding this comment.
The period number in period_log in a nexus file is 1 based
There was a problem hiding this comment.
so i mean from a mantid file reading point of view it is using 1 based periods when it used period_log to find period for an event
| We will use 0-indexing everywhere (in hardware, udp format, flatbuffers, nexus files) other than the period PVs served by `kafka_dae_control` and `kafka_dae_diagnostics` which will subtract and add one respectively to set and display period numbers. | ||
|
|
||
| There was a consideration to reserve period 0 as an "unknown" period, similar to how traditionally the 0th detector was reserved for malformed data, however this doesn't make sense in the streaming world and an event without a period should just be discarded by the hardware or UDP to Kafka layer. | ||
|
|
There was a problem hiding this comment.
period 0 was traditionally used on the user facing GUI to mean "current period" i.e. you woudl setthe display to use period 0 and it would display whatever was being collected in the current period on the graph
There was a problem hiding this comment.
the current period being whatever the period card was set to
There was a problem hiding this comment.
was it - i thought 1 was, so you could have 1/1 periods rather than 0/1 periods
There was a problem hiding this comment.
I think Freddie means PVs like SPEC:0:1:X (where the format is SPEC:period:detector) where that zero is interpreted as current period. I think this is mainly a concern for kafka_dae_diagnostics.
There was a problem hiding this comment.
yes things like Tom says, also on PVs used by e.g. live view OPI screen - you can choose period you want to display in live view, if you choose period 0 there it displays picture for current period so saves user needing to toggle between flipper on and flipper off etc.
|
|
||
| In the previous system, periods are 1-indexed to a user. This means that a dashboard can display things like `1/1 periods` if using a single period, or for example `9/12` periods to show a run that is in progress and currently collecting data to period 9. This means that PVs that interface the client and/or `genie_python` need to display as 1-indexed. Conversely, the hardware was 0-indexed, so the ICP had to subtract 1 from the served PV. | ||
|
|
||
| As well as hardware, periods in a NeXus file were previously 0-indexed, which made array slicing more straightforward as most programming languages similarly index starting at 0. |
There was a problem hiding this comment.
Having checked, and after a discussion with @FreddieAkeroyd, the data in /raw_data_1/framelog/period_log is 1-indexed. So I think filewriter will also need to do the +1 when it writes data (we have to treat the .nxs effectively as a "user interface") to remain backwards compatible.
This doesn't change the rest of the decision that we can be zero-indexed in hardware and in flatbuffers etc, just that the nexus writer will need a bit of logic for this.
There was a problem hiding this comment.
also anything writing a multi-dimensional period based histogram to a nexus file will need to know to set the axis labels array for period number to be one based.
|
|
||
| There was a consideration to reserve period 0 as an "unknown" period, similar to how traditionally the 0th detector was reserved for malformed data, however this doesn't make sense in the streaming world and an event without a period should just be discarded by the hardware or UDP to Kafka layer. | ||
|
|
||
| ## Consequences |
There was a problem hiding this comment.
In this section can we list more explicitly that:
- Hardware will use zero-indexed
- UDP messages from detectors will use zero-indexed
- Flatbuffers messages, both before and after event_aggregator, will use zero-indexed
kafka_dae_controlwill use zero-indexed everywhere except user-facing PVs, where it will need to do a conversionkafka_dae_diagnosticswill use zero-indexed everywhere except user-facing PVs, where it will need to do a conversion- Filewriter will need to do a conversion internally to remain consistent with old format, such that flatbuffers messages saying "period 0" end up in a frame log called "period 1".
There was a problem hiding this comment.
what are we considering as a "user-facing PVs" - any PV without explicit access control on it, or any pv that doesn't start/end with an _, or any not documented as user facing?
There was a problem hiding this comment.
any pv that doesn't start/end with an _
This feels like a reasonable definition to me. I think we've used conventions like IN:INST:_SOMETHING:SOME_PARAMETER in the past if everything in _SOMETHING is private, so maybe
PV that has no component starting/ending with a _
might be more precise, but not that bothered.
|
let's get this in for now, can always change later |
3 participants
No description provided.