Data streaming ADR for period indexing #227
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # 7 - Periods | ||
|
|
||
| ## Status | ||
|
|
||
| Accepted. | ||
|
|
||
| ## Context | ||
|
|
||
| Periods are used to split up runs. This may be because a scientist wants to scan over several points and keep events separated before data analysis, or it may be for other arbitrary reasons. | ||
|
|
||
| In the previous system, periods are 1-indexed to a user. This means that a dashboard can display things like `1/1 periods` if using a single period, or for example `9/12` periods to show a run that is in progress and currently collecting data to period 9. This means that PVs that interface the client and/or `genie_python` need to display as 1-indexed. NeXus files, specifically the `/raw_data_1/framelog/period_log/value` dataset, follow this logic. Conversely, the hardware was 0-indexed, so the ICP had to subtract/add 1 from the served PVs. | ||
|
|
||
| ## Decision | ||
|
|
||
| We will use 0-indexing everywhere internal and 1-indexing for user facing period controls and readbacks: | ||
| - The streaming hardware will use zero-indexed periods. | ||
| - UDP messages from detectors will use zero-indexed periods. | ||
| - Flatbuffers messages, both before and after event_aggregator, will be zero-indexed | ||
| - `kafka_dae_control` will use zero-indexed periods everywhere except user-facing PVs (PVs that have no component starting/ending with a `_`), where it will need to do a conversion | ||
| - `kafka_dae_diagnostics` will use zero-indexed periods everywhere except user-facing PVs(see above), where it will need to do a conversion | ||
| - The filewriter will need to do a conversion internally to remain consistent with old format, such that flatbuffers messages saying "period 0" end up in a frame log called "period 1". | ||
|
|
||
| There was a consideration to reserve period 0 as an "unknown" period, similar to how traditionally the 0th detector was reserved for malformed data, however this doesn't make sense in the streaming world and an event without a period should just be discarded by the hardware or UDP to Kafka layer. | ||
|
|
||
| ## Consequences | ||
|
Member
There was a problem hiding this comment. In this section can we list more explicitly that:
Member
There was a problem hiding this comment. what are we considering as a "user-facing PVs" - any PV without explicit access control on it, or any pv that doesn't start/end with an
Member
There was a problem hiding this comment.
This feels like a reasonable definition to me. I think we've used conventions like
might be more precise, but not that bothered. |
||
| - We can use 0-indexed periods as a standard for everything that isn't user facing (and NeXus files which _are_ user facing) which means dealing with arrays is much simpler | ||
| - We need to be careful to make sure that the user is shown 1-indexed periods when viewing and setting periods | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
period 0 was traditionally used on the user facing GUI to mean "current period" i.e. you woudl setthe display to use period 0 and it would display whatever was being collected in the current period on the graph
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the current period being whatever the period card was set to
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
was it - i thought 1 was, so you could have 1/1 periods rather than 0/1 periods
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think Freddie means PVs like
SPEC:0:1:X(where the format isSPEC:period:detector) where that zero is interpreted as current period. I think this is mainly a concern forkafka_dae_diagnostics.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes things like Tom says, also on PVs used by e.g. live view OPI screen - you can choose period you want to display in live view, if you choose period 0 there it displays picture for current period so saves user needing to toggle between flipper on and flipper off etc.