From dfe219a8fc6e0a7d0dcbe92cc2dd81ea81588b24 Mon Sep 17 00:00:00 2001 From: "Charles Graham, SWT" Date: Tue, 25 Aug 2026 19:00:30 +0000 Subject: [PATCH 1/3] docs: propose duplicate timeseries value handling Signed-off-by: Charles Graham, SWT --- .../0014-duplicate-timeseries-values.rst | 124 ++++++++++++++++++ docs/source/decisions/index.rst | 1 + 2 files changed, 125 insertions(+) create mode 100644 docs/source/decisions/0014-duplicate-timeseries-values.rst diff --git a/docs/source/decisions/0014-duplicate-timeseries-values.rst b/docs/source/decisions/0014-duplicate-timeseries-values.rst new file mode 100644 index 000000000..bf92cdfe8 --- /dev/null +++ b/docs/source/decisions/0014-duplicate-timeseries-values.rst @@ -0,0 +1,124 @@ +############################################## +Duplicate Time-Series Values in Write Requests +############################################## + + +Summary +======== + +This ADR proposes a consistent API policy for handling multiple time-series +values that resolve to the same CWMS storage minute in a single write request. +The time-series write endpoints will expose a ``use-if-multiple`` query +parameter with four strategies: ``error``, ``first``, ``last``, and +``average``. The default will be ``error``. + + +Context +======== + +CWMS stores time-series timestamps at minute precision. A request can therefore +contain records that have identical timestamps or distinct sub-minute +timestamps that resolve to the same storage minute. Passing those records to +the database without first resolving the collision causes the write to fail. + +Handling these collisions only in individual clients produces inconsistent +behavior. The API should define and enforce the policy so that direct API users +and downstream libraries have the same choices and default behavior. + + +Proposal +======== + +The ``POST /timeseries`` and ``PATCH /timeseries/{timeseries}`` endpoints will +accept an optional ``use-if-multiple`` query parameter. Parameter values will +use the lowercase names below. An unsupported value will produce a ``400 Bad +Request`` response. + +Records will be grouped by the minute to which CWMS will store their timestamp. +Within each group, ``first`` and ``last`` refer to the order of records in the +request payload. + +.. list-table:: Duplicate value strategies + :header-rows: 1 + :widths: 20 30 50 + + * - Value + - Behavior + - Notes + * - ``error`` + - Reject the request when any storage minute has more than one record. + - This is the default. The response will be ``400 Bad Request`` and will + identify that multiple values were supplied for the same minute. The + request will be validated before storage so that no values from the + request are written. + * - ``first`` + - Store the first record supplied for each storage minute and discard later + records for that minute. + - The selected record's value and quality code are kept together. + * - ``last`` + - Store the last record supplied for each storage minute and discard earlier + records for that minute. + - The selected record's value and quality code are kept together. + * - ``average`` + - Store the arithmetic mean of the non-null values supplied for each storage + minute. + - If all values in the group are null, the resolved value is null. The + quality-code policy for an averaged value must be settled before this ADR + is accepted. + +Duplicate handling is independent of ``store-rule``. The +``use-if-multiple`` parameter resolves collisions within one incoming payload; +``store-rule`` continues to control how the resolved records interact with data +that is already stored. + + +Opinions +======== + +Opinion 1 +--------- + +Summary: Adopt the four strategies and default described in this proposal. + +Charles Graham + +Defining duplicate handling at the API boundary gives every caller the same +behavior. Defaulting to ``error`` avoids silently discarding or changing data, +while the other strategies allow callers to make an explicit choice when their +source data can contain collisions. + + +Consequences +============ + +* Existing callers that omit ``use-if-multiple`` retain the current fail-safe + behavior when duplicate storage minutes are submitted. +* Downstream libraries can expose the API strategies rather than implementing + collision handling independently. +* ``first`` and ``last`` make request order significant and must therefore be + implemented without reordering records before selection. +* The OpenAPI description and generated clients will eventually need to expose + the parameter, but those implementation changes are outside this ADR-only + pull request. + + +Questions Before Acceptance +=========================== + +* Which quality code should be stored for a value produced by ``average``? +* If write formats later include data-entry dates, how should an averaged + record's data-entry date be selected? +* Should a successful non-``error`` request report how many records were + discarded or combined, and if so, through which response field or header? + + +Decision Status +=============== + +(Status: proposed) + + +References +========== + +Issue/Discussion: https://github.com/USACE/cwms-data-api/issues/1783 diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst index a01accde0..caa4d6d27 100644 --- a/docs/source/decisions/index.rst +++ b/docs/source/decisions/index.rst @@ -30,3 +30,4 @@ Some decisions may also be a proposal and marked appropriately. JMS Queue Message Structure <./0011-queue-messages.rst> Vertical Datum Storage <./0012-vertical-datum-storage.rst> CDA User Lists <./0013-cda-user-lists.md> + Duplicate Time-Series Values <./0014-duplicate-timeseries-values.rst> From 06ec9b238bb59ec2743840f727889c976992b65a Mon Sep 17 00:00:00 2001 From: "Charles Graham, SWT" Date: Tue, 25 Aug 2026 20:23:40 +0000 Subject: [PATCH 2/3] docs: clarify duplicate timestamp scope Signed-off-by: Charles Graham, SWT --- .../0014-duplicate-timeseries-values.rst | 50 ++++++++++++------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/docs/source/decisions/0014-duplicate-timeseries-values.rst b/docs/source/decisions/0014-duplicate-timeseries-values.rst index bf92cdfe8..36fb6f939 100644 --- a/docs/source/decisions/0014-duplicate-timeseries-values.rst +++ b/docs/source/decisions/0014-duplicate-timeseries-values.rst @@ -7,21 +7,25 @@ Summary ======== This ADR proposes a consistent API policy for handling multiple time-series -values that resolve to the same CWMS storage minute in a single write request. +values that resolve to the same effective CWMS storage timestamp in a single +write request. The time-series write endpoints will expose a ``use-if-multiple`` query parameter with four strategies: ``error``, ``first``, ``last``, and -``average``. The default will be ``error``. +``average``. The default, if nothing is specified, will be ``error``. Context ======== -CWMS stores time-series timestamps at minute precision. A request can therefore -contain records that have identical timestamps or distinct sub-minute -timestamps that resolve to the same storage minute. Passing those records to -the database without first resolving the collision causes the write to fail. +The current CDA storage path normalizes incoming time-series timestamps to +minute precision before checking for duplicates. This applies regardless of +whether the time-series interval is one minute, one hour, one day, or longer. A +request can therefore contain records with identical timestamps or distinct +sub-minute timestamps that resolve to the same effective storage timestamp. +Passing those records to the database without first resolving the collision +causes the write to fail. -Handling these collisions only in individual clients produces inconsistent +Handling these collisions only in individual clients can produce inconsistent behavior. The API should define and enforce the policy so that direct API users and downstream libraries have the same choices and default behavior. @@ -29,14 +33,21 @@ and downstream libraries have the same choices and default behavior. Proposal ======== -The ``POST /timeseries`` and ``PATCH /timeseries/{timeseries}`` endpoints will +The ``POST /timeseries`` and ``PATCH /timeseries/{timeseries}`` endpoints accept an optional ``use-if-multiple`` query parameter. Parameter values will use the lowercase names below. An unsupported value will produce a ``400 Bad Request`` response. -Records will be grouped by the minute to which CWMS will store their timestamp. -Within each group, ``first`` and ``last`` refer to the order of records in the -request payload. +Records will be grouped by their effective CWMS storage timestamp. For the +current CDA storage path, this means the timestamp after normalization to +minute precision. Within each group, ``first`` and ``last`` refer to the order +of records in the request payload. + +This policy does not group records merely because they fall within the same +named time-series interval. For example, two records in the same hour are not +duplicates under this policy if they retain different effective storage +timestamps. Rounding or bucketing records into hourly, daily, monthly, or other +intervals would be separate API behavior and is not defined by this ADR. .. list-table:: Duplicate value strategies :header-rows: 1 @@ -46,22 +57,23 @@ request payload. - Behavior - Notes * - ``error`` - - Reject the request when any storage minute has more than one record. + - Reject the request when any effective storage timestamp has more than one + record. - This is the default. The response will be ``400 Bad Request`` and will identify that multiple values were supplied for the same minute. The request will be validated before storage so that no values from the request are written. * - ``first`` - - Store the first record supplied for each storage minute and discard later - records for that minute. + - Store the first record supplied for each effective storage timestamp and + discard later records for that timestamp. - The selected record's value and quality code are kept together. * - ``last`` - - Store the last record supplied for each storage minute and discard earlier - records for that minute. + - Store the last record supplied for each effective storage timestamp and + discard earlier records for that timestamp. - The selected record's value and quality code are kept together. * - ``average`` - Store the arithmetic mean of the non-null values supplied for each storage - minute. + timestamp. - If all values in the group are null, the resolved value is null. The quality-code policy for an averaged value must be settled before this ADR is accepted. @@ -92,7 +104,7 @@ Consequences ============ * Existing callers that omit ``use-if-multiple`` retain the current fail-safe - behavior when duplicate storage minutes are submitted. + behavior when duplicate effective storage timestamps are submitted. * Downstream libraries can expose the API strategies rather than implementing collision handling independently. * ``first`` and ``last`` make request order significant and must therefore be @@ -105,6 +117,8 @@ Consequences Questions Before Acceptance =========================== +* Should a separate API option support rounding or bucketing timestamps into + the named time-series interval before applying ``use-if-multiple``? * Which quality code should be stored for a value produced by ``average``? * If write formats later include data-entry dates, how should an averaged record's data-entry date be selected? From 813b44fcdd23460c82cc626c899bb8c83029d810 Mon Sep 17 00:00:00 2001 From: "Charles Graham, SWT" Date: Tue, 25 Aug 2026 20:25:32 +0000 Subject: [PATCH 3/3] docs: remove duplicate value ADR status Signed-off-by: Charles Graham, SWT --- docs/source/decisions/0014-duplicate-timeseries-values.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/source/decisions/0014-duplicate-timeseries-values.rst b/docs/source/decisions/0014-duplicate-timeseries-values.rst index 36fb6f939..cd1dea958 100644 --- a/docs/source/decisions/0014-duplicate-timeseries-values.rst +++ b/docs/source/decisions/0014-duplicate-timeseries-values.rst @@ -126,12 +126,6 @@ Questions Before Acceptance discarded or combined, and if so, through which response field or header? -Decision Status -=============== - -(Status: proposed) - - References ==========