Skip to content

Fix paging when the ordering field is not a plain UTC string - #75

Merged
rolandbanks merged 1 commit into
mainfrom
fix/issue-74-paging-field-parsing
Aug 4, 2026
Merged

Fix paging when the ordering field is not a plain UTC string#75
rolandbanks merged 1 commit into
mainfrom
fix/issue-74-paging-field-parsing

Conversation

@rolandbanks

Copy link
Copy Markdown
Contributor

Addresses #74, originally reported by @jamesmanning in #25.

The problem

Paging read the ordering field with ToString() and concatenated "Z" onto it. That breaks in two ways once sysparm_display_value is involved.

With sysparm_display_value=all every field is returned as a { display_value, value } object, so ToString() yields JSON and the parse throws, taking the whole query with it. This is the failure reported in #25.

Less obviously, Newtonsoft recognises ISO-8601 text while deserialising and converts it to a DateTime before the client ever sees it, shifting it to the host's local timezone. ToString() then renders it in the host's culture:

returned:  2026-01-02T05:00:00+05:00     true instant: 2026-01-02 00:00:00Z
ToString:  02/01/2026 07:00:00           shifted to host local, rendered dd/MM/yyyy
parsed:    2026-02-01 07:00:00Z          invariant culture reads 02/01 as 1 February

The paging window ends up a month out, with no error raised. Being host-dependent, this behaves differently on a UTC build agent than on a developer machine, which makes it easy to miss.

Appending "Z" is separately wrong for any value that already carries an offset, since it produces an unparseable string.

The change

ParsePagingFieldValue now:

  1. Unwraps { display_value, value }, preferring the raw value, which carries the underlying UTC timestamp
  2. Takes Date tokens as dates, avoiding the ToString() round-trip that introduced the host's culture and timezone
  3. Parses any remaining string with InvariantCulture and AssumeUniversal | AdjustToUniversal, instead of concatenating "Z"
  4. Raises a ServiceNowApiException naming the field and the offending value when the ordering field cannot be interpreted, rather than a bare FormatException surfacing from inside a Max()

Testing

Adds PagingFieldParsingTests (5 cases) and extracts StubServiceNowHandler into its own file so both paging test classes can share it. The stub now records request URIs, so a test can assert on the paging window actually sent rather than only on the rows returned.

These are credential-free and run in CI.

Verified against the previous behaviour: reverting the parse makes 4 of the 5 new cases fail. The fifth, PlainUtcPagingField_StillPagesAsBefore, passes either way, which is the point: the common case is unchanged.

Verified against a live instance: the same multi-page query run with sysparm_display_value unset, =true and =all now returns the same 5,410 rows in all three modes. The all run is the one that previously died.

Notes

Paging read the ordering field with ToString() and concatenated "Z". That failed with sysparm_display_value=all, where each field is a { display_value, value } object so ToString() yields JSON, and it silently corrupted values that Newtonsoft had already converted to DateTime during deserialisation: those render in the host's culture and timezone, so an en-GB host turned 2026-01-02T05:00:00+05:00 into a window of 2026-02-01 07:00:00. Date tokens are now taken as dates, the raw value is preferred over the display value, remaining strings are parsed with the invariant culture and AssumeUniversal, and an unusable ordering field raises a ServiceNowApiException naming the field and value. Adds PagingFieldParsingTests; 4 of its 5 cases were verified to fail against the previous behaviour. Verified against a live instance: with the fix, display-value modes none/true/all all return the same 5,410 rows. Reported by @jamesmanning in #25.
@rolandbanks
rolandbanks merged commit f22eca3 into main Aug 4, 2026
2 of 5 checks passed
@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 22 complexity · 7 duplication

Metric Results
Complexity 22
Duplication 7

View in Codacy

AI Reviewer: run a review on demand. To trigger the first review automatically, go to your organization or repository integration settings. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant