Skip to content

feat(api): Batch StreamForecastData responses#163

Merged
devsjc merged 2 commits into
mainfrom
devsjc/batched-streaming
May 6, 2026
Merged

feat(api): Batch StreamForecastData responses#163
devsjc merged 2 commits into
mainfrom
devsjc/batched-streaming

Conversation

@devsjc
Copy link
Copy Markdown
Contributor

@devsjc devsjc commented May 6, 2026

Contribution Checklist

  • Have you followed the Open Climate Fix Contribution Guidelines?
  • Have you referenced the Issue this PR addresses, where applicable?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added a summary of the changes?
  • Have you written new tests for your changes, where applicable?
  • Have you successfully run make lint with your changes locally?
  • Have you successfully run make test with your changes locally?

Warning

PRs may be closed if all the above boxes are not checked.

Changes in this Pull Request

Modifies the StreamForecastData route to return its values in batches. This should be more stable and reduce CPU usage a bit. It does introduce a breaking change to the API, but since the Stream route is not on any hot path, I don't mind doing this. This is still in pre-1.0.0 after all.

The change is reflected in the example.py file, and boils down to data from the stream being under a values field, when previously it was returned top-level.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 6, 2026

Benchmark Results

Benchmark results
?   	github.com/openclimatefix/data-platform/cmd	[no test files]
?   	github.com/openclimatefix/data-platform/internal/gen/ocf/dp	[no test files]
?   	github.com/openclimatefix/data-platform/internal/interceptors	[no test files]
PASS
ok  	github.com/openclimatefix/data-platform/internal/server/dummy	0.005s
{"level":"debug","time":"2026-05-06T15:38:11Z","message":"Completed migrations"}
goos: linux
goarch: amd64
pkg: github.com/openclimatefix/data-platform/internal/server/postgres
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
BenchmarkPostgresClient/small/GetForecastAsTimeseries-4         	      61	  18344513 ns/op
BenchmarkPostgresClient/small/GetForecastAtTimestamp-4          	     307	   3769575 ns/op
BenchmarkPostgresClient/small/GetObservationsAsTimeseries-4     	    1090	   1060037 ns/op
BenchmarkPostgresClient/small/CreateForecast-4                  	     100	  11638655 ns/op
BenchmarkPostgresClient/small/StreamForecastData-4              	      14	  83146088 ns/op
PASS
ok  	github.com/openclimatefix/data-platform/internal/server/postgres	66.469s
?   	github.com/openclimatefix/data-platform/internal/server/postgres/gen	[no test files]
Benchmark vs base branch
goos: linux
goarch: amd64
pkg: github.com/openclimatefix/data-platform/internal/server/postgres
cpu: AMD EPYC 7763 64-Core Processor                
                                                   │ bench-main.txt │
                                                   │     sec/op     │
PostgresClient/small/GetForecastAsTimeseries-4         16.32m ± ∞ ¹
PostgresClient/small/GetForecastAtTimestamp-4          4.162m ± ∞ ¹
PostgresClient/small/GetObservationsAsTimeseries-4     1.265m ± ∞ ¹
PostgresClient/small/CreateForecast-4                  13.20m ± ∞ ¹
PostgresClient/small/StreamForecastData-4              96.92m ± ∞ ¹
geomean                                                10.19m
¹ need >= 6 samples for confidence interval at level 0.95

cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                                                   │ bench-devsjc-batched-streaming.txt │
                                                   │               sec/op               │
PostgresClient/small/GetForecastAsTimeseries-4                             18.34m ± ∞ ¹
PostgresClient/small/GetForecastAtTimestamp-4                              3.770m ± ∞ ¹
PostgresClient/small/GetObservationsAsTimeseries-4                         1.060m ± ∞ ¹
PostgresClient/small/CreateForecast-4                                      11.64m ± ∞ ¹
PostgresClient/small/StreamForecastData-4                                  83.15m ± ∞ ¹
geomean                                                                    9.336m
¹ need >= 6 samples for confidence interval at level 0.95

@devsjc devsjc merged commit 41df950 into main May 6, 2026
4 checks passed
@devsjc devsjc deleted the devsjc/batched-streaming branch May 6, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant