Repository navigation
Conversation
Fixes CI failures: black --check was failing on three files, and actions/checkout, actions/cache, setup-miniconda, setup-node, and codecov-action were pinned to old majors that target the now deprecated Node 20 runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mambaforge installers are no longer published upstream, causing a 404 on download. Miniforge3 (the action's default) now bundles mamba by default and use-mamba: true is already set, so this is a no-op behaviorally. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open meteo migration
0018_remove_zentra_add_open_meteo_fields depended on the old 0017_remove_aggregateddepthprediction_tile_size_and_more migration, which had already been squashed and superseded by 0003_riverchannel via 0001_squashed_0017/0002_remove_depthprediction_bounding_box_and_more. On a fresh database Django rewires that stale dependency onto the squash node directly, creating two leaf migrations and failing CI with "Conflicting migrations detected". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…cibility
The Unit Tests workflow was failing at the setup-miniconda step. Both conda
environment files listed `defaults` alongside `conda-forge`, and conda 25.3+
(shipped in current Miniforge3) refuses to use repo.anaconda.com channels
without an interactively accepted Terms of Service:
CondaToSNonInteractiveError: Terms of Service have not been accepted
for the following channels: .../pkgs/main, .../pkgs/r
setup-miniconda's `conda-remove-defaults` does not help here: it only strips
`defaults` when implicitly added, and treats naming it in the environment file
as an explicit opt-in. Removing the channel is the fix; everything resolves
from conda-forge alone.
Also fixed, each independently verified:
* njsscan used github/codeql-action/upload-sarif@v2, discontinued 2025-01-10.
Moved to v3.
* "Check built assets are up-to-date" never worked. With
`working-directory: manyfews` the pathspec `manyfews/webapp/static` resolved
to manyfews/manyfews/webapp/static, matched nothing, and passed regardless of
what the build produced. Corrected to `webapp/static`.
* package.json required webpack ^5.76.0 while package-lock.json pinned 5.70.0,
so `npm ci` would refuse outright and `npm install` silently resolved a
different bundle than the committed one. Regenerated the lockfile and rebuilt;
CI and the Dockerfile now use `npm ci` so the build is reproducible. Verified
`npm ci && npm run build` reproduces the committed bundle byte-for-byte.
* Dropped node-sass. It is unused (sass-loader resolves dart-sass) and supports
only Node <=17, so it would have broken the Dockerfile's node:lts-alpine
stage once Docker builds started running again.
* Dockerfile build_python used continuumio/miniconda3, which configures the
Anaconda defaults channel and hits the same ToS wall. Switched to
condaforge/miniforge3.
* black was pinned three different ways (CI ~= 22.3, pre-commit 24.10.0, devel
env 22.3.0), so running pre-commit locally produced code CI rejected. Aligned
all three on 24.10.0 and reformatted the three affected files.
* Workflow hygiene: bumped the stale docker workflow actions, chromedriver
v2 -> v3, Node 16 -> 22 with npm caching, and replaced the conda cache (wrong
path, key referencing a nonexistent get-date step, malformed CACHE_NUMBER)
with package caching on ~/conda_pkgs_dir. Dropped use-only-tar-bz2, which was
pinning the solve to 2022-era builds, and use-mamba, which is a no-op now that
Miniforge3 no longer bundles mamba.
Bootstrap (5.1.3 -> 5.3.8) and jQuery (3.6.0 -> 3.7.1) moved within their
declared ^ ranges during the lockfile regeneration. The Selenium tests exercise
Bootstrap collapse and modal behaviour and were not run as part of this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These three failures are present in every Unit Tests run in this repo's history, including the run before the recent CI config changes. They are not caused by the Bootstrap/jQuery version bump. 1. calculations.tests.FloodCalculationTests.test_predict_depth predict_depth was decorated @jit(nopython=False) and takes a Django model as its second argument. Numba 0.59 removed @jit's object-mode fallback, so this now attempts a nopython compile and fails: TypingError: Cannot determine Numba type of <class 'calculations.models.FloodModelParameters'> Split the numeric work into _depth_centiles, a nopython kernel taking plain floats and a float64 array, leaving the model attribute reads in a thin Python wrapper. This is what the code was reaching for with the existing "FIXME: getattr is slow" note, and it now compiles in nopython mode rather than relying on a fallback that no longer exists. The polynomial is evaluated in Horner form instead of via np.polynomial.Polynomial (unsupported in nopython). Verified numerically identical to the previous implementation on the repo's own three test cases and on 500 randomised cases covering negative coefficients, None coefficients and the minQ threshold. One deliberate behaviour change: depths are accumulated in float64 rather than np.zeros_like(flow_values). The old code truncated results to integers when handed an integer flow array. No caller does that; the test that passes an int array only produces whole numbers, so its expectations are unchanged. 2. webapp.tests.WebAppTestCase.test_users 3. webapp.tests.WebAppTestCase.test_alerts Both are races. WebElement.click() does not reliably block until the next page has loaded, so reading current_url immediately afterwards can observe the old page - which is why these failed at a different assertion on each run (test_users at line 157 one run, line 144 the next). test_alerts fetched the alerts table and then its rows as two separate round trips, and the page could reload in between, raising StaleElementReferenceException. Added assertOnPage(), which waits for the expected URL before asserting, and alert_table_rows(), which re-finds the table and retries while it is being re-rendered. Waiting for the password_reset done page also removes the assumption that the reset email has been sent by the time the outbox is read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_users now gets far enough to reach the password-reset flow, which no CI run has ever executed. It fails looking up id_username on /accounts/login/, and NoSuchElementException alone does not say what the browser is actually displaying - the login template renders that field unconditionally, so the browser is evidently not on the page we think it is. wait_for_element reports current_url, title and the start of the page source on timeout, so the next run identifies the actual landing page rather than leaving it to guesswork. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The diagnostic added in the previous commit showed the browser sitting on
/accounts/reset/done/ while the test was looking for the login form:
'id_username' never appeared.
current_url: http://localhost:38993/accounts/reset/done/
Clicking 'Change password' starts a navigation that click() does not wait for,
so the get() to /accounts/login/ was issued mid-flight and the reset form's
pending redirect landed afterwards, replacing it.
Wait for the reset-complete page before navigating on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates ManyFEWS to remove Zentra Cloud + NOAA GEFS dependencies and instead use Open-Meteo for both historical weather (state updates/spin-up) and ensemble forecasts, while also addressing intermittent UI test failures and modernizing CI/build tooling.
Changes:
- Replace Zentra/GEFS ingestion with a new Open-Meteo ingestion layer (historical archive + ensemble forecast) and propagate that through tasks, models, migrations, and tests.
- Make Selenium tests more reliable by adding explicit wait helpers and improving failure diagnostics; fix Twilio verification semantics and add a unit test for status handling.
- Update CI/build configuration (conda-forge-only, newer GitHub Actions,
npm ci, Black version sync) and remove unused/deprecated dependencies.
Reviewed changes
Copilot reviewed 34 out of 38 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Updates project description and team listing to reflect Open-Meteo usage. |
| manyfews/webapp/tests.py | Adds Twilio verification tests and stabilizes Selenium flows with explicit waits/helpers. |
| manyfews/webapp/static/index-bundle.js.LICENSE.txt | Updates bundled license metadata after frontend dependency updates. |
| manyfews/webapp/load.py | Removes Zentra device creation from test data loader path. |
| manyfews/webapp/fixtures/initial_data.json | Removes Zentra-related seed objects (device + periodic task). |
| manyfews/webapp/alerts.py | Fixes verification logic to treat only Twilio "approved" as verified. |
| manyfews/package.json | Removes node-sass and relies on sass. |
| manyfews/manyfews/urls.py | Minor formatting tweak. |
| manyfews/manyfews/settings.py | Removes Zentra/GEFS settings and adds Open-Meteo configuration + stricter secrets handling. |
| manyfews/manyfews/.env.CI | Updates CI env to include SECRET_KEY and Open-Meteo settings. |
| manyfews/calculations/zentra.py | Removes Zentra ingestion implementation. |
| manyfews/calculations/zentra_devices.py | Removes Zentra device discovery/bindings. |
| manyfews/calculations/tests.py | Reworks task tests to mock Open-Meteo and validate ensemble member + bucket counts. |
| manyfews/calculations/tasks.py | Updates model setup/update tasks to use Open-Meteo historical + ensemble forecast; removes Zentra device import task. |
| manyfews/calculations/open_meteo.py | New module implementing Open-Meteo historical + ensemble ingestion and 6-hour bucketing. |
| manyfews/calculations/models.py | Removes Zentra models; adds Open-Meteo forecast metadata fields and AggregatedWeatherReading. |
| manyfews/calculations/migrations/0018_remove_zentra_add_open_meteo_fields.py | Introduces new weather schema, adds forecast metadata, drops Zentra tables. |
| manyfews/calculations/generate_river_flows.py | Generalizes “GEFS” weather input to “weather data” and supports ensemble member selection. |
| manyfews/calculations/gefs.py | Removes GEFS download/parse implementation. |
| manyfews/calculations/flood_risk.py | Dedupe scheduling per forecast_time and aggregate river-flow samples across ensemble members; refactors depth-centile computation for nopython compilation. |
| manyfews/calculations/fixtures/ZentraDevice.json | Removes Zentra fixture. |
| manyfews/calculations/bulk_create_manager.py | Minor formatting tweak. |
| manyfews/calculations/admin.py | Removes ZentraDevice admin registration. |
| docs/DEVELOPMENT.md | Updates development instructions to remove Zentra account requirement. |
| Dockerfile | Updates build images/tooling (miniforge, npm ci, Debian base), and adjusts build-time env vars. |
| config/manyFEWS.devel.yml | Removes defaults channel, updates Black version to align with CI/pre-commit. |
| config/manyFEWS.base.yml | Moves to conda-forge-only and updates dependency set to match Open-Meteo/tenacity/requests. |
| .pre-commit-config.yaml | Updates Black version and documents cross-file version sync. |
| .github/workflows/run_unitTest_GenerateRiverFlows.yml | Updates CI (actions versions, conda caching strategy, Node 22, npm ci, static pathspec fix). |
| .github/workflows/njsscan.yml | Updates action versions. |
| .github/workflows/build_docker_images.yml | Updates Docker action versions. |
| .github/workflows/black.yml | Updates action versions and Black version pin. |
| .github/azure/docker-compose.backend.yml | Removes Zentra env wiring and adds Open-Meteo env vars. |
| .github/azure/azure-pipelines.yml | Removes Zentra vars and updates comments/vars to Open-Meteo equivalents. |
Suppressed comments (2)
manyfews/calculations/tasks.py:153
- Checking only
== 0can miss partial/incomplete historical data for yesterday (e.g., if a prior run inserted fewer than the expected number of 6-hour buckets). Since the model expects a full day’s buckets, trigger a refetch unless the expected count is present.
if aggregateDataLength == 0:
# Get the last day’s data from Open-Meteo's historical archive
prepareOpenMeteoHistorical(start_date=yday[0], end_date=yday[0])
manyfews/calculations/flood_risk.py:58
len(forecast_times)evaluates the whole queryset into Python just to check emptiness. Use.exists()for an efficient DB-level existence check.
if len(forecast_times) == 0:
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
10
to
+14
| | Project Member | Contact address | Role | Unit | | ||
| |--------------------------------------------------|----------------------------------------------------------------------|----------------------------------|-------------------------------------------------------------------------------------| | ||
|
|
||
|
|
||
| | Prof. Simon Mathias | [simon.mathias@durham.ac.uk](mailto:simon.mathias@durham.ac.uk) | Project Lead (PI) | [Department of Engineering](https://www.durham.ac.uk/departments/academic/engineering/) | |
Comment on lines
134
to
138
| aggregateDataLength = len( | ||
| AggregatedZentraReading.objects.filter(date__range=(yday[0], yday[1])).filter( | ||
| AggregatedWeatherReading.objects.filter(date__range=(yday[0], yday[1])).filter( | ||
| location=location | ||
| ) | ||
| ) |
| logger.info(f"Loading parameters from {filename}") | ||
|
|
||
| total_rows = sum(1 for _ in open(filename)) | ||
| total_rows = sum(1 for _ in open(filename, encoding="utf-8-sig")) |
Comment on lines
+50
to
+53
| RiverFlowCalculationOutput.objects.filter( | ||
| prediction_date=latest_prediction_date, | ||
| forecast_time__lte=today + timedelta(days=16), | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.