diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 545e44e..50d4c44 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -88,6 +88,42 @@ jobs: - name: API contract is committed (docs/contracts/openapi.json) run: python scripts/export_openapi.py --check + desktop-install: + name: Default install (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + timeout-minutes: 20 + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + env: + OMP_NUM_THREADS: "2" + OPENBLAS_NUM_THREADS: "2" + MKL_NUM_THREADS: "2" + MPLBACKEND: Agg + steps: + - uses: actions/checkout@v4 + - uses: astral-sh/setup-uv@v6 + with: + python-version: "3.12" + - uses: actions/setup-node@v4 + with: + node-version: "22" + cache: npm + cache-dependency-path: frontend/package-lock.json + - name: Build the bundled Studio page + run: | + npm --prefix frontend ci + npm --prefix frontend run build + - name: Install defaults using automatic PyTorch backend selection + run: | + uv venv --python 3.12 + uv pip install --python .venv -e ".[dev]" --torch-backend=auto + - name: Check default dependencies and available compute + run: uv run --no-project --python .venv python scripts/check_install.py + - name: Real browser fallback remains reachable + run: uv run --no-project --python .venv python -m pytest tests/integration/test_launcher_browser_fallback.py -q + frontend: name: Frontend (types, lint, unit, build, browser journeys) runs-on: ubuntu-latest diff --git a/README.md b/README.md index 217e592..0345dd0 100644 --- a/README.md +++ b/README.md @@ -37,54 +37,63 @@ OpenDPD is a PyTorch framework for power amplifier (PA) modeling and digital pre ## What's new -**OpenDPD 2.2.4** separates PSD plots by signal-chain position: **DPD Input**, **DPD Output / PA Input**, and **PA Output**. Output references and with/without-DPD comparisons share only the PA Output plot, with compact legends and independent chart controls. +**OpenDPD 2.2.5** adds LaTeX equations with interactive PA parameter highlighting, shared or per-channel OFDMA settings, and **ILC + ILA DPD** with a separate waveform-specific Ideal benchmark. Next-step controls sit above settings, and metric calculation methods have clearer names. -**Signal Generator** creates a PA Input Dataset with separate CSV and metadata downloads. The new **PA Library** offers nine mathematical Virtual PAs with editable formulas and linked parameter controls. Explicitly simulate the output, create a paired synthetic dataset, and continue to PA/DPD training and testing. An expandable workflow diagram follows your progress. +**Signal Generator → PA Library → PA training → DPD training/testing.** Generate a PA input waveform, simulate its output with one of nine Virtual PAs, or upload existing input/output CSV data. Results label synthetic, surrogate and measured evidence and show separate PSD charts at each signal-chain position. -See the [2.2.4 release notes](https://github.com/lab-emi/OpenDPD/blob/main/docs/releases/release-notes-2.2.4.md) and [Signal Generator guide](https://lab-emi.github.io/OpenDPD/guides/signal-generator/). Standard presets are uncoded engineering stimuli; Wi-Fi 8 is experimental. Local Studio also adds research comparisons, publication figures and reproduction, measurement sessions, Sweep Board, hardware cost evidence and optional dataset contribution PRs for human review. - -- CUDA replay for supported native models reduces dispatch overhead while retaining the existing optimizer, precision, batches and scheduler. -- Quick/full training defaults are 10/150 epochs; plots update once per epoch; DPD previews capture the intermediate signal in a bounded shadow-model forward pass. -- See the [2.2.1 performance measurements](https://github.com/lab-emi/OpenDPD/blob/main/docs/performance/studio-2.2.1.md) and [release notes](https://github.com/lab-emi/OpenDPD/blob/main/docs/releases/release-notes-2.2.1.md). -- **Guided experiments:** explore built-in I/Q data, train and test PA/DPD models, and choose from the original backbone registry. -- **Live feedback:** separate epoch and batch progress bars, NMSE and other task metrics, live signal plots, reconnectable experiments and a Stop control. -- **Download models while training:** save the best checkpoint so far; after training, download the selected final model. Compare compatible runs and export reports. -- **Browser and local workbench:** nine interface languages, English by default, CUDA when available, touch-friendly plots and system light/dark themes. - -**Bring your own CSV:** upload UTF-8 CSV with two complex columns or four real I/Q columns, up to 25 MiB and 1,000,000 paired samples. Every row is validated in quarantine before preview; rejected uploads are deleted. Code, package and checkpoint uploads are unavailable in the public app. - -For a hosted installation, the [public Studio deployment guide](https://lab-emi.github.io/OpenDPD/architecture/public-studio/) covers GitHub Pages, a Cloudflare Tunnel and isolated local VM compute, with temporary sessions and automatic file deletion within 24 hours. +[2.2.5 release notes](https://lab-emi.github.io/OpenDPD/releases/release-notes-2.2.5/) · [Signal Generator](https://lab-emi.github.io/OpenDPD/guides/signal-generator/) · [ILC guide](https://lab-emi.github.io/OpenDPD/guides/ilc-dpd/). Standard presets are engineering stimuli, not conformance waveforms; Wi-Fi 8 is experimental. [Feature history](docs/whats-new.md) · [Verified platform status](docs/releases/support-matrix.md) ## Get started with Studio -**[Open the hosted Studio now](https://opendpd.com/studio/)**, or install the packaged local app with **Python 3.10–3.13**: +Use **[Studio on the web](https://opendpd.com/studio/)**, or install locally: + +**1. Install [uv](https://docs.astral.sh/uv/getting-started/installation/)** (then open a new terminal). + +macOS / Linux: + +```sh +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +Windows PowerShell: + +```powershell +powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" +``` + +**2. Create an environment, install, and launch** (same commands on all three platforms): -```bash -python -m pip install "opendpd[gui]==2.2.4" -opendpd gui +```sh +mkdir opendpd-lab +cd opendpd-lab +uv venv --python 3.12 +uv pip install --python .venv "opendpd==2.2.5" --torch-backend=auto +uv run --no-project --python .venv opendpd gui ``` -The wheel includes the frontend; Node.js is not needed. For development from source, also install Git and Node.js 22.22+: +PyTorch, Studio and pywebview install together; no Node.js is needed. uv selects a PyTorch backend for the detected platform/drivers. Studio prefers available CUDA or Apple MPS, then CPU. Keep the terminal running. **A `127.0.0.1` link opens on the computer running OpenDPD**; for SSH, use the [port-forwarding instructions](docs/install.md#ssh-or-another-computer). + +Click **Get Started → Signal Generator**, or **Use an existing dataset**. See [Installation](docs/install.md) for drivers, Linux system libraries, native-window troubleshooting and pip; [Studio walkthrough](docs/tutorials/gui-quickstart.md) for your first experiment. + +
+Develop from source (Git and Node.js 22.22+ required) -```bash +```sh git clone https://github.com/lab-emi/OpenDPD.git cd OpenDPD -python -m venv .venv -source .venv/bin/activate -python -m pip install -e ".[gui]" +uv venv --python 3.12 +uv pip install --python .venv -e ".[dev]" --torch-backend=auto npm --prefix frontend ci npm --prefix frontend run build -opendpd gui +uv run --no-project --python .venv opendpd gui ``` -This opens Studio locally in your browser. For Windows, a native desktop window, GPU setup, or a core-only installation, see [Installation](docs/install.md). - -Click **Get Started → Signal Generator** to create a waveform, then **Choose Virtual PA** to simulate a paired dataset. Or choose **Use an existing dataset → DPA_200MHz** to go directly to PA Training. Use **Starting settings → Quick trial** to check the pipeline, then choose **Full training** for a longer experiment. Quick trial defaults to **10 epochs**; full training defaults to **150 epochs**. Plots update **once per epoch**; DPD previews show x, u and PA(u) from the same bounded validation probe. Advanced settings offer an optional batch preview interval with a red warning because extra previews can severely slow training. +
## The PA → DPD workflow @@ -99,7 +108,7 @@ Click **Get Started → Signal Generator** to create a waveform, then **Choose V [PA Library guide](docs/guides/virtual-pa-library.md) · [Reading signal-chain PSD plots](docs/guides/signal-chain-spectra.md) -![Studio 2.2.4: independent signal-chain PSD plots](pics/studio-psd-chain.png) +![Studio 2.2.5: independent signal-chain PSD plots](pics/studio-psd-chain.png) ## Choose your next step diff --git a/docs/about.md b/docs/about.md index 76867e2..b675400 100644 --- a/docs/about.md +++ b/docs/about.md @@ -13,7 +13,7 @@ OpenDPD is developed by the [Lab of Efficient Machine Intelligence](https://www. | Ruishen Yang | Contributor | | Qian Wu | Contributor | -See the [GitHub contributors](https://github.com/lab-emi/OpenDPD/graphs/contributors) for the evolving contribution history. Studio's About page also links to the team and repository activity. +See the [GitHub contributors](https://github.com/lab-emi/OpenDPD/graphs/contributors) for the evolving contribution history. Studio's About page presents the team, lab, repository link and installed version; it no longer embeds GitHub activity. ## Research and citation diff --git a/docs/architecture/adr/0002-native-window-shell.md b/docs/architecture/adr/0002-native-window-shell.md index 5b29e23..2238164 100644 --- a/docs/architecture/adr/0002-native-window-shell.md +++ b/docs/architecture/adr/0002-native-window-shell.md @@ -40,3 +40,7 @@ security, no silent scientific change. ## Verification `tests/unit/test_window.py`, `tests/unit/test_launcher.py` (window mode, fallback, refusal, reuse, CLI flags, doctor); the macOS evidence in `docs/releases/support-matrix.md`. + +## 2.2.5 installation update + +The default package now includes pywebview and the Studio server. Platform markers select Qt on Linux and native bindings on macOS/Windows. The old extras are compatibility aliases. OS libraries and GPU drivers remain host requirements; the recommended uv workflow selects a compatible PyTorch wheel with `--torch-backend=auto`. SSH sessions receive loopback/port-forward guidance. See [Installation](../../install.md). diff --git a/docs/architecture/threat-model.md b/docs/architecture/threat-model.md index 36bcec2..1d1e65a 100644 --- a/docs/architecture/threat-model.md +++ b/docs/architecture/threat-model.md @@ -19,7 +19,7 @@ ASGI application (`tests/integration/test_hardening.py` unless stated). |---|---|---|---| | Malicious web page drives the local API | `fetch("http://127.0.0.1:port/api/v1/runs", {method:"POST"})` | session cookie required (SameSite=Strict, HttpOnly); writes need `X-OpenDPD-CSRF` that only same-origin scripts can read; no CORS headers, preflight refused; Origin/Referer must match Host | `tests/integration/test_studio_api.py::test_cross_origin_and_csrf` | | DNS rebinding | attacker host resolves to 127.0.0.1 | Host header must be a loopback name | `test_studio_api.py::test_host_header_enforced` | -| Script injection into the page (XSS) | run names, notes, dataset ids, log lines, package contents rendered in the UI | React escapes every string; no `dangerouslySetInnerHTML`/`innerHTML`/`eval` in the frontend sources; server-side diagnostic pages HTML-escape their text; a Content-Security-Policy on every response allows scripts only from this origin (no inline scripts, no `eval`, no CDN), plus `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `frame-ancestors 'none'`, `Referrer-Policy: same-origin` | `test_every_response_carries_the_security_headers`, `test_diagnostic_pages_escape_their_text`, `test_frontend_sources_have_no_html_sinks`, `test_user_text_is_stored_and_returned_as_data`; the real-server journey (`frontend/e2e/live.spec.ts`) fails on any console error, which is where CSP violations surface | +| Script injection into the page (XSS) | run names, notes, dataset ids, log lines, package contents rendered in the UI | React escapes user strings; the sole reviewed HTML sink is the bundled KaTeX renderer, which escapes source text and trusts only catalog coefficient classes; arbitrary URLs, resources and HTML/style commands are disabled; server-side diagnostic pages HTML-escape their text; a Content-Security-Policy on every response allows scripts only from this origin (no inline scripts, no `eval`, no CDN), plus `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `frame-ancestors 'none'`, `Referrer-Policy: same-origin` | `test_every_response_carries_the_security_headers`, `test_diagnostic_pages_escape_their_text`, `test_frontend_html_sinks_are_confined_to_the_audited_math_renderer`, `MathFormula.test.tsx`, `test_user_text_is_stored_and_returned_as_data`; the real-server journey (`frontend/e2e/live.spec.ts`) fails on any console error, which is where CSP violations surface | | Guessing the session | brute force | 256-bit random ids; bootstrap token printed only to the local console/URL; sessions die with the process | `tests/unit/test_security.py` | | Path traversal / symlink escape on download | `GET /artifacts/{run}/{id}` | downloads by registered artifact id; the resolved path (symlinks followed) must stay inside the run directory | `test_studio_api.py::test_artifact_download_by_id_only`, `test_artifact_symlink_outside_the_run_is_refused` | | Reading files outside the import roots | `../` or a symlink inside a root pointing elsewhere | every source path is resolved and must stay inside its root; a symlink that leaves the root is neither listed nor readable | `tests/unit/test_datasets_service.py::test_import_roots_refuse_traversal_and_unknown_roots`, `test_import_root_symlink_escapes_are_invisible_and_unreadable` | diff --git a/docs/contracts/openapi.json b/docs/contracts/openapi.json index 42df263..1a19787 100644 --- a/docs/contracts/openapi.json +++ b/docs/contracts/openapi.json @@ -717,7 +717,8 @@ "kind": { "enum": [ "surrogate_without_dpd", - "measured_without_dpd" + "measured_without_dpd", + "ilc_ideal" ], "title": "Kind", "type": "string" @@ -3502,6 +3503,18 @@ } ] }, + "ilc": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "title": "Ilc" + }, "is_mock": { "default": false, "title": "Is Mock", @@ -5246,6 +5259,17 @@ "title": "Seed", "type": "integer" }, + "shared_channel_settings": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Shared Channel Settings" + }, "snr_db": { "anyOf": [ { @@ -7797,7 +7821,8 @@ "gradient", "gradient_dla", "least_squares", - "ila_least_squares" + "ila_least_squares", + "ilc_ila" ], "type": "string" }, @@ -8284,6 +8309,11 @@ "title": "Symbol", "type": "string" }, + "symbol_latex": { + "default": "", + "title": "Symbol Latex", + "type": "string" + }, "unit": { "default": "", "title": "Unit", @@ -10382,7 +10412,7 @@ "title": "Csrf Token" }, "version": { - "default": "2.2.4", + "default": "2.2.5", "title": "Version", "type": "string" } @@ -12214,6 +12244,13 @@ "title": "Equations", "type": "array" }, + "equations_latex": { + "items": { + "type": "string" + }, + "title": "Equations Latex", + "type": "array" + }, "limitations": { "$ref": "#/components/schemas/PALocalizedText" }, @@ -12565,7 +12602,7 @@ }, "info": { "title": "OpenDPD Studio API", - "version": "2.2.4" + "version": "2.2.5" }, "openapi": "3.1.0", "paths": { @@ -16464,6 +16501,46 @@ ] } }, + "/api/v1/signal-generator/signals/{signal_id}/archive": { + "post": { + "operationId": "archive_api_v1_signal_generator_signals__signal_id__archive_post", + "parameters": [ + { + "in": "path", + "name": "signal_id", + "required": true, + "schema": { + "title": "Signal Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": {} + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Archive", + "tags": [ + "signal generator" + ] + } + }, "/api/v1/signal-generator/signals/{signal_id}/dataset": { "post": { "deprecated": true, @@ -16637,6 +16714,48 @@ ] } }, + "/api/v1/signal-generator/signals/{signal_id}/restore": { + "post": { + "operationId": "restore_api_v1_signal_generator_signals__signal_id__restore_post", + "parameters": [ + { + "in": "path", + "name": "signal_id", + "required": true, + "schema": { + "title": "Signal Id", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/PAInputDataset" + } + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Restore", + "tags": [ + "signal generator" + ] + } + }, "/api/v1/signal-generator/validate": { "post": { "operationId": "validate_api_v1_signal_generator_validate_post", @@ -17013,6 +17132,18 @@ "/api/v1/system/about": { "get": { "operationId": "system_about_api_v1_system_about_get", + "parameters": [ + { + "in": "query", + "name": "activity", + "required": false, + "schema": { + "default": true, + "title": "Activity", + "type": "boolean" + } + } + ], "responses": { "200": { "content": { @@ -17025,6 +17156,16 @@ } }, "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" } }, "summary": "System About", diff --git a/docs/documentation.md b/docs/documentation.md index 782dc16..3542b27 100644 --- a/docs/documentation.md +++ b/docs/documentation.md @@ -7,7 +7,7 @@ The GitHub README is the short entry point; the documentation site is the place | Content | Source of truth | Reuse | | --- | --- | --- | | Product introduction, current highlights, homepage screenshot | Root `README.md` sections `intro`, `studio-features`, `hero` | Included by the site home page. | -| Minimal source-install command block | Root `README.md`, `source-install` section | Included by `docs/install.md`; edit once. | +| Minimal source-install command block | Root `README.md`, `source-install` section | Shown in README; the installation guide adds platform and troubleshooting details. | | Setup options and troubleshooting | `docs/install.md` | Linked from README and tutorials. | | Task walkthroughs and explanations | `docs/tutorials/`, `training.md`, `visualization.md`, `advanced.md`, `faq.md` | Read directly on GitHub or through the site. | | Dataset format and Python examples | `datasets/README.md`, `examples/README.md` | Included by `docs/datasets.md` and `docs/examples.md`. | @@ -19,7 +19,7 @@ The GitHub README is the short entry point; the documentation site is the place | Shared images | `pics/` | Published by `docs/hooks/assets.py`; keep one copy. | | Studio and EMI logos | `frontend/src/assets/`; root `README.md`, `brand` section | The site includes the same brand section and publishes the original SVGs through the asset hook. | -`docs/index.md`, `docs/datasets.md`, `docs/examples.md` and `docs/benchmark/index.md` are site entry pages. They compose canonical content with `pymdownx.snippets`; they are not a second place to edit that content. The installation page reuses only the short command block. When reading its Markdown on GitHub, that block is available in the root README. +`docs/index.md`, `docs/datasets.md`, `docs/examples.md` and `docs/benchmark/index.md` are site entry pages. They compose canonical content with `pymdownx.snippets`; they are not a second place to edit that content. Keep the short uv commands in README and the installation guide consistent. ## Writing and linking @@ -60,6 +60,6 @@ Reuse the transparent SVGs from `frontend/src/assets/`, including their light an The README's `brand` snippet uses GitHub-compatible `