perf+feat: pipeline optimization, inspectr fast loading, plot UI improvements by astafan8 · Pull Request #441 · toolsforexperiments/plottr

astafan8 · 2026-04-15T11:51:33Z

Summary

Performance optimizations for the DataDict pipeline, faster inspectr database loading, plot UI improvements, enable faster pyqtgraph backend and bring it to match features of the matplotlib backend including LaTeX label support.

Performance: DataDict pipeline (core)

copy(deep=True/False) — Use ndarray.copy() instead of deepcopy (14.8x faster). Added deep argument following xarray convention.
is_invalid() dtype fast-path — Skip a == None for numeric dtypes, use np.isnan() directly. 44x faster on 963k complex128 arrays, cascading to 2.8x faster datadict_to_meshgrid.
structure() / _build_structure() — Internal helper skips validation for known-good callers.
validate() monotonicity — Direct min/max instead of np.unique + np.sign.
label() — Removed redundant validate() call; uses .get() with defaults for robustness.
mask_invalid() / extract() — Eliminated redundant copies.
largest_numtype() — Direct dtype check (15,000x faster).
_find_switches() — Vectorized percentile + filter (2.6x faster grid guessing).
datasets_are_equal() — Short-circuit on shape mismatch.
Node process() — Deferred structure() call; only recomputed when structure actually changes.
XYSelector — Removed cascading copy. datadict_to_meshgrid uses copy=False.
Plot splitting — dataclasses.replace instead of deepcopy for PlotItem.

Performance: Inspectr database loading

Fast SQL overview — New plottr.data.qcodes_db_overview module with single JOIN query (~14ms vs minutes for large DBs). Will be upstreamed to QCoDeS shortly. As part of that, records counter has been changed to leverage counting of rows from results table or take shape info from run_description depending on situation (this change might contain regressions but hopefully is not too critical).
Lazy snapshot tree — Built on-expand, not on-load (951ms to 0.3ms per click).
Collapsed by default — Info pane starts collapsed; smooth pixel scrolling.
Loading progress overlay — Live "Loading database... (N/M datasets)" feedback.

Performance: Plot backends

metadataShape default — Both backends default to GridOption.metadataShape when QCodes shape metadata exists, skipping expensive guess_grid_from_sweep_direction.
Matplotlib replot optimization — _inSetData flag suppresses redundant _plotData calls from toolbar signals during setData().

Fixes: Pyqtgraph backend

Axis orientation — Transpose z data in setImage() to match matplotlib convention. Fixes 90 degree rotated image plots.
Grid resize — Reset row/column stretch factors when subplot count changes. Fixes plots staying small after reducing from many to few subplots.
Complex mode switching — Reset imagData flag before checking data. All complex representations (Real, Re/Im, Split Re/Im, Mag/Phase) available for 1D and 2D data.
Minimum widget size — Set 40x40 minimum on PlotBase to reduce QFont::setPointSize warnings.
Grid layout for pyqtgraph — QGridLayout with equal stretch, matching matplotlib's near-square subplot arrangement.
plottr.utils.latex — LaTeX-to-HTML conversion for Qt rich text (Greek letters via unicodeit, subscripts, superscripts, fractions, square roots). Only triggers on actual LaTeX syntax. Plain strings pass through unchanged.

Fixes: Data handling

Missing data files — ds_to_datadicts skips dependents whose parameter tree is missing from the cache (e.g., metadata-only DBs where the .nc file is absent) instead of crashing with KeyError.
Warning banner for missing data — When a dataset has no data (empty or missing .nc file), a prominent yellow banner is shown at the top of the autoplot window with the exact missing file path(s). The banner auto-removes once data arrives (e.g., when monitoring a dataset being filled).
Generic showWarningBanner / removeWarningBanner — Public methods on AutoPlotMainWindow for displaying any dataset warning or error. Text is selectable and copy-pastable.

UI: Data selector buttons

Select all — Selects all dependent variables. Single signal emission (batch).
Select first only — Selects only the first dependent (default view).
Select all 1D — Selects only 1D dependents. Only visible when 1D variables exist.
Select all 2D — Selects only 2D dependents. Only visible when 2D variables exist.

UI: Plot layout and controls

Plot backend selector — Combo box in inspectr toolbar to switch between matplotlib and pyqtgraph. Selection is persisted between launches via QSettings.
Scrollable plot area — Toggle + min-height spinbox in both backends. Off by default.
Wider default window — Inspectr opens at 1400x900.
Matplotlib colormap selector — Combo box in toolbar with popular colormaps first, then all matplotlib colormaps. Changes take effect immediately.

Code quality and typing

find_scale_and_prefix import — Updated to qcodes.plotting.axis_labels with fallback chain for older qcodes and when qcodes is not installed.
Mypy clean for both PyQt5 and PyQt6 — Per-module warn_unused_ignores override for modules with cross-backend type: ignore comments.
Timestamp parsing — datetime.fromisoformat instead of string slicing.

Tests (new)

test_datadict_copy_semantics.py — 64 copy/isolation tests
test_pipeline_coverage.py — 63 pipeline tests with hypothesis
test_round2_optimizations.py — 32 optimization tests
test_gridder_comprehensive.py — 62 gridder tests
test_latex.py — 38 LaTeX conversion tests
test_plotting.py — 17 new tests: axis orientation, complex splitting, mpl first-plot, pyqtgraph complex modes
test_data_selector.py — 8 new tests: selection buttons
test_qcodes_data.py — 12 new tests: records counter, dataset refresh (including incomplete datasets), backend persistence, loadFullDB UI reset, missing data handling

Documentation

PERFORMANCE_PLAN.md — Profiling analysis, benchmarks, and future optimization suggestions.

Major performance improvements to plottr's data pipeline: - Rewrite copy() with targeted per-key semantics (14.8x faster for meshgrid) - Add copy(deep=False) API for shallow copies (xarray convention) - Optimize MeshgridDataDict.validate() monotonicity check (1.5x faster) - Add _build_structure() to skip redundant validation in internal callers - Fast-path mask_invalid() to skip clean data (65,000x memory reduction) - Fix cascading copies in XYSelector (was copying twice via inheritance) - Pass copy=False in DataGridder to avoid redundant array duplication - Optimize datasets_are_equal() with shape short-circuit - Fix bug: copy() now properly deep-copies global mutable metadata Adds 127 new tests covering copy semantics, pipeline integrity, various data shapes/dtypes, and edge cases (hypothesis property-based testing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Comprehensive analysis of remaining performance improvements across: - HDF5 loading (reads full dataset for shape metadata - critical fix) - Node.process() redundant structure() call on every update - Complex plot rendering deepcopy overhead - Signal emission overhead (7 signals per node per update) - largest_numtype() iterating every array element as Python objects - Various numpy anti-patterns (np.append in loops, unnecessary copies) - Architectural improvements (change detection, memoization) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Round 2 performance improvements: - largest_numtype(): use numpy dtype instead of iterating every element as a Python object (~15,000x faster for numeric arrays) - Node.process(): defer structure() call to only when structure changes (50x faster steady-state updates for large meshgrids) - is_invalid(): skip unnecessary np.zeros allocation for non-float arrays - guess_grid_from_sweep_direction(): convert once with np.asarray, not 4x - remove_invalid_entries(): replace O(n^2) np.append with list+concatenate Also fixes crash on inhomogeneous index arrays (pre-existing bug) - meshgrid_to_datadict/datadict_to_dataframe: ravel() instead of flatten() - _splitComplexData(): dataclasses.replace instead of deepcopy Adds 32 new tests (205 total passing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename loop variable 'd' to 'diffs' to avoid shadowing the outer loop variable from 'for d in self.dependents()'. Add explicit type annotations for ndarray variables to satisfy mypy's type narrowing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarked the full plottr pipeline (load -> DataSelector -> DataGridder -> XYSelector) on 23 QCodes datasets of varying shapes and sizes. Pipeline total: 1478 ms -> 1025 ms = 1.44x overall speedup. Largest gains on big datasets: stability_diagram 1.81x, large_3d_scan 1.65x. No regressions on any dataset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pipeline total: 6,550 ms -> 3,465 ms = 1.89x overall speedup on 8 large datasets (4M-point 1D, 800x800 2D, 100x100x80 3D, interrupted, multi-dep). Consistent ~2x speedup across 1D/2D shapes, ~1.7x on 3D. Loading times unchanged (QCodes SQLite I/O dominated). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New benchmark measures both cold start (new flowchart) and steady state (persistent flowchart, simulating live monitoring refresh). Uses 5 repeats with warmup, reports median. Results on 31 datasets (23 small + 8 large): - Large datasets: cold 1.88x, steady 1.77x faster - Small datasets: cold 1.43x, steady 1.69x faster - Steady-state on small data shows up to 2.11x speedup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarks real user actions (switch dep, swap axes, toggle subtract avg, slide dimension, toggle grid) on large datasets with per-node time breakdown. Key findings: - DataSelector: 10-17x faster (largest_numtype O(1), copy optimized) - SubtractAverage: 6-29x faster (copy 15x faster, mask_invalid skips clean) - ScaleUnits: 7-15x faster (copy 15x faster) - XYSelector: 1.5-2.3x (cascading copy removed) - DataGridder: 1.1x (dominated by actual gridding computation) Action-level: toggle_subtract_avg 9-10x, swap_xy 3.3x, switch_dep 2.3x, data_refresh 2.2x, slide_dimension 1.5-1.6x. DataGridder is now the dominant cost (58% of pipeline) and is the next frontier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The shape-guessing algorithm (_find_switches -> find_direction_period -> guess_grid_from_sweep_direction) was the #1 bottleneck after rounds 1-2. Optimizations: - Compute is_invalid() once instead of 3 times per call - Single np.percentile([lo, hi]) call instead of two separate sorts - Direct numpy subtraction instead of MaskedArray creation - Vectorized boolean mask instead of Python list comprehension - np.nanmean for NaN-safe sweep direction detection - Cached np.std in guess_grid_from_sweep_direction Results (800x800 = 640K pts): - _find_switches: 80ms -> 31ms (2.6x) - datadict_to_meshgrid: 175ms -> 71ms (2.5x) - Cumulative pipeline speedup vs master: 2.8-3.5x Adds 62 comprehensive gridder tests covering all GridOption paths, edge cases, various shapes, noisy axes, incomplete data. All 267 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarked on production quantum device datasets: - QDstability (223 MB, 16 deps): cold 3.56x, steady 2.93x faster - TopogapStage2 (152 MB, 21 deps, 4D): cold 2.47x, steady 2.7x faster - QDtuning (14 MB, 16 deps): steady 2.73x faster - DataSelector node: 12-13x faster on these multi-dependent datasets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Lazy snapshot loading in RunInfo: - Snapshot tree widget items are built only when the user expands the 'QCoDeS Snapshot' section, not on every click - Saves ~951ms per click on datasets with 5.9 MB snapshots (3,554x faster) - Info pane shows collapsed by default instead of expandAll() Incremental DB refresh: - refreshDB() now loads only new runs since last refresh using the start parameter of get_runs_from_db() - Merges incremental results into existing dataframe - First load still loads everything All 267 tests pass. No mypy errors introduced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add get_runs_from_db_fast() which uses load_by_id() directly per run, bypassing the O(N^2) experiments() + data_sets() enumeration. For 1496 runs: old approach takes 15+ minutes, new takes ~5 seconds. Incremental refresh loads only new runs since last known run_id. LoadDBProcess now uses get_runs_from_db_fast with start_run_id parameter for both initial load and incremental refresh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

RunList now shows contextual overlay messages: - 'Loading database... (N/M datasets)' with live progress during load - 'Select a date on the left to browse datasets.' when idle - 'No datasets found in this database.' for empty DBs - 'No datasets match the current filter.' when star/cross filters hide all Progress is reported from the worker thread via progressUpdated signal, updated every 10 datasets for smooth display without overhead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Check actual RunList widget state (topLevelItemCount) instead of _selected_dates to decide whether to show the hint text. This handles same-file reload, empty date selection, and filter edge cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Data structure and Metadata sections now collapsed by default (user expands what they need) - Set ScrollPerPixel on RunInfo tree widget so tall rows (e.g., long exception tracebacks in metadata) can be scrolled smoothly instead of jumping to the next row Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a combo box in the toolbar to switch between matplotlib and pyqtgraph backends. Default is matplotlib. The selection applies to all newly opened plot windows. Existing windows keep their backend. The combo box respects the --plotWidgetClass passed via constructor (e.g., from script_pyqtgraph entrypoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New module plottr/data/qcodes_db_overview.py: - get_db_overview(): single SQL JOIN query for all run metadata - Skips snapshot and run_description blobs entirely - Reads inspectr_tag directly as a column from runs table - 6x faster than load_by_id, ~1000x faster than experiments() enumeration - Intended for eventual contribution to QCoDeS Inspectr LoadDBProcess now uses SQL path by default with automatic fallback to qcodes API (get_runs_from_db_fast) if SQL fails. Also: default window size widened from 640x640 to 960x640. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace the single-column QSplitter with a QGridLayout that arranges subplots on a near-square grid, using the same formula as matplotlib: nrows = int(n ** 0.5 + 0.5) ncols = ceil(n / nrows) This makes pyqtgraph behave like matplotlib when plotting many dependents: plots are arranged in columns (e.g., 4 plots = 2x2, 6 = 2x3, 16 = 4x4) instead of stacking vertically. A scroll area wraps the grid so very many plots remain accessible. Each plot has a minimum height of 250px to stay readable. 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add 'Scrollable' checkbox in both pyqtgraph and matplotlib toolbars: - Enabled by default - When many subplots exist, the plot area expands beyond the window and becomes scrollable, keeping each subplot readable - Can be unchecked to fit everything into the visible window PyQtGraph: min plot height reduced from 250px to 75px. Matplotlib: canvas wraps in QScrollArea, min height set per grid row. 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Both backends: - Scrollable is now OFF by default - Added a 'px' spinbox next to the Scrollable checkbox showing the minimum height per subplot row (default 75px pyqtgraph, 100px mpl) - Spinbox is only enabled when Scrollable is checked - Minimum value is 40px - Changing the spinbox value triggers replot 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The inspectr backend selector passes plotWidgetClass to autoplotQcodesDataset, but the function signature was missing this parameter on the branch. Also passes it through to QCAutoPlotMainWindow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Extract 'Select a date...' string into _SELECT_DATE_HINT constant - Replace string-magic backend detection with explicit _PLOT_BACKENDS mapping (display name -> class) - _backend_name_for_class() for reverse lookup - Unknown plotWidgetClass added to combo with its class name as label - _onBackendChanged uses the mapping instead of hardcoded imports 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- _split_timestamp(): proper datetime parsing instead of string slicing for splitting qcodes timestamp strings into date/time components. Applied to both get_ds_info() and _ds_to_info_dict(). - get_runs_from_db_fast(): removed unnecessary initialise_or_create_database_at call, use same read_only pattern as get_runs_from_db. - qcodes_db_overview: use conn_from_dbpath_or_conn from qcodes instead of raw sqlite3.connect. Remove unused get_last_run_id function. - mpl/widgets: remove dead _scrollable attribute and fix setScrollable which had identical code in both branches. 280 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New module plottr/utils/latex.py using unicodeit (now a required dependency): - Greek letters: \alpha -> α, \Omega -> Ω - Math symbols: \hbar -> ℏ, \partial -> ∂, \int -> ∫ - Subscripts: V_{gate} -> Vgate (HTML for text, Unicode for digits) - Superscripts: x^{2} -> x² (Unicode), e^{iπ} -> eiπ - Fractions: \frac{dI}{dV} -> dI/dV - Square root: \sqrt{x} -> √x - Dollar delimiters stripped Applied to pyqtgraph axis labels in FigureMaker.formatSubPlot(). Falls through gracefully on plain text (no LaTeX = no change). 35 new tests including hypothesis property-based testing. 315 tests pass, 0 mypy errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Convert subscripts and superscripts to HTML tags BEFORE running unicodeit, so they become 11 and 2 instead of Unicode ₁₁ and ². HTML tags render more consistently in Qt rich text. unicodeit still converts Greek letters and symbols inside the tags. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Plain strings with underscores (e.g. gate_voltage, channel_1_amplitude) now pass through unchanged. Conversion only triggers when the string contains backslash commands (\alpha), dollar delimiters ($...$), or braced sub/superscripts (_{...}, ^{...}). Also drops single-char bare sub/sup patterns (_x, ^x) which were too aggressive on ordinary identifiers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…optimization # Conflicts: # plottr/apps/inspectr.py # plottr/data/datadict.py # plottr/node/autonode.py # plottr/node/scaleunits.py # plottr/plot/mpl/autoplot.py # test_requirements.txt

Master cleaned up inspectr type:ignore comments, so the per-module override is no longer needed for that module. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

CI installs PyQt5, not PyQt6. Use plottr's Qt abstraction layer (plottr.QtCore, plottr.QtWidgets) for cross-binding compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…optimization # Conflicts: # test_requirements.txt

refresh: refreshDB() now always does a full DB re-read instead of incremental loading. The old incremental path (start_run_id > latest) would never update existing rows in the dataframe, so the records counter for incomplete datasets being filled with new data would stay stale until the user closed and reopened inspectr. Full re-read is fast (~10ms via SQL JOIN) so the optimization wasn't worth the correctness cost. Existing merge logic via dbdf.update() correctly applies the fresh values to existing rows. backend persistence: User's plot backend choice (matplotlib / pyqtgraph) is now saved via QSettings and restored on next launch. QSettings was chosen for the cleanest cross-platform persistence (registry on Windows, plist on macOS, ini on Linux) and zero external dependencies. Add 4 new tests: refresh updates incomplete records, save/load backend choice, invalid value handling, launch uses saved backend, combo change persists choice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ds_to_datadicts: Skip dependents whose parameter tree is missing from the cache instead of raising KeyError. This commonly happens when the dataset's .nc data file is missing (metadata-only DB downloaded without the companion data files). QCAutoPlotMainWindow: Show a clear status bar message ('No data available for run N ...') when the dataset has no results, instead of leaving the user with an unexplained empty window. This is not a regression from our changes — it's a pre-existing issue that becomes more visible now that users can browse large metadata-only DBs quickly via the fast SQL overview. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

autoplot: When a dataset has no data (number_of_results == 0), check its export_info for missing .nc files and show a status bar message with the exact missing file path(s). This replaces the silent empty window for metadata-only DBs where the .nc companion files are absent. ds_to_datadicts: Skip dependents whose parameter tree is missing from the cache instead of raising KeyError — returns an empty dict for datasets with no loadable data. inspectr: Clear the run list before reloading a DB so that stale items from a previous load don't show through the overlay text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

autoplot: Show the missing-data-file message as a large centered label in the plot area (in addition to the status bar) so it is impossible to miss. inspectr: When the user opens the same DB file that is already loaded, skip the reload entirely instead of showing a transient 'Loading database...' overlay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

loadFullDB was skipping when path == self.filepath, but this also blocked the initial load during __init__ (where filepath is set from the dbPath constructor argument before loadFullDB is called). Now only skips when data is already loaded (self.dbdf is not None). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

loadFullDB now always treats the request as a fresh load — clears the run list, resets latestRunId, and re-reads the full database. Same flow whether the file is new or already open. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

loadFullDB now clears dateList, runList, AND sets dbdf=None before starting the background load. Previously only runList was cleared, so the old date selection persisted and immediately repopulated the run list from stale data — making it look like nothing changed. Add test that verifies loadFullDB clears all three (dbdf, dateList, runList) immediately, then repopulates after the load completes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Display the missing-data-file warning as a yellow banner at the top of the autoplot window (above the plot area) for maximum visibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When a dataset starts empty (no results yet) and the autoplot window shows the 'No data available' banner, subsequent monitor-triggered refreshes that find new data now remove the banner and restore the normal plot area. Uses _removeNoDataBanner in QCAutoPlotMainWindow. refreshData override. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…inWindow Move the banner logic from QCAutoPlotMainWindow private methods to the parent AutoPlotMainWindow as public showWarningBanner() and removeWarningBanner(). Any subclass or future use can now show warnings/errors about datasets with a single method call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

autoplot: - _no_data_message takes only ds (uses ds.run_id, ds.running) - State-specific messages: running → 'started but no data yet', missing files → shows paths, default → generic message - export_paths is dict[str, str], removed isinstance list check - Removed empty setDefaults override from QCAutoPlotMainWindow - Removed dataset-specific comments from warning banner code inspectr: - Removed start_run_id from LoadDBProcess (always 1, was a relic) - Removed script_pyqtgraph entry point - Simplified refreshDB docstring - Merged redundant elif/else in DBLoaded qcodes_db_overview: - Completed datasets: prefer shapes from run_description - Active datasets: prefer results table row count - Simplified comment on missing tables qcodes_dataset: - Use tqdm (when available) for smart progress frequency - Falls back to every-10th-item without tqdm data_display: - blockSignals(False) in finally clause of setBatchSelectedData Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

astafan8 · 2026-05-21T14:56:43Z

@marcosfrenkel @wpfff this large improvement PR is on it's final days before merge (doing last user-acceptance testing), so i plan for a release next week. Let know if the changes here break anything for monitr, i'll be happy to fix those. After merging this, we'll get to the next PRs (that have been open for some time) :)

marcosfrenkel · 2026-05-27T19:11:48Z

+            if data.meta_val('qcodes_shape') is not None:
+                self.fc.nodes()['Grid'].grid = GridOption.metadataShape, {}
+            else:
+                self.fc.nodes()['Grid'].grid = GridOption.guessShape, {}


Not sure why but this is making it such that in a "normal" ddh5 file, the default grid option is No grid making it annoying to have to click on the grid every time.

marcosfrenkel · 2026-05-27T20:09:24Z

Sorry for the delay, I just got time to look at this. Other than the comment I left in plottr/apps/autoplot.py it seems to be al working correctly.

jenshnielsen · 2026-05-28T07:39:23Z

 LOGGER = logging.getLogger('plottr.apps.autoplot')


+def _no_data_message(ds: Any) -> str:


Suggested change

def _no_data_message(ds: Any) -> str:

def _no_data_message(ds: DataSetProtcol) -> str:

jenshnielsen · 2026-05-28T07:40:05Z

+        return f"{header} — dataset has started but does not contain data yet."
+
+    # Check for missing exported data files
+    missing_files: List[str] = []


We have dropped python < 3.11 so no need for these legacy types (but that can also be fixed in another pr)

jenshnielsen · 2026-05-28T07:42:48Z

+        return '', ''
+
+
+def get_db_overview(db_path: str,


This feels like something that should be contributed upstream to qcodes

jenshnielsen · 2026-05-28T07:43:43Z

+    from qcodes.plotting.axis_labels import find_scale_and_prefix
+except ImportError:
+    try:
+        # fallback for qcodes < 0.46 where the function lived under utils


Bump min version of qcodes and drop this path imho

Actually I think this whole block is at least partially reverting changes from a pr that I merged

jenshnielsen · 2026-05-28T07:47:23Z

+    "plottr.node.autonode",
+    "plottr.node.scaleunits",
+]
+warn_unused_ignores = false


Consider if this can be fixed by not pinning the stubs to some specific very old version

Mikhail Astafev and others added 17 commits April 15, 2026 13:51

astafan8 force-pushed the perf/datadict-copy-optimization branch from a9843a4 to 4263563 Compare April 20, 2026 12:21

Mikhail Astafev and others added 5 commits April 20, 2026 14:27

fix: reduce matplotlib scrollable min height to 100px per row

c663ec7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs: reorganize PERFORMANCE_PLAN.md into implemented vs future

f948fb6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

astafan8 changed the title ~~perf: optimize DataDict copy, validate, and pipeline data flow~~ perf+feat: pipeline optimization, inspectr fast loading, plot UI improvements Apr 20, 2026

Mikhail Astafev and others added 6 commits April 20, 2026 16:01

Mikhail Astafev and others added 16 commits May 1, 2026 17:27

Merge remote-tracking branch 'origin/master' into perf/datadict-copy-…

e6a31e1

…optimization # Conflicts: # plottr/apps/inspectr.py # plottr/data/datadict.py # plottr/node/autonode.py # plottr/node/scaleunits.py # plottr/plot/mpl/autoplot.py # test_requirements.txt

chore: remove inspectr from mypy warn_unused_ignores override

8ec0349

Master cleaned up inspectr type:ignore comments, so the per-module override is no longer needed for that module. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: use plottr Qt imports instead of PyQt6 in tests

15f7b3a

CI installs PyQt5, not PyQt6. Use plottr's Qt abstraction layer (plottr.QtCore, plottr.QtWidgets) for cross-binding compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: ParamSpecBase import compat for older qcodes in CI

99122f9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/master' into perf/datadict-copy-…

805d43a

…optimization # Conflicts: # test_requirements.txt

fix: move no-data banner to top of autoplot window

d4149ce

Display the missing-data-file warning as a yellow banner at the top of the autoplot window (above the plot area) for maximum visibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: make no-data banner text selectable and copy-pastable

42e6888

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

astafan8 commented May 21, 2026

View reviewed changes

astafan8 self-assigned this May 21, 2026

astafan8 requested review from jenshnielsen, marcosfrenkel and wpfff May 21, 2026 14:57

marcosfrenkel requested changes May 27, 2026

View reviewed changes

jenshnielsen reviewed May 28, 2026

View reviewed changes

		LOGGER = logging.getLogger('plottr.apps.autoplot')


		def _no_data_message(ds: Any) -> str:

	def _no_data_message(ds: Any) -> str:
	def _no_data_message(ds: DataSetProtcol) -> str:

Conversation

astafan8 commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Performance: DataDict pipeline (core)

Performance: Inspectr database loading

Performance: Plot backends

Fixes: Pyqtgraph backend

Fixes: Data handling

UI: Data selector buttons

UI: Plot layout and controls

Code quality and typing

Tests (new)

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

astafan8 commented May 21, 2026

Uh oh!

marcosfrenkel May 27, 2026

Choose a reason for hiding this comment

Uh oh!

marcosfrenkel commented May 27, 2026

Uh oh!

jenshnielsen May 28, 2026

Choose a reason for hiding this comment

Uh oh!

jenshnielsen May 28, 2026

Choose a reason for hiding this comment

Uh oh!

jenshnielsen May 28, 2026

Choose a reason for hiding this comment

Uh oh!

jenshnielsen May 28, 2026

Choose a reason for hiding this comment

Uh oh!

jenshnielsen May 28, 2026

Choose a reason for hiding this comment

Uh oh!

jenshnielsen May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

astafan8 commented Apr 15, 2026 •

edited

Loading