Spectrogram & PSD plot types, EEG datasource, other-source cleanup - #76
Merged
Conversation
…ht and focused on display
… to each of the source file (issue 75)
…tasource (issue 74) - find_files keeps one file per stem, preferring parquet, so a folder holding both data.csv and data.parquet no longer loads the same signals twice - 'other' now symlinks each source file into clinical_scope_output/, extending the traceability guarantee previously limited to philips_waves - an other::<stem> section can carry a trace_options block (mode, line_width, ...) overriding the datasource default - xlsx global groups emit qualified refs for other:: signals, which could not resolve before
… 74) They performed no format-specific parsing, so they were 'other' with extra machinery. Now that each file in other/ carries its own configuration and time_shift, they no longer earn a module. Example data moved into other/, configs rekeyed to other::<stem>, and a folder still named after a removed source is reported with a warning. Also corrects signal names in the demo workbook that never matched the demo data (a pre-existing mismatch this migration made visible).
AlexisJanin
added
enhancement
AlexisJanin
changed the title
Extract shared type-check and window/overlap-override boilerplate from Signal.spectrogram_from_signal/psd_from_signal; move hovertemplate dB/Hz precision into constants.Spectral; drop an issue-number reference from a comment; close test gaps around the sentinel-only-column warning loop (only trace_mode was covered; timezone had no coverage at all), trace_options isolation across datasources, and characterize today's unvalidated pass-through of trace_mode/line_width/opacity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
example/ mixed shipped demo material with test-only fixtures. It now holds
only what a user or the release bundle touches; test fixtures moved under
tests/data/.
Moves (history preserved):
- Patient_difficult_format -> tests/data/patients/
- example_database_options.{json,xlsx} -> tests/data/option_files/
Removals:
- Patient_other: unreferenced; two files were byte-identical to demo_patient,
and its stem-collision case is already covered synthetically in test_other.py
- example_database_options_other.{json,xlsx}: superseded by the demo config
- template_database_options.json: restated the tutorial reference with no
consumer and had silently drifted three features behind
example/demo_database/database_options.json is new: a generated twin of the
xlsx, so library users who never launch the app have a runnable config to
point at. The xlsx gained sections for fluxmed_signals, fluxmed_parameters,
mindray_respi_waves and mindray_respi_numerics -- the demo shipped data for
these but never plotted them -- plus range/visible/display coverage. Every
column of the signals sheet is now exercised by a real example.
New guards in tests/unit/test_example_assets.py: the folder template matches
the registry (this is how edf/ went missing), each template folder keeps its
.gitkeep, the demo json matches its xlsx, and the demo config configures every
datasource it ships.
Docs: tutorial repointed at the demo configs and its display/visible rows now
state how they differ; CLAUDE.md corrects the database_options key name to
'signals'; README names a real config path; the new-datasource skill requires
a demo config section; the organize-patient-folder evals no longer target a
patient folder deleted a release ago.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The demo's anonymisation left two stamping conventions in the raw files (naive 08:12:33 and 06:12:33 UTC), and each source's DATA_SOURCE_DEFAULT_TIMEZONE read them differently, so one recording landed in three clusters two hours apart. Loading the demo showed sources that never overlapped. Data shifted +2h so it agrees with the sources that were already right: - mindray_scope: all seven CSVs re-stamped; filenames renamed to keep the range they advertise truthful (they only supply the signal name to the loader, which reads timestamps from the file contents) - servo_u: the 'Log start' header of Curves_00000000.sta, which is the sole reference -- timestamps are built as Log start + Time(ms), and the T(h:m:s.ms) column is dropped - other/numerics.parquet: tz-aware index 06:12:33Z -> 08:12:33Z, since an already-aware index ignores the database_options timezone override Config instead of data where the stored index is naive: other::waves moves from Europe/Paris to UTC. The demo now exercises four timezone paths at once -- explicit UTC, explicit non-European (eit stays Asia/Karachi, landing 21 minutes after the others), implicit source default, and already-aware-so-override-ignored. Ten of eleven sources now start at 10:12:33 CEST; eit remains offset so the demo still has something for time_shift to correct. patient_options_example.json and the README quickstart now carry the demo's real eit day and edf recording_start, so 'try it immediately' holds. Snapshots regenerated for mindray_scope and servo_u. The naive/aware bound equivalence test pinned two windows seven seconds into the servo_u recording; both move +2h with the data, keeping the same instant expressed two ways. Full suite green: 809 passed, ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Covers #71, #72, #73, #74, #75.
#71 — Spectrogram plot type
Third plot type after
time_seriesandloop: time × frequency, power as colour (go.Heatmap), built on a newspectral.py(pure numpy STFT, grid-uniformity validation, decimation refusal). Config in JSON and the xlsxspectrogramssheet,db_rangeUser-option fallback, annotation support, demo EEG data.68edcf5Spectrogram feature first implementation#72 — EEG
.edfdatasourceNew
eegdatasource reading EDF headers and per-channel-rate int16 records directly, no dependency added. Replaces theother-as-CSV workaround #71 used to get EEG data in.45bf512.edf datasource added74648bbFixing bad eeg data in the example, keep good signals#73 — PSD plot type
Power-vs-frequency line plot reusing
spectral.py's Welch averaging; rendered as ago.Scatter(the Loop case, not the heatmap case).fab5b1cpsd plot type added#75 — Per-file options for every
otherfile (closed)Each file inside
other/gets its ownother::<stem>box in the patient-options UI (owntime_shift, own group by file), instead of one flat box shared by the whole datasource.7601ccaExtending other data source support by using database option specific to each of the source file#74 — Remove datasources with no format-specific parsing
philips_waves,philips_numericsandsyringedid no real parsing — with #75's per-file config in place, they wereotherwith extra machinery and are now removed; their example data moved underother::<stem>. Per-filetrace_options(mode/line_width/opacity/marker_symbol) was added so a removed source doesn't lose its distinct line style, then exposed on the xlsx sentinel row so it isn't JSON-only.c7dfc58Per-file trace style, source symlinks and stem dedup for the other datasource60a4adbRemove philips_waves, philips_numerics and syringe datasourcesf7ee7edUpdate docs, skills and the pushdown benchmark for the datasource removal4a234faAdd TraceOption exposure onto xlsx fileAlso
a9d9d10Improving contribution rules about ClinicalScope processing being light and focused on displayTest plan
pytestgreen (pre-existingmodebardisable/plotly-version failures insignal_container.py:1086aside, unrelated to this PR)ruff check/ruff format --checkclean on touched files