Skip to content

Warn on DynamicTable read validation issues#822

Merged
ehennestad merged 3 commits into
mainfrom
codex/dynamic-table-read-warnings
May 19, 2026
Merged

Warn on DynamicTable read validation issues#822
ehennestad merged 3 commits into
mainfrom
codex/dynamic-table-read-warnings

Conversation

@ehennestad
Copy link
Copy Markdown
Collaborator

@ehennestad ehennestad commented May 6, 2026

Motivation

Following PR #818, DynamicTable validation now detects two metadata problems that can appear in existing NWB files: materialized columns missing from colnames, and duplicate entries in colnames.

Those checks are useful when creating or exporting new data, but applying them as hard errors while reading existing files breaks compatibility with files that were previously readable. This change makes parsed file reads permissive for these DynamicTable column-name issues while keeping user-side construction and export strict.

The implementation adds a generic validation context for read-time object construction, so this behavior can also support the broader read-validation policy discussed in related issue #776.

How to test the behavior?

Reading/parsing legacy DynamicTable metadata warns instead of failing.

io.createParsedType( ...
    'some/time_intervals/path', ...
    'types.core.TimeIntervals', ...
    'description', 'legacy time intervals table', ...
    'colnames', {'stop_time'}, ...
    'start_time', types.hdmf_common.VectorData( ...
        'description', 'start time column', ...
        'data', single((1:3)')), ...
    'stop_time', types.hdmf_common.VectorData( ...
        'description', 'stop time column', ...
        'data', single((2:4)')), ...
    'not_in_colnames', types.hdmf_common.VectorData( ...
        'description', 'not in colnames', ...
        'data', single((3:5)')));

Before:

Error using io.createParsedType (line 48)
Failed to create object of type "types.core.TimeIntervals" in file location "some/time_intervals/path".

Caused by:
    Error using assert
    All materialized DynamicTable columns must be listed in `colnames`.
    Missing from `colnames`: not_in_colnames

    Error in types.util.dynamictable.checkConfig (line 44)
        assert(isempty(missingColumnNames), ...
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    Error in types.core.TimeIntervals (line 80)
                types.util.dynamictable.checkConfig(obj);
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    Error in io.createParsedType (line 28)
            typeInstance = feval(typeName, varargin{:});

After:

Warning: All materialized DynamicTable columns must be listed in `colnames`.
Missing from `colnames`: start_time, not_in_colnames 
> In types.util.dynamictable.checkConfig>handleColumnNamesMismatch (line 133)
In types.util.dynamictable.checkConfig (line 47)
In types.core/TimeIntervals (line 80)
In io.createParsedType (line 32) 

Creating new invalid DynamicTable metadata remains strict.

types.hdmf_common.DynamicTable( ...
    'description', 'new dynamic table', ...
    'colnames', {'columnA', 'columnA'});

Before/After:

Error using types.util.dynamictable.validateUniqueColnames (line 29)
Column names in `colnames` must be unique. Duplicate column name: `columnA`.

Error in types.util.dynamictable.validateColnames (line 8)
    types.util.dynamictable.validateUniqueColnames(colnames);
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in types.hdmf_common.DynamicTable/validate_colnames (line 82)
        val = types.util.dynamictable.validateColnames(val);
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in types.hdmf_common.DynamicTable/set.colnames (line 66)
        obj.colnames = obj.validate_colnames(val);
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in types.hdmf_common.DynamicTable (line 52)
        obj.colnames = p.Results.colnames;
        ^^^^^^^^^^^^

Improve error/warning message on unique column validation

types.hdmf_common.DynamicTable(...
    'description', 'legacy dynamic table', ...
    'colnames', {'columnA', 'columnA'}, ...
    'columnA', types.hdmf_common.VectorData( ...
        'description', 'column a', ...
        'data', (1:3)'));

Before:

 Error using assert
Column names in `colnames` must be unique.

After:

 Error using types.util.dynamictable.validateUniqueColnames (line 29)
Column names in `colnames` must be unique. Duplicate column name: `columnA`.

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

@ehennestad ehennestad requested a review from Copilot May 6, 2026 16:02
@ehennestad ehennestad force-pushed the codex/dynamic-table-read-warnings branch from 18f5515 to 4addedc Compare May 6, 2026 16:08
@ehennestad ehennestad marked this pull request as ready for review May 6, 2026 16:17
@ehennestad ehennestad enabled auto-merge May 6, 2026 16:22
@ehennestad ehennestad requested review from Copilot and removed request for Copilot May 6, 2026 16:26
@codecov
Copy link
Copy Markdown

codecov Bot commented May 6, 2026

Codecov Report

❌ Patch coverage is 95.31250% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.42%. Comparing base (a1f3824) to head (ea11288).

Files with missing lines Patch % Lines
+types/+util/+dynamictable/normalizeColnames.m 84.61% 2 Missing ⚠️
...types/+util/+dynamictable/validateUniqueColnames.m 94.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #822      +/-   ##
==========================================
+ Coverage   95.26%   95.42%   +0.15%     
==========================================
  Files         212      215       +3     
  Lines        7624     7648      +24     
==========================================
+ Hits         7263     7298      +35     
+ Misses        361      350      -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ehennestad ehennestad requested a review from bendichter May 6, 2026 17:10
@ehennestad ehennestad force-pushed the codex/dynamic-table-read-warnings branch from 4addedc to bb65a9a Compare May 6, 2026 17:32
@@ -0,0 +1,17 @@
function validateUniqueColnames(colnames)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if you said which name was duplicated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4c81649

@ehennestad ehennestad added this pull request to the merge queue May 19, 2026
Merged via the queue into main with commit 3e8407b May 19, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants