Generalize entities

[gkiar]

Downstream/On top of #85

PR Contribution Summary

This PR generalizes the indexing layer from subject-only discovery to entity-based crawling, and replaces hardcoded patterns with schema-derived logic wherever possible.

Architecture: Subject → Entity

• _index_bids_subject_dir → _index_bids_entity_dir — indexes any entity directory (sub-*, tpl-*, etc.)
• _find_bids_subject_dirs → _find_bids_entity_dirs — discovers any entity type at a dataset root
• _is_bids_subject_dir → _is_bids_entity_dir — checks arbitrary entity type by name
• format_bids_path now uses schema-derived directory hierarchy (TPL → Cohort → Sub → Ses → Datatype)
• All schema-related discovery functions centralized in _entities.py

Template and Cohort Support

• tpl-* and cohort-* directories are indexed alongside sub-*
• _is_bids_dataset derivative checks look for subject OR template entity dirs
• Verified with real TemplateFlow datasets (1590 files, 30 templates)

Schema-Driven Discovery

• get_entity_child_dirs(dataset_type, parent_rule) — reads valid entity subdirectories from rules.directories
• get_file_entity_prefixes() — root-level entity name prefixes derived from schema
• get_all_root_entity_types() — deduplicated root entity types across all dataset types
• get_all_dataset_types() — enumerates schema-defined dataset types
• _BIDS_JSON_SIDECAR_EXCEPTION_SUFFIXES — derived from rules.files (currently coordsystem, description)
• _BIDS_DATATYPE_PATTERN — built from entity names at schema init
• _ensure_dict() helper — centralizes bidsschematools Namespace→dict conversion

Derivative Detection

• _is_bids_dataset() and _get_dataset_type() detect derivative datasets without dataset_description.json by checking inside derivatives/ for valid entity subdirectories
• Correctly rejects combined sub-*_ses-* directories (spec-invalid)
• Fallback iterates all dataset types when the detected type yields no entity dirs

Generic Filtering and .bidsignore

• include_subjects → generic filters dict mapping any entity name to glob patterns
• --filter / -f CLI argument replaces --subjects (deprecated, backward-compatible)
• .bidsignore support via _is_bidsignored with cached upward search
• Filters forwarded through batch_index_dataset to workers

Dataset Metadata Columns

• dataset_name, dataset_type, bids_version added to Arrow schema, populated from dataset_description.json
• clear_schema_caches() exposed as public API for schema reload safety

Code Cleanup

• Removed dead code: get_all_entity_prefixes, get_required_entity_types
• Deduplicated _get_subdir_names() for oneOf expansion
• _read_dataset_description with @lru_cache to deduplicate reads
• Simplified _resolve_entity_dirs — extracts entity discovery into _discover_entity_dirs
• Updated stale comments and removed redundant wrapper functions

Testing

• test_derivative_detection — 5 scenarios including no-description derivatives and invalid combined entity dirs
• test_index_dataset_filters — single, multi-value, glob, and cross-entity AND filters
• test_batch_index_dataset_filters — filter forwarding through parallel workers
• test_index_dataset_bidsignore — .bidsignore exclusion
• Template integration tests gated by @templateflow_available
• Renamed test_is_bids_subject_dir → test_is_bids_entity_dir
• test_find_bids_datasets is now skipped (@pytest.mark.skip); the rglob("dataset_description.json") baseline no longer matches the schema-correct derivative detection

Impact

We should now be less fragile in schema updates, and can correctly index derivative datasets using entity types other than subject and session (namely template and cohort), meaning this can be used across a wide range of the field's projects.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	7	0	100%
__main__.py	64	5	92%	101, 127, 155, 159, 163
_entities.py	112	1	99%	129
_indexing.py	228	7	96%	154, 163–164, 179, 357, 407, 445
_logging.py	31	4	87%	30, 37, 39–40
_metadata.py	48	4	91%	39–40, 66, 71
_pathlib.py	17	3	82%	12–13, 15
_version.py	11	0	100%
pybids
__init__.py	4	0	100%
_bidsfile.py	38	13	65%	71–73, 77–79, 83–85, 89–91, 95
_layout.py	156	45	71%	63, 72, 81, 104, 114–115, 118, 140–141, 156–157, 173–174, 177–181, 186, 188–189, 192–193, 228, 233, 241, 322–324, 389–394, 396, 399–404, 406, 462, 482
_utils.py	13	5	61%	47–50, 52
TOTAL	729	87	88%

Tests	Skipped	Failures	Errors	Time
100	1 💤	0 ❌	0 🔥	23.612s ⏱️

[gkiar]

Thanks for the feedback, @effigies ! I think we're in pretty good shape now. I know @kaitj is clearing a few things off his plate before he evaluates the upstream PR to serialize the indexer which this depends on, then we can get his review here too. Getting there.

In the meantime, would it make sense to rebase your PR from this one, to get ahead of that?

[effigies]

Sure, if this will go in first.

[effigies]

This strikes me as overly complicated. I think you just want to verify that desc['BIDSVersion'] exists, and then it's BIDS. IMO, if someone wants to index something without a visible dataset_description.json, you can provide a force option. But I also don't have a lot of context for why you were writing these heuristics in the first place.

[gkiar]

I believe the context is wanting to index derivatives datasets which don't need to have a dataset_description.json file, and having those still show up as valid BIDS. Whether the dataset is raw or derived changes the entity options and ordering, so this is intended to help identify what entities can appear at the top level.

[kaitj]

Did a dive into the majority of the changed code, but probably want to do a second pass at it at some point to spend more time figuring out how the new changes work together.

There are a few parts of the codebase where I think it could benefit from either validation (e.g. the arguments passed to the cli) or additional warnings when used (e.g. deprecation of arguments).

Also looks like it will likely need another rebase (or the serial-index branch does)

[kaitj]

This could probably use some validation to enforce the pattern shown so that it doesn't silently fail if an invalid entity-value pair is provided. Maybe a custom type with the validation logic.

[kaitj]

Should probably throw a deprecation warning if something is passed to the arg (in addition to the help string).

[kaitj]

Should some sort of warning be thrown here? This seems like a catch-all if a pattern can't be resolved.

[kaitj]

I think this is minor depending on how large this stack grows, but could implement this using collections.deque instead of a list.

[kaitj]

Similar to the use in __main__, probably want to throw a deprecation warning

[kaitj]

I wonder if we want to keep these together with the dataset information (prior to entities)?