Skip to content

Drop index template fields once the schema no longer references them#1303

Open
jwils wants to merge 2 commits into
mainfrom
joshuaw/drop-unreferenced-template-fields
Open

Drop index template fields once the schema no longer references them#1303
jwils wants to merge 2 commits into
mainfrom
joshuaw/drop-unreferenced-template-fields

Conversation

@jwils

@jwils jwils commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #1206.

Why

Index templates preserve every field forever (a guard added after a near-SEV caused by dropping a template field while deployed indexers still wrote it), so fields removed from the schema accumulate as stale template cruft with no garbage-collection path.

This implements the approach proposed in the #1206 review discussion: preserve a template field as long as it exists in some form in the schema definition--as a current field, a renamed_from field, or a deleted_field--and drop it only once the last reference is removed.

That rule avoids the SEV footgun without relying on vigilant artifact-diff review: eg-schema_def requires a deleted_field/renamed_from declaration for as long as any versioned JSON schema references the field, and warns once the declaration becomes unnecessary. By the time an operator can remove the declaration, no JSON schema version references the field--so every deployed indexer drops it from old events, and no replayable event can produce it.

How

  • RuntimeMetadata::IndexDefinition gains field_paths_protected_from_removal: full index field paths derived from deleted_field and renamed_from declarations on the indexed type and all embedded object types (union/interface subtypes included). Empty lists are pruned from dumped YAML, so existing schema artifacts do not change.

  • DatastoreCore::IndexDefinition exposes it like the other runtime metadata attributes.

  • MappingUpdate.build_mapping_update accepts an optional protected_field_paths set: when provided, a field missing from the desired mapping is preserved only if its path (or a descendant's path) is protected; other missing fields drop out of the built update. ForIndexTemplate passes the protected set; concrete indices continue to preserve everything (the datastore provides no way to remove their fields).

  • Dropped fields are called out explicitly in the admin output, in addition to appearing as deletions in the diff:

    Removed 2 stale field(s) from index template `widgets` that are no longer referenced by the schema: name, options.size.
    New rollover indices created from this template will reject documents that still contain these fields (mappings are `dynamic: strict`).
    

Known limitation

A deleted field's name_in_index is not recorded anywhere (deleted_field only captures the GraphQL field name, and old versioned JSON schemas have their ElasticGraph metadata nulled on deletion), so protection matches template fields by the declared name. A removed field that used a custom name_in_index would not match its template field, which would therefore be dropped while still referenced. If we want to close that gap, deleted_field could optionally accept name_in_index: (or we could capture it from the prior artifacts at dump time)--happy to tackle that here or in a follow-up.

Verification

  • script/lint
  • script/spellcheck
  • script/type_check
  • script/quick_build
  • bundle exec rake schema_artifacts:dump confirms no artifact changes for schemas without declarations

@jwils
jwils force-pushed the joshuaw/hide-delete-changes-admin-lambda branch from 2b3c876 to bc2d25d Compare July 8, 2026 21:33
@jwils
jwils force-pushed the joshuaw/drop-unreferenced-template-fields branch from c507876 to 5d8214f Compare July 8, 2026 21:38
jwils added a commit that referenced this pull request Jul 9, 2026
## Summary
- Move mapping-removal noise handling into `elasticgraph-admin` so all
admin users benefit.
- Preserve current mapping fields in index update payloads and
diffs--for concrete indices *and* index templates--hiding
unsupported/unsafe field removals while still showing new fields and
mapping parameter changes.
- Keep settings diffs complete, and report new index templates as
created instead of dumping the full config diff.

This is purely an output improvement: no behavior changes. Index
templates keep preserving fields that have been removed from the schema
(as they always have); the removals just no longer clutter every admin
run's diff. Actually dropping stale template fields safely is
implemented separately in the stacked PR #1303, per review feedback.

## Context
Mapping removals are unsupported by Elasticsearch/OpenSearch concrete
indices, so showing removed fields every admin run just makes the diffs
harder to read. Index templates technically allow field removal, but
dropping a field that an older deployed indexer can still write causes
indexing failures on new rollover indices (mappings are `dynamic:
strict`), so templates keep their existing fields and the removals are
likewise hidden from the diff output.

This addresses #766 and the review feedback to improve
`elasticgraph-admin` directly instead of adding an
`elasticgraph-admin_lambda` opt-in.

## Verification
- `script/lint`
- `script/spellcheck`
- `script/type_check`
- `script/quick_build`
Base automatically changed from joshuaw/hide-delete-changes-admin-lambda to main July 9, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant