JSON-absent required complex collection materializes as null under ComplexProperty().ToJson(), then throws InvalidOperationException in PrepareToSave on save

[philcarbone]

Bug description

When a complex type is mapped to a JSON column via ComplexProperty().ToJson(), and the stored JSON is missing the key for a nested complex collection (e.g. a row written before that collection property was added — schema evolution), EF Core materializes that collection as null rather than running the CLR initializer / producing an empty collection.

This has two downstream consequences:

1. Read hazard. Any consumer that touches the collection (.Add, iterate, .Count) throws NullReferenceException. The CLR property initializer (= []) is ignored during materialization.
2. Save hazard. Loading such a row tracked, mutating anything on the entity, and calling SaveChanges() throws InvalidOperationException from PrepareToSave.CheckForNullComplexProperties (see stack trace below).

The insert path is unaffected — only pre-existing rows whose stored JSON predates the collection field are poison-on-touch, which makes this a classic schema-evolution failure that only surfaces in production after a deploy.

Expected behavior: a required complex collection that is absent from the stored JSON should materialize as an empty collection (consistent with the CLR initializer and with how System.Text.Json POCO deserialization behaves for an absent array property), so that reads return [] and SaveChanges() does not throw for a collection that is legitimately empty. At minimum, a row that round-trips (load → save) should not become un-saveable purely because the collection key was absent in the stored JSON.

Workaround we're using: coalesce the null complex collections to empty after materialization (via the ChangeTracker.Tracked event / a post-materialization pass) before the entity is read by consumers or saved.

Related: #37162 (milestone 10.0.1) and #37890 (milestone 10.0.6) fix adjacent complex-type-null cases but do not cover the absent-collection case — this still reproduces on 10.0.10. Also related: #37224 (dup of #37162), #36402.

Your code

// net10.0 console app; Npgsql.EntityFrameworkCore.PostgreSQL 10.0.0 (EF Core 10.0.2; also reproduces on 10.0.10).
// Needs a local PostgreSQL (any recent version).
using Microsoft.EntityFrameworkCore;

await using (var ctx = new AppDbContext()) {
    await ctx.Database.EnsureDeletedAsync();
    await ctx.Database.EnsureCreatedAsync();

    // 1. Insert a row with the CURRENT model shape.
    ctx.Add(new Order {
        Id = Guid.NewGuid(),
        Data = new OrderDetails {
            Shipments = [new Shipment { Carrier = "A", Parcels = [new Parcel { TrackingCode = "p1" }] }],
        },
    });
    await ctx.SaveChangesAsync();

    // 2. Simulate an OLD-SHAPE row: strip the key of a later-added nested collection
    //    (i.e. a row written before Parcel.InspectionTagIds existed).
    await ctx.Database.ExecuteSqlRawAsync(
        "UPDATE orders SET data = data #- '{Shipments,0,Parcels,0,InspectionTagIds}'");
}

await using (var ctx = new AppDbContext()) {
    // 3. READ hazard: the JSON-absent collection materializes as null (expected: []).
    var order = await ctx.Orders.AsNoTracking().SingleAsync();
    Console.WriteLine(order.Data.Shipments[0].Parcels[0].InspectionTagIds is null); // True
}

await using (var ctx = new AppDbContext()) {
    // 4. SAVE hazard: tracked load + any mutation + SaveChanges throws InvalidOperationException
    //    in PrepareToSave.CheckForNullComplexProperties.
    var order = await ctx.Orders.SingleAsync();
    order.Data.Shipments[0].Parcels.Add(new Parcel { TrackingCode = "p2", InspectionTagIds = [] });
    await ctx.SaveChangesAsync(); // throws
}

public class AppDbContext : DbContext {
    public DbSet<Order> Orders => Set<Order>();

    protected override void OnConfiguring(DbContextOptionsBuilder options)
        => options.UseNpgsql("Host=localhost;Port=5432;Database=efrepro;Username=postgres;Password=postgres");

    protected override void OnModelCreating(ModelBuilder modelBuilder)
        => modelBuilder.Entity<Order>(e => {
            e.ToTable("orders");
            e.HasKey(x => x.Id);
            e.ComplexProperty(x => x.Data, d => d.ToJson("data"));
        });
}

public class Order {
    public Guid Id { get; set; }
    public OrderDetails Data { get; set; } = new();
}

public class OrderDetails {
    public List<Shipment> Shipments { get; set; } = [];
}

public class Shipment {
    public string Carrier { get; set; } = "";
    public List<Parcel> Parcels { get; set; } = [];
}

public class Parcel {
    public string TrackingCode { get; set; } = "";
    public List<Guid> InspectionTagIds { get; set; } = [];   // the later-added field, absent from old rows
}

Stack traces

System.InvalidOperationException: The complex type property '<ComplexType>.<CollectionProperty>' is configured as required (non-nullable) but has a null value when saving changes.
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntryBase.<PrepareToSave>g__CheckForNullComplexProperties|99_2(<>c__DisplayClass99_0&)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntryBase.PrepareToSave()
   (property name matches whichever required complex collection materialized as null from the JSON-absent key)

Verbose output

No response

EF Core version

10.0.2 (reproduces unchanged on 10.0.10 — every released 10.0.x patch)

Database provider

Npgsql.EntityFrameworkCore.PostgreSQL 10.0.0

Target framework

.NET 10.0

Operating system

macOS (Apple Silicon); PostgreSQL in Docker

IDE

No response

[philcarbone]

Additional empirically-verified facet (same environment as the report: EF Core 10.0.2–10.0.10, Npgsql provider 10.0.0, net10.0):

A complex collection configured via ComplexCollection(...) with HasJsonPropertyName(...) materializes as null even when the stored JSON contains the key with an empty array. With a mapping like:

e.ComplexProperty(x => x.Audit, a => {
    a.ToJson("audit");
    a.ComplexCollection(p => p.Entries, en => en.HasJsonPropertyName("en"));
});

a row whose stored column is {"en": []} reads back with Entries == null — and then hits the same PrepareToSave failure described above on the next tracked save.

For contrast (verified on the same stack):

• an empty primitive collection (List<Guid>) with its key present ("Ids": []) round-trips to [] correctly;
• an empty nested complex collection mapped by convention inside a ToJson() complex property ("Parcels": []) also round-trips to [] correctly.

So the null-materialization is triggered both by the absent key (original report) and by present-but-empty collections under the explicit ComplexCollection + HasJsonPropertyName configuration. The second case is nastier operationally: rewriting the row (e.g. a projection rebuild) produces perfectly-shaped JSON and still reads back null.

[philcarbone]

Update: reproduced inside the EF Core test suite itself (SQLite provider), identically on both release/10.0 and main — so this is provider-agnostic relational core behavior, not something in my original environment's provider.

I added three tests to ComplexCollectionJsonUpdateTestBase — the existing WidgetWithDeepJson fixture already models this exact shape (Deep → Mid → Items[] → Inner[]/Others[], with = [] CLR initializers). An "old-shape" row (persisted before the type gained the Others sub-collection) is simulated by rewriting the stored document with raw SQL so the key is absent:

await context.Database.ExecuteSqlAsync(
    $$$"""UPDATE "Widgets" SET "Deep" = '{"Mid":{"Items":[{"Title":"Item1","Inner":[{"Value":"inner-0"}]}]}}' WHERE "Id" = 1""");

Results (identical on release/10.0 @ 95ed66f and current main):

1. Materialization — after loading, widget.Deep.Mid.Items[0].Others is null; the = [] initializer is ignored for the JSON-absent complex collection (Assert.NotNull fails).

2. Save — loading such a row and changing an unrelated scalar (Title), then SaveChangesAsync():

   System.InvalidOperationException : The property 'Items' belongs to the type
   'WidgetWithDeepJson.Deep#DeepData.Mid#MiddleData', but is being used with an
   instance of type 'WidgetWithDeepJson.Deep#DeepData.Mid#MiddleData.Items#DeepItem'.
      at Microsoft.EntityFrameworkCore.Metadata.Internal.TypeBaseExtensions.CheckContains[T](...)
      at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntryBase.GetCurrentValue(...)
      at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntryBase.<PrepareToSave>g__CheckForNullComplexProperties|...
   

   i.e. the change tracker's own save-path validation trips over the materializer's output — the row is effectively poison-on-touch: readable state is corrupt and any tracked save throws.

3. Scoping note: the "key present with an empty array under HasJsonPropertyName" facet from my earlier comment did not reproduce on SQLite (test passes on both branches). That part is likely specific to my original provider; I'll pursue it separately and keep this issue focused on the absent-key defect.

I'd like to contribute the fix. There seem to be two candidate semantics — which would the team accept?

• (a) Skip the assignment when the JSON key is absent — the materialized instance keeps whatever its CLR initializer/constructor produced ([] here). More general (respects any initializer) and preserves the distinction between an explicit JSON null and an absent key.
• (b) Coalesce absent complex collections to an empty collection during materialization.

If one of these gets a green light, I'll open a PR against main with the fix plus these regression tests (they slot in right next to the #38299 regression test in ComplexCollectionJsonUpdateTestBase). I'd also ask for servicing consideration for 10.0.x: any application that adds a collection to an existing JSON-mapped complex type finds all its pre-existing rows unloadable-then-unsavable, with no workaround at the model level — same family as the complex-collection servicing fixes that shipped in 10.0.3 (#37373), 10.0.6 (#37725), and 10.0.10 (#38299).

[AndriySvyryd]

The materializer that EF uses does not preserve default property values, so option b would be more in line with the current behavior.