Await time-series mapping loads and structured search resolution

.agents/skills/foundatio-repositories/SKILL.md

@@ -132,6 +132,7 @@ IReadOnlyRepository<T>
 - **`ExistsAsync(query)` is a dirty read**: Uses the Search API (`size: 0`), NOT the realtime Document Exists API. After a write without `ImmediateConsistency`, it can return stale results.
 - **`ExistsAsync(id)` is real-time even with soft deletes**: Uses the GET API with a source filter for `IsDeleted`.
 - **Register repositories as singletons**: Repository instances maintain internal state (index configuration, cache references).
+- **Keep mapping resolvers long-lived**: Daily/monthly query mapping loads use async names-and-aliases discovery plus one partition mapping request. Concurrent lookups share a load per index and process; disposing the index cancels its active mapping load. See [index lifecycle](references/index-lifecycle.md#mapping-resolver-cache) for refresh behavior.
 - **`FieldEquals` with multiple values is OR**: `.FieldEquals(e => e.Field, "A", "B")` produces an OR filter, not AND.
 - **`FieldContains` is token matching, NOT wildcard**: `FieldContains(f => f.Name, "Er")` will NOT match "Eric". Use `FilterExpression("field:pattern*")` for prefix/wildcard matching.
 - **`FieldNot` is AND-NOT**: Multiple conditions inside `FieldNot` mean NOT A AND NOT B. For NOT (A AND B), nest `FieldAnd` inside `FieldNot`.
@@ -143,3 +144,4 @@ IReadOnlyRepository<T>
 - **Patches do not fire `DocumentsSaving`/`DocumentsSaved`**: Patch operations only fire `DocumentsChanged`.
 - **Patches do not detect soft-delete transitions**: Even if a patch sets `IsDeleted = true`, the `ChangeType` is always `Saved`. Soft-delete detection requires `SaveAsync` with `OriginalsEnabled = true`.
 - **Large documents can make `ReindexAsync` trip Elasticsearch's indexing pressure limit**: The default reindex batch size (1000 docs) can produce a bulk sub-request bigger than a node's `indexing_pressure.memory.limit` (10% of heap), causing `es_rejected_execution_exception` ("rejected execution of coordinating operation"). Set `ReindexBatchSize`/`ReindexRequestsPerSecond` (must be positive and finite, or `ReindexAsync` throws `ArgumentOutOfRangeException`; a `null` work item throws `ArgumentNullException`) on the index to throttle. Task-status polling backs off exponentially with jitter (1s → 30s cap, +/-25%) on failure. A low `ReindexRequestsPerSecond` also extends the reindex's stall-detection timeout (default 10 minutes) so a healthy but slow, throttled reindex isn't cancelled as falsely "stalled". See [index-lifecycle.md](references/index-lifecycle.md#reindexasync).
+- In asynchronous query builders, await mapping resolver APIs and `GetResolvedFieldsAsync` / `ResolveFieldNameAsync` / `ResolveFieldSortAsync`; synchronous helpers can block on a cold or missing field even with an async loader.

.agents/skills/foundatio-repositories/references/index-lifecycle.md

@@ -337,10 +337,23 @@ var results = await repository.FindAsync(q => q.Index("logs-last-7-days"));
 
 | Cache layer | Lifetime | How to invalidate |
 |---|---|---|
-| `ElasticMappingResolver` field cache | Auto-refreshes ~60 seconds | `index.MappingResolver.RefreshMapping()` |
+| `ElasticMappingResolver` field cache | Snapshot lifetime; unresolved fields trigger reloads with a five-second cooldown | `index.MappingResolver.RefreshMapping()` |
 | `_isEnsured` flag (Index/VersionedIndex) | Process lifetime | App restart or index deletion |
 | `_ensuredDates` (DailyIndex) | Process lifetime per-date | `DeleteAsync(name)` or `Dispose()` |
-| `ConfigureIndexesAsync` cache marker | 5 minutes (distributed) | Expires automatically; or `ConfigureIndexesAsync(force: true)` |
+| `ConfigureIndexesAsync` cache marker | 5 minutes (distributed) | Expires automatically; pass explicit indexes to bypass the configuration-level lock and cache |
+
+Daily/monthly resolvers asynchronously discover the newest partition using names and aliases only, then load
+that partition's mapping. Concurrent lookups share one load per resolver. Index disposal cancels active mapping
+I/O without initializing an unused resolver. Keep indexes long-lived and do not invalidate after every write.
+The cooldown is not a freshness guarantee, and successful lookups do not trigger periodic refreshes. Use
+`RefreshMapping()` after known mapping changes, including changes to an already-resolved alias. Mappings from
+historical partitions are not merged into the newest partition's mapping.
+
+A caller's cancellation token cancels its wait without canceling a shared load needed by other callers.
+Disposing the resolver cancels the shared mapping I/O. Failed or empty reloads retain the last usable snapshot;
+explicit `RefreshMapping()` invalidates that snapshot so the next lookup loads again.
+
+Structured sort, field-condition, include/exclude, date-range, and paging query builders await mapping resolution. Async custom builders should use `GetResolvedFieldsAsync`, `ResolveFieldNameAsync`, and `ResolveFieldSortAsync` to preserve boosts and sort settings. Protected GET/multi-GET request configuration hooks remain synchronous for compatibility.
 
 ## Index Operations
 
@@ -351,13 +364,17 @@ Creates indexes and updates mappings. Protected by distributed lock + cache mark
 ```csharp
 await configuration.ConfigureIndexesAsync();
 
-// Bypass cache marker (after structural changes)
-await configuration.ConfigureIndexesAsync(force: true);
+// Explicit indexes bypass the configuration-level lock and cache marker.
+// Disable enqueueing when a queue worker is not configured; reindex separately.
+await configuration.ConfigureIndexesAsync(configuration.Indexes, beginReindexingOutdated: false);
 
-// Configure specific indexes (bypasses lock and cache)
-await configuration.ConfigureIndexesAsync([myIndex]);
+// Configure specific indexes using the same explicit-selection path.
+await configuration.ConfigureIndexesAsync([myIndex], beginReindexingOutdated: false);
 ```
 
+There is no `force` parameter. Explicit selection does not update existing daily/monthly partition mappings;
+those indexes still follow the manual mapping lifecycle described above.
+
 ### MaintainIndexesAsync
 
 Updates aliases for time-series indexes, deletes expired indexes:
@@ -476,7 +493,7 @@ public class EmployeeIndex : VersionedIndex<Employee>
 | Error | `Error updating index ({name}) mappings.` | PUT Mapping failed |
 | Error | `Error updating index ({name}) mappings. Changing existing fields requires a new index version.` | Tried to change existing field type on VersionedIndex |
 | Warning | `Adding new analyzer/tokenizer/filter to existing index (requires close/reopen)` | New analysis component needs index close/reopen |
-| Error | `Error getting task status while reindexing: {OldIndex} -> {NewIndex}` | Task status poll failed (e.g. `es_rejected_execution_exception` from indexing pressure); retried with exponential backoff (1s, doubling, capped at 30s) |
+| Error | `Error getting task status while reindexing: {OldIndex} -> {NewIndex}` | Task status poll failed (e.g. `es_rejected_execution_exception` from indexing pressure); retried with exponential backoff (1s, doubling, capped to 30s) |
 | Error | `Failed to get the status {N} times in a row for reindex task ... reindexing {OldIndex} -> {NewIndex}` | Status polling gave up after `MAX_STATUS_FAILS` (10) consecutive failures; reindex progress can no longer be tracked, but the server-side `_reindex` task keeps running |
 
 DailyIndex never emits mapping errors from the built-in configuration path (since `ConfigureAsync` is a no-op).

.agents/skills/foundatio-repositories/references/patterns.md

@@ -290,14 +290,15 @@ All index configurations use `.Dynamic(false)`, which disables Elasticsearch's a
 After adding a mapping for a previously unmapped field, only **newly saved/indexed documents** will be searchable on that field. To make existing documents searchable:
 
 - **Foundatio migration** -- create a `MigrationBase` subclass that uses `PatchAllAsync` or `BatchProcessAsync` to touch all affected documents (recommended for production).
-- **`PatchAllAsync`** with a no-op `ScriptPatch` (e.g., `ctx.op = 'none'` -- still triggers re-index of `_source`).
 - **Elasticsearch Update By Query API** with no script: `POST /{index}/_update_by_query` re-indexes every document in place.
 
+A script that explicitly skips a write does **not** backfill a new mapping. Do not use `ctx.op = 'none'` for this purpose; update-by-query uses `ctx.op = 'noop'` to skip a document. Leave the indexing operation enabled when re-indexing `_source`.
+
 For `DailyIndex`/`MonthlyIndex`, you must also apply the mapping to existing physical indexes before the update-by-query will help.
 
 **Trade-off for Daily/Monthly indexes**: Rolling forward (doing nothing to old partitions and waiting for retention to cycle out old data) is often the cheapest strategy.
 
-**Mapping resolver cache**: After applying a manual PUT Mapping, the in-process `ElasticMappingResolver` auto-refreshes from the server within ~60 seconds. To force immediate recognition, call `index.MappingResolver.RefreshMapping()`.
+**Mapping resolver cache**: The in-process `ElasticMappingResolver` retains a snapshot; it does not refresh on a 60-second timer. Unresolved fields can trigger reloads with a five-second cooldown, which is not a freshness guarantee. After Elasticsearch acknowledges a known mapping change, call `index.MappingResolver.RefreshMapping()` so the next lookup reloads the mapping. This is also required for changes to an already-resolved alias. Resolver invalidation does not backfill existing documents. See [Mapping Resolver Cache](index-lifecycle.md#mapping-resolver-cache).
 
 ### Checklist: Adding a Queryable Model Field
 

.github/workflows/build.yml

@@ -8,3 +8,5 @@ jobs:
     with:
       new-test-runner: true
       minimum-major-minor: '8.0'
+      # Node-wide scroll assertions require a cluster without Kibana background work.
+      compose-command: docker compose up --detach --wait --wait-timeout 120 elasticsearch

docs/guide/index-management.md

@@ -500,7 +500,7 @@ Neither of the following reindexes time-series data: `MaintainIndexesJob` (alias
 **Across different indexes** it depends on how you trigger it: `configuration.ReindexAsync()` processes indexes **sequentially** (one index fully finishes before the next starts), while `ElasticMigrationJob` reindexes them **in parallel** (`Task.WhenAll`, one task per outdated index). Either way each index is internally sequential, and a **distributed lock keyed on the alias** (`reindex:audit`) guarantees a given index is never reindexed by two runners at once — even across multiple application instances (pods, workers). The lock is held for 20 minutes and auto-renewed on every progress callback, so long partition copies keep it alive.
 
 ::: tip Predictable, bounded disk usage per index
-Within one index the upgrade only ever duplicates **one partition at a time**, so bumping a single index (e.g. `audit`) needs roughly one extra partition of headroom regardless of how many partitions it has. If several indexes reindex in parallel (via `ElasticMigrationJob`), peak extra disk is about the sum of one in-flight partition per concurrently-migrating index. Wall-clock time scales with partition count; run during off-peak hours if needed.
+Within one index the upgrade only ever duplicates **one partition** at a time, so bumping a single index (e.g. `audit`) needs roughly one extra partition of headroom regardless of how many partitions it has. If several indexes reindex in parallel (via `ElasticMigrationJob`), peak extra disk is about the sum of one in-flight partition per concurrently-migrating index. Wall-clock time scales with partition count; run during off-peak hours if needed.
 :::
 
 #### Multiple versions and interrupted upgrades
@@ -677,7 +677,7 @@ When a single script applies, it is sent directly to Elasticsearch. When multipl
 
 ```javascript
 void f000(def ctx) { /* v2 rename script */ }
-void f001(def ctx) { /* v2 remove script */ }
+void f001(def ctx) { /* v3 remove script */ }
 void f002(def ctx) { /* v3 custom script */ }
 f000(ctx); f001(ctx); f002(ctx);
 ```
@@ -870,14 +870,36 @@ POST /logs-v1-2025.05.*/_update_by_query?conflicts=proceed
 
 ### Mapping Resolver Cache (Query-Time Mapping Awareness)
 
+Structured search builders for sorts, field conditions, includes/excludes, date ranges, and search-after
+paging await mapping resolution. Custom async builders can use `GetResolvedFieldsAsync`,
+`ResolveFieldNameAsync`, and `ResolveFieldSortAsync` to preserve field boosts and sort settings while
+awaiting mapping I/O. Existing synchronous resolver helpers and protected GET/multi-GET request
+configuration hooks retain their synchronous behavior.
+
 The repository framework does **not** cache the PUT Mapping request/response (that's purely server-side). However, the **query parser** uses an `ElasticMappingResolver` that caches field-to-type resolution for building queries, sorting, and aggregations. This resolver combines two sources:
 
 1. **Code mapping** — derived from your `ConfigureIndexMapping` method at startup (immutable for the process lifetime)
-2. **Server mapping** — fetched from the Elasticsearch GET Mapping API, cached in memory and **automatically refreshed at most once per minute**
+2. **Server mapping** — loaded on first use, then reloaded when a field cannot be resolved, with a five-second cooldown between automatic reload attempts. Successful lookups do not trigger periodic reloads.
+
+Daily and monthly indexes discover the newest partition using an index request limited to names and aliases,
+then retrieve the full mapping of that single partition. Both requests use asynchronous I/O on asynchronous
+query paths. Concurrent lookups share one load through the index's long-lived resolver. Synchronous resolver
+calls remain supported, but block while the asynchronous load completes. Disposing an index disposes its
+initialized resolver and cancels outstanding mapping I/O without creating an unused resolver.
+
+A caller's cancellation token cancels its wait without canceling a shared load needed by other callers.
+Failed or empty reloads retain the last usable snapshot; explicit invalidation clears it.
+
+Each reload discovers the latest partition again so newly created partitions are visible without restarting
+the application. This does not merge mappings across historical partitions. Keep indexes and their resolvers
+long-lived; creating one per request bypasses their caches and multiplies metadata requests. Automatic reload
+limits apply independently to each resolver in each process. Do not call `RefreshMapping()` after every write.
 
 #### What this means after a manual PUT Mapping
 
-If you manually apply a mapping change (e.g., `PUT /index/_mapping` via the Elasticsearch API or a script), the `ElasticMappingResolver` will automatically pick it up within ~60 seconds on the next field resolution. You typically do not need to do anything in application code.
+Requests for an unresolved field can discover new mappings after the cooldown. The cooldown is not a freshness
+guarantee: slow or failed requests can extend the stale window. Changes to an already-resolved field or alias
+require explicit invalidation.
 
 If you need immediate recognition (e.g., in tests or a migration script that queries the new field right after applying the mapping), call:
 
@@ -891,16 +913,16 @@ This clears the cached server mapping and forces the next `GetMapping()` call to
 
 | Cache layer | Lifetime | How to invalidate |
 |---|---|---|
-| `ElasticMappingResolver` field cache | Auto-refreshes from server every ~60 seconds | `index.MappingResolver.RefreshMapping()` |
+| `ElasticMappingResolver` field cache | Snapshot lifetime; unresolved fields trigger reloads with a five-second cooldown | `index.MappingResolver.RefreshMapping()` |
 | `_isEnsured` flag (`Index<T>` / `VersionedIndex<T>`) | Process lifetime (one-time flag) | Deleting the index resets it; otherwise persists until app restart |
 | `_ensuredDates` (`DailyIndex<T>`) | Process lifetime per-date | Cleared on `DeleteAsync(name)` or `Dispose()`; otherwise persists until app restart |
-| `ConfigureIndexesAsync` cache marker | 5 minutes (distributed via `ICacheClient`) | Automatically expires; or call `ConfigureIndexesAsync(force: true)` |
+| `ConfigureIndexesAsync` cache marker | 5 minutes (distributed via `ICacheClient`) | Automatically expires; pass explicit indexes to bypass the configuration-level lock and cache marker |
 
 #### No cluster-side action needed
 
 Elasticsearch itself has no mapping cache you need to invalidate — once a PUT Mapping succeeds, the mapping is immediately active for new indexing and queries. The only caching is in-process within the .NET application:
 
-- **For queries**: The `ElasticMappingResolver` auto-refreshes. If you need it sooner, call `RefreshMapping()`.
+- **For queries**: Unresolved fields can trigger mapping reloads. For a known mapping change that must be visible immediately, call `RefreshMapping()` after Elasticsearch acknowledges the change.
 - **For writes**: The `_isEnsured` / `_ensuredDates` flags only control whether `ConfigureAsync` runs again. They don't prevent writes to the index — they just skip redundant index creation/mapping calls. Manual PUT Mapping changes are orthogonal to these flags.
 
 ### In-Place Analysis Updates (analyzers, tokenizers, filters)
@@ -1122,10 +1144,17 @@ await configuration.ConfigureIndexesAsync();
 // Subsequent calls within 5 minutes skip (fast path)
 await configuration.ConfigureIndexesAsync();
 
-// Passing explicit indexes bypasses the lock and cache marker
-await configuration.ConfigureIndexesAsync([myIndex]);
+// Passing explicit indexes bypasses the configuration-level lock and cache marker.
+await configuration.ConfigureIndexesAsync([myIndex], beginReindexingOutdated: false);
+
+// Or explicitly configure all registered indexes without enqueueing reindex work.
+await configuration.ConfigureIndexesAsync(configuration.Indexes, beginReindexingOutdated: false);
 ```
 
+There is no `force` parameter. Explicit selection does not change daily/monthly mapping behavior:
+existing partitions still require a manual PUT Mapping. Run `ReindexAsync()` separately when a version
+upgrade is required and no reindex queue worker is configured.
+
 ### Maintain Indexes
 
 Run maintenance tasks: