Skip to content

Commit 530c1df

Browse files
Elon Muskclaude
andauthored
fix(objectql): the Archiver honours a declared ttl instead of archiving by created_at age alone (#10541)
* fix(objectql): the Archiver honours a declared `ttl` (#10347) A lifecycle declaring both `ttl` and `archive` parses, but `reapObject` returns into `archiveObject` before the ttl branch is reachable, so the declared per-row expiry never ran and the Archiver moved rows by `created_at` age alone — declared not enforced. Maintainer ruling 2026-08-20: what the author declared is what executes. `archiveObject` now selects candidates by the ttl cutoff on `ttl.field` when `ttl` is declared, and by `created_at`/`archive.after` otherwise, so archive-only lifecycles are byte-for-byte unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019yDEhPBC3tcGkW9bkce1HM * chore(changeset): the Archiver honours a declared `ttl` (#10347) Also names the filed follow-up (#10527) at the site that defers it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019yDEhPBC3tcGkW9bkce1HM --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 2c9d60f commit 530c1df

3 files changed

Lines changed: 254 additions & 3 deletions

File tree

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
"@objectstack/objectql": patch
3+
---
4+
5+
**Behaviour change:** a `lifecycle` that declares both `ttl` and `archive` now
6+
has its **`ttl` enforced** — the Archiver selects the rows it moves by the
7+
declared TTL cutoff (`ttl.field` past `ttl.expireAfter`) instead of by
8+
`created_at` age (#10347).
9+
10+
That pair has always parsed — ADR-0057 §3.5 is satisfied because `ttl` is a
11+
bounding policy, and the `archive.after === retention.maxAge` refine only fires
12+
when `retention` is present — but it did nothing: `LifecycleService.reapObject`
13+
returns into `archiveObject` before its `ttl` branch is reachable, so no reap on
14+
`ttl.field` ever ran and the Archiver copied and hot-deleted by `created_at` age
15+
alone. Declared, not enforced. What the author wrote is now what executes; they
16+
no longer have to discover that the two keys cannot usefully be written
17+
together.
18+
19+
**Lifecycles that declare `archive` without `ttl` are unaffected** — they keep
20+
selecting rows by `created_at` past `archive.after`, unchanged. Every
21+
archive-declaring object shipped with the platform (`sys_audit_log`,
22+
`sys_metadata_audit`) is that shape, so no bundled object changes behaviour.
23+
24+
Two details of the new selection, both deliberate:
25+
26+
- A row whose `ttl.field` is **null or absent is retained, not archived**. `$lt`
27+
is a positive comparison and a value that is not there satisfies none of them
28+
(the platform-wide null answer settled in #5298/#5299), which is also the
29+
right reading: a row with no expiry stamp has not been given one, and treating
30+
"absent" as "expired at the epoch" would archive exactly the rows whose expiry
31+
the author has not yet decided.
32+
- The cold-side `archive.keep` prune is unchanged. It bounds how long **archived**
33+
rows survive in cold storage, not which hot rows are due, and it still measures
34+
from `created_at` under either policy.
35+
36+
If you declare `retention` beside `ttl` and `archive`, the TTL cutoff is what
37+
selects: the age window no longer separately bounds the hot store for that
38+
triple. Whether the Archiver should honour both windows is a separate open
39+
question, filed as #10527 rather than decided here.

packages/objectql/src/lifecycle/lifecycle-service.test.ts

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -946,6 +946,169 @@ describe('LifecycleService.sweep — Archiver (P3)', () => {
946946
expect(hot.bulkDeleted).toEqual([]);
947947
expect(report.skipped).toEqual([{ object: 'sys_audit_log', reason: 'archive-pending' }]);
948948
});
949+
950+
/* ------------------------------------------------------------------ *
951+
* [#10347] The Archiver honours a declared `ttl`.
952+
*
953+
* The property under test is "the declared ttl cutoff GOVERNED which rows
954+
* moved", and it is invisible to a suite that only asserts rows were
955+
* archived — such a suite passes identically against the `created_at`-only
956+
* behaviour this card changes. Two things make the cases below able to fail:
957+
*
958+
* 1. `filteringHotStore` really evaluates the `where` the Archiver sends.
959+
* `hotStore()` above deliberately ignores it (its subjects are batching
960+
* and teardown, not selection), so a control built on that fake returns
961+
* every row under either policy and can never distinguish them.
962+
* 2. The rows are chosen so the two policies DISAGREE about which are due,
963+
* in both directions — one row `created_at` age would move and the ttl
964+
* would not, and one the reverse.
965+
* ------------------------------------------------------------------ */
966+
967+
/** The card's own example. Both windows are '90d' ON PURPOSE: the cutoff
968+
* INSTANT is then identical under either policy, so the only thing that can
969+
* separate them is which COLUMN is read. */
970+
const TTL_ARCHIVE_OBJ: LifecycleObjectLike = {
971+
name: 'sys_audit_log',
972+
lifecycle: {
973+
class: 'audit',
974+
ttl: { field: 'expires_at', expireAfter: '90d' },
975+
archive: { after: '90d', to: 'archive', keep: '7y' },
976+
} as any,
977+
};
978+
979+
const DAY = 86_400_000;
980+
const at = (deltaMs: number) => new Date(FIXED_NOW + deltaMs).toISOString();
981+
982+
/**
983+
* Four rows. `created_at` age and the `expires_at` ttl disagree on the first
984+
* two in opposite directions; the last two carry no expiry stamp at all
985+
* (null, then the key absent) while being old enough for the age policy.
986+
*/
987+
const disagreeingRows = () => [
988+
// Age says move it — 400 days old. The ttl says it has not expired yet.
989+
{ id: 'old-unexpired', created_at: at(-400 * DAY), expires_at: at(+30 * DAY) },
990+
// The reverse: one day old, so age keeps it — but its stamp expired 400
991+
// days ago, so the declared ttl says it is due.
992+
{ id: 'young-expired', created_at: at(-DAY), expires_at: at(-400 * DAY) },
993+
// No expiry stamp: old by age, undecided by ttl.
994+
{ id: 'null-stamp', created_at: at(-400 * DAY), expires_at: null },
995+
{ id: 'absent-stamp', created_at: at(-400 * DAY) },
996+
];
997+
998+
/**
999+
* A hot store that EVALUATES the archiver's `where` and records it. `$lt` is
1000+
* applied with the platform's settled null answer — a value that is not
1001+
* there satisfies no positive comparison (#5298/#5299, every backend's
1002+
* `nullValueSatisfiesOperator` ends `default: return false`) — so the
1003+
* null/absent rows are decided by that contract, not by a JS accident.
1004+
*/
1005+
function filteringHotStore(rows: Array<Record<string, unknown>>) {
1006+
const wheres: Array<Record<string, any>> = [];
1007+
const bulkDeleted: Array<Array<string | number>> = [];
1008+
let remaining = [...rows];
1009+
const matches = (row: Record<string, unknown>, where: Record<string, any>) =>
1010+
Object.entries(where).every(([field, cond]) => {
1011+
const value = row[field];
1012+
if (value === null || value === undefined) return false;
1013+
return String(value) < String(cond.$lt);
1014+
});
1015+
return {
1016+
wheres,
1017+
bulkDeleted,
1018+
remaining: () => remaining.map((r) => r.id),
1019+
driver: {
1020+
name: 'default',
1021+
find: async (_object: string, query: any) => {
1022+
wheres.push(query.where);
1023+
return remaining.filter((r) => matches(r, query.where)).slice(0, query.limit ?? remaining.length);
1024+
},
1025+
upsert: async () => ({}),
1026+
bulkDelete: async (_object: string, ids: Array<string | number>) => {
1027+
bulkDeleted.push(ids);
1028+
remaining = remaining.filter((r) => !ids.includes(r.id as string));
1029+
},
1030+
deleteMany: async () => 0,
1031+
},
1032+
};
1033+
}
1034+
1035+
it('DISCRIMINATING CONTROL: a declared ttl decides which rows move — not created_at age', async () => {
1036+
const cold = coldStore();
1037+
const hot = filteringHotStore(disagreeingRows());
1038+
const { engine } = captureEngine([TTL_ARCHIVE_OBJ], {
1039+
driver: hot.driver,
1040+
datasources: { archive: cold.driver },
1041+
});
1042+
1043+
const report = await service(engine).sweep();
1044+
1045+
// The candidate read is issued against the DECLARED ttl field.
1046+
expect(hot.wheres).toEqual([{ expires_at: { $lt: isoCutoff('90d') } }]);
1047+
// Only the expired row moves. `old-unexpired` is what makes this a control:
1048+
// the `created_at`-only Archiver copies it, and the ttl the author declared
1049+
// says it is not due for another 30 days.
1050+
expect(cold.upserts.map((r) => r.id)).toEqual(['young-expired']);
1051+
expect(hot.bulkDeleted).toEqual([['young-expired']]);
1052+
expect(hot.remaining()).toEqual(['old-unexpired', 'null-stamp', 'absent-stamp']);
1053+
1054+
const entry = report.swept.find((e) => e.policy === 'archive');
1055+
expect(entry?.archived).toBe(1);
1056+
expect(entry?.cutoff).toBe(isoCutoff('90d'));
1057+
expect(report.skipped).toEqual([]);
1058+
});
1059+
1060+
it('a row whose ttl.field is null or absent is NOT due at the epoch — it is retained', async () => {
1061+
// Stated as its own case because it is a DECISION, not a side effect: a row
1062+
// with no expiry stamp has not been given one, and archiving it would move
1063+
// exactly the rows whose expiry the author has not decided yet. The three
1064+
// rows here are all past `archive.after` by age, so a fix that reached for
1065+
// `created_at` — or read a missing stamp as 0 — would copy all three.
1066+
const cold = coldStore();
1067+
const hot = filteringHotStore([
1068+
{ id: 'null-stamp', created_at: at(-400 * DAY), expires_at: null },
1069+
{ id: 'absent-stamp', created_at: at(-400 * DAY) },
1070+
{ id: 'expired', created_at: at(-400 * DAY), expires_at: at(-91 * DAY) },
1071+
]);
1072+
const { engine } = captureEngine([TTL_ARCHIVE_OBJ], {
1073+
driver: hot.driver,
1074+
datasources: { archive: cold.driver },
1075+
});
1076+
1077+
const report = await service(engine).sweep();
1078+
1079+
expect(cold.upserts.map((r) => r.id)).toEqual(['expired']);
1080+
expect(hot.remaining()).toEqual(['null-stamp', 'absent-stamp']);
1081+
expect(report.swept.find((e) => e.policy === 'archive')?.archived).toBe(1);
1082+
});
1083+
1084+
it('POSITIVE CONTROL: archive WITHOUT ttl still moves rows by created_at age', async () => {
1085+
// Every archive-declaring object shipped today is this shape (`sys_audit_log`,
1086+
// `sys_metadata_audit`: retention + archive, no ttl). It is fed the SAME
1087+
// rows as the discriminating control, so the two cases answer differently
1088+
// on the same input: a fix leaking into this path would copy
1089+
// `young-expired` (which has an expired stamp) and skip the two stampless
1090+
// rows, and this expectation would fail.
1091+
const cold = coldStore();
1092+
const hot = filteringHotStore(disagreeingRows());
1093+
const { engine } = captureEngine([AUDIT_OBJ], {
1094+
driver: hot.driver,
1095+
datasources: { archive: cold.driver },
1096+
});
1097+
1098+
const report = await service(engine).sweep();
1099+
1100+
expect(hot.wheres).toEqual([{ created_at: { $lt: isoCutoff('90d') } }]);
1101+
expect(cold.upserts.map((r) => r.id)).toEqual(['old-unexpired', 'null-stamp', 'absent-stamp']);
1102+
expect(hot.bulkDeleted).toEqual([['old-unexpired', 'null-stamp', 'absent-stamp']]);
1103+
expect(hot.remaining()).toEqual(['young-expired']);
1104+
1105+
const entry = report.swept.find((e) => e.policy === 'archive');
1106+
expect(entry?.archived).toBe(3);
1107+
expect(entry?.cutoff).toBe(isoCutoff('90d'));
1108+
// The cold-side `keep` prune is a bound on the ARCHIVE, not on which hot
1109+
// rows are due: it stays on `created_at` under either policy.
1110+
expect(cold.coldDeletes).toEqual([{ where: { created_at: { $lt: isoCutoff('7y') } } }]);
1111+
});
9491112
});
9501113

9511114
describe('LifecycleService.sweep — space reclaim', () => {

packages/objectql/src/lifecycle/lifecycle-service.ts

Lines changed: 52 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,9 @@ import type {
2121
* oldest shard. Until a driver advertises rotation support, declared
2222
* rotation falls back to an age-based reap bounded by `shards × unit`.
2323
* - **Archiver** (P3): copies audit-class cold rows to the declared archive
24-
* datasource, then deletes them from the hot store. **Safety rule:** an
24+
* datasource, then deletes them from the hot store. Cold is `created_at`
25+
* past `archive.after`, or [#10347] `ttl.field` past `ttl.expireAfter`
26+
* when the object declares a `ttl` beside its `archive`. **Safety rule:** an
2527
* object that declares `archive` is never hot-deleted unless the archive
2628
* copy succeeded — a compliance ledger must not be dropped unarchived.
2729
*
@@ -910,6 +912,13 @@ export class LifecycleService {
910912
// deletion happens ONLY for rows the Archiver has copied to the cold
911913
// store; when the archive datasource isn't registered, rows are retained
912914
// (never dropped unarchived) and the object is reported as skipped.
915+
//
916+
// [#10347] This return is not a policy DROP. A lifecycle may declare `ttl`
917+
// beside `archive` — that pair parses — and until this card the ttl branch
918+
// below was simply unreachable for it, so the declared per-row expiry never
919+
// ran anywhere. {@link archiveObject} now applies that window itself (see
920+
// its `dueField` note), so the hand-off carries the policy instead of
921+
// discarding it.
913922
if (lc.archive) {
914923
return this.archiveObject(engine, obj, lc, report);
915924
}
@@ -1124,6 +1133,9 @@ export class LifecycleService {
11241133
/**
11251134
* Archiver (ADR-0057 §3.3 / P3): copy rows past `archive.after` from the
11261135
* hot store to the archive datasource, then delete the copied rows hot.
1136+
* [#10347] When the object ALSO declares `ttl`, the declared per-row expiry
1137+
* is what selects candidates — `ttl.field` past its `expireAfter` window —
1138+
* instead of `created_at` past `archive.after`.
11271139
* Batched (500 × 20 per sweep) so a large backlog drains across sweeps
11281140
* without one long-locking pass. Copies are per-row idempotent upserts, so
11291141
* a sweep interrupted between copy and hot-delete re-converges. When
@@ -1158,7 +1170,44 @@ export class LifecycleService {
11581170
await cold.syncSchema(object, obj);
11591171
}
11601172

1161-
const cutoff = new Date(this.now() - parseLifecycleDuration(archive.after)).toISOString();
1173+
// [#10347] WHICH ROWS ARE DUE. `archive` alone moves rows by age from
1174+
// `created_at`, bounded by `archive.after` — unchanged. But a lifecycle may
1175+
// also declare `ttl` beside `archive`: ADR-0057 §3.5's refine is satisfied
1176+
// (`ttl` IS a bounding policy) and the `archive.after === retention.maxAge`
1177+
// refine only fires when `retention` is present, so the pair parses. It used
1178+
// to do nothing at all — `reapObject` returns into this method before its
1179+
// ttl branch — leaving the author with a declared expiry that never ran
1180+
// while the Archiver moved rows by `created_at` age alone.
1181+
//
1182+
// Maintainer ruling 2026-08-20: what the author declared is what executes.
1183+
// When `ttl` is declared the Archiver selects candidates by the TTL cutoff
1184+
// on `ttl.field` and hands them to the same copy → hot-delete pair, so the
1185+
// expiry the author wrote decides which rows move. Nothing else about the
1186+
// Archiver changes: `archive`-only objects keep selecting by
1187+
// `created_at`/`archive.after`, and the cold-side `keep` prune below is a
1188+
// bound on the ARCHIVE (how long cold rows survive), not on which hot rows
1189+
// are due — it stays on `created_at` either way.
1190+
//
1191+
// A row whose `ttl.field` is NULL or absent is NOT due, and is retained.
1192+
// `$lt` is a positive comparison, and a value that is not there satisfies
1193+
// none of them — the platform-wide answer settled in #5298/#5299 and
1194+
// spelled out in every backend's `nullValueSatisfiesOperator`
1195+
// (`default: return false`). That is also the answer this method wants: a
1196+
// row with no expiry stamp has not been GIVEN one, so reading "absent" as
1197+
// "expired at the epoch" would archive exactly the rows whose expiry the
1198+
// author has not decided yet — against the retain-first posture that makes
1199+
// this method refuse to hot-delete anything the cold store has not taken.
1200+
//
1201+
// ⚠️ Deliberately NOT decided here: `retention` declared beside `ttl` +
1202+
// `archive`. The ttl cutoff selects, so the age window (`archive.after`,
1203+
// which the spec pins equal to `retention.maxAge`) no longer separately
1204+
// bounds the hot store for that triple. Whether the Archiver should union
1205+
// the two windows, or the triple be refused at parse time (a
1206+
// `packages/spec` accept-set question, outside this card's fence), is
1207+
// #10527 rather than a choice this diff makes silently.
1208+
const dueField = lc.ttl ? lc.ttl.field : 'created_at';
1209+
const dueWindow = lc.ttl ? lc.ttl.expireAfter : archive.after;
1210+
const cutoff = new Date(this.now() - parseLifecycleDuration(dueWindow)).toISOString();
11621211
let archived = 0;
11631212
for (let batch = 0; batch < ARCHIVE_MAX_BATCHES_PER_SWEEP; batch++) {
11641213
// [#4747] Leg boundary, per batch — the same check the reap loop makes
@@ -1175,7 +1224,7 @@ export class LifecycleService {
11751224
// store unchanged.
11761225
if (this.abort.aborted) break;
11771226
const rows = await hot.find(object, {
1178-
where: { created_at: { $lt: cutoff } },
1227+
where: { [dueField]: { $lt: cutoff } },
11791228
limit: ARCHIVE_BATCH_SIZE,
11801229
});
11811230
if (!rows.length) break;

0 commit comments

Comments
 (0)