Skip to content

OME: SkillClusterUpdated persists a 1024-dim vector, inflating run_record ~18x #398

Description

@Kendrick-Song

What

SkillClusterUpdated.case_vector (added in #393) carries the triggering case's 1024-dim embedding so extract_agent_skill avoids a second embedding call. The event payload is persisted verbatim, so the vector lands in two stores:

  • run_record.event_payload (_stores/run_record.py, column is TEXT NOT NULL)
  • the APScheduler jobstore, while a job is queued (engine.py:686 serializes the event into args)

Measured with a realistic payload:

with case_vector : 14,074 bytes
without          :    765 bytes   -> 18.4x, +13.3 KB/record

At the default max_records_per_strategy = 1000, the skill_cluster_updated ring buffer is ~14 MB instead of ~0.8 MB. It has no audit value — nobody reads 1024 floats out of a run record.

The vector is only consumed when a cluster holds more skills than MAX_SKILLS_IN_PROMPT (extract_agent_skill.py:265-273), which for a highly-aggregated artifact like a skill should be the minority of runs. Most of the time the 13 KB rides along unused.

Why it wasn't fixed in #393

The obvious fix — strip large fields when persisting to run_record — is not local, and this is the part worth carrying forward:

Crash recovery replays the persisted payload to rebuild the event (_background/crash_recovery.py:63 -> engine.py:749 model_validate_json). A trimmed persisted copy deserializes with case_vector is None, which is a legal value that silently routes the recovered run down the md-ordering fallback branch while the original run took the LanceDB-ranked branch. Same event id, different selection, no error. So trimming needs either a payload/audit split with recovery reading the untrimmed copy, or an explicit "vector unavailable after recovery" signal.

Options

  1. Audit projection. Give BaseEvent a hook (or pass model_dump_json(exclude=...)) used only for the run_record write, keeping the recovery path on the full payload. Fixes the audit bloat, leaves the jobstore copy.
  2. Don't carry the vector; re-embed in the > MAX_SKILLS_IN_PROMPT branch. Costs one embedding call on the minority path. Note this does not reintroduce the eventual-consistency dependency fix(memory): rescue skill extraction, disable foresight, tighten APIs #393 removed: that branch already reads LanceDB (find_topk_relevant_in_cluster) and already degrades to md ordering when the index is stale.
  3. Keep as-is and document the sizing (done in CHANGELOG 1.2.3 + fix(memory): rescue skill extraction, disable foresight, tighten APIs #393 body).

Option 2 is the smaller diff and removes the field entirely; option 1 preserves the saved call. Either needs the recovery-divergence question answered first.

Not urgent

Bounded by the ring buffer, so it does not grow without limit. Filed so ome.db sizing is traceable to a decision rather than an accident.

Found during post-merge-review of #393.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions