Repository navigation
perf: autoresearch optimization pass — transact & query fast paths - #42
Merged
Merged
Conversation
apply_tx ran four tempid-only passes (try_new_tempid_entities, ambiguous/bulk tempid-entity guards, tempid_add_groups) on every transaction even when no op mentioned a tempid — pure Add/Entity_id transactions paid several full tx_ops scans plus Hashtbl allocations for results that can only be empty. Compute two cheap reachability flags (has_tempid_entity, has_tempid_add) and gate each pass on them; behavior is identical because each gated check can only return false/empty for tempid-free transactions. bench (ocaml native, size 1000): add-1 21.4 -> 19.6 ms, add-all 26.6 -> 24.3 ms; other metrics unchanged.
Two per-transaction validations ran unconditional scans whose inputs are often empty/singleton: - the in-tx conflict checks (duplicate_fact / duplicate_unique / duplicate_cardinality_one_fact) allocated three hashtables and scanned facts even when 0-1 facts could never conflict; keep only conflicts_with_existing (the db-side check) for singleton txs - validate_tempid_usage scanned tx_data removing eids from value_tempids even when no value-position tempid ever appeared — skip when the table is empty bench (ocaml native, size 1000): add-1 19.6 -> 17.0 ms; other metrics unchanged.
A fact on an entity with e > db.max_datom_e can neither retract nor duplicate an existing db fact, so both cardinality branches of tx_data_for_fact reduce to [d] — emit it directly instead of paying an index seek, a cardinality lookup and a same-fact scan per fact. Also build tx_data via rev_append fold instead of concat_map to drop the intermediate mapped list. bench (ocaml native, size 1000): bulk-explicit 62.0->59.9ms (-3%), bulk-add-ops 59.5->57.0ms (-4%); measured fp-tx-data phase 12->0.4ms on a 10k-datom bulk tx. Other metrics unchanged.
Pins logseq/persistent-sorted-set-ocaml d4a8ca5 (perf/add-insert-fast-path): a no-split leaf/branch insert emits the updated node directly instead of walking the generic chunking pipelines, saving a few intermediate allocations per PSet.add. Tree shape and stored bytes are unchanged (storage parity hash verified identical across native/jsoo/cljs). bench (10 runs): add-1 17.0 -> 16.1ms (~-5%).
Retract (Entity_id, attr, value) ops used to fall back to apply_ops, which refreshes the index trees once per op. They now run on the fast path: ops emit tx_data in order against a staged per-(e,a) effective view (db datoms plus pending adds minus pending retracts), so a retract correctly sees what earlier ops in the same transaction added or removed, and the index refresh stays batched at the end. Component-attribute cascades, RetractEntity, and RetractAttr still take the slow path; txs without retract ops keep the plain per-fact emission. bench (retract+add attr update, 4000 txs): ~13-19us -> ~7.5-10us (-40%).
For the same-entity-constant fast path (queries like [:find ?e ?l ?a :where [?e :name "Ivan"] [?e :last-name ?l] [?e :age ?a] [?e :sex :male]]), upstream materializes every constant's datoms and each value var's full attribute list into per-entity arrays — O(total datoms per attribute) regardless of selectivity. This change picks per-attribute access plans: constants that an avet seek can serve stay materialized; every other non-avet constant and every value var either materializes (when the driver is large relative to the attribute) or is served by a single bounded eavt e-prefix slice per surviving entity (when driver_count * 4 <= attr datom count). All of an entity's verify/slot lookups then share one index slice. q4 (4-clause same-entity query, 2000-entity db): ~0.9-1.0ms -> ~0.22ms. Tests and cljs parity hash unchanged.
Entity attr materialization (Logseq page load: 100 blocks/page) paid the full datoms accessor (arg resolution, seq machinery, duplicate merge, filter_pred) per seek — once for the entity-existence check, once for the forward-attr slice, and once more per ref value's visibility check. The materialization also populated a per-entity lookup Hashtbl and ran a final attr sort even though the eavt slice already emits attr-ordered groups. - datoms_by_entity / datoms_by_entity_attr in the entity and pull contexts now use bounded PSet.slice seeks when duplicate bookkeeping and filter_pred are absent; the full accessor remains otherwise. - A materialized entity answers forward-attr lookups via List.assoc_opt on its attrs instead of copying every attr into lookup_cache. - sorted_forward_entity_attrs drops the redundant sort. Divergence from upstream for speed: same observable values — the index slice and the accessor return identical datoms when the guard applies. get-page-data ~0.72 -> ~0.41ms; q/pull metrics unchanged.
A pull-in-find query ran result-row dedup with every pulled row hashing to the same bucket (query_result_hash Result_pull _ = 1), so the O(n) hash dedup degraded into pairwise deep compares of full pulled entities — O(n²) on q143's [(pull ?b [*]) ...] over 2000 blocks. - query_result_hash now hashes the pulled entity id (same eid + same selector always yields the same pulled content, so equality checks still decide correctness). - collect_find_specs / aggregate_rows memoize pull results per (selector object, entity id) for the row loop: repeated entities pay one pull each; Find_pull_form selectors are parsed once per loop instead of once per row. Divergence from upstream for speed: identical result sets — pull is deterministic per (db, selector, eid) inside one evaluation. logseq_queries.edn on the synthetic Logseq db (lq_prof): q143 44.6 -> ~10.1ms (-77%); q132/q176 unchanged (their cost is where-eval, not pull).
missing?, get-else, get-some and friends probe (e, a) datoms per row through the full datoms accessor (arg resolution, seq machinery, duplicate merge, filter_pred). The query-evaluator context now routes bounded eavt lookups through the lean index slices when duplicate bookkeeping and filter_pred are absent — identical datoms, ~half the per-row cost of a [(missing? $ ?p attr)] clause over 2k rows.
missing?/get-else/get-some probe (e, a) presence through attr_value_for_query once per row; rows sharing an entity repeat the same lookup. A per-db Hashtbl memo (reset when the db object changes — dbs are immutable so results stay valid) collapses those to one probe per distinct (e, a): the missing? clause over ~2k joined rows with ~20 distinct entities drops from ~2.9ms to ~1.7ms.
A deterministic *Value clause (string transforms, identity) whose input terms only reference vars bound in the single seed binding evaluates to the same extension for every row — in Logseq search queries that is [(lower-case ?query) ?lower] once per candidate block. Hoist such clauses to one evaluation on the seed before the clause machinery runs; printing, random and user-callable clauses stay per-row. q160 -4.9→-3.9ms (-20%).
pattern_datoms on QAttr with a bound entity ran the full aevt accessor per probe — the same-entity relation path issues one such probe per candidate entity (has_pattern / verify clauses). Route e-bounded attr probes to the entity_attr_index_datoms slice under the usual duplicate/filter_pred guards. Same-entity joins (title+page, uuid+page) drop from ~3.3ms to ~1.9ms (-40%); stream_join's right side also folds without seq allocation.
Same last-occurrence-wins semantics; the fold reuses the caller's pull_key sort downstream so the intermediate order is unobservable. pull-wildcard ~3.2->~2.9us/entity.
Or/SourceOr/OrJoin branches like (or [?v :block/title ?x] ...) went through eval_clauses / eval_relation_clauses, materializing the whole attr into a relation and joining — ~1ms per branch on a 286-binding outer. When a branch is a single pattern whose entity term is bound in every row, resolve it as one bounded index probe per row via eval_clause instead; fall back to the generic path otherwise. Or-join q110 (with attr arg bound) ~2.2ms -> ~44us on the Logseq fixture; same guard applies to SourceClause and not-join paths that share eval_projected_clauses_for_bindings.
Rule clauses whose bodies are plain clauses expand in place before the entity-set collect: params substitute the invocation terms, body-local vars get fresh #N names so they cannot collide with the outer query, and multiple same-name rules wrap expansions in an Or. Nested invocations expand recursively (depth-capped); recursive or unhandled bodies leave the Rule clause behind so the collect falls back to the generic engine unchanged. Measured: (self-ref ?b ?r) 226->153us, (page ?b ?name) -12%, recursive rules verified to stay on the generic path.
[?e ?attr], [?b ?p ?v] etc. where the attr var is input-bound to a concrete attr name now rewrite to plain QAttr patterns at matcher entry — the whole fast-path family (entity sets, n-var finds, value projection) applies unchanged. Unbound attr vars still go generic. Measured: [?b ?p ?v] add-cli query 2520->97us, [?e ?attr] -65%.
[(get-else $ ?e :attr ?default) ?v] binds ?v once — it is a free projection, not an entity constraint. The collect skips such clauses when the output var is single-use, and value_var_projection emits one scalar per entity: the attr value when present, else the (constant or input-bound) default. Refs unwrap to Result_entity. Measured: get-else coll/rel -20~24% on the page fixture, all MATCH.
[?e :attr ?v] with ?v bound by a [?v ...] collection input is the set of entities whose attr value is a member — evaluated as one aevt scan with a hash-set membership check instead of a full join pass. [(contains? ?coll ?e)] stays generic: it is a filter on an already-bound var, not a generator — the clause-order-dependent unbound-key drop cannot be expressed as an entity-set entry. Measured: [?b :block/tags ?cls] coll input 89->26us (-71%), MATCH.
[(clojure.string/includes? ?v ?needle)] and the starts-with / ends-with variants, plus [(= ?v c)] and [(not= ?v c)] with a resolved constant, become a `Str entry: one aevt scan keeping datoms whose string value satisfies every predicate bound to that var. The block-content rule — [?b :block/title ?c] [(includes? ?c ?query)] — hits this after rule inlining. Measured: includes?-title scan -66%; eq/not-eq/in/out MATCH.
[(contains? ?coll ?v)] with ?v bound by an earlier [for_var :attr ?v] pattern folds into the entity-set machinery, following value_contains exactly: Set -> member set, Map -> key set, List/Vector/Tuple -> Int64 index range [0,len), scalars -> empty, collection-input elements -> union of each element contribution, multiple contains? -> intersection. A contains? before the binding pattern drops all rows, matching the generic left-to-right eval. Measured: contains?-on-title -79%; all 12 probes MATCH incl. the non-vacuous set-membership case (4=4).
…losure Three linked extensions that together cover the expanded logseq property/task rule family: - resolve_ident_entity_vars: a var bound by [?x :db/ident K] where K is constant resolves to its (unique) entity; entity-position uses become QEntity e, the ident clause drops out, and clauses left without variables are evaluated as ground checks (missing ident or failed check => empty result). Attr/value positions stay unresolved, matching the generic engine which does not resolve entities in attr position. - or-bound contains?: a var bound only inside single-pattern or branches [for_var :attr var] and constrained by contains? folds into `Or of `InColl branches (contains_entry_of computes the allowed-value intersection once). - cross-ref component closure: the clauses constraining a ref edge value var are its connected component (a contains? on a var bound only inside the or counts too); the inner collect is trusted for soundness and v must not be a find var. Consumed clauses are skipped in the outer walk. Measured: ident+missing?-ground+or+contains? end-to-end 155=155 MATCH, -22%; all prior probes still MATCH.
After ident resolution, an or branch can carry var-free clauses ([?pe :block/tags :Property] with ?pe substituted). Evaluate them per branch: a ground clause that fails kills its branch, a branch reduced to all-true clauses constrains nothing so the whole or is always satisfied and contributes no entity filter. Also map [(missing? $ ?e :a)] inside a branch to a `Neg entry, matching the top-level handling. Probes: multi-clause branches, missing?-in-branch, ground-in-branch all MATCH fast-vs-generic.
Upstream binds the cross var through the ref edge and then filters it, so the inner candidate universe is the entities referenced by the edge attr — not the whole db. Two consequences: - ValueIn now seeds eval_collected with the edge targets. That is both cheaper (the seed is ~20 pages, not every entity) and makes inner sets that are pure Neg correct: [?b :page ?p] [(missing? $ ?p :x)] previously could not collect (unseeded all-Neg -> empty) and fell back to the generic engine at ~3ms; now the positive-binding guard is gone and the same query lands the fast path at ~163us (-94%). - try_ref_target builds src->targets maps in one aevt pass per edge attr instead of a per-src eavt accessor call (2105 lookups -> one scan); ref-target pull 1790us -> 893us. - ref_target_pull_relation likewise replaces a per-datom entity_has_attr accessor call with one aevt pass building a has-missing-attr bitmap.
The entity-set fast paths built over the last rounds only fired for q_return callers — Datascript.q (the parsed-query entry used by the edn runner and real consumers) went straight to the generic engine. q now shadows its earlier binding inside Query with the same ladder q_return uses, unwrapping each result back to rows. Guards that keep the ladder semantics-exact for AST-built queries that skip the parser: - inputs arity is checked the same way bind_query_inputs does, so too-many/too-few :in arguments still raise via the generic path - not/missing? clauses must have their vars bound by preceding clauses (in-order walk, or-branches checked with a bound-set copy) - or-branches must bind the same free-var set on every branch - reverse attrs (_a) are understood by the set/value/presence helpers instead of silently matching nothing: an entity "has" _a when some other entity references it through a Measured through q (was → now): q176 6231→117us, q188 1939→95us, q132 3859→940us, q186 2637→1524us, q184 2540→1536us, q143 8987→7065us. Full probe suite fast-vs-generic: 0 mismatches; test suite green.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An autoresearch-style optimization pass over the transact and query paths — ~50 measured rounds, each keeping only changes that improved a real metric without regressing others. The storage/serialization format is byte-for-byte unchanged (native/jsoo/cljs parity hashes identical), the full test suite is green, and every query fast path is verified row-for-row equivalent to the generic engine (0 mismatches across the probe suite).
Measured on a synthetic Logseq-shaped db (2000 blocks / ~14.6k datoms), vs
main:Transact
add-1(single-datom tx): 21.4 → ~16.1ms (-25%) — skips the tempid machinery when no tempids exist, early-out for small txs, staged effective view forRetractopsretract+add(attr-update txs): 13-19µs → ~7.4µs (-40%)add/removeemit nodes directly on the no-split path (pin bump toperf/add-insert-fast-path)Query — an entity-set fast-path ladder (
simple_attr_entity_ids+ collection/relation/scalar/pull projections) now wired intoDatascript.qitself, not justq_return:q176(join + missing? + range + pull): 14ms → ~120µs (-99%)q132(join + missing? + pull): 10ms → ~940µs (-91%)q188(uuid + page): 3.2ms → ~95µs (-97%)q143((pull ?b [*])over all blocks): 44.6 → ~7.1ms (-84%)q4same-entity 4-clause: -75%;get-page-data: -43%or/contains?/[?x :db/ident K]/cross-ref edges, incl.(task)/(property)shapes): collected throughexpand_rule_clauses+ branch-local ground evalSemantics preserved for AST-built queries that bypass the parser:
:inarity,not/missing?var-boundness ordering,orfree-var agreement, and reverse attr (_a) evaluation all fall through to the generic engine (which raises) or are evaluated identically.The experiment ledger lives in
.auto/(prompt.md,ideas.md,log.jsonl) — each round has a keep/revert record with its before/after measurement.Link to Devin session: https://app.devin.ai/sessions/a699c982a95844238c1dcd1ff6cc8d5d
Open in Devin Desktop: https://app.devin.ai/desktop/session/a699c982a95844238c1dcd1ff6cc8d5d?variant=devin
Requested by: @tiensonqin