Skip to content

docs: from-code interview brief, role prep, and one recorded limitation - #16

Merged
Odiethebest merged 3 commits into
mainfrom
docs/interview-tech-brief
Aug 1, 2026
Merged

Odiethebest merged 3 commits into
mainfrom
docs/interview-tech-brief

Conversation

@Odiethebest

Copy link
Copy Markdown
Owner

Two interview-preparation documents rebuilt from the source, plus one limitation
the audit that produced them turned up.

Nothing in apps/, packages/, infra/ or scripts/ changes. The only
non-docs/ change is a single entry appended to one run's limitations array.

What is here

Commit
f5d0088 docs(eval) — record the taxonomy x source confound in the recorded run's limitations
23568f4 docs(interview) — DCODE_TECH_BRIEF.md, 1235 lines, rebuilt from code
9faa279 docs(interview) — DCODE_AGENT_ROLE_PREP.md, 615 lines, the same facts reorganised for a role

Method

Every figure and every file:line was re-executed before being written down.
Each number carries a source level: A = committed artifact field, B =
committed source constant, C = recomputed this pass, D = depends on the
running Postgres or the local .env and cannot be checked from the repo. The
ledger also gives, per number, the sentence that has to be said alongside it —
and an explicit list of figures that are misleading on their own and should not
be quoted at all.

What the audit found

Four things the existing docs did not have:

  • chunks.tsv and its GIN index are written by nothing. Sparse retrieval is
    application-level Okapi BM25 in dcode_shared.bm25. The column is a leftover
    from before ts_rank was found not to be BM25, and it currently costs an
    index maintained on every INSERT for a column that is always NULL.
  • @overload stubs shadow real definitions. _build_symbols dedupes by
    qualified name and keeps the first record, which is the first overload stub.
    9 symbols and 20 chunks (2.8%) resolve to a three-line ... body — verified
    live: iter_slices resolves to line 614, not 621.
  • The recorded run ran on an older call graph. Re-running the current worker
    code offline over the same corpus produces 316 call edges against the 303 in
    the database (+4.3%). The other three edge types are byte-identical, which
    localises the difference to call resolution alone and confirms — with a
    number — what provenance.json already said qualitatively.
  • The call-edge funnel, measured rather than asserted: 2615 ast.Call sites
    visited, 316 resolved, 12.1%. The largest unresolved class is 1076 obj.X()
    receivers with no known type — the capability this analysis explicitly
    declines.

Two claims in the existing artifacts were checked and hold: the
graph_tools.py table (B3.5 scoring 0/0/0 against B4's 4/3/5 under the
corrected origin set) reproduces digit for digit from committed bytes, and the
three retrieval metrics reproduce by hand on q-009.

Appendix A of the brief carries 25 drift entries with a suggested fix for
each. Two of them were closed while writing it — the confound in f5d0088, and
the missing provenance for the a11y live-region regression, which bisects to
09e2446 adding it and 7bc6abe removing it along with the retired query page.

About the limitations entry

provenance.json is not harness output — dcode_eval.run writes six other
files and never this one — so it is hand-maintained and safe to append to.
scripts/sync_eval_artifacts.py reads only display,
groundedness_guardrail, verdict_written_at and sparse_retrieval from it,
never limitations, so the artifact drift check is unaffected.

The entry is prefixed ADDED AFTER THE RUN because line 2 of that file claims
it was recorded immediately after the process completed, and that claim has to
stay true. Nothing was re-executed; the figures are recomputed from bytes
already in the directory and no recorded metric changed.

Checks

make lint passes end to end — ruff, eslint, and
sync_eval_artifacts.py --check all green.

Read-only otherwise: no code, no evaluation artifact and no run was touched
beyond that one entry.

…tions

Found during a from-code audit of the recorded run, not during the run.
All 5 L1 questions are `manual`; all 17 `graph_reverse` questions are L2 or
L3, and their ground-truth sets are about twice as large (mean 4.29 targets
across 2.94 files, against 2.12 across 1.38). So "B4 leads on L3" and "B4
leads on the reverse-constructed batch" are not distinguishable with this
suite -- the two dimensions are collinear.

Measured, so the entry states a direction rather than only a worry: the
graph's isolated margin (B4 - B3.5) is +0.028 on the manual L2/L3 subset
against +0.018 on `graph_reverse`, and the share of questions citing at
least one graph-origin chunk is 9/16 manual against 10/17 `graph_reverse`.
The confound does not appear to run in the direction that would flatter B4.
The entry says explicitly that this measures whether the questions that
exist favour the graph, and cannot measure the flows the graph misses.

Prefixed ADDED AFTER THE RUN because line 2 of this file claims it was
recorded immediately after the process completed, and that claim has to
stay true. Nothing was re-executed: the figures are recomputed from bytes
already in this directory and no recorded metric changed.

Safe against the artifact drift check: provenance.json is not harness
output (dcode_eval.run writes six other files), and
scripts/sync_eval_artifacts.py reads only `display`,
`groundedness_guardrail`, `verdict_written_at` and `sparse_retrieval` from
it -- never `limitations`. `make lint` passes.
1235 lines rebuilt from the source, not from the existing docs. Every
figure and every file:line was re-executed before being written down, and
each number carries a source level: A = committed artifact field, B =
committed source constant, C = recomputed this pass, D = depends on the
running Postgres or the local .env and cannot be checked from the repo.

Structure: current-state inventory; both end-to-end chains hop by hop with
the cross-boundary points marked; the nine core algorithms in
intuition -> LaTeX -> code mapping -> parameters -> complexity -> failure
modes; a quotable-number ledger where every row also carries the sentence
that must be said alongside it; and an unquotable list for the figures that
are structurally misleading on their own.

Four things the audit turned up that the docs did not have:

- `chunks.tsv` and its GIN index are written by nothing. Sparse retrieval
  is application-level Okapi BM25. The column is a leftover, not a feature.
- `@overload` stubs shadow real definitions. `_build_symbols` dedupes by
  qualified name and keeps the first record, which is the first overload
  stub -- 9 symbols and 20 chunks (2.8%) point at a three-line `...` body.
  Verified live: `iter_slices` resolves to line 614, not 621.
- The recorded run ran on an older call graph. Re-running the current
  worker code offline over the same corpus produces 316 call edges against
  the 303 in the database (+4.3%); the other three edge types are
  identical, which localises the difference to call resolution alone.
- The call-edge funnel, measured rather than asserted: 2615 ast.Call sites
  visited, 316 resolved, 12.1%. The largest unresolved class is 1076
  `obj.X()` receivers with no known type -- the capability this analysis
  explicitly declines.

Appendix A carries 25 drift entries with a suggested fix for each. Two of
them were closed while writing this: the taxonomy x source confound (see
the preceding commit) and the missing provenance for the a11y live-region
regression, which git bisects to 09e2446 adding it and 7bc6abe removing it
with the retired query page.

Read-only pass: no code, no evaluation artifact and no run was touched
beyond the one limitations entry committed separately.
…neering

Same facts as DCODE_TECH_BRIEF.md, reorganised by what a hiring side
actually evaluates rather than by how the system is built. The brief stays
the source of truth; this is a view over it, and appendix B maps every
section back so a follow-up question can be traced.

Narrative order is deliberate: honesty contract -> bounded-agent control
-> evaluation methodology. The call graph is not the opener. It is the part
my own ablation measured down, so leading with it means opening on the
weakest link; its correct place is as evidence that I do not overrate a
component because I spent effort on it.

The non-ML boundary is stated explicitly rather than left ambiguous. There
is no training, no loss, no RL, no fine-tuning -- every model is an
off-the-shelf call. The static call-graph resolution has real difficulty
but it is program analysis, not machine learning, and section 3.5 answers
"is this an algorithms project?" with "no" and then gives the positioning
I do stand behind: evaluation and verifiability infrastructure for agents,
which is an upstream dependency of the training side rather than a
substitute for it.

Two defects carried over from the brief are fixed here, and appendix C
records all seven such corrections:

- "multi-step evidence gathering is six times the call graph" only holds
  on L3 (+0.1466 against +0.0226). On L2 the two are indistinguishable
  (+0.0221 against +0.0218, a ratio of 1.01). Every occurrence of "six
  times" now carries the L3 qualifier, and the cheat sheet's ablation
  table lists both rows so the L2 line cannot be dropped by accident.
- The brief's eight-minute version quoted httpx figures that were not in
  its own ledger. They are now measured and added to appendix A at level
  D, with the constraints that make them safe to say: 963 is the sum of
  four edge types (calls is 398), the figures depend on a running Postgres
  and cannot be checked from the repo, and they enter no conclusion.

Also new: a section on what to say when a number will not come to mind,
which substitutes derivation and magnitude for the exact value -- the
failure mode it addresses is going vague under pressure, which reads worse
than saying how the number is computed and offering to look it up.
@Odiethebest
Odiethebest merged commit 11d0b67 into main Aug 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant