Repository navigation
docs: from-code interview brief, role prep, and one recorded limitation - #16
Merged
Merged
Conversation
…tions Found during a from-code audit of the recorded run, not during the run. All 5 L1 questions are `manual`; all 17 `graph_reverse` questions are L2 or L3, and their ground-truth sets are about twice as large (mean 4.29 targets across 2.94 files, against 2.12 across 1.38). So "B4 leads on L3" and "B4 leads on the reverse-constructed batch" are not distinguishable with this suite -- the two dimensions are collinear. Measured, so the entry states a direction rather than only a worry: the graph's isolated margin (B4 - B3.5) is +0.028 on the manual L2/L3 subset against +0.018 on `graph_reverse`, and the share of questions citing at least one graph-origin chunk is 9/16 manual against 10/17 `graph_reverse`. The confound does not appear to run in the direction that would flatter B4. The entry says explicitly that this measures whether the questions that exist favour the graph, and cannot measure the flows the graph misses. Prefixed ADDED AFTER THE RUN because line 2 of this file claims it was recorded immediately after the process completed, and that claim has to stay true. Nothing was re-executed: the figures are recomputed from bytes already in this directory and no recorded metric changed. Safe against the artifact drift check: provenance.json is not harness output (dcode_eval.run writes six other files), and scripts/sync_eval_artifacts.py reads only `display`, `groundedness_guardrail`, `verdict_written_at` and `sparse_retrieval` from it -- never `limitations`. `make lint` passes.
1235 lines rebuilt from the source, not from the existing docs. Every figure and every file:line was re-executed before being written down, and each number carries a source level: A = committed artifact field, B = committed source constant, C = recomputed this pass, D = depends on the running Postgres or the local .env and cannot be checked from the repo. Structure: current-state inventory; both end-to-end chains hop by hop with the cross-boundary points marked; the nine core algorithms in intuition -> LaTeX -> code mapping -> parameters -> complexity -> failure modes; a quotable-number ledger where every row also carries the sentence that must be said alongside it; and an unquotable list for the figures that are structurally misleading on their own. Four things the audit turned up that the docs did not have: - `chunks.tsv` and its GIN index are written by nothing. Sparse retrieval is application-level Okapi BM25. The column is a leftover, not a feature. - `@overload` stubs shadow real definitions. `_build_symbols` dedupes by qualified name and keeps the first record, which is the first overload stub -- 9 symbols and 20 chunks (2.8%) point at a three-line `...` body. Verified live: `iter_slices` resolves to line 614, not 621. - The recorded run ran on an older call graph. Re-running the current worker code offline over the same corpus produces 316 call edges against the 303 in the database (+4.3%); the other three edge types are identical, which localises the difference to call resolution alone. - The call-edge funnel, measured rather than asserted: 2615 ast.Call sites visited, 316 resolved, 12.1%. The largest unresolved class is 1076 `obj.X()` receivers with no known type -- the capability this analysis explicitly declines. Appendix A carries 25 drift entries with a suggested fix for each. Two of them were closed while writing this: the taxonomy x source confound (see the preceding commit) and the missing provenance for the a11y live-region regression, which git bisects to 09e2446 adding it and 7bc6abe removing it with the retired query page. Read-only pass: no code, no evaluation artifact and no run was touched beyond the one limitations entry committed separately.
…neering Same facts as DCODE_TECH_BRIEF.md, reorganised by what a hiring side actually evaluates rather than by how the system is built. The brief stays the source of truth; this is a view over it, and appendix B maps every section back so a follow-up question can be traced. Narrative order is deliberate: honesty contract -> bounded-agent control -> evaluation methodology. The call graph is not the opener. It is the part my own ablation measured down, so leading with it means opening on the weakest link; its correct place is as evidence that I do not overrate a component because I spent effort on it. The non-ML boundary is stated explicitly rather than left ambiguous. There is no training, no loss, no RL, no fine-tuning -- every model is an off-the-shelf call. The static call-graph resolution has real difficulty but it is program analysis, not machine learning, and section 3.5 answers "is this an algorithms project?" with "no" and then gives the positioning I do stand behind: evaluation and verifiability infrastructure for agents, which is an upstream dependency of the training side rather than a substitute for it. Two defects carried over from the brief are fixed here, and appendix C records all seven such corrections: - "multi-step evidence gathering is six times the call graph" only holds on L3 (+0.1466 against +0.0226). On L2 the two are indistinguishable (+0.0221 against +0.0218, a ratio of 1.01). Every occurrence of "six times" now carries the L3 qualifier, and the cheat sheet's ablation table lists both rows so the L2 line cannot be dropped by accident. - The brief's eight-minute version quoted httpx figures that were not in its own ledger. They are now measured and added to appendix A at level D, with the constraints that make them safe to say: 963 is the sum of four edge types (calls is 398), the figures depend on a running Postgres and cannot be checked from the repo, and they enter no conclusion. Also new: a section on what to say when a number will not come to mind, which substitutes derivation and magnitude for the exact value -- the failure mode it addresses is going vague under pressure, which reads worse than saying how the number is computed and offering to look it up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two interview-preparation documents rebuilt from the source, plus one limitation
the audit that produced them turned up.
Nothing in
apps/,packages/,infra/orscripts/changes. The onlynon-
docs/change is a single entry appended to one run'slimitationsarray.What is here
f5d0088docs(eval)— record the taxonomy x source confound in the recorded run's limitations23568f4docs(interview)—DCODE_TECH_BRIEF.md, 1235 lines, rebuilt from code9faa279docs(interview)—DCODE_AGENT_ROLE_PREP.md, 615 lines, the same facts reorganised for a roleMethod
Every figure and every
file:linewas re-executed before being written down.Each number carries a source level: A = committed artifact field, B =
committed source constant, C = recomputed this pass, D = depends on the
running Postgres or the local
.envand cannot be checked from the repo. Theledger also gives, per number, the sentence that has to be said alongside it —
and an explicit list of figures that are misleading on their own and should not
be quoted at all.
What the audit found
Four things the existing docs did not have:
chunks.tsvand its GIN index are written by nothing. Sparse retrieval isapplication-level Okapi BM25 in
dcode_shared.bm25. The column is a leftoverfrom before
ts_rankwas found not to be BM25, and it currently costs anindex maintained on every INSERT for a column that is always NULL.
@overloadstubs shadow real definitions._build_symbolsdedupes byqualified name and keeps the first record, which is the first overload stub.
9 symbols and 20 chunks (2.8%) resolve to a three-line
...body — verifiedlive:
iter_slicesresolves to line 614, not 621.code offline over the same corpus produces 316 call edges against the 303 in
the database (+4.3%). The other three edge types are byte-identical, which
localises the difference to call resolution alone and confirms — with a
number — what
provenance.jsonalready said qualitatively.ast.Callsitesvisited, 316 resolved, 12.1%. The largest unresolved class is 1076
obj.X()receivers with no known type — the capability this analysis explicitly
declines.
Two claims in the existing artifacts were checked and hold: the
graph_tools.pytable (B3.5 scoring 0/0/0 against B4's 4/3/5 under thecorrected origin set) reproduces digit for digit from committed bytes, and the
three retrieval metrics reproduce by hand on q-009.
Appendix Aof the brief carries 25 drift entries with a suggested fix foreach. Two of them were closed while writing it — the confound in
f5d0088, andthe missing provenance for the a11y live-region regression, which bisects to
09e2446adding it and7bc6aberemoving it along with the retired query page.About the limitations entry
provenance.jsonis not harness output —dcode_eval.runwrites six otherfiles and never this one — so it is hand-maintained and safe to append to.
scripts/sync_eval_artifacts.pyreads onlydisplay,groundedness_guardrail,verdict_written_atandsparse_retrievalfrom it,never
limitations, so the artifact drift check is unaffected.The entry is prefixed
ADDED AFTER THE RUNbecause line 2 of that file claimsit was recorded immediately after the process completed, and that claim has to
stay true. Nothing was re-executed; the figures are recomputed from bytes
already in the directory and no recorded metric changed.
Checks
make lintpasses end to end — ruff, eslint, andsync_eval_artifacts.py --checkall green.Read-only otherwise: no code, no evaluation artifact and no run was touched
beyond that one entry.