Agentic Documentation System

A local, no-API-cost agent that reads a codebase, drafts documentation, pauses for human approval, and incrementally re-documents only what actually changed. Built in raw Python with Ollama — no agent frameworks (no LangGraph).

Why

Documentation for a live codebase isn't a one-time artifact — code keeps changing, and regenerating everything on every change doesn't scale. This system treats documentation as a pipeline with a human checkpoint in the middle: draft → pause → approve/edit/reject → commit, with per-batch granularity so a single wrong section never forces regenerating a whole file.

Stack

Python · Ollama (qwen2.5-coder:3b-instruct-q4_K_M) · ast module for code parsing · Windows Task Scheduler for automation

Fully local, CPU-only (16GB RAM, no GPU).

Architecture

[READ] → [SPLIT via ast] → [BATCH] → [DRAFT via Ollama] → [SAVE state to disk, exit]
                                                                  ↓
                                            (human reviews — process is not running)
                                                                  ↓
                                    [RESUME: --approve / --edit / --reject] → [COMMIT to docs.md]

State lives on disk as JSON between drafting and human review — no background process waits for input. Resuming is just re-running the script with a flag.

Key engineering decisions

• Model selection benchmarked, not assumed: three local models tested on the same task. qwen2.5-coder:3b chosen over a same-size general model (which fabricated a function's return values) and a 7B coder model (accurate but too slow for iterative use).
• AST-based chunking, not full-file context: feeding an entire file to the model caused confident hallucination (local Ollama calls described as "external API calls"). Splitting into function/class-level chunks via Python's ast module eliminated it — smaller, focused context is what makes the local model reliable.
• Per-section reject, not full regeneration: rejecting one section re-runs only that section's model call with feedback appended to its own history; every other section stays untouched. Cuts a correction from ~25 minutes to about one batch's drafting time.
• Hash-based incremental re-documentation: each committed section carries a content hash. Unchanged code is skipped entirely — zero model calls, zero cost — and only sections whose underlying code actually changed get redrafted.
• Import graph as free structural context: a static ast pass builds a project-wide dependency graph (who imports whom, both directions) with zero model calls. Injecting a one-sentence structural fact ("this file is imported by X, Y, Z") into the prompt measurably improved the model's ability to describe a function's role in the system — without needing multi-file context.
• Fixed-schedule automation over debounce: chose scheduled runs (Task Scheduler, twice daily) over triggering on every file save, to avoid resource contention with active development on a RAM-constrained machine.

Tiered model architecture (the core finding)

Different documentation tasks need different model capabilities, and those capabilities don't scale together:

Task	Context needed	Result with 3B model
Per-function documentation	Small, single-subject	Reliable — in production
Module-level overview (constants, scripts)	Medium, structure vs. content	Unreliable — hallucinates
System-level synthesis (cross-file)	Large, architectural reasoning	Unreliable — hallucinates

The system is built to accommodate this: each tier is a separate batch type in the pipeline. The first tier runs today; the other two are designed as slots — a runtime warning fires when a task needing tier 2/3 is detected, and upgrading is a one-line model swap for that batch type, not a redesign.

Known limitations (documented, not hidden)

• Module-level and cross-file synthesis with the current 3B model is disabled, not fixed — it reliably hallucinates on large, heterogeneous, or purely structural content (turned a variable into an invented class, answered embedded test questions instead of documenting them). Code is retained as a marker for where a larger model slots in later.
• Captures import relationships, not call relationships — doesn't know if an imported module is used in one function or ten.
• Python-specific by construction (ast is Python's parser). Other languages need a pluggable parsing layer, not a rewrite of the splitting logic.
• No email/notification loop yet — a scheduled run drafts and saves a summary file, but a human still has to check it and approve manually. Human-in-the-loop is preserved by design; auto-approval was never the goal.

Setup

pip install ollama
ollama pull qwen2.5-coder:3b-instruct-q4_K_M

python doc_agent.py --draft-folder /path/to/project   # discover, chunk, draft
python doc_agent.py --approve                          # commit all pending sections
python doc_agent.py --edit <batch_id> "feedback"        # revise one section
python doc_agent.py --reject <batch_id> "reason"         # redraft one section only

Status

Core pipeline (AST chunking, per-batch drafting, disk-based pause/resume, per-section reject, hash-based incremental updates, multi-file support with import-graph context, scheduled automation) is built and verified against a real six-file production codebase. Module-overview and system-level synthesis are deliberately parked pending a larger model, per the tiered architecture above.