diff --git a/.github/workflows/verify-examples.yml b/.github/workflows/verify-examples.yml new file mode 100644 index 0000000..e2b25df --- /dev/null +++ b/.github/workflows/verify-examples.yml @@ -0,0 +1,17 @@ +name: verify-examples + +on: + push: + branches: [master, develop] + pull_request: + +jobs: + verify: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Verify every good example against its source root + run: bash scripts/verify-examples.sh diff --git a/README.md b/README.md index bd1716c..f47ffb7 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,7 @@ Pure Python stdlib — no install step, runs anywhere Python 3.10+ is available. | Prompt | Purpose | |--------|---------| +| `00-verification-core.md` | Canonical verification tags + verifier step (shared by all prompts) | | `01-architecture-overview.md` | System components & structure (with framework detection) | | `01a-overlay-model-systems.md` | Additional detection for ML/AI model systems | | `02-code-flows.md` | Execution path tracing | @@ -93,21 +94,22 @@ Pure Python stdlib — no install step, runs anywhere Python 3.10+ is available. ## Framework-Specific Examples -Each framework has its own mini reference app and good/bad documentation examples: +Each framework has its own mini reference app (or vendored source) and good/bad documentation examples: -| Framework | Mini App | Description | -|-----------|----------|-------------| -| Laravel | `examples/laravel/slotbooker/` | Booking system with MVC, events, services | +| Framework | Source | Description | +|-----------|--------|-------------| +| Laravel | `examples/laravel/slotbooker/` | Booking system with MVC, events, services (+ code-flow example pair) | | FastAPI | `examples/fastapi/tasktracker/` | Task management API with repositories, Pydantic | -| React | `examples/react/` | *(coming soon)* | -| Vue | `examples/vue/` | *(coming soon)* | -| Livewire | `examples/livewire/` | *(coming soon)* | -| Flask | `examples/flask/` | *(coming soon)* | +| React | `examples/react/expense-tracker/` | SPA with hooks, routing | +| Next.js | `examples/nextjs/linkboard/` | App Router: server/client components, API routes | +| Vue | `examples/vue/kanban-board/` | Pinia store, optimistic updates | +| Livewire | `examples/livewire/approval-flow/` | Laravel + Livewire components | +| Model-centric (ML/AI) | `examples/model-systems/whisper/` | Vendored openai/whisper source — pairs with the `01a` overlay | -Each framework folder contains: -- A mini reference app demonstrating that framework's patterns -- `good-architecture-doc-example.md` - Properly verified documentation -- `bad-architecture-doc-example.md` - Common hallucination patterns to avoid +Each folder contains: +- The mini app / vendored source the docs cite +- `good-architecture-doc-example.md` - Properly verified documentation (**must pass `verify.py`** — enforced by `scripts/verify-examples.sh` in CI) +- `bad-architecture-doc-example.md` - Common hallucination patterns, annotated with ❌ callouts explaining each failure ### Package/Library Examples @@ -126,33 +128,31 @@ Package examples are in `examples/packages/{package}/` with the same good/bad do ``` agent-system-mapper/ ├── prompts/ # AI agent prompts (what gets installed) +│ ├── 00-verification-core.md # Canonical tags + verifier step (shared) │ ├── 01-architecture-overview.md # With framework detection │ ├── 01a-overlay-model-systems.md # ML/AI model detection overlay │ ├── 02-code-flows.md │ ├── 02a-recommend-code-flows.md # Analyze & recommend flows │ ├── 03-data-models.md │ ├── 04-diagrams.md -│ └── 05-test-surface.md # Test candidates from flows -├── examples/ # Framework-specific examples -│ ├── laravel/ -│ │ ├── slotbooker/ # Laravel mini app -│ │ ├── good-architecture-doc-example.md -│ │ └── bad-architecture-doc-example.md -│ ├── fastapi/ -│ │ ├── tasktracker/ # FastAPI mini app -│ │ ├── good-architecture-doc-example.md -│ │ └── bad-architecture-doc-example.md -│ ├── react/ # (coming soon) -│ ├── vue/ # (coming soon) -│ ├── livewire/ # (coming soon) -│ ├── flask/ # (coming soon) -│ ├── packages/ # Library/package examples -│ │ └── requests/ # Python HTTP client -│ └── test-surface/ # Test surface examples (framework-agnostic) -│ ├── good-test-surface-example.md -│ └── bad-test-surface-example.md -├── guides/ # Methodology guides -│ └── 01-architecture-overview.md +│ ├── 05-test-surface.md # Test candidates from flows +│ └── lsp/ # LSP-optimized variants +├── examples/ # Every good example must pass verify.py (CI-enforced) +│ ├── laravel/ # slotbooker mini app + architecture AND code-flow pairs +│ ├── fastapi/ # tasktracker mini app + pair +│ ├── react/ # expense-tracker mini app + pair +│ ├── nextjs/ # linkboard mini app (App Router) + pair +│ ├── vue/ # kanban-board mini app + pair +│ ├── livewire/ # approval-flow mini app + pair +│ ├── model-systems/ # vendored openai/whisper + pair (01a overlay) +│ ├── packages/ +│ │ └── requests/ # vendored requests source + pair +│ ├── verifier/ # self-verifying example (documents verify.py) +│ └── test-surface/ # test surface pair (cites slotbooker) +├── skills/ # Claude Code slash commands (/map-arch, /map-verify, ...) +├── scripts/ +│ └── verify-examples.sh # CI guard: every good example must PASS +├── verify.py # Two-phase doc verifier └── install.sh # Installation script ``` @@ -160,7 +160,7 @@ agent-system-mapper/ ## Getting Started (Contributors) -1. Read the guides in `guides/` to understand the methodology +1. Read the prompts in `prompts/` to understand the methodology (`guides/` is superseded) 2. Examine mini apps in `examples/{framework}/` as reference implementations 3. Review good vs bad examples to understand hallucination patterns 4. Test prompts against mini apps to validate changes @@ -176,8 +176,10 @@ The architecture prompt auto-detects frameworks using these patterns: | Laravel | `composer.json` with `laravel/framework` | | FastAPI | `requirements.txt` with `fastapi` | | React | `package.json` with `react` | +| Next.js | `package.json` with `next` | | Vue | `package.json` with `vue` | | Livewire | Laravel + `livewire/livewire` in `composer.json` | -| Flask | `requirements.txt` with `flask` | +| Model-centric (ML/AI) | Weights files, `torch`/`transformers` deps — loads the `01a` overlay | -If your framework isn't supported yet, use Laravel examples as a baseline. +If your framework isn't supported yet, use the packages/requests examples as a +generic baseline (or Laravel for web frameworks) and adapt terminology. diff --git a/SKILLS.md b/SKILLS.md index f7b9b0f..59e78ef 100644 --- a/SKILLS.md +++ b/SKILLS.md @@ -38,9 +38,11 @@ curl -sL "https://raw.githubusercontent.com/peak-flow/agent-system-mapper/master | `/map-flows [name]` | Document a specific code flow | Architecture doc | | `/map-flows-lsp [name]` | Code flow (LSP, 60% fewer tokens) | Architecture doc + LSP | | `/map-recommend` | Recommend which flows to document | Architecture doc | +| `/map-recommend-lsp` | Flow recommendations (LSP-verified entry points + complexity) | Architecture doc + LSP | | `/map-data` | Document data models and schema | Prompts installed | | `/map-diagrams` | Generate Mermaid diagrams | At least one doc exists | | `/map-tests` | Derive test candidates from flows | Code flow doc | +| `/map-verify [doc]` | Verify citations + quoted code; a doc is done only at PASS | Any generated doc | --- @@ -53,17 +55,22 @@ The recommended workflow is: ↓ 2. /map-arch → Document architecture (or /map-arch-lsp) ↓ -3. /map-recommend → Get prioritized list of flows to document +3. /map-verify → Verify the doc (citations + quoted code); fix until PASS ↓ -4. /map-flows [name] → Document each recommended flow (or /map-flows-lsp) +4. /map-recommend → Get prioritized list of flows to document ↓ -5. /map-data → Document data models +5. /map-flows [name] → Document each recommended flow (or /map-flows-lsp) ↓ -6. /map-diagrams → Generate visual diagrams +6. /map-data → Document data models ↓ -7. /map-tests → Derive test candidates from flows +7. /map-diagrams → Generate visual diagrams + ↓ +8. /map-tests → Derive test candidates from flows ``` +Run `/map-verify` after every doc-producing step (arch, flows, data, tests) — +a doc is only "done" when `verify.py` exits 0. + --- ## Skill Details @@ -249,7 +256,7 @@ Derives test candidates from documented code flows. | Aspect | Standard (`/map-arch`) | LSP (`/map-arch-lsp`) | |--------|------------------------|----------------------| -| Token usage | 15-26k | 7-12k | +| Token usage | 8-12k (whole pipeline: 15-26k) | 4-6k (whole pipeline: 7-12k) | | Requirements | None | LSP server | | Best for | All languages | TypeScript, Python, PHP | | Tradeoff | Works everywhere | 50% fewer tokens | diff --git a/examples/fastapi/bad-architecture-doc-example.md b/examples/fastapi/bad-architecture-doc-example.md index 58bcb01..321504e 100644 --- a/examples/fastapi/bad-architecture-doc-example.md +++ b/examples/fastapi/bad-architecture-doc-example.md @@ -1,9 +1,13 @@ # TaskTracker Architecture Overview +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns. Each ❌ callout explains a failure. See good-architecture-doc-example.md for the correct approach. + ## What This System Does TaskTracker is a comprehensive project management API built with FastAPI. It allows users to create projects, manage tasks, assign work to team members, and track progress with notifications. +> **❌ PROBLEMS:** Pure uncited prose. No metadata block (commit, path, date), no verification summary, and not a single verification tag in the entire document. "Comprehensive" is marketing language, and "track progress with notifications" oversells reality — the only notification mechanism is a single outbound webhook (`app/services/notification_service.py:43-59`). A reader has no way to check any of this. + ## Components ### API Layer @@ -12,12 +16,16 @@ The system uses FastAPI routers to handle HTTP requests. Each resource has its o - Projects router manages project lifecycle - Tasks router handles task management with status updates +> **❌ PROBLEMS:** This is the trap of *plausible* hallucination — these routers happen to exist (`app/api/users.py:12`, `app/api/projects.py:13`, `app/api/tasks.py:13`), but nothing here is cited, so the reader cannot tell this section apart from the fabricated ones below. Uncited-but-true is indistinguishable from uncited-and-false. + ### Database Layer Uses SQLAlchemy ORM with async support for high-performance database operations. The models are well-structured with proper relationships: - User has many Projects and Tasks - Project has many Tasks - Task belongs to Project and User +> **❌ PROBLEMS:** "Async support" is fabricated. The engine is created with the synchronous `create_engine` (`app/core/database.py:11-14`) and a plain `sessionmaker` (`app/core/database.py:16`); every route handler is a sync `def`, not `async def` (e.g. `app/api/tasks.py:17`). "High-performance" is unfounded editorializing — the code itself flags "No connection pooling configured" (`app/core/database.py:10`). The relationship bullets are roughly right (`app/models/user.py:22-23`) but carry no citations, so they add no verifiable information. + ### Service Layer Business logic is handled by services: - TaskService handles task creation with notifications @@ -25,9 +33,13 @@ Business logic is handled by services: - UserService manages user authentication - NotificationService sends emails and push notifications +> **❌ PROBLEMS:** Half of this service roster is invented by naming symmetry. Only two services exist: `TaskService` (`app/services/task_service.py:14`) and `NotificationService` (`app/services/notification_service.py:13`). There is no `ProjectService` and no `UserService` — `app/services/` contains exactly `task_service.py`, `notification_service.py`, and `__init__.py`. "UserService manages user authentication" is doubly false: the class doesn't exist and neither does authentication. And `NotificationService` sends neither emails nor push notifications — it POSTs a JSON webhook via httpx (`app/services/notification_service.py:43-59`). + ### Authentication The API uses JWT authentication with refresh tokens. Users authenticate via the /auth/login endpoint and receive access tokens. +> **❌ PROBLEMS:** This entire section is fabricated. Searches for "jwt", "token", "login", and "auth" in `app/` and `main.py` return nothing. The only mounted routers are `/users`, `/projects`, and `/tasks` (`main.py:19-21`) plus `GET /health` (`main.py:24-25`) — there is no `/auth/login` endpoint and every endpoint is completely open. A good doc states this as a `[NOT_FOUND]` with the search terms; this doc invents the opposite. + ## Data Flow 1. Request comes in through FastAPI router @@ -37,6 +49,8 @@ The API uses JWT authentication with refresh tokens. Users authenticate via the 5. Repository layer handles database operations 6. Response is serialized via Pydantic models +> **❌ PROBLEMS:** Step-by-step execution tracing is a banned pattern in an architecture overview — prompt 01's Section 3 rules say "MUST NOT trace step-by-step execution"; detailed traces belong in the code-flows document. An architecture doc describes surfaces and what moves in tables (see Section 3 of the good example). This numbered walkthrough is also generic FastAPI boilerplate that would "document" any FastAPI app equally well, and it silently assumes every request goes through a service layer — in reality only the task routes use `TaskService` (`app/api/tasks.py:105`, `app/api/tasks.py:125`); user and project routes call repositories directly (`app/api/users.py:18`, `app/api/projects.py:19`). + ## External Integrations - **Email Service**: Sends transactional emails via SendGrid @@ -44,6 +58,8 @@ The API uses JWT authentication with refresh tokens. Users authenticate via the - **Webhook System**: Notifies external services of task events - **Redis Cache**: Caches frequently accessed data +> **❌ PROBLEMS:** Three of these four integrations do not exist. Searches for "sendgrid", "smtp", "firebase", "redis", and "cache" in `app/` return nothing. The only real integration is the webhook (`app/services/notification_service.py:43-59`), configured by `NOTIFICATION_WEBHOOK_URL` and `NOTIFICATION_ENABLED` (`app/core/config.py:17-18`). One true bullet buried in three hallucinated ones is worse than useless — the reader cannot tell which is which. + ## Key Patterns - Repository pattern for data access @@ -51,6 +67,8 @@ The API uses JWT authentication with refresh tokens. Users authenticate via the - Async/await for non-blocking operations - Pydantic models for validation +> **❌ PROBLEMS:** "Async/await for non-blocking operations" is false — there is not a single `async def` in `app/` or `main.py`; the notification path even uses a blocking `httpx.Client` (`app/services/notification_service.py:54`). The other three bullets are real (`app/repositories/task_repository.py:11`, `app/api/users.py:16`, `app/schemas/task.py:10`) but uncited, repeating the pattern of mixing verifiable truth and fabrication with no way to distinguish them. + ## Database Schema | Table | Description | @@ -60,3 +78,21 @@ The API uses JWT authentication with refresh tokens. Users authenticate via the | tasks | Tasks within projects | | notifications | Notification queue | | audit_log | Tracks all changes | + +> **❌ PROBLEMS:** Two of five tables are invented. Only three models declare `__tablename__`: users (`app/models/user.py:14`), projects (`app/models/project.py:22`), and tasks (`app/models/task.py:30`). There is no `notifications` table and no `audit_log` table — `app/models/` contains only `user.py`, `project.py`, `task.py`, and `__init__.py`. The users row is also wrong in detail: the User model has no password column, only id, email, name, and created_at (`app/models/user.py:16-19`) — consistent with the fact that no authentication exists. + +--- + +## Why This Example is BAD + +Each numbered item pairs the false claim with the verifiable reality in `examples/fastapi/tasktracker/`: + +1. **"JWT authentication with refresh tokens" and a `/auth/login` endpoint** → No authentication of any kind exists. The only mounted routers are `/users`, `/projects`, `/tasks` (`main.py:19-21`) plus `GET /health` (`main.py:24-25`); searches for "jwt", "token", "login", "auth" in `app/` return nothing. +2. **"SQLAlchemy ORM with async support"** → The database layer is synchronous: `create_engine` (`app/core/database.py:11-14`), `sessionmaker` (`app/core/database.py:16`), and sync `def` handlers throughout (e.g. `app/api/tasks.py:17`). +3. **Service roster of TaskService, ProjectService, UserService, NotificationService** → Only `TaskService` (`app/services/task_service.py:14`) and `NotificationService` (`app/services/notification_service.py:13`) exist; `ProjectService` and `UserService` are invented. +4. **"Sends transactional emails via SendGrid" and "Firebase Cloud Messaging" push** → `NotificationService` only POSTs a JSON webhook with httpx (`app/services/notification_service.py:43-59`); no email or push code exists anywhere. +5. **"Redis Cache: Caches frequently accessed data"** → No caching layer; searches for "redis" and "cache" in `app/` return nothing. +6. **users table with "email and password"** → The User model has no password column: id, email, name, created_at only (`app/models/user.py:16-19`). +7. **`notifications` and `audit_log` tables** → Do not exist; the only tables are users (`app/models/user.py:14`), projects (`app/models/project.py:22`), and tasks (`app/models/task.py:30`). +8. **Step-by-step "Data Flow" section** → A banned pattern in the architecture overview (prompt 01 Section 3: "MUST NOT trace step-by-step execution"); detailed tracing belongs in the code-flows document, and Section 3 of the good example shows the table-based alternative. +9. **Zero verification tags** → Not one `[VERIFIED: path:line]` or `[NOT_FOUND]` in the whole document, so nothing is checkable — `verify.py` finds no citations to resolve, and every claim (true or false) reads identically. Cite or admit; there is no middle ground. diff --git a/examples/fastapi/good-architecture-doc-example.md b/examples/fastapi/good-architecture-doc-example.md index ca722d1..e5215dc 100644 --- a/examples/fastapi/good-architecture-doc-example.md +++ b/examples/fastapi/good-architecture-doc-example.md @@ -5,23 +5,43 @@ |-------|-------| | Repository | `agent-system-mapper` | | Path | `examples/fastapi/tasktracker/` | -| Commit | `ecd9f00` | -| Documented | `2025-12-21` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | | Verification Status | `Verified` | +Verify with: + +```bash +python3 verify.py examples/fastapi/good-architecture-doc-example.md --repo-root examples/fastapi/tasktracker +``` + ## Verification Summary -- [VERIFIED]: 28 claims -- [INFERRED]: 2 claims -- [NOT_FOUND]: 5 items (auth, email, redis, audit_log, async db) -- [ASSUMED]: 1 item (standard FastAPI conventions) +- `[VERIFIED]`: 77 claims (104 file:line citations, all resolving) +- `[INFERRED]`: 2 claims +- `[NOT_FOUND]`: 11 items (auth, email delivery, redis/cache, audit/notification tables, async DB, middleware, frontend assets, CLI entry points, migrations, unpinned httpx x2) +- `[ASSUMED]`: 0 items + +These counts are transcribed from the `Tag counts` line that `verify.py` prints for this document. + +--- + +## 0. System Classification + +| Field | Value | +|-------|-------| +| Category | Traditional Code | +| Type | Framework backend (FastAPI REST API) | +| Evidence | `fastapi==0.104.1` pinned [VERIFIED: requirements.txt:1]; app constructed via `FastAPI(...)` [VERIFIED: main.py:12] | +| Overlay Loaded | No | +| Confidence | `[VERIFIED]` | --- -## System Purpose +## 1. System Purpose -TaskTracker is a **task and project management REST API** built with FastAPI. +TaskTracker is a small task- and project-management REST API built with FastAPI. It exposes open (unauthenticated) CRUD endpoints for users, projects, and tasks, enforces a per-project task limit, and emits webhook notifications when tasks are created or completed. -[VERIFIED: `main.py:10-14`] +[VERIFIED: main.py:12-16] ```python app = FastAPI( title="TaskTracker API", @@ -30,136 +50,130 @@ app = FastAPI( ) ``` -The API provides CRUD operations for: -- Users [VERIFIED: `app/api/users.py`] -- Projects [VERIFIED: `app/api/projects.py`] -- Tasks [VERIFIED: `app/api/tasks.py`] +One router exists per resource: users [VERIFIED: app/api/users.py:12], projects [VERIFIED: app/api/projects.py:13], and tasks [VERIFIED: app/api/tasks.py:13]. --- -## Component Map +## 2. Component Map -| Component | Location | Responsibility | Verified | +| Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| FastAPI App | `main.py` | Application entry, router mounting | [VERIFIED] | -| API Routers | `app/api/` | HTTP request handlers | [VERIFIED] | -| SQLAlchemy Models | `app/models/` | Database entities | [VERIFIED] | -| Pydantic Schemas | `app/schemas/` | Request/response validation | [VERIFIED] | -| Repositories | `app/repositories/` | Data access layer | [VERIFIED] | -| Services | `app/services/` | Business logic | [VERIFIED] | -| Config | `app/core/config.py` | Environment settings | [VERIFIED] | -| Database | `app/core/database.py` | DB session management | [VERIFIED] | +| FastAPI app | `main.py` | App construction, router mounting, `/health` endpoint | [VERIFIED: main.py:12-27] | +| API routers | `app/api/` | HTTP handlers for users, projects, tasks | [VERIFIED: app/api/users.py:12, app/api/projects.py:13, app/api/tasks.py:13] | +| SQLAlchemy models | `app/models/` | `User`, `Project`, `Task` ORM entities | [VERIFIED: app/models/user.py:11, app/models/project.py:19, app/models/task.py:27] | +| Pydantic schemas | `app/schemas/` | Request/response validation DTOs | [VERIFIED: app/schemas/user.py:9, app/schemas/project.py:10, app/schemas/task.py:10] | +| Repositories | `app/repositories/` | Data access layer around the ORM | [VERIFIED: app/repositories/user_repository.py:11, app/repositories/project_repository.py:11, app/repositories/task_repository.py:11] | +| Services | `app/services/` | Business logic and webhook notifications | [VERIFIED: app/services/task_service.py:14, app/services/notification_service.py:13] | +| Config | `app/core/config.py` | Env-driven settings singleton | [VERIFIED: app/core/config.py:10-24] | +| Database | `app/core/database.py` | Engine, session factory, `get_db` dependency | [VERIFIED: app/core/database.py:11-30] | -[NOT_FOUND: searched "auth", "login", "jwt", "token" in app/] -No authentication layer exists. API endpoints are open. +### Service Wiring -[NOT_FOUND: searched "email", "sendgrid", "smtp" in app/] -No email service. Only webhook notifications exist. +`TaskService` composes the repositories and the notification service via plain constructor injection: -[NOT_FOUND: searched "redis", "cache" in app/] -No caching layer implemented. - ---- - -## File Structure - -``` -tasktracker/ -├── main.py # FastAPI app initialization [VERIFIED] -├── requirements.txt # Dependencies [VERIFIED] -└── app/ - ├── api/ # Route handlers - │ ├── users.py # User CRUD [VERIFIED] - │ ├── projects.py # Project CRUD [VERIFIED] - │ └── tasks.py # Task CRUD [VERIFIED] - ├── core/ - │ ├── config.py # Settings from env [VERIFIED] - │ └── database.py # SQLAlchemy setup [VERIFIED] - ├── models/ # SQLAlchemy ORM models - │ ├── user.py # User entity [VERIFIED] - │ ├── project.py # Project entity [VERIFIED] - │ └── task.py # Task entity [VERIFIED] - ├── schemas/ # Pydantic models - │ ├── user.py # User DTOs [VERIFIED] - │ ├── project.py # Project DTOs [VERIFIED] - │ └── task.py # Task DTOs [VERIFIED] - ├── repositories/ # Data access - │ ├── user_repository.py [VERIFIED] - │ ├── project_repository.py [VERIFIED] - │ └── task_repository.py [VERIFIED] - └── services/ # Business logic - ├── task_service.py [VERIFIED] - └── notification_service.py [VERIFIED] +[VERIFIED: app/services/task_service.py:20-24] +```python + def __init__(self, db: Session): + self.db = db + self.task_repo = TaskRepository(db) + self.project_repo = ProjectRepository(db) + self.notification_service = NotificationService() ``` ---- +### Domain Entities -## Data Models +| Entity | Table | Key Columns | Evidence | +|--------|-------|-------------|----------| +| `User` | `users` | id, email, name, created_at | [VERIFIED: app/models/user.py:14-19] | +| `Project` | `projects` | id, name, description, status, owner_id, created_at, updated_at | [VERIFIED: app/models/project.py:22-30] | +| `Task` | `tasks` | id, title, description, status, priority, project_id, assignee_id, due_date, created_at, updated_at | [VERIFIED: app/models/task.py:30-41] | -### Relationships +[NOT_FOUND: searched "audit", "log", "notification" in app/models/ — the only `__tablename__` declarations are users, projects, and tasks] -[VERIFIED: `app/models/user.py:17-19`] -```python -# User has many projects and assigned tasks -projects = relationship("Project", back_populates="owner") -assigned_tasks = relationship("Task", back_populates="assignee") -``` +Relationships are declared directly on the models — for example on `User`: -[VERIFIED: `app/models/project.py:29-30`] +[VERIFIED: app/models/user.py:21-23] ```python -# Project belongs to owner, has many tasks -owner = relationship("User", back_populates="projects") -tasks = relationship("Task", back_populates="project", cascade="all, delete-orphan") + # Relationships + projects = relationship("Project", back_populates="owner") + assigned_tasks = relationship("Task", back_populates="assignee") ``` -[VERIFIED: `app/models/task.py:37-38`] -```python -# Task belongs to project and optionally to assignee -project = relationship("Project", back_populates="tasks") -assignee = relationship("User", back_populates="assigned_tasks") -``` +`Project.tasks` cascades deletes to its tasks [VERIFIED: app/models/project.py:34], and `Task` links back to both `Project` and `User` [VERIFIED: app/models/task.py:44-45]. -### Database Tables +### What Does Not Exist -| Table | Columns | Source | -|-------|---------|--------| -| users | id, email, name, created_at | [VERIFIED: `app/models/user.py:13-16`] | -| projects | id, name, description, status, owner_id, created_at, updated_at | [VERIFIED: `app/models/project.py:21-27`] | -| tasks | id, title, description, status, priority, project_id, assignee_id, due_date, created_at, updated_at | [VERIFIED: `app/models/task.py:27-35`] | +[NOT_FOUND: searched "auth", "login", "jwt", "token" in app/ and main.py — no authentication layer; every endpoint is open] -[NOT_FOUND: searched "audit", "log" in app/models/] -No audit_log table exists. +[NOT_FOUND: searched "smtp", "sendgrid", "send_email", "mailer" in app/ — no email delivery; the only mail-adjacent hits are the `User.email` column and the `get_by_email` lookup] + +[NOT_FOUND: searched "redis", "cache" in app/ and main.py — no caching layer] --- -## Entry Points +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) -### HTTP Endpoints +### 3.1 Primary Execution Surfaces -[VERIFIED: `main.py:16-18`] -```python -app.include_router(users.router, prefix="/users", tags=["users"]) -app.include_router(projects.router, prefix="/projects", tags=["projects"]) -app.include_router(tasks.router, prefix="/tasks", tags=["tasks"]) -``` +| Entry Surface | Type | Primary Components Involved | Evidence | +|--------------|------|-----------------------------|----------| +| `/users` routes | HTTP API | users router → `UserRepository` | [VERIFIED: main.py:19, app/api/users.py:15-63] | +| `/projects` routes | HTTP API | projects router → `ProjectRepository`, `UserRepository` | [VERIFIED: main.py:20, app/api/projects.py:16-107] | +| `/tasks` routes | HTTP API | tasks router → `TaskService`, `TaskRepository` | [VERIFIED: main.py:21, app/api/tasks.py:16-150] | +| `GET /health` | HTTP API | inline handler in `main.py` | [VERIFIED: main.py:24-27] | + +[NOT_FOUND: searched for templates/, static/, "fetch(", "axios" under tasktracker/ — no frontend assets and no CLI entry points; HTTP is the only execution surface] + +The app is presumably served with `uvicorn main:app` — uvicorn is pinned but no launch script or `__main__` block exists [INFERRED: uvicorn pinned at requirements.txt:2; main.py ends at the health handler with no launcher]. + +### 3.2 High-Level Data Movement (Non-Procedural) + +| Stage | Input Type | Output Type | Participating Components | +|-------|------------|-------------|--------------------------| +| Request validation | HTTP JSON | Pydantic schema instance (`UserCreate`, `ProjectCreate`, `TaskCreate`, …) | routers + `app/schemas/` | +| Business rules | Schema instance | ORM entity + side effects | `TaskService` (project existence check, task limit) | +| Persistence | Schema instance / ORM entity | Committed rows | repositories + `SessionLocal` | +| Notification egress | Task event | Webhook JSON payload | `NotificationService` | +| Response serialization | ORM entity | Response schema JSON (`TaskResponse`, …) | routers + `app/schemas/` | + +### 3.3 Pointers to Code Flow Documentation -| Prefix | Router | Operations | -|--------|--------|------------| -| `/users` | `app/api/users.py` | list, get, create, update, delete | -| `/projects` | `app/api/projects.py` | list, get, create, update, delete | -| `/tasks` | `app/api/tasks.py` | list, get, create, update, delete, overdue | -| `/health` | `main.py:21-24` | health check | +Candidates for detailed flow tracing (see `02-code-flows.md`): + +- **Task creation** — `POST /tasks/` → `TaskService.create_task` (limit check + notification) +- **Task completion notification** — `PUT /tasks/{task_id}` → `TaskService.update_task_status` +- **Overdue task listing** — `GET /tasks/overdue` → `TaskRepository.get_overdue` + +Detailed execution paths deliberately belong in `02-code-flows.md`, not here. + +### Section 3 Self-Check +- [x] No method bodies longer than 3 lines quoted in this section +- [x] No loops or conditionals described +- [x] Movements described as conceptual stages +- [x] Defers detailed tracing to `02-code-flows.md` --- -## Key Patterns +## 4. File/Folder Conventions -### Dependency Injection for Database +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| `app/api/*.py` | One router module per resource | [VERIFIED: app/api/users.py:2, app/api/projects.py:2, app/api/tasks.py:2] | +| `app/models/*.py` | One SQLAlchemy model module per entity | [VERIFIED: app/models/user.py:2, app/models/project.py:2, app/models/task.py:2] | +| `app/schemas/*.py` | Pydantic Create/Update/Response triad per entity | [VERIFIED: app/schemas/task.py:10, 20, 30] | +| `app/repositories/*_repository.py` | Data-access class per entity | [VERIFIED: app/repositories/task_repository.py:2, app/repositories/user_repository.py:2] | +| `app/services/*_service.py` | Business-logic classes | [VERIFIED: app/services/task_service.py:2, app/services/notification_service.py:2] | +| `app/core/` | Cross-cutting config and database setup | [VERIFIED: app/core/config.py:2, app/core/database.py:2] | -[VERIFIED: `app/core/database.py:19-27`] +Database sessions are injected into every handler through FastAPI's `Depends(get_db)` [VERIFIED: app/api/users.py:16, app/api/tasks.py:20]: + +[VERIFIED: app/core/database.py:21-30] ```python def get_db(): - """Dependency that provides database session.""" + """ + Dependency that provides database session. + Yields session and ensures cleanup. + """ db = SessionLocal() try: yield db @@ -167,146 +181,179 @@ def get_db(): db.close() ``` -Used in endpoints: -[VERIFIED: `app/api/users.py:18`] -```python -def list_users(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)): -``` +--- -### Repository Pattern +## 5. External Dependencies -Repositories encapsulate data access: -[VERIFIED: `app/repositories/task_repository.py:13-17`] -```python -class TaskRepository: - def __init__(self, db: Session): - self.db = db +| Dependency | Purpose | Evidence | +|------------|---------|----------| +| `fastapi==0.104.1` | Web framework | [VERIFIED: requirements.txt:1] | +| `uvicorn==0.24.0` | ASGI server (no launch script in repo) | [VERIFIED: requirements.txt:2] | +| `sqlalchemy==2.0.23` | ORM, synchronous engine | [VERIFIED: requirements.txt:3, app/core/database.py:11] | +| `pydantic==2.5.2` | Validation | [VERIFIED: requirements.txt:4] | +| `python-dotenv==1.0.0` | `.env` loading | [VERIFIED: requirements.txt:5, app/core/config.py:5] | +| `httpx` | Webhook HTTP client — imported but **not pinned** | [VERIFIED: app/services/notification_service.py:4] | - def get_by_id(self, task_id: int) -> Optional[Task]: -``` +[NOT_FOUND: searched "httpx" in requirements.txt — the import in app/services/notification_service.py has no matching pinned dependency] -### Service Layer +The only external service integration is an outbound notification webhook, configured entirely by environment variables: -Business logic separated from routes: -[VERIFIED: `app/services/task_service.py:17-23`] +[VERIFIED: app/core/config.py:16-18] ```python -class TaskService: - def __init__(self, db: Session): - self.db = db - self.task_repo = TaskRepository(db) - self.project_repo = ProjectRepository(db) - self.notification_service = NotificationService() + # Notification settings - external webhook + NOTIFICATION_WEBHOOK_URL: str = os.getenv("NOTIFICATION_WEBHOOK_URL", "") + NOTIFICATION_ENABLED: bool = os.getenv("NOTIFICATION_ENABLED", "false").lower() == "true" ``` +Notifications fire on task creation [VERIFIED: app/services/task_service.py:47] and on completion [VERIFIED: app/services/task_service.py:62], and POST to the configured URL [VERIFIED: app/services/notification_service.py:54-58]. + --- -## External Integrations +## 6. Known Issues & Risks -### Webhook Notifications +### 6.1 Magic number duplicated -[VERIFIED: `app/services/notification_service.py:14-15`] +The per-project task limit is defined in two places: + +[VERIFIED: app/core/config.py:20-21] ```python -class NotificationService: - """Service for sending notifications via webhook.""" + # Wart: Magic number for task limit, duplicated in TaskService + MAX_TASKS_PER_PROJECT: int = 100 ``` -Configuration: -[VERIFIED: `app/core/config.py:13-14`] +[VERIFIED: app/services/task_service.py:17-18] ```python -NOTIFICATION_WEBHOOK_URL: str = os.getenv("NOTIFICATION_WEBHOOK_URL", "") -NOTIFICATION_ENABLED: bool = os.getenv("NOTIFICATION_ENABLED", "false").lower() == "true" + # Wart: Magic number duplicated from config.py + MAX_TASKS_PER_PROJECT = 100 ``` -Events that trigger notifications: -- Task created [VERIFIED: `app/services/task_service.py:40`] -- Task completed [VERIFIED: `app/services/task_service.py:51`] +`TaskService` reads its own copy when enforcing the limit, so changing the config value alone has no effect [VERIFIED: app/services/task_service.py:38]. ---- +### 6.2 Synchronous HTTP call in the notification path -## Known Issues / Warts - -### 1. Magic Number Duplication - -[VERIFIED: `app/core/config.py:17`] +[VERIFIED: app/services/notification_service.py:52-59] ```python -MAX_TASKS_PER_PROJECT: int = 100 + try: + # Wart: Should use async httpx in production + with httpx.Client(timeout=5.0) as client: + response = client.post( + settings.NOTIFICATION_WEBHOOK_URL, + json=payload + ) + response.raise_for_status() ``` -[VERIFIED: `app/services/task_service.py:19`] +The blocking `httpx.Client` call runs inside request handling; under an async server this can stall the event loop. + +### 6.3 Notification failures are swallowed + +[VERIFIED: app/services/notification_service.py:60-62] ```python -MAX_TASKS_PER_PROJECT = 100 # Wart: Magic number duplicated from config.py + except httpx.HTTPError as e: + # Wart: Silently fails, no retry + logger.error(f"Failed to send notification: {e}") ``` -Same value defined in two places - should use config. +No retry and no dead-letter handling; the API response succeeds even when the webhook never arrives. -### 2. Synchronous HTTP in Notification Service +### 6.4 Unpaginated default task listing with dead code -[VERIFIED: `app/services/notification_service.py:45-49`] +[VERIFIED: app/api/tasks.py:33-37] ```python -# Wart: Synchronous HTTP call in async context. -with httpx.Client(timeout=5.0) as client: - response = client.post(...) + # Wart: No pagination on default list + tasks = repo.get_by_project(1) if False else [] + # Actually get all - but this is expensive + from app.models.task import Task + tasks = db.query(Task).limit(100).all() ``` -Uses sync HTTP client instead of async - can block event loop. +The unfiltered `GET /tasks/` branch contains dead code (`if False`) and an inline import, and caps results at a hardcoded 100 with no offset support. -### 3. No Retry Logic for Notifications +### 6.5 Tables created at import time -[VERIFIED: `app/services/notification_service.py:52-53`] +[VERIFIED: main.py:8-10] ```python -except httpx.HTTPError as e: - # Wart: Silently fails, no retry - logger.error(f"Failed to send notification: {e}") +# Create tables on startup - not recommended for production +# but fine for this example app +Base.metadata.create_all(bind=engine) ``` -Notification failures are logged but not retried. +[NOT_FOUND: searched "alembic", "migration" in tasktracker/ — no migration tooling; schema changes require manual handling] -### 4. No Pagination on Default Task List +### 6.6 No connection pooling configured -[VERIFIED: `app/api/tasks.py:28-31`] +[VERIFIED: app/core/database.py:10-14] ```python -# Wart: No pagination on default list -tasks = repo.get_by_project(1) if False else [] -# Actually get all - but this is expensive +# Wart: No connection pooling configured for SQLite +engine = create_engine( + settings.DATABASE_URL, + connect_args={"check_same_thread": False} # SQLite specific +) ``` -Default task list has no proper pagination when no filters applied. +### 6.7 Undeclared runtime dependency + +`httpx` is imported [VERIFIED: app/services/notification_service.py:4] but absent from `requirements.txt` [NOT_FOUND: searched "httpx" in requirements.txt], so a fresh install from the pinned requirements fails at startup [INFERRED: main.py imports the api package, which transitively imports the notification service and its httpx import]. + +--- + +## 7. Entry Points Summary -### 5. Tables Created on Startup +All routers are mounted with URL prefixes in `main.py`: -[VERIFIED: `main.py:8-9`] +[VERIFIED: main.py:19-21] ```python -# Create tables on startup - not recommended for production -Base.metadata.create_all(bind=engine) +app.include_router(users.router, prefix="/users", tags=["users"]) +app.include_router(projects.router, prefix="/projects", tags=["projects"]) +app.include_router(tasks.router, prefix="/tasks", tags=["tasks"]) ``` -Should use migrations (Alembic) instead. +| Route/Entry | Method | Handler | Middleware | Verified | +|-------------|--------|---------|------------|----------| +| `/users/` | GET | `list_users` | none | [VERIFIED: app/api/users.py:15-16] | +| `/users/{user_id}` | GET | `get_user` | none | [VERIFIED: app/api/users.py:22-23] | +| `/users/` | POST | `create_user` | none | [VERIFIED: app/api/users.py:32-33] | +| `/users/{user_id}` | PUT | `update_user` | none | [VERIFIED: app/api/users.py:45-46] | +| `/users/{user_id}` | DELETE | `delete_user` | none | [VERIFIED: app/api/users.py:55-56] | +| `/projects/` | GET | `list_projects` | none | [VERIFIED: app/api/projects.py:16-17] | +| `/projects/{project_id}` | GET | `get_project` | none | [VERIFIED: app/api/projects.py:37-38] | +| `/projects/` | POST | `create_project` | none | [VERIFIED: app/api/projects.py:56-57] | +| `/projects/{project_id}` | PUT | `update_project` | none | [VERIFIED: app/api/projects.py:79-80] | +| `/projects/{project_id}` | DELETE | `delete_project` | none | [VERIFIED: app/api/projects.py:99-100] | +| `/tasks/` | GET | `list_tasks` | none | [VERIFIED: app/api/tasks.py:16-17] | +| `/tasks/overdue` | GET | `list_overdue_tasks` | none | [VERIFIED: app/api/tasks.py:57-58] | +| `/tasks/{task_id}` | GET | `get_task` | none | [VERIFIED: app/api/tasks.py:80-81] | +| `/tasks/` | POST | `create_task` | none | [VERIFIED: app/api/tasks.py:102-103] | +| `/tasks/{task_id}` | PUT | `update_task` | none | [VERIFIED: app/api/tasks.py:122-123] | +| `/tasks/{task_id}` | DELETE | `delete_task` | none | [VERIFIED: app/api/tasks.py:142-143] | +| `/health` | GET | `health_check` | none | [VERIFIED: main.py:24-25] | + +[NOT_FOUND: searched "middleware", "CORSMiddleware", "add_middleware" in app/ and main.py — no middleware is registered anywhere, hence "none" in every row] --- -## Configuration +## 8. Technology Stack Summary -[VERIFIED: `app/core/config.py:8-17`] +| Layer | Technology | Evidence | +|-------|------------|----------| +| Backend framework | FastAPI 0.104.1 | [VERIFIED: requirements.txt:1] | +| ASGI server | Uvicorn 0.24.0 | [VERIFIED: requirements.txt:2] | +| ORM | SQLAlchemy 2.0.23, synchronous `create_engine` + `sessionmaker` | [VERIFIED: requirements.txt:3, app/core/database.py:11-16] | +| Validation | Pydantic 2.5.2 | [VERIFIED: requirements.txt:4, app/schemas/task.py:4] | +| Primary database | SQLite (default `DATABASE_URL`) | [VERIFIED: app/core/config.py:14] | +| Outbound HTTP | httpx sync `Client` (unpinned) | [VERIFIED: app/services/notification_service.py:4, 54] | +| Frontend framework | None | [NOT_FOUND: no templates/, static/, or JS assets in tasktracker/] | +| External services | Single notification webhook | [VERIFIED: app/core/config.py:17] | -| Setting | Source | Default | -|---------|--------|---------| -| DATABASE_URL | env | `sqlite:///./tasktracker.db` | -| NOTIFICATION_WEBHOOK_URL | env | empty string | -| NOTIFICATION_ENABLED | env | `false` | -| MAX_TASKS_PER_PROJECT | hardcoded | `100` | - -[NOT_FOUND: searched "async" in database.py] -Database is synchronous SQLite, not async. +[NOT_FOUND: searched "asyncio", "AsyncSession", "create_async_engine", "async def" in app/ and main.py — the entire request path is synchronous] --- -## What This System Does NOT Have - -Based on searches finding no results: +## Why This Example is GOOD -1. **No Authentication** - All endpoints are open -2. **No Email Service** - Only webhook notifications -3. **No Caching** - No Redis or in-memory cache -4. **No Audit Logging** - No change tracking -5. **No Async Database** - Uses sync SQLAlchemy -6. **No Migrations** - Tables created on startup +1. **Every positive claim carries a `file:line` citation** that resolves against `examples/fastapi/tasktracker/` — run the verify command at the top of this document and every citation checks out. +2. **Quoted code is copy-pasted, not paraphrased** — each fenced block matches the cited slice exactly, so the verifier's phase-2 quote check passes. +3. **Absences are proven, not assumed** — every `[NOT_FOUND]` records the actual search terms and scope used (auth, email delivery, redis, audit_log, async DB, middleware, migrations), so a reader can re-run the same searches. +4. **No step-by-step execution traces** — Section 3 uses discovery tables and defers all tracing to `02-code-flows.md`, exactly as prompt 01 requires. +5. **Warts are documented with evidence** — the duplicated magic number, sync HTTP, swallowed failures, dead code, and the unpinned `httpx` import are all cited, not hand-waved. +6. **Counts are honest** — the Verification Summary numbers are transcribed from `verify.py`'s tag-count output for this document, not estimated. diff --git a/examples/laravel/bad-code-flow-doc-example.md b/examples/laravel/bad-code-flow-doc-example.md new file mode 100644 index 0000000..04f48e9 --- /dev/null +++ b/examples/laravel/bad-code-flow-doc-example.md @@ -0,0 +1,56 @@ +# Create Booking Code Flow (SlotBooker) + +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns. Each ❌ callout explains a failure. See the good example for the correct approach. + +## Metadata +| Field | Value | +|-------|-------| +| Repository | `slotbooker` | +| Documented | `2026-08-03` | + +> **❌ PROBLEMS:** No commit hash, no trigger, no end state, and no "Verify with:" command — nothing ties this document to a code state, so nobody can check it. The good example pins commit `9a69c14` and ships a runnable `verify.py` command. Running the verifier on THIS document reports zero checkable citations and fails: unverifiable docs fail by default. + +--- + +## The Booking Flow + +1. The user submits the booking form, which is validated client-side and then by the `StoreBookingRequest` form request class. +2. `BookingController::store()` passes the validated data to `BookingService::createBooking()`. +3. `BookingService` re-checks availability with `TimeSlot::hasAvailability()` inside a database transaction, locking the slot row with `lockForUpdate()`. +4. The booking is saved with status `confirmed`. +5. A `BookingConfirmationMail` is queued and emailed to the user. +6. An SMS reminder is scheduled via Twilio for 24 hours before the slot. +7. A `SyncBookingToCalendarJob` is pushed onto the `calendar` queue and retried up to 3 times using the `retry_attempts` config value. +8. The user is shown the confirmation page with a link to the synced calendar event. + +> **❌ PROBLEMS:** Not one step has a `file:line` citation or a quoted line of code, and most of them are fiction: +> - **Step 1** — there is no `StoreBookingRequest` and no validation at all: the controller reads raw input at `app/Http/Controllers/BookingController.php:39` and `:53`, and its own comment at `:48` says "no validation on notes field, could be XSS". +> - **Step 2** — `BookingService` does not exist. The only service is `app/Services/CalendarService.php`; booking logic is inline in `BookingController::store()` at `app/Http/Controllers/BookingController.php:37-64`. +> - **Step 3** — no transaction, no `lockForUpdate()`. The capacity check is a plain count at `app/Http/Controllers/BookingController.php:43-46`, duplicating `TimeSlot::hasAvailability()` (`app/Models/TimeSlot.php:31-42`) — the controller's own comment at `app/Http/Controllers/BookingController.php:42` even flags the duplication. +> - **Step 4** — the booking is created with status `'pending'` at `app/Http/Controllers/BookingController.php:52` and only flipped to `'confirmed'` at `:60`, *after* the event fires at `:57`. "Saved with status confirmed" hides a real ordering wart. +> - **Steps 5-6** — pure hallucination. There is no mail, no `BookingConfirmationMail`, no SMS, no Twilio anywhere under `app/`. The plausible-sounding notification steps are exactly what an agent invents when it documents from convention instead of code. +> - **Step 7** — there are no jobs and no queues. Sync happens in a synchronous event listener (`app/Providers/CalendarServiceProvider.php:34`, `app/Listeners/SyncToExternalCalendar.php:22-34`), and `retry_attempts` is defined at `config/calendar.php:28` but never read — the config file's own comment at `:22-23` says so. +> - **Step 8** — there is no confirmation page and no calendar link. The controller redirects back to `booking.index` with a flash message at `app/Http/Controllers/BookingController.php:62-63`. + +--- + +## Error Handling + +If the calendar sync fails, the job is retried automatically and the user is notified by email that their booking could not be synced. All errors are handled gracefully. + +> **❌ PROBLEMS:** The opposite of the real behavior. `CalendarService::syncBooking()` has no try/catch — the comment at `app/Services/CalendarService.php:29` says "no try/catch here - errors bubble up" — and on an API failure it just logs and returns `null` (`app/Services/CalendarService.php:44-51`). The caller then confirms the booking anyway (`app/Http/Controllers/BookingController.php:60`), so a failed sync is silent: no retry, no notification, and the user sees "Booking confirmed!". "Handled gracefully" is an unverifiable comfort phrase, not a finding. + +--- + +## Why This Example is BAD + +1. **"Validated by `StoreBookingRequest`"** → no validation exists; raw `$request->input()` at `app/Http/Controllers/BookingController.php:39`, `:53`, with the XSS wart noted at `:48`. +2. **"`BookingService::createBooking()`"** → `BookingService` does not exist; the logic is inline in `app/Http/Controllers/BookingController.php:37-64`. This is the exact hallucination the architecture bad example warns about. +3. **"Transaction with `lockForUpdate()`"** → no transaction or lock anywhere; just a count at `app/Http/Controllers/BookingController.php:43-46`, and the bookings migration comment at `database/migrations/2024_01_01_000003_create_bookings_table.php:27-28` flags the missing unique constraint the lock story papers over. +4. **"Saved with status `confirmed`"** → created `'pending'` (`app/Http/Controllers/BookingController.php:52`), event fired at `:57`, then updated to `'confirmed'` at `:60`. The invented version erases a real bug surface. +5. **"Confirmation email queued"** → no mail code exists anywhere in `app/` — a real trace records this as a NOT_FOUND, like the good example does. +6. **"SMS via Twilio"** → no SMS integration exists anywhere in `app/`. +7. **"`SyncBookingToCalendarJob` on the `calendar` queue, 3 retries"** → sync is a synchronous listener (`app/Listeners/SyncToExternalCalendar.php:22-34`) registered via `Event::subscribe` (`app/Providers/CalendarServiceProvider.php:34`); `retry_attempts` (`config/calendar.php:28`) is dead config that nothing reads. +8. **"Confirmation page with calendar link"** → a redirect with a flash message (`app/Http/Controllers/BookingController.php:62-63`); the view never shows any calendar link. +9. **"Errors handled gracefully"** → errors bubble up uncaught (`app/Services/CalendarService.php:29`) or are logged and swallowed (`app/Services/CalendarService.php:44-51`) while the booking is confirmed regardless. +10. **No citations, no quotes, no tags, no commit** → nothing in this document can be checked. Every numbered step *sounds* right for a Laravel booking app, which is precisely why uncited flow docs are dangerous: a reader cannot tell steps 1-3 (plausible fiction) from steps 5-7 (pure invention). diff --git a/examples/laravel/good-architecture-doc-example.md b/examples/laravel/good-architecture-doc-example.md index f51034f..edd8871 100644 --- a/examples/laravel/good-architecture-doc-example.md +++ b/examples/laravel/good-architecture-doc-example.md @@ -1,359 +1,327 @@ # SlotBooker Architecture Overview +> **This is an example of GOOD architecture documentation.** +> Every citation resolves against the mini Laravel app in `examples/laravel/slotbooker/`, +> so this example is machine-verifiable wherever the mapper is installed. + ## Metadata | Field | Value | |-------|-------| | Repository | `agent-system-mapper` | -| Commit | `e043013` | -| Documented | `2025-01-15` | +| Path | `examples/laravel/slotbooker` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | | Verification Status | `Verified` | +**Verify with:** +```bash +python3 verify.py examples/laravel/good-architecture-doc-example.md --repo-root examples/laravel/slotbooker +``` + ## Verification Summary -- [VERIFIED]: 25 claims -- [INFERRED]: 3 claims -- [NOT_FOUND]: 4 items (email, SMS, BookingService, extra tables) -- [ASSUMED]: 2 items (Laravel conventions) +- `[VERIFIED]`: 91 tags — 133/133 citations resolve; 8/8 quoted blocks match their source +- `[INFERRED]`: 2 tags +- `[NOT_FOUND]`: 7 items (BookingService, email, SMS, unused config reads, framework manifest) +- `[ASSUMED]`: 2 items (Laravel conventions) --- -## 1. System Purpose +## 0. System Classification -SlotBooker is a booking system that allows users to reserve time slots and syncs those bookings to an external calendar API. - -[VERIFIED: routes/web.php:16-26] The system exposes four routes: -- `GET /booking` - view available slots -- `POST /booking` - create booking -- `POST /booking/{booking}/cancel` - cancel booking -- `GET /api/slots/availability` - AJAX availability check +| Field | Value | +|-------|-------| +| Category | Traditional Code | +| Type | Framework backend (Laravel-style MVC with events/listeners) | +| Evidence | Eloquent models extend `Illuminate\Database\Eloquent\Model` [VERIFIED: app/Models/Booking.php:5-7]; routes use the `Route` facade [VERIFIED: routes/web.php:4]; events wired via `Event::subscribe` [VERIFIED: app/Providers/CalendarServiceProvider.php:34] | +| Overlay Loaded | No | +| Confidence | `[VERIFIED]` for the code patterns; [NOT_FOUND: no composer.json or artisan in examples/laravel/slotbooker/ — the example app ships application code only, so the framework version cannot be pinned] | --- -## 2. Component Map +## 1. System Purpose -| Component | Location | Responsibility | Verified | -|-----------|----------|----------------|----------| -| Models | `app/Models/` | Data entities and relationships | [VERIFIED] | -| Migrations | `database/migrations/` | Schema definitions (3 tables) | [VERIFIED] | -| Controller | `app/Http/Controllers/BookingController.php` | HTTP request handling | [VERIFIED] | -| Service | `app/Services/CalendarService.php` | External API integration | [VERIFIED] | -| Contract | `app/Contracts/CalendarServiceInterface.php` | Service abstraction | [VERIFIED] | -| Events | `app/Events/` | Domain events | [VERIFIED] | -| Listener | `app/Listeners/SyncToExternalCalendar.php` | Event handling | [VERIFIED] | -| Provider | `app/Providers/CalendarServiceProvider.php` | Dependency wiring | [VERIFIED] | -| Config | `config/calendar.php` | External API settings | [VERIFIED] | -| View | `resources/views/booking.blade.php` | UI template | [VERIFIED] | -| JavaScript | `public/js/booking.js` | Client-side behavior | [VERIFIED] | - -[NOT_FOUND: searched "BookingService" in app/] No BookingService exists. Booking logic lives directly in BookingController. +SlotBooker is a small booking system: users view available time slots, create bookings, cancel bookings, and the system syncs each booking to an external calendar API through an event listener. ---- +The system exposes four routes [VERIFIED: routes/web.php:17-30]: +- `GET /booking` — view available slots and the user's bookings +- `POST /booking` — create a booking +- `POST /booking/{booking}/cancel` — cancel a booking +- `GET /api/slots/availability` — AJAX availability check -## 3. Data Models +--- -### User (`app/Models/User.php`) +## 2. Component Map -[VERIFIED: app/Models/User.php:10-12] +| Component | Location | Responsibility | Evidence | +|-----------|----------|----------------|----------| +| BookingController | `app/Http/Controllers/BookingController.php` | HTTP handling for all four routes: `index`, `store`, `cancel`, `checkAvailability` | [VERIFIED: app/Http/Controllers/BookingController.php:16, 37, 70, 95] | +| User model | `app/Models/User.php` | User entity; `hasMany` bookings | [VERIFIED: app/Models/User.php:7, 14-17] | +| Booking model | `app/Models/Booking.php` | Booking entity; status constants; cancellation rule; sync marker | [VERIFIED: app/Models/Booking.php:7, 18-20, 36-47, 52-56] | +| TimeSlot model | `app/Models/TimeSlot.php` | Slot entity; availability and capacity logic | [VERIFIED: app/Models/TimeSlot.php:7, 31-42] | +| Migrations | `database/migrations/` | Schema for `users`, `time_slots`, `bookings` (3 tables) | [VERIFIED: database/migrations/2024_01_01_000001_create_users_table.php:14, database/migrations/2024_01_01_000003_create_bookings_table.php:14] | +| CalendarService | `app/Services/CalendarService.php` | External calendar API integration (`syncBooking`, `removeBooking`) | [VERIFIED: app/Services/CalendarService.php:25, 57] | +| CalendarServiceInterface | `app/Contracts/CalendarServiceInterface.php` | Service abstraction bound in the provider | [VERIFIED: app/Contracts/CalendarServiceInterface.php:7, 15, 23] | +| Events | `app/Events/` | `BookingCreated`, `BookingCancelled` domain events wrapping a Booking | [VERIFIED: app/Events/BookingCreated.php:9, app/Events/BookingCancelled.php:9] | +| SyncToExternalCalendar | `app/Listeners/SyncToExternalCalendar.php` | Event subscriber; calls CalendarService on create/cancel | [VERIFIED: app/Listeners/SyncToExternalCalendar.php:22, 39, 59-65] | +| CalendarServiceProvider | `app/Providers/CalendarServiceProvider.php` | Binds interface to implementation; registers the event subscriber | [VERIFIED: app/Providers/CalendarServiceProvider.php:19-22, 34] | +| Config | `config/calendar.php` | External API settings plus (unused) sync and booking-rule settings | [VERIFIED: config/calendar.php:14-15, 27-28, 41-42] | +| Blade view | `resources/views/booking.blade.php` | Booking page UI: slot cards, user bookings, cancel modal | [VERIFIED: resources/views/booking.blade.php:19, 47, 77] | +| JavaScript | `public/js/booking.js` | Availability AJAX, cancel modal wiring, 30-second polling | [VERIFIED: public/js/booking.js:17-27, 57-65, 94-98] | + +[NOT_FOUND: searched "BookingService", "NotificationService" in app/] No dedicated BookingService exists — booking logic lives directly in `BookingController::store` and `BookingController::cancel`. + +### Core Data Models + +#### `User` — app/Models/User.php + +[VERIFIED: app/Models/User.php:9] ```php -protected $fillable = ['name', 'email', 'phone']; + protected $fillable = ['name', 'email', 'phone']; ``` Relationships: -- [VERIFIED: app/Models/User.php:17-20] `hasMany(Booking::class)` +- `bookings()` → `hasMany(Booking::class)` [VERIFIED: app/Models/User.php:14-17] +- `activeBookings()` filters on the string literal `'confirmed'` instead of the constant [VERIFIED: app/Models/User.php:23-26] -### Booking (`app/Models/Booking.php`) +#### `Booking` — app/Models/Booking.php -[VERIFIED: app/Models/Booking.php:10-16] +[VERIFIED: app/Models/Booking.php:9-15] ```php -protected $fillable = [ - 'user_id', - 'time_slot_id', - 'status', - 'notes', - 'external_calendar_id', -]; + protected $fillable = [ + 'user_id', + 'time_slot_id', + 'status', + 'notes', + 'external_calendar_id', // stored after sync + ]; ``` -Status constants defined: -- [VERIFIED: app/Models/Booking.php:19-21] `STATUS_PENDING`, `STATUS_CONFIRMED`, `STATUS_CANCELLED` +Status constants [VERIFIED: app/Models/Booking.php:18-20]: +```php + const STATUS_PENDING = 'pending'; + const STATUS_CONFIRMED = 'confirmed'; + const STATUS_CANCELLED = 'cancelled'; +``` -[INFERRED] However, some code uses string literals instead of constants (e.g., `'confirmed'` at BookingController.php:53). +[INFERRED] The constants exist but several call sites use raw string literals instead — see §6 Known Issues & Risks. -Relationships: -- [VERIFIED: app/Models/Booking.php:23-26] `belongsTo(User::class)` -- [VERIFIED: app/Models/Booking.php:28-31] `belongsTo(TimeSlot::class)` +Relationships and business rules: +- `user()` → `belongsTo(User::class)` [VERIFIED: app/Models/Booking.php:22-25] +- `timeSlot()` → `belongsTo(TimeSlot::class)` [VERIFIED: app/Models/Booking.php:27-30] +- `canCancel()` — blocks cancellation within 24 hours of the slot; the `24` is hardcoded [VERIFIED: app/Models/Booking.php:36-47] +- `markSynced($externalId)` — stores the external calendar id after a successful sync [VERIFIED: app/Models/Booking.php:52-56] -### TimeSlot (`app/Models/TimeSlot.php`) +#### `TimeSlot` — app/Models/TimeSlot.php -[VERIFIED: app/Models/TimeSlot.php:10-15] +[VERIFIED: app/Models/TimeSlot.php:9-14] ```php -protected $fillable = [ - 'start_time', - 'end_time', - 'capacity', - 'is_available', -]; + protected $fillable = [ + 'start_time', + 'end_time', + 'capacity', + 'is_available', + ]; ``` -Relationships: -- [VERIFIED: app/Models/TimeSlot.php:23-26] `hasMany(Booking::class)` +Relationships and helpers: +- `bookings()` → `hasMany(Booking::class)` [VERIFIED: app/Models/TimeSlot.php:22-25] +- `hasAvailability()` — capacity check counting `'confirmed'` bookings [VERIFIED: app/Models/TimeSlot.php:31-42] +- `spotsLeft()` — remaining capacity for display [VERIFIED: app/Models/TimeSlot.php:47-51] +- `scopeAvailableFuture()` — future, available slots [VERIFIED: app/Models/TimeSlot.php:56-61] --- -## 4. Data Flow: Booking Creation +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) -``` -User clicks "Book This Slot" - │ - ▼ -[routes/web.php:20] -POST /booking → BookingController@store - │ - ▼ -[BookingController.php:38-55] -- Finds TimeSlot -- Checks capacity (duplicated from model) -- Creates Booking with status 'pending' -- Fires BookingCreated event -- Updates status to 'confirmed' - │ - ▼ -[Events/BookingCreated.php] -Event dispatched with Booking instance - │ - ▼ -[Listeners/SyncToExternalCalendar.php:24-35] -handleCreated() called - │ - ▼ -[Services/CalendarService.php:27-46] -- Builds payload with booking data -- POST to external API -- Returns external_id if successful - │ - ▼ -[Listeners/SyncToExternalCalendar.php:32-33] -booking.markSynced(external_id) - │ - ▼ -[BookingController.php:57-58] -Redirect with success message -``` +This section identifies **where execution enters the system** and **which components participate**. Step-by-step tracing is deliberately deferred to the Code Flow documentation (`02-code-flows.md`; the create-booking flow is traced in `good-code-flow-doc-example.md`). -[VERIFIED: Each step above references actual file:line] +### 3.1 Primary Execution Surfaces ---- +| Entry Surface | Type | Primary Components Involved | Evidence | +|--------------|------|-----------------------------|----------| +| `GET /booking` | Web Route | BookingController::index, TimeSlot (scope), booking.blade.php | [VERIFIED: routes/web.php:17-18, app/Http/Controllers/BookingController.php:16-31] | +| `POST /booking` | Web Route (form submit) | BookingController::store, Booking, BookingCreated event | [VERIFIED: routes/web.php:21-22, app/Http/Controllers/BookingController.php:37-64] | +| `POST /booking/{booking}/cancel` | Web Route (form submit via JS modal) | BookingController::cancel, Booking::canCancel, BookingCancelled event | [VERIFIED: routes/web.php:25-26, app/Http/Controllers/BookingController.php:70-89] | +| `GET /api/slots/availability` | JSON API (AJAX) | BookingController::checkAvailability, TimeSlot | [VERIFIED: routes/web.php:29-30, app/Http/Controllers/BookingController.php:95-109] | +| `BookingCreated` / `BookingCancelled` events | Event | SyncToExternalCalendar subscriber, CalendarService | [VERIFIED: app/Providers/CalendarServiceProvider.php:34, app/Listeners/SyncToExternalCalendar.php:59-65] | -## 5. Data Flow: Booking Cancellation +### 3.2 High-Level Data Movement (Non-Procedural) -``` -User clicks "Cancel" button - │ - ▼ -[public/js/booking.js:47-54] -confirmCancel() shows modal, sets form action - │ - ▼ -[routes/web.php:23] -POST /booking/{booking}/cancel → BookingController@cancel - │ - ▼ -[BookingController.php:65-82] -- Verifies ownership (user_id check) -- Calls booking.canCancel() -- Updates status to 'cancelled' -- Fires BookingCancelled event - │ - ▼ -[Models/Booking.php:33-44] -canCancel() checks: -- Status is not 'cancelled' -- More than 24 hours until slot - │ - ▼ -[Listeners/SyncToExternalCalendar.php:40-52] -handleCancelled() calls calendarService.removeBooking() - │ - ▼ -[Services/CalendarService.php:51-67] -DELETE request to external API -``` +| Stage | Input Type | Output Type | Participating Components | +|------|------------|-------------|--------------------------| +| Booking request handling | HTTP form POST (`time_slot_id`, `notes`) | Booking record (status `pending`, then `confirmed`) | BookingController, Booking [VERIFIED: app/Http/Controllers/BookingController.php:39, 49-54, 60] | +| Event dispatch | Booking record | Event payload carrying the Booking | BookingController, BookingCreated/BookingCancelled [VERIFIED: app/Http/Controllers/BookingController.php:57, 85] | +| Calendar sync | Booking (+ related TimeSlot, User) | HTTP POST/DELETE to external API; `external_calendar_id` stored | SyncToExternalCalendar, CalendarService, Booking [VERIFIED: app/Services/CalendarService.php:30-33, 63-65, app/Models/Booking.php:52-56] | +| Availability read | `slot_id` query param | JSON (`available`, `spots_left`, `start_time`) | BookingController, TimeSlot [VERIFIED: app/Http/Controllers/BookingController.php:104-108] | +| Page render | Slots + user bookings | HTML (Blade) | BookingController, booking.blade.php [VERIFIED: app/Http/Controllers/BookingController.php:27-30] | ---- +### 3.3 Pointers to Code Flow Documentation -## 6. External Dependencies +Candidates for detailed flow tracing (see `02-code-flows.md`): -### External Calendar API -[VERIFIED: config/calendar.php:14-15] -```php -'api_url' => env('CALENDAR_API_URL', 'https://api.example-calendar.com/v1'), -'api_key' => env('CALENDAR_API_KEY'), -``` +- **Create Booking** — form submit → `BookingController::store` → `BookingCreated` → `SyncToExternalCalendar::handleCreated` → external POST (traced in `good-code-flow-doc-example.md`) +- **Cancel Booking** — modal form → `BookingController::cancel` → `BookingCancelled` → `SyncToExternalCalendar::handleCancelled` → external DELETE +- **Availability Check** — `booking.js` fetch → `BookingController::checkAvailability` → JSON response + +### Section 3 Self-Check +- [x] No method bodies quoted; no loops or conditionals described +- [x] Movements described as conceptual stages, not steps +- [x] Detailed tracing deferred to `02-code-flows.md` -Called from: -- [VERIFIED: Services/CalendarService.php:28-31] POST `/events` for sync -- [VERIFIED: Services/CalendarService.php:60] DELETE `/events/{id}` for removal +--- -### Environment Variables Required -- `CALENDAR_API_URL` - External API base URL -- `CALENDAR_API_KEY` - Authentication key +## 3b. Frontend → Backend Interaction Map -[NOT_FOUND: searched "mail\|email" in app/] No email service integration. -[NOT_FOUND: searched "sms\|twilio\|nexmo" in app/] No SMS integration. +| Frontend Source | Trigger Type | Backend Target | Handler / Method | Evidence | +|-----------------|--------------|----------------|------------------|----------| +| `resources/views/booking.blade.php` | form submit (`POST /booking`) | BookingController | `store()` | [VERIFIED: resources/views/booking.blade.php:33-38, routes/web.php:21-22] | +| `resources/views/booking.blade.php` | form submit (cancel modal; action set by JS) | BookingController | `cancel()` | [VERIFIED: resources/views/booking.blade.php:80-84, public/js/booking.js:62] | +| `resources/views/booking.blade.php` | inline `onclick` | `public/js/booking.js` | `confirmCancel(bookingId)` | [VERIFIED: resources/views/booking.blade.php:63, public/js/booking.js:57-65] | +| `public/js/booking.js` | `fetch()` request | BookingController | `checkAvailability()` | [VERIFIED: public/js/booking.js:19, routes/web.php:29-30] | +| `public/js/booking.js` | 30-second polling (`setInterval`) | BookingController | `checkAvailability()` per slot card | [VERIFIED: public/js/booking.js:94-98, app/Http/Controllers/BookingController.php:95] | --- -## 7. File/Folder Conventions +## 4. File/Folder Conventions | Pattern | Location | Example | |---------|----------|---------| -| Models | `app/Models/` | `User.php`, `Booking.php` | -| Controllers | `app/Http/Controllers/` | `BookingController.php` | -| Services | `app/Services/` | `CalendarService.php` | -| Contracts | `app/Contracts/` | `CalendarServiceInterface.php` | -| Events | `app/Events/` | `BookingCreated.php` | -| Listeners | `app/Listeners/` | `SyncToExternalCalendar.php` | -| Providers | `app/Providers/` | `CalendarServiceProvider.php` | -| Views | `resources/views/` | `booking.blade.php` | -| JS | `public/js/` | `booking.js` | -| Config | `config/` | `calendar.php` | - -[ASSUMED: Laravel convention] Routes in `routes/web.php` for web routes. +| Models | `app/Models/` | `User.php`, `Booking.php`, `TimeSlot.php` [VERIFIED: app/Models/Booking.php:3] | +| Controllers | `app/Http/Controllers/` | `BookingController.php` [VERIFIED: app/Http/Controllers/BookingController.php:3] | +| Services | `app/Services/` | `CalendarService.php` [VERIFIED: app/Services/CalendarService.php:3] | +| Contracts | `app/Contracts/` | `CalendarServiceInterface.php` [VERIFIED: app/Contracts/CalendarServiceInterface.php:3] | +| Events | `app/Events/` | `BookingCreated.php`, `BookingCancelled.php` [VERIFIED: app/Events/BookingCreated.php:3] | +| Listeners | `app/Listeners/` | `SyncToExternalCalendar.php` [VERIFIED: app/Listeners/SyncToExternalCalendar.php:3] | +| Providers | `app/Providers/` | `CalendarServiceProvider.php` [VERIFIED: app/Providers/CalendarServiceProvider.php:3] | +| Views | `resources/views/` | `booking.blade.php` [VERIFIED: resources/views/booking.blade.php:1] | +| JS | `public/js/` | `booking.js` [VERIFIED: public/js/booking.js:1] | +| Config | `config/` | `calendar.php` [VERIFIED: config/calendar.php:1] | +| Migrations | `database/migrations/` | 3 `create_*_table.php` files [VERIFIED: database/migrations/2024_01_01_000002_create_time_slots_table.php:1] | + +[ASSUMED: Laravel convention] Web routes live in `routes/web.php`; the file's own header comment says auth middleware is assumed to be applied globally rather than attached per-route [VERIFIED: routes/web.php:11-12]. --- -## 8. Service Provider Wiring +## 5. External Dependencies -[VERIFIED: app/Providers/CalendarServiceProvider.php:18-21] -```php -$this->app->bind( - CalendarServiceInterface::class, - CalendarService::class -); -``` - -[VERIFIED: app/Providers/CalendarServiceProvider.php:31] -```php -Event::subscribe(SyncToExternalCalendar::class); -``` - -[ASSUMED: Laravel convention] Provider registered in `config/app.php` providers array. - ---- - -## 9. Frontend-Backend Interaction +### External Calendar API -### AJAX Availability Check +The only external dependency is a calendar HTTP API, configured in `config/calendar.php` and called from `CalendarService` via the `Http` facade [VERIFIED: app/Services/CalendarService.php:7, 30-33]. -[VERIFIED: public/js/booking.js:17-25] -```javascript -function checkSlotAvailability(slotId) { - fetch('/api/slots/availability?slot_id=' + slotId) - .then(response => response.json()) - .then(data => { - updateSlotDisplay(slotId, data); - }); -} +[VERIFIED: config/calendar.php:14-15] +```php + 'api_url' => env('CALENDAR_API_URL', 'https://api.example-calendar.com/v1'), + 'api_key' => env('CALENDAR_API_KEY'), ``` -[VERIFIED: app/Http/Controllers/BookingController.php:89-100] -Returns JSON with `available`, `spots_left`, `start_time`. +Call sites: +- `POST {api_url}/events` — booking sync [VERIFIED: app/Services/CalendarService.php:30-33] +- `DELETE {api_url}/events/{external_calendar_id}` — booking removal [VERIFIED: app/Services/CalendarService.php:63-65] -### Auto-refresh Polling - -[VERIFIED: public/js/booking.js:73-78] -```javascript -setInterval(function() { - document.querySelectorAll('.slot-card').forEach(card => { - checkSlotAvailability(card.dataset.slotId); - }); -}, 30000); -``` +Environment variables required: `CALENDAR_API_URL`, `CALENDAR_API_KEY` [VERIFIED: config/calendar.php:14-15]. -[INFERRED] Polling continues even when browser tab is inactive (no visibility check). +[NOT_FOUND: searched "mail", "Mail::", "Notification" in app/] No email integration. +[NOT_FOUND: searched "sms", "twilio", "nexmo", "vonage" in app/] No SMS integration. --- -## 10. Known Issues & Technical Debt +## 6. Known Issues & Risks -### Duplicated Logic +### Duplicated 24-hour cancellation rule -**24-hour cancellation rule:** -- [VERIFIED: app/Models/Booking.php:40] Hardcoded `>= 24` -- [VERIFIED: public/js/booking.js:11] Hardcoded `CANCEL_HOURS_BEFORE = 24` -- [VERIFIED: config/calendar.php:39] Defined as `'cancel_hours_before' => 24` but NOT USED +The 24-hour rule lives in three places, one of which is dead config: +- `Booking::canCancel()` hardcodes `>= 24` [VERIFIED: app/Models/Booking.php:46] +- `booking.js` hardcodes `CANCEL_HOURS_BEFORE = 24` [VERIFIED: public/js/booking.js:11] +- `config/calendar.php` defines `'cancel_hours_before' => 24` but nothing reads it [VERIFIED: config/calendar.php:41] -**Capacity checking:** -- [VERIFIED: app/Models/TimeSlot.php:33-40] `hasAvailability()` method -- [VERIFIED: app/Http/Controllers/BookingController.php:42-45] Duplicated in controller +### Duplicated capacity check -### Missing Error Handling +- `TimeSlot::hasAvailability()` implements the capacity rule [VERIFIED: app/Models/TimeSlot.php:31-42] +- The controller re-implements the same count inline; the file's own comment flags the duplication [VERIFIED: app/Http/Controllers/BookingController.php:42-44] -[VERIFIED: app/Services/CalendarService.php:27-31] +### Missing error handling on calendar sync + +[VERIFIED: app/Services/CalendarService.php:29-33] ```php -$response = Http::withHeaders([...])->post($this->apiUrl . '/events', $payload); + // NOTE: no try/catch here - errors bubble up + $response = Http::withHeaders([ + 'Authorization' => 'Bearer ' . $this->apiKey, + 'Content-Type' => 'application/json', + ])->post($this->apiUrl . '/events', $payload); ``` -No try/catch. HTTP errors will throw unhandled exceptions. +HTTP transport failures propagate uncaught; API-level failures are logged and swallowed (`return null`) with no retry and no user-visible surfacing [VERIFIED: app/Services/CalendarService.php:44-51]. + +### Event fired before status is final -[VERIFIED: app/Services/CalendarService.php:43-47] -Sync failure is logged but not surfaced to user or retried. +`BookingCreated` fires while the booking still has status `'pending'`; the controller flips it to `'confirmed'` only afterwards, regardless of sync outcome [VERIFIED: app/Http/Controllers/BookingController.php:57, 60]. -### Unused Configuration +### Unused configuration -[VERIFIED: config/calendar.php:26-27] +[VERIFIED: config/calendar.php:27-28] ```php -'sync_timeout' => env('CALENDAR_SYNC_TIMEOUT', 30), -'retry_attempts' => env('CALENDAR_RETRY_ATTEMPTS', 3), + 'sync_timeout' => env('CALENDAR_SYNC_TIMEOUT', 30), + 'retry_attempts' => env('CALENDAR_RETRY_ATTEMPTS', 3), ``` -[NOT_FOUND: searched "sync_timeout\|retry_attempts" in app/Services/] -These config values are defined but never read. +[NOT_FOUND: searched "sync_timeout", "retry_attempts" in app/] Defined but never read. +[NOT_FOUND: searched "max_bookings_per_user" in app/] `'max_bookings_per_user' => 5` is defined at config/calendar.php:42 but not enforced anywhere. -### Potential XSS +### Unvalidated notes field (potential XSS) -[VERIFIED: app/Http/Controllers/BookingController.php:49] +[VERIFIED: app/Http/Controllers/BookingController.php:53] ```php -'notes' => $request->input('notes'), + 'notes' => $request->input('notes'), ``` -No sanitization on notes field. Stored directly to database. +No validation or sanitization; stored raw. [INFERRED] If `notes` is ever rendered unescaped, XSS is possible — the current Blade view does not render it at all. -[INFERRED] If notes are displayed without escaping, XSS is possible. +### Status constants vs. string literals -### Status Constant Inconsistency +Constants are defined [VERIFIED: app/Models/Booking.php:18-20] but raw strings are used at: +- `'status' => 'pending'` in `store()` [VERIFIED: app/Http/Controllers/BookingController.php:52] +- `where('status', 'confirmed')` in the capacity checks [VERIFIED: app/Http/Controllers/BookingController.php:43, app/Models/TimeSlot.php:38] +- `where('status', 'confirmed')` in `User::activeBookings()` [VERIFIED: app/Models/User.php:25] +- `$this->status === 'cancelled'` in `canCancel()` [VERIFIED: app/Models/Booking.php:38] -[VERIFIED: app/Models/Booking.php:19-21] Constants defined: -```php -const STATUS_PENDING = 'pending'; -const STATUS_CONFIRMED = 'confirmed'; -const STATUS_CANCELLED = 'cancelled'; -``` +### No double-booking guard -But literal strings used in: -- [VERIFIED: app/Http/Controllers/BookingController.php:48] `'status' => 'pending'` -- [VERIFIED: app/Models/TimeSlot.php:37] `->where('status', 'confirmed')` +The bookings migration has no unique constraint on (`user_id`, `time_slot_id`); the migration's own comment flags the gap [VERIFIED: database/migrations/2024_01_01_000003_create_bookings_table.php:27-28]. --- -## Why This Example is GOOD - -1. **Every claim has a verification tag** - Reader knows what's proven vs assumed +## 7. Entry Points Summary -2. **File:line citations** - Can checkout commit `e043013` and verify each claim +| Route/Entry | Method | Handler | Middleware | Verified | +|-------------|--------|---------|------------|----------| +| `/booking` | GET | `BookingController@index` | none attached here | [VERIFIED: routes/web.php:17-18] | +| `/booking` | POST | `BookingController@store` | none attached here | [VERIFIED: routes/web.php:21-22] | +| `/booking/{booking}/cancel` | POST | `BookingController@cancel` | none attached here | [VERIFIED: routes/web.php:25-26] | +| `/api/slots/availability` | GET | `BookingController@checkAvailability` | none attached here | [VERIFIED: routes/web.php:29-30] | +| `BookingCreated` event | Event | `SyncToExternalCalendar::handleCreated` | — | [VERIFIED: app/Listeners/SyncToExternalCalendar.php:62] | +| `BookingCancelled` event | Event | `SyncToExternalCalendar::handleCancelled` | — | [VERIFIED: app/Listeners/SyncToExternalCalendar.php:63] | -3. **Actual code quoted** - Not descriptions, but the real code +[ASSUMED: Laravel convention] The routes file comment says auth middleware is applied globally in `RouteServiceProvider`, but no middleware is attached in `routes/web.php` itself [VERIFIED: routes/web.php:11-12]. -4. **NOT_FOUND explicitly stated** - Documents what DOESN'T exist (email, SMS, BookingService) +--- -5. **Issues surfaced** - Found real problems: duplicated logic, missing error handling, unused config +## 8. Technology Stack Summary -6. **Metadata locked to commit** - Documentation tied to specific code state +| Layer | Technology | Evidence | +|-------|------------|----------| +| Backend Framework | Laravel-style (Eloquent models, Blade views, `Illuminate\*` facades) | [VERIFIED: app/Http/Controllers/BookingController.php:9, app/Models/Booking.php:5] | +| Frontend | Server-rendered Blade + vanilla JavaScript | [VERIFIED: resources/views/booking.blade.php:89, public/js/booking.js:1] | +| Database | Migration-defined relational schema (3 tables) | [VERIFIED: database/migrations/2024_01_01_000002_create_time_slots_table.php:14] | +| HTTP Client | `Illuminate\Support\Facades\Http` | [VERIFIED: app/Services/CalendarService.php:7] | +| External Services | External calendar API (env-configured) | [VERIFIED: config/calendar.php:14-15] | -7. **Verification summary** - Quick assessment of documentation reliability +[NOT_FOUND: no composer.json, package.json, or artisan in examples/laravel/slotbooker/] The example app ships application code only, so the exact framework version is inferred from the `Illuminate\*` imports rather than a manifest. -8. **Both flows documented** - Shows booking creation AND cancellation paths +--- -9. **Frontend-backend connection** - Documents how JS interacts with API +## Why This Example is GOOD -10. **Actionable for AI agents** - An agent reading this can: - - Know exactly where to make changes - - Understand what doesn't exist (won't hallucinate BookingService) - - See the actual patterns used - - Identify risks before modifying +1. **Every claim has a verification tag** — the reader always knows what is proven vs. inferred vs. absent. +2. **File:line citations re-derived from source** — checkout commit `9a69c14` and every citation lands on the claimed code. +3. **Quoted code is exact** — the verifier's phase 2 compares each fenced block against the cited slice; paraphrased quotes fail. +4. **NOT_FOUND documents absences** — no BookingService, no email, no SMS. An agent reading this will not hallucinate them. +5. **Section 3 uses discovery tables, not execution traces** — no ASCII arrow flows; step-by-step tracing is deferred to the code-flow doc where it belongs. +6. **Known issues surfaced** — duplicated rules, silent sync failure, unused config, missing unique constraint, constant/literal drift. +7. **Machine-checkable** — the "Verify with" command above exits 0; the doc is only "done" while that stays true. diff --git a/examples/laravel/good-code-flow-doc-example.md b/examples/laravel/good-code-flow-doc-example.md new file mode 100644 index 0000000..bfeef6c --- /dev/null +++ b/examples/laravel/good-code-flow-doc-example.md @@ -0,0 +1,325 @@ +# Create Booking Code Flow (SlotBooker) + +> **This is an example of GOOD code flow documentation.** +> Every step cites real code in `examples/laravel/slotbooker/` and every quoted +> block is an exact copy of the cited slice, so the verifier can check it. + +## Metadata +| Field | Value | +|-------|-------| +| Repository | `agent-system-mapper` | +| Path | `examples/laravel/slotbooker` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | +| Trigger | User submits the "Book This Slot" form (`POST /booking`) | +| End State | Booking row with status `confirmed`; external calendar event created; redirect with success flash | + +**Verify with:** +```bash +python3 verify.py examples/laravel/good-code-flow-doc-example.md --repo-root examples/laravel/slotbooker +``` + +## Verification Summary +- `[VERIFIED]`: 31 tags — 36/36 citations resolve; 12/12 quoted blocks match their source +- `[INFERRED]`: 1 tag +- `[NOT_FOUND]`: 3 items (no request validation class, no `$listen` array, no email) + +--- + +## Flow Diagram + +``` +[Form submit: POST /booking] booking.blade.php:33-38 + │ + ▼ +routes/web.php:21 ──→ BookingController::store() BookingController.php:37 + │ + ├──→ TimeSlot::findOrFail($slotId) BookingController.php:40 + ├──→ inline capacity check (duplicated) BookingController.php:43-46 + ├──→ Booking::create([... 'pending' ...]) BookingController.php:49-54 + │ + ├──⚡ event(new BookingCreated($booking)) BookingController.php:57 + │ │ (synchronous — subscriber, not queued) + │ ▼ + │ SyncToExternalCalendar::handleCreated() SyncToExternalCalendar.php:22-34 + │ │ + │ ├──→ CalendarService::syncBooking() CalendarService.php:25-52 + │ │ │ + │ │ └──→ HTTP POST {api_url}/events CalendarService.php:30-33 + │ │ + │ └──→ Booking::markSynced($externalId) Booking.php:52-56 + │ + ├──→ $booking->update(['status' => 'confirmed']) BookingController.php:60 + │ + ▼ +[Redirect to booking.index with success flash] BookingController.php:62-63 +``` + +Symbols: `│ ▼` synchronous flow, `⚡` event dispatch, `├──→` method call. + +--- + +## Detailed Flow + +### Step 1: Form Submission + +[VERIFIED: resources/views/booking.blade.php:33-38] +```html +
+ @csrf + + + +
+``` + +Rendered once per available slot. The form action is hardcoded instead of using `route()` — the template's own comment flags this [VERIFIED: resources/views/booking.blade.php:32]. + +**Submits:** `time_slot_id` (hidden), `notes` (optional text), CSRF token +**To:** `POST /booking` + +--- + +### Step 2: Route Match + +[VERIFIED: routes/web.php:21-22] +```php +Route::post('/booking', [BookingController::class, 'store']) + ->name('booking.store'); +``` + +**Calls:** `BookingController::store()` + +--- + +### Step 3: Controller — Capacity Check and Booking Creation + +[VERIFIED: app/Http/Controllers/BookingController.php:37-54] +```php + public function store(Request $request) + { + $slotId = $request->input('time_slot_id'); + $slot = TimeSlot::findOrFail($slotId); + + // Wart: duplicates capacity check from TimeSlot::hasAvailability() + $confirmedCount = $slot->bookings()->where('status', 'confirmed')->count(); + if ($confirmedCount >= $slot->capacity) { + return back()->with('error', 'This slot is no longer available'); + } + + // Wart: no validation on notes field, could be XSS + $booking = Booking::create([ + 'user_id' => auth()->id(), + 'time_slot_id' => $slotId, + 'status' => 'pending', // Wart: should use constant + 'notes' => $request->input('notes'), + ]); +``` + +**Data in:** `{time_slot_id: int, notes: string|null}` — raw `$request->input()`, no validation layer +**Data out:** `Booking` model instance with status `'pending'` +**Alternate exit:** slot full → `back()` with error flash (line 45); unknown slot id → `findOrFail` throws (line 40) + +[NOT_FOUND: searched "validate", "FormRequest", "rules(" in app/Http/] No request validation class exists for this flow. + +--- + +### Step 4: Event Dispatch + +[VERIFIED: app/Http/Controllers/BookingController.php:56-57] +```php + // Fire event - listener will sync to external calendar + event(new BookingCreated($booking)); +``` + +The event object just wraps the model: + +[VERIFIED: app/Events/BookingCreated.php:13-18] +```php + public Booking $booking; + + public function __construct(Booking $booking) + { + $this->booking = $booking; + } +``` + +**Dispatch type:** synchronous (`⚡`). Neither the event nor the listener implements a queue interface, so listeners run inside the request [VERIFIED: app/Events/BookingCreated.php:9-11, app/Listeners/SyncToExternalCalendar.php:10]. + +--- + +### Step 5: Listener Wiring (Subscriber Pattern) + +The listener is registered as an event subscriber in the provider's `boot()`: + +[VERIFIED: app/Providers/CalendarServiceProvider.php:34] +```php + Event::subscribe(SyncToExternalCalendar::class); +``` + +[VERIFIED: app/Listeners/SyncToExternalCalendar.php:59-65] +```php + public function subscribe($events): array + { + return [ + BookingCreated::class => 'handleCreated', + BookingCancelled::class => 'handleCancelled', + ]; + } +``` + +[NOT_FOUND: searched "protected $listen" in app/Providers/] No EventServiceProvider `$listen` array — the subscriber is the only wiring, and the listener's own comment notes this is less discoverable. + +--- + +### Step 6: Listener Handles the Event + +[VERIFIED: app/Listeners/SyncToExternalCalendar.php:22-34] +```php + public function handleCreated(BookingCreated $event): void + { + $booking = $event->booking; + + Log::info('Syncing new booking to calendar', ['booking_id' => $booking->id]); + + $externalId = $this->calendarService->syncBooking($booking); + + if ($externalId) { + $booking->markSynced($externalId); + } + // Wart: if sync fails, we just log it (in service) but don't retry or notify anyone + } +``` + +**Calls:** `CalendarService::syncBooking()` via `CalendarServiceInterface`, constructor-injected [VERIFIED: app/Listeners/SyncToExternalCalendar.php:14-17] and bound in the provider [VERIFIED: app/Providers/CalendarServiceProvider.php:19-22] +**Data in:** `BookingCreated` event carrying a Booking with status `'pending'` +**Data out:** none (void) — side effects only + +--- + +### Step 7: External API Call + +[VERIFIED: app/Services/CalendarService.php:25-52] +```php + public function syncBooking(Booking $booking): ?string + { + $payload = $this->buildPayload($booking); + + // NOTE: no try/catch here - errors bubble up + $response = Http::withHeaders([ + 'Authorization' => 'Bearer ' . $this->apiKey, + 'Content-Type' => 'application/json', + ])->post($this->apiUrl . '/events', $payload); + + if ($response->successful()) { + $externalId = $response->json('id'); + Log::info('Booking synced to calendar', [ + 'booking_id' => $booking->id, + 'external_id' => $externalId, + ]); + return $externalId; + } + + // Wart: we log but don't throw, caller doesn't know it failed + Log::error('Calendar sync failed', [ + 'booking_id' => $booking->id, + 'status' => $response->status(), + 'body' => $response->body(), + ]); + + return null; + } +``` + +The payload is built by a private helper: + +[VERIFIED: app/Services/CalendarService.php:86-95] +```php + return [ + 'title' => 'Booking: ' . $user->name, + 'start' => $slot->start_time->format('Y-m-d\TH:i:s'), + 'end' => $slot->end_time->format('Y-m-d\TH:i:s'), + 'attendee_email' => $user->email, + 'metadata' => [ + 'booking_id' => $booking->id, + 'source' => 'slotbooker', + ], + ]; +``` + +**Endpoint:** `POST {config('calendar.api_url')}/events`, Bearer auth from `calendar.api_key` [VERIFIED: config/calendar.php:14-15] +**Data out:** external event id (`$response->json('id')`) on success; `null` on API failure — the caller cannot distinguish failure from success-without-id + +--- + +### Step 8: Sync Result Stored; Controller Finishes + +On a successful sync the listener persists the external id: + +[VERIFIED: app/Models/Booking.php:52-56] +```php + public function markSynced($externalId) + { + $this->external_calendar_id = $externalId; + $this->save(); + } +``` + +Control returns to the controller (the listener ran synchronously), which confirms and redirects **regardless of sync outcome**: + +[VERIFIED: app/Http/Controllers/BookingController.php:59-63] +```php + // Wart: assuming event succeeded, mark as confirmed + $booking->update(['status' => 'confirmed']); + + return redirect()->route('booking.index') + ->with('success', 'Booking confirmed!'); +``` + +**Data out:** HTTP redirect to `booking.index` with a `success` flash message. [INFERRED] `redirect()` produces an HTTP 302 response. + +--- + +## External Calls + +| Call | Where | Endpoint | Payload | Response | +|------|-------|----------|---------|----------| +| Calendar event create | [VERIFIED: app/Services/CalendarService.php:30-33] | `POST {calendar.api_url}/events` | `{title, start, end, attendee_email, metadata: {booking_id, source}}` [VERIFIED: app/Services/CalendarService.php:86-95] | JSON body with `id` on success [VERIFIED: app/Services/CalendarService.php:36] | + +Database writes in this flow: +- Booking insert with status `'pending'` [VERIFIED: app/Http/Controllers/BookingController.php:49-54] +- `external_calendar_id` saved by `markSynced` [VERIFIED: app/Models/Booking.php:54-55] +- Status update to `'confirmed'` [VERIFIED: app/Http/Controllers/BookingController.php:60] + +--- + +## Events Fired + +| Event | Fired At | Listeners | +|-------|----------|-----------| +| `BookingCreated` | [VERIFIED: app/Http/Controllers/BookingController.php:57] | `SyncToExternalCalendar::handleCreated` [VERIFIED: app/Listeners/SyncToExternalCalendar.php:62] | + +`BookingCancelled` also exists but is fired only by the cancel flow [VERIFIED: app/Http/Controllers/BookingController.php:85] — out of scope for this document. + +--- + +## Known Issues + +1. **Redundant status dance** — the booking is created `'pending'` and then unconditionally updated to `'confirmed'` a few lines later, with no branch in between [VERIFIED: app/Http/Controllers/BookingController.php:52, 60] +2. **Event fires before final state** — the listener (and the external calendar) observe a `'pending'` booking [VERIFIED: app/Http/Controllers/BookingController.php:57, 60] +3. **Silent sync failure** — `syncBooking()` returns `null` on API failure; the controller confirms anyway, so the user sees "Booking confirmed!" with no calendar event [VERIFIED: app/Services/CalendarService.php:44-51, app/Http/Controllers/BookingController.php:60-63] +4. **No transport error handling** — an HTTP exception aborts the request after the booking row exists but before confirmation, leaving the booking stuck at `'pending'` [VERIFIED: app/Services/CalendarService.php:29-33] +5. **Overbooking window** — the inline capacity check counts only `'confirmed'` bookings and there is no unique constraint on (`user_id`, `time_slot_id`) [VERIFIED: app/Http/Controllers/BookingController.php:43, database/migrations/2024_01_01_000003_create_bookings_table.php:27-28] +6. [NOT_FOUND: searched "mail", "Mail::", "Notification", "notify" in app/] No confirmation email or notification is sent anywhere in this flow. + +--- + +## Why This Example is GOOD + +1. **Every step has `[VERIFIED: file:line]`** with the actual code pasted, not paraphrased. +2. **The event chain is traced, not assumed** — dispatch → subscriber registration → handler method, each with evidence. +3. **Sync vs. async is explicit** — the listener is shown to run in-request because no queue interface exists. +4. **Data shapes at boundaries** — what the form submits, what the API receives, what comes back. +5. **Dead ends are documented** — no validation class, no `$listen` array, no email: `[NOT_FOUND]` with the searches that were run. +6. **Known issues fall out of the trace** — the pending→confirmed dance and the silent sync failure are visible only because the real path was followed. +7. **Machine-checkable** — the "Verify with" command exits 0. diff --git a/examples/livewire/bad-architecture-doc-example.md b/examples/livewire/bad-architecture-doc-example.md index 613988e..50ed9e9 100644 --- a/examples/livewire/bad-architecture-doc-example.md +++ b/examples/livewire/bad-architecture-doc-example.md @@ -1,9 +1,13 @@ # ApprovalFlow Architecture Overview +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns. Each ❌ callout explains a failure. See good-architecture-doc-example.md for the correct approach. + ## What This System Does ApprovalFlow is a comprehensive multi-step approval workflow system built with Laravel Livewire. It handles expense requests, content moderation, and internal change requests with full role-based access control. +> **❌ PROBLEMS:** No citations, and the scope is invented. There is one generic `Request` model with `title`, `description`, `amount` fields (`app/Models/Request.php:13-22`) and no request-type column in the schema (`database/migrations/2024_01_01_000002_create_requests_table.php:11-25`) — "content moderation" and "internal change requests" do not exist anywhere in the code. "Multi-step" is also wrong: a single reviewer approves or rejects (`app/Models/Request.php:84-107`); there are no approval chains. + ## Technology Stack - Laravel 10 with Livewire 3 @@ -13,6 +17,8 @@ ApprovalFlow is a comprehensive multi-step approval workflow system built with L - Redis for caching and queue - Pusher for real-time updates +> **❌ PROBLEMS:** Four of six items are hallucinated from framework habit. `composer.json:4-11` declares only `php`, `laravel/framework`, `livewire/livewire`, and `laravel/pint` — there is no Alpine.js, no TailwindCSS, no Redis, no Pusher, and no `package.json` at all. Views use plain CSS classes (`resources/views/livewire/request-list.blade.php:3-8`). The migrations use the driver-agnostic Schema builder; MySQL is never named. A good doc would write `[NOT_FOUND: searched "alpine", "tailwind", "redis", "pusher"]` for each. + ## Components ### Livewire Components @@ -24,17 +30,23 @@ ApprovalFlow is a comprehensive multi-step approval workflow system built with L - NotificationBell - Real-time notification dropdown - UserProfile - User settings and preferences +> **❌ PROBLEMS:** `NotificationBell` and `UserProfile` do not exist — `app/Livewire/` contains exactly five classes: `ApprovalActions.php`, `CommentSection.php`, `RequestDetail.php`, `RequestForm.php`, `RequestList.php`. "Real-time updates" is false everywhere it appears: components refresh only via Livewire `$listeners` after a user action (`app/Livewire/RequestList.php:21`), not via push. Comments are flat, not threaded — there is no parent/reply column (`database/migrations/2024_01_01_000003_create_comments_table.php:11-20`). And "with confirmation" is only half true: reject opens a modal (`resources/views/livewire/approval-actions.blade.php:20-43`) but approve fires immediately (`resources/views/livewire/approval-actions.blade.php:9-11`). + ### Services - ApprovalService - Orchestrates the approval workflow - NotificationService - Sends emails, SMS, and push notifications - AuditService - Tracks all system changes - PDFService - Generates approval certificates +> **❌ PROBLEMS:** The entire service layer is hallucinated. `app/` contains only `Enums`, `Events`, `Listeners`, `Livewire`, `Models`, `Policies` — there is no `app/Services/` directory and none of these four classes exist. Approval logic lives in the model (`app/Models/Request.php:84-107`), audit writing in `AuditLog::logStatusChange` (`app/Models/AuditLog.php:54-67`) called from the listener (`app/Listeners/SendStatusNotification.php:18-23`), and "notifications" are `Log::info` stubs (`app/Listeners/SendStatusNotification.php:35-40`). Nothing generates PDFs. + ### Jobs - SendApprovalNotification - Queued email sending - GenerateReportJob - Weekly summary reports - CleanupOldRequests - Archive old completed requests +> **❌ PROBLEMS:** There is no `app/Jobs/` directory and no queued job anywhere in the example. All three classes are invented. The only queue-adjacent code is the `SerializesModels` trait import on the event (`app/Events/RequestStatusChanged.php:8`), which proves nothing about queues being used. + ## Data Flow 1. User creates request via Livewire form @@ -45,6 +57,8 @@ ApprovalFlow is a comprehensive multi-step approval workflow system built with L 6. Requester notified of outcome 7. PDF certificate generated if approved +> **❌ PROBLEMS:** Two failures at once. First, a numbered step-by-step trace belongs in code-flow documentation, not an architecture overview — the methodology requires tables describing what moves, not execution steps. Second, steps 4, 6, and 7 are false: "notification" is a log line, `Log::info("New request pending review: ...")` (`app/Listeners/SendStatusNotification.php:39`) and `Log::info("Request {id} was {action}")` (`app/Listeners/SendStatusNotification.php:46`); no Pusher exists; nothing generates a PDF certificate. + ## Authentication Uses Laravel Sanctum with: @@ -53,6 +67,8 @@ Uses Laravel Sanctum with: - API tokens for mobile app - OAuth for SSO integration +> **❌ PROBLEMS:** Entirely fabricated. `composer.json:4-11` does not include Sanctum, and there is no MFA, token, or OAuth code anywhere. The example only calls the `Auth` facade (e.g. `app/Livewire/ApprovalActions.php:22`) and the routes file comment defers auth to a `RouteServiceProvider` that is not part of this example (`routes/web.php:14`). There is no mobile app and no API route (`routes/web.php:19-28` defines four GET web routes only). + ## Real-time Features - Live updates when request status changes @@ -60,6 +76,8 @@ Uses Laravel Sanctum with: - Presence indicators showing who's viewing - Typing indicators in comments +> **❌ PROBLEMS:** None of this exists. Searching the whole app for "pusher", "broadcast", "websocket", "presence", "typing" finds nothing. Updates happen only when a component dispatches a Livewire event during a user's own request cycle (`app/Livewire/ApprovalActions.php:54`, `app/Livewire/CommentSection.php:46`) and siblings re-render via `$refresh` listeners (`app/Livewire/RequestDetail.php:12-15`). No other browser is ever notified. + ## Database Schema | Table | Description | @@ -71,3 +89,19 @@ Uses Laravel Sanctum with: | notifications | User notifications | | attachments | File uploads | | approval_chains | Multi-level approval rules | + +> **❌ PROBLEMS:** The last three tables are invented. `database/migrations/` contains exactly four migrations: `2024_01_01_000001_create_users_table.php`, `..._000002_create_requests_table.php`, `..._000003_create_comments_table.php`, `..._000004_create_audit_logs_table.php`. There are no `notifications`, `attachments`, or `approval_chains` tables — padding a schema table with plausible-sounding rows is exactly the hallucination the methodology exists to prevent. + +## Why This Example is BAD + +1. **No metadata, no commit hash, no verification tags anywhere** — not a single claim carries `[VERIFIED: path:line]`, so nothing can be checked and `verify.py` has nothing to verify. → The good example resolves 104/104 citations. +2. **Invented technology stack**: Alpine.js, TailwindCSS, MySQL, Redis, Pusher → reality: `composer.json:4-11` declares only Laravel, Livewire, and Pint; no JS dependency file exists. +3. **Invented components**: NotificationBell, UserProfile → reality: five component classes in `app/Livewire/` (`RequestList.php:11`, `RequestForm.php:10`, `RequestDetail.php:8`, `ApprovalActions.php:9`, `CommentSection.php:10`). +4. **Invented service layer**: ApprovalService, NotificationService, AuditService, PDFService → reality: no `app/Services/`; logic sits in models (`app/Models/Request.php:66-107`) and the listener (`app/Listeners/SendStatusNotification.php:13-33`). +5. **Invented jobs**: SendApprovalNotification, GenerateReportJob, CleanupOldRequests → reality: no `app/Jobs/` directory exists. +6. **False notification claims**: "Reviewers notified via Pusher", emails/SMS → reality: log lines only (`app/Listeners/SendStatusNotification.php:39`, `app/Listeners/SendStatusNotification.php:46`), acknowledged by the wart comment at `app/Listeners/SendStatusNotification.php:26`. +7. **Invented authentication**: Sanctum, MFA, API tokens, OAuth → reality: bare `Auth::user()` calls (`app/Livewire/RequestList.php:35`) and a comment pointing at a provider that isn't in the example (`routes/web.php:14`). +8. **Invented real-time features**: live updates, presence, typing indicators → reality: request-cycle Livewire events only (`app/Livewire/ApprovalActions.php:54`). +9. **Padded database schema**: notifications, attachments, approval_chains tables → reality: four migrations only (`database/migrations/2024_01_01_000001_create_users_table.php` through `..._000004_create_audit_logs_table.php`). +10. **Step-by-step "Data Flow" trace in an architecture doc** — execution tracing belongs in code-flow documentation; the overview must stay at discovery level (tables, not numbered steps). +11. **No `[NOT_FOUND]` admissions** — a trustworthy doc records what it searched for and failed to find; this one asserts instead of admitting. diff --git a/examples/livewire/good-architecture-doc-example.md b/examples/livewire/good-architecture-doc-example.md index 4921f73..236e1b7 100644 --- a/examples/livewire/good-architecture-doc-example.md +++ b/examples/livewire/good-architecture-doc-example.md @@ -5,106 +5,94 @@ |-------|-------| | Repository | `agent-system-mapper` | | Path | `examples/livewire/approval-flow/` | -| Commit | `5d83fc5` | -| Documented | `2025-12-21` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | | Verification Status | `Verified` | +**Verify with:** +```bash +python3 verify.py examples/livewire/good-architecture-doc-example.md --repo-root examples/livewire/approval-flow +``` + ## Verification Summary -- [VERIFIED]: 42 claims -- [INFERRED]: 3 claims -- [NOT_FOUND]: 8 items (Alpine.js, Pusher, Redis, PDF, SMS, OAuth, attachments, approval chains) -- [ASSUMED]: 1 item (Laravel conventions) +- `[VERIFIED]`: 88 tags (104 `file:line` citations, all resolving; 17 quoted blocks, all matching) +- `[INFERRED]`: 1 claim (database driver) +- `[NOT_FOUND]`: 13 items (services, extra components, extra tables, real-time, attachments, approval chains, email/SMS, Alpine.js, CSS framework, API routes, cancellation path, external services, auth scaffolding) +- `[ASSUMED]`: 1 item (Laravel auth conventions) --- -## System Classification +## 0. System Classification + | Field | Value | |-------|-------| -| Type | Laravel Livewire hybrid (server-rendered with reactive components) | -| Evidence | `composer.json` with `livewire/livewire`, `app/Livewire/` components | +| Category | Traditional Code | +| Type | Laravel Livewire hybrid (server-rendered pages with reactive components) | +| Evidence | `composer.json` requires `livewire/livewire` [VERIFIED: composer.json:7]; five Livewire component classes under `app/Livewire/` [VERIFIED: app/Livewire/RequestList.php:11, app/Livewire/RequestForm.php:10] | +| Overlay Loaded | No | | Confidence | `[VERIFIED]` | --- -## System Purpose +## 1. System Purpose -ApprovalFlow is a **multi-step approval workflow system** for managing requests that require review and approval. +ApprovalFlow is a **multi-step approval workflow system**: requesters draft requests with a title, description, and dollar amount; reviewers and admins review, approve, or reject them; every status change is written to an audit trail. It is a Laravel 10 + Livewire 3 application. -[VERIFIED: `composer.json:6-7`] +[VERIFIED: composer.json:5-7] ```json -"laravel/framework": "^10.0", -"livewire/livewire": "^3.0" + "php": "^8.2", + "laravel/framework": "^10.0", + "livewire/livewire": "^3.0" ``` -Key features: -- Submit requests for approval [VERIFIED: `app/Models/Request.php:64-75`] -- Role-based review permissions [VERIFIED: `app/Enums/UserRole.php`] -- Multi-status workflow [VERIFIED: `app/Enums/RequestStatus.php`] -- Comments with internal/public visibility [VERIFIED: `app/Models/Comment.php`] -- Audit trail [VERIFIED: `app/Models/AuditLog.php`] - ---- - -## Component Map +Key capabilities: -| Component | Location | Responsibility | Verified | -|-----------|----------|----------------|----------| -| RequestList | `app/Livewire/RequestList.php` | Paginated list with filters | [VERIFIED] | -| RequestForm | `app/Livewire/RequestForm.php` | Create/edit requests | [VERIFIED] | -| RequestDetail | `app/Livewire/RequestDetail.php` | View single request | [VERIFIED] | -| ApprovalActions | `app/Livewire/ApprovalActions.php` | Approve/reject controls | [VERIFIED] | -| CommentSection | `app/Livewire/CommentSection.php` | Add/view comments | [VERIFIED] | -| Request model | `app/Models/Request.php` | Request entity + workflow | [VERIFIED] | -| User model | `app/Models/User.php` | User with role | [VERIFIED] | -| Comment model | `app/Models/Comment.php` | Comment entity | [VERIFIED] | -| AuditLog model | `app/Models/AuditLog.php` | Activity logging | [VERIFIED] | -| RequestPolicy | `app/Policies/RequestPolicy.php` | Authorization rules | [VERIFIED] | - -[NOT_FOUND: searched "NotificationBell", "UserProfile" in app/Livewire/] -No NotificationBell or UserProfile components. - -[NOT_FOUND: searched "Service" in app/] -No service classes - business logic is in models and Livewire components. +- Draft → submit-for-review lifecycle on the Request model [VERIFIED: app/Models/Request.php:66-79] +- Approve / reject by reviewers [VERIFIED: app/Models/Request.php:84-93, app/Models/Request.php:98-107] +- Role-based review permissions via a backed enum [VERIFIED: app/Enums/UserRole.php:23-26] +- Six-state status workflow [VERIFIED: app/Enums/RequestStatus.php:5-12] +- Comments with internal/public visibility [VERIFIED: app/Models/Comment.php:36-43] +- Audit trail of status changes [VERIFIED: app/Models/AuditLog.php:54-67] --- -## Livewire Component Architecture - -### Component Communication - -Components communicate via Livewire events: - -[VERIFIED: `app/Livewire/RequestList.php:18`] -```php -protected $listeners = ['requestUpdated' => '$refresh']; -``` - -[VERIFIED: `app/Livewire/ApprovalActions.php:45`] -```php -$this->dispatch('requestUpdated'); -``` - -[VERIFIED: `app/Livewire/RequestDetail.php:12-15`] -```php -protected $listeners = [ - 'requestUpdated' => '$refresh', - 'commentAdded' => '$refresh', -]; -``` - -### Component Hierarchy (on detail page) +## 2. Component Map -``` -RequestDetail -├── ApprovalActions (emits: requestUpdated) -└── CommentSection (emits: commentAdded) -``` - ---- - -## Status Workflow - -[VERIFIED: `app/Enums/RequestStatus.php:5-11`] +| Component | Location | Responsibility | Evidence | +|-----------|----------|----------------|----------| +| RequestList | `app/Livewire/RequestList.php` | Paginated list with search + status filter | [VERIFIED: app/Livewire/RequestList.php:11-18] | +| RequestForm | `app/Livewire/RequestForm.php` | Create/edit requests, submit for review | [VERIFIED: app/Livewire/RequestForm.php:10-22] | +| RequestDetail | `app/Livewire/RequestDetail.php` | View single request with nested components | [VERIFIED: app/Livewire/RequestDetail.php:8-15] | +| ApprovalActions | `app/Livewire/ApprovalActions.php` | Start review, approve, reject controls | [VERIFIED: app/Livewire/ApprovalActions.php:9-13] | +| CommentSection | `app/Livewire/CommentSection.php` | Add/view comments with visibility filter | [VERIFIED: app/Livewire/CommentSection.php:10-18] | +| Request model | `app/Models/Request.php` | Request entity + workflow methods | [VERIFIED: app/Models/Request.php:11-29] | +| User model | `app/Models/User.php` | User with role helpers | [VERIFIED: app/Models/User.php:9-15] | +| Comment model | `app/Models/Comment.php` | Comment entity + visibility rule | [VERIFIED: app/Models/Comment.php:8-14] | +| AuditLog model | `app/Models/AuditLog.php` | Activity logging (create-only) | [VERIFIED: app/Models/AuditLog.php:8-24] | +| RequestPolicy | `app/Policies/RequestPolicy.php` | view/update/delete/review authorization | [VERIFIED: app/Policies/RequestPolicy.php:8-13] | +| RequestStatusChanged | `app/Events/RequestStatusChanged.php` | Event carrying old + new status | [VERIFIED: app/Events/RequestStatusChanged.php:10-18] | +| SendStatusNotification | `app/Listeners/SendStatusNotification.php` | Writes audit log, logs "notifications" | [VERIFIED: app/Listeners/SendStatusNotification.php:11-23] | +| RequestStatus enum | `app/Enums/RequestStatus.php` | Status cases + transition guards | [VERIFIED: app/Enums/RequestStatus.php:5-12] | +| UserRole enum | `app/Enums/UserRole.php` | Role cases + permission helpers | [VERIFIED: app/Enums/UserRole.php:5-9] | + +[NOT_FOUND: searched "NotificationBell", "UserProfile" in app/Livewire/ — only the five components listed above exist] + +[NOT_FOUND: searched "Service" in app/ — no service classes; app/ contains only Enums, Events, Listeners, Livewire, Models, Policies. Business logic lives in models and Livewire components] + +### Database Schema (summary — details belong in Data Models documentation) + +| Table | Columns | Evidence | +|-------|---------|----------| +| users | id, name, email, role, timestamps | [VERIFIED: database/migrations/2024_01_01_000001_create_users_table.php:11-17] | +| requests | id, title, description, amount, requester_id, reviewer_id, status, submitted_at, reviewed_at, timestamps | [VERIFIED: database/migrations/2024_01_01_000002_create_requests_table.php:11-25] | +| comments | id, request_id, user_id, body, is_internal, timestamps | [VERIFIED: database/migrations/2024_01_01_000003_create_comments_table.php:11-20] | +| audit_logs | id, request_id, user_id, action, old_value, new_value, metadata, created_at | [VERIFIED: database/migrations/2024_01_01_000004_create_audit_logs_table.php:11-22] | + +[NOT_FOUND: searched "attachments", "approval_chains", "notifications" in database/migrations/ — only the four tables above exist] + +### Status Workflow + +[VERIFIED: app/Enums/RequestStatus.php:5-12] ```php enum RequestStatus: string { @@ -114,219 +102,274 @@ enum RequestStatus: string case APPROVED = 'approved'; case REJECTED = 'rejected'; case CANCELLED = 'cancelled'; -} ``` -Workflow transitions: -``` -DRAFT → PENDING (submit) -PENDING → UNDER_REVIEW (reviewer starts) -UNDER_REVIEW → APPROVED | REJECTED (reviewer decides) -DRAFT | PENDING → CANCELLED (requester cancels) -``` +Editing is only allowed in DRAFT or PENDING: -[VERIFIED: `app/Models/Request.php:103-107`] +[VERIFIED: app/Enums/RequestStatus.php:41-44] ```php -public function canEdit(): bool -{ - return $this->status->isEditable(); -} + public function isEditable(): bool + { + return in_array($this, [self::DRAFT, self::PENDING]); + } ``` -[VERIFIED: `app/Enums/RequestStatus.php:38-41`] -```php -public function isEditable(): bool -{ - return in_array($this, [self::DRAFT, self::PENDING]); -} -``` +The model delegates to the enum guard [VERIFIED: app/Models/Request.php:112-115]; reviewing is allowed from PENDING or UNDER_REVIEW [VERIFIED: app/Models/Request.php:120-124]. Transition writers: `submit()` sets PENDING [VERIFIED: app/Models/Request.php:74], `approve()` sets APPROVED [VERIFIED: app/Models/Request.php:87], `reject()` sets REJECTED [VERIFIED: app/Models/Request.php:101], and `startReview()` in the component sets UNDER_REVIEW [VERIFIED: app/Livewire/ApprovalActions.php:29-32]. ---- +[NOT_FOUND: searched "CANCELLED" in app/ — the enum defines CANCELLED but no code path assigns it; cancellation is not implemented] -## Role-Based Access +### Role-Based Access -[VERIFIED: `app/Enums/UserRole.php:5-9`] +[VERIFIED: app/Enums/UserRole.php:5-9] ```php enum UserRole: string { case REQUESTER = 'requester'; case REVIEWER = 'reviewer'; case ADMIN = 'admin'; -} ``` -Permissions: +[VERIFIED: app/Enums/UserRole.php:23-26] +```php + public function canApprove(): bool + { + return in_array($this, [self::REVIEWER, self::ADMIN]); + } +``` + +`canViewAll()` grants reviewers/admins visibility of all requests [VERIFIED: app/Enums/UserRole.php:31-34]; the User model exposes both helpers [VERIFIED: app/Models/User.php:44-47, app/Models/User.php:52-55]. + +### Event System -[VERIFIED: `app/Enums/UserRole.php:22-25`] +[VERIFIED: app/Events/RequestStatusChanged.php:14-18] ```php -public function canApprove(): bool -{ - return in_array($this, [self::REVIEWER, self::ADMIN]); -} + public function __construct( + public Request $request, + public RequestStatus $oldStatus, + public RequestStatus $newStatus, + ) {} ``` -[VERIFIED: `app/Enums/UserRole.php:30-33`] +Fired on every status transition, e.g. in `submit()`: + +[VERIFIED: app/Models/Request.php:78] ```php -public function canViewAll(): bool -{ - return in_array($this, [self::REVIEWER, self::ADMIN]); -} + event(new RequestStatusChanged($this, $oldStatus, $this->status)); ``` +The listener writes the audit log entry: + +[VERIFIED: app/Listeners/SendStatusNotification.php:17-23] +```php + // Log the status change + AuditLog::logStatusChange( + $request, + Auth::user(), + $event->oldStatus->value, + $event->newStatus->value + ); +``` + +--- + +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) + +### 3.1 Primary Execution Surfaces + +| Entry Surface | Type | Primary Components Involved | Evidence | +|---------------|------|-----------------------------|----------| +| `GET /` | Web Route (full-page Livewire) | RequestList, Request model | [VERIFIED: routes/web.php:19] | +| `GET /requests/create` | Web Route | RequestForm, Request model | [VERIFIED: routes/web.php:22] | +| `GET /requests/{request}` | Web Route | RequestDetail, ApprovalActions, CommentSection | [VERIFIED: routes/web.php:25] | +| `GET /requests/{request}/edit` | Web Route | RequestForm | [VERIFIED: routes/web.php:28] | +| Livewire actions (`wire:submit` / `wire:click` / `wire:model.live`) | Livewire AJAX round-trip | RequestForm, ApprovalActions, CommentSection, RequestList | See §3b for the per-interaction map | + +### 3.2 High-Level Data Movement (Non-Procedural) + +| Stage | Input Type | Output Type | Participating Components | +|-------|------------|-------------|--------------------------| +| Request creation | Validated form fields | Request record (status `draft`) | RequestForm, Request model | +| Submission | Draft request | Pending request + RequestStatusChanged event | RequestForm, Request model | +| Review decision | Pending/under-review request | Approved or rejected request + event | ApprovalActions, Request model | +| Audit logging | RequestStatusChanged event | AuditLog record | SendStatusNotification, AuditLog model | +| Commenting | Comment form fields | Comment record + `commentAdded` event | CommentSection, Comment model | +| List/detail refresh | `requestUpdated` / `commentAdded` events | Re-rendered component HTML | RequestList, RequestDetail | + +### 3.3 Pointers to Code Flow Documentation + +Detailed execution paths are deliberately **not** traced here — see `02-code-flows.md` for: + +- **Submit Request flow** — entry `RequestForm::submit()` [VERIFIED: app/Livewire/RequestForm.php:71-88] +- **Approve/Reject flow** — entry `ApprovalActions::approve()` / `reject()` [VERIFIED: app/Livewire/ApprovalActions.php:37-55, app/Livewire/ApprovalActions.php:68-85] +- **Add Comment flow** — entry `CommentSection::addComment()` [VERIFIED: app/Livewire/CommentSection.php:25-49] + --- -## Frontend → Backend Interaction Map +## 3b. Frontend → Backend Interaction Map + +Each row is a distinct frontend-triggered backend entry point (discovery only; behavior belongs in Code Flow documentation). | Frontend Source | Trigger Type | Backend Target | Handler / Method | Evidence | |-----------------|--------------|----------------|------------------|----------| -| request-list.blade.php | wire:model.live | RequestList.php | search/filter | [VERIFIED:request-list.blade.php:5-6] | -| request-form.blade.php | wire:submit | RequestForm.php | save() | [VERIFIED:request-form.blade.php:2] | -| request-form.blade.php | wire:click | RequestForm.php | submit() | [VERIFIED:request-form.blade.php:45] | -| approval-actions.blade.php | wire:click | ApprovalActions.php | approve() | [VERIFIED:approval-actions.blade.php:10] | -| approval-actions.blade.php | wire:click | ApprovalActions.php | reject() | [VERIFIED:approval-actions.blade.php:29] | -| comment-section.blade.php | wire:submit | CommentSection.php | addComment() | [VERIFIED:comment-section.blade.php:3] | +| `request-list.blade.php` | `wire:model.live` | RequestList.php | `search` / `statusFilter` updates | [VERIFIED: resources/views/livewire/request-list.blade.php:6, 11] | +| `request-form.blade.php` | `wire:submit` | RequestForm.php | `save()` | [VERIFIED: resources/views/livewire/request-form.blade.php:2] | +| `request-form.blade.php` | `wire:click` | RequestForm.php | `submit()` | [VERIFIED: resources/views/livewire/request-form.blade.php:50] | +| `approval-actions.blade.php` | `wire:click` | ApprovalActions.php | `startReview()` | [VERIFIED: resources/views/livewire/approval-actions.blade.php:5] | +| `approval-actions.blade.php` | `wire:click` | ApprovalActions.php | `approve()` | [VERIFIED: resources/views/livewire/approval-actions.blade.php:9] | +| `approval-actions.blade.php` | `wire:click` | ApprovalActions.php | `openRejectModal()` / `closeRejectModal()` | [VERIFIED: resources/views/livewire/approval-actions.blade.php:12, 34] | +| `approval-actions.blade.php` | `wire:click` | ApprovalActions.php | `reject()` | [VERIFIED: resources/views/livewire/approval-actions.blade.php:37] | +| `comment-section.blade.php` | `wire:submit` | CommentSection.php | `addComment()` | [VERIFIED: resources/views/livewire/comment-section.blade.php:5] | ---- +### Component-to-Component Events -## Event System +RequestDetail's view nests the other two interactive components [VERIFIED: resources/views/livewire/request-detail.blade.php:40, 44]. They coordinate through Livewire events: -[VERIFIED: `app/Events/RequestStatusChanged.php:10-15`] +[VERIFIED: app/Livewire/RequestList.php:21] ```php -public function __construct( - public Request $request, - public RequestStatus $oldStatus, - public RequestStatus $newStatus, -) {} + protected $listeners = ['requestUpdated' => '$refresh']; ``` -Fired when status changes: -[VERIFIED: `app/Models/Request.php:74`] +[VERIFIED: app/Livewire/RequestDetail.php:12-15] ```php -event(new RequestStatusChanged($this, $oldStatus, $this->status)); + protected $listeners = [ + 'requestUpdated' => '$refresh', + 'commentAdded' => '$refresh', + ]; ``` -Listener creates audit log: -[VERIFIED: `app/Listeners/SendStatusNotification.php:17-23`] +[VERIFIED: app/Livewire/ApprovalActions.php:54] ```php -public function handle(RequestStatusChanged $event): void -{ - // Log the status change - AuditLog::logStatusChange( - $request, - Auth::user(), - $event->oldStatus->value, - $event->newStatus->value - ); + $this->dispatch('requestUpdated'); +``` + +[VERIFIED: app/Livewire/CommentSection.php:46] +```php + $this->dispatch('commentAdded'); ``` --- -## Database Schema +## 4. File/Folder Conventions + +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| `app/Livewire/*.php` | One class per page or nested component | [VERIFIED: app/Livewire/RequestList.php:11, app/Livewire/RequestDetail.php:8] | +| `resources/views/livewire/*.blade.php` | Kebab-case Blade view per component, resolved via `view('livewire.request-list', ...)` | [VERIFIED: app/Livewire/RequestList.php:60, app/Livewire/RequestForm.php:92] | +| `app/Models/` | Eloquent models carrying workflow methods | [VERIFIED: app/Models/Request.php:11, app/Models/User.php:9] | +| `app/Enums/` | Backed enums for status and role | [VERIFIED: app/Enums/RequestStatus.php:5, app/Enums/UserRole.php:5] | +| `app/Events/` + `app/Listeners/` | One event + one listener pair for status changes | [VERIFIED: app/Events/RequestStatusChanged.php:10, app/Listeners/SendStatusNotification.php:11] | +| `app/Policies/` | Authorization rules per model | [VERIFIED: app/Policies/RequestPolicy.php:8] | +| `database/migrations/` | Four dated migrations, one table each | [VERIFIED: database/migrations/2024_01_01_000001_create_users_table.php:11, database/migrations/2024_01_01_000004_create_audit_logs_table.php:11] | +| `routes/web.php` | All routes map directly to Livewire component classes | [VERIFIED: routes/web.php:19-28] | + +--- + +## 5. External Dependencies -| Table | Columns | Source | -|-------|---------|--------| -| users | id, name, email, role, timestamps | [VERIFIED: migrations/000001] | -| requests | id, title, description, amount, requester_id, reviewer_id, status, submitted_at, reviewed_at, timestamps | [VERIFIED: migrations/000002] | -| comments | id, request_id, user_id, body, is_internal, timestamps | [VERIFIED: migrations/000003] | -| audit_logs | id, request_id, user_id, action, old_value, new_value, metadata, created_at | [VERIFIED: migrations/000004] | +| Dependency | Purpose | Evidence | +|------------|---------|----------| +| `laravel/framework` ^10.0 | Application framework | [VERIFIED: composer.json:6] | +| `livewire/livewire` ^3.0 | Reactive server-rendered components | [VERIFIED: composer.json:7] | +| `laravel/pint` ^1.0 (dev) | Code style | [VERIFIED: composer.json:10] | -[NOT_FOUND: searched "attachments", "approval_chains" in migrations/] -No attachments or approval_chains tables. +[NOT_FOUND: searched "Http::", "env(" in app/ — no outbound HTTP calls and no environment-driven service configuration] + +[NOT_FOUND: searched "pusher", "redis", "sanctum" in approval-flow/ — no real-time, cache/queue, or API-auth dependencies declared] --- -## Known Issues / Warts +## 6. Known Issues & Risks -### 1. Business Logic Duplication +### 6.1 Business Logic Duplication -[VERIFIED: `app/Models/Request.php:65`] +[VERIFIED: app/Models/Request.php:68] ```php -// Wart: Business rule check duplicated in Livewire component + // Wart: Business rule check duplicated in Livewire component ``` -Same editable check appears in: -- `app/Models/Request.php:103-106` -- `app/Livewire/RequestForm.php:41-45` +The same editable check appears in the model and the form component [VERIFIED: app/Models/Request.php:112-115, app/Livewire/RequestForm.php:41-44]. -### 2. Role Check Duplication +### 6.2 Role Check Duplication -[VERIFIED: `app/Livewire/RequestList.php:34`] +[VERIFIED: app/Livewire/RequestList.php:40] ```php -// Wart: This logic duplicated in RequestPolicy + // Wart: This logic duplicated in RequestPolicy ``` -Same view permission logic in: -- `app/Livewire/RequestList.php:35-37` -- `app/Policies/RequestPolicy.php:17-22` +The same view-permission logic appears in the component query and the policy [VERIFIED: app/Livewire/RequestList.php:41-43, app/Policies/RequestPolicy.php:21-22]. -### 3. Rejection Reason Not Saved +### 6.3 Rejection Reason Collected But Never Saved -[VERIFIED: `app/Livewire/ApprovalActions.php:71-72`] +[VERIFIED: app/Livewire/ApprovalActions.php:77-78] ```php -// Wart: Rejection reason not actually saved anywhere -// Should create a comment with the reason + // Wart: Rejection reason not actually saved anywhere + // Should create a comment with the reason ``` -### 4. Notifications Not Implemented +The reject modal binds a `rejectionReason` textarea that is discarded [VERIFIED: resources/views/livewire/approval-actions.blade.php:26-31, app/Livewire/ApprovalActions.php:62-66]. + +### 6.4 Notifications Are Log Lines Only -[VERIFIED: `app/Listeners/SendStatusNotification.php:36-38`] +[VERIFIED: app/Listeners/SendStatusNotification.php:35-40] ```php -private function notifyReviewers($request): void -{ - // Wart: Should get all reviewers and notify them - // Currently just logs - Log::info("New request pending review: {$request->id}"); -} + private function notifyReviewers($request): void + { + // Wart: Should get all reviewers and notify them + // Currently just logs + Log::info("New request pending review: {$request->id}"); + } ``` -### 5. Comment Audit Missing +### 6.5 Comments Bypass the Audit Trail -[VERIFIED: `app/Livewire/CommentSection.php:42`] +[VERIFIED: app/Livewire/CommentSection.php:48] ```php -// Wart: Should also log to audit trail + // Wart: Should also log to audit trail ``` ---- - -## Entry Points +### 6.6 Features Confirmed Absent -[VERIFIED: `routes/web.php:16-23`] - -| Route | Method | Handler | Verified | -|-------|--------|---------|----------| -| `/` | GET | RequestList::class | [VERIFIED] | -| `/requests/create` | GET | RequestForm::class | [VERIFIED] | -| `/requests/{request}` | GET | RequestDetail::class | [VERIFIED] | -| `/requests/{request}/edit` | GET | RequestForm::class | [VERIFIED] | +- [NOT_FOUND: searched "pusher", "redis", "broadcast", "websocket" in approval-flow/ — no real-time updates] +- [NOT_FOUND: searched "attachment", "upload", "approval_chain" in approval-flow/ — no file attachments, no multi-level approval; single reviewer only] +- [NOT_FOUND: searched "Mail::", "Notification::", "sms" in app/ — no email/SMS delivery; the listener only writes log lines] +- [NOT_FOUND: searched "alpine", "x-data", "tailwind" in approval-flow/ — no Alpine.js, no CSS framework; views use plain CSS classes] +- [NOT_FOUND: searched "Route::post", "api" in routes/ — no POST or API endpoints; all four routes are GET Livewire pages] --- -## Technology Stack Summary +## 7. Entry Points Summary -| Layer | Technology | -|-------|------------| -| Backend Framework | Laravel 10 [VERIFIED: composer.json] | -| Frontend Framework | Livewire 3 [VERIFIED: composer.json] | -| Database | SQLite/MySQL (via Laravel) [INFERRED: migrations use Schema] | -| Authentication | Laravel built-in [ASSUMED: standard Laravel] | +| Route/Entry | Method | Handler | Middleware | Verified | +|-------------|--------|---------|------------|----------| +| `/` | GET | `RequestList::class` | see note below | [VERIFIED: routes/web.php:19] | +| `/requests/create` | GET | `RequestForm::class` | see note below | [VERIFIED: routes/web.php:22] | +| `/requests/{request}` | GET | `RequestDetail::class` | see note below | [VERIFIED: routes/web.php:25] | +| `/requests/{request}/edit` | GET | `RequestForm::class` | see note below | [VERIFIED: routes/web.php:28] | -[NOT_FOUND: searched "alpine", "tailwind" in approval-flow/] -No Alpine.js or TailwindCSS configuration found - views use plain CSS classes. +Middleware note: the routes file comment says authentication is "applied in RouteServiceProvider" [VERIFIED: routes/web.php:14], but that provider is not part of this example. [NOT_FOUND: searched "RouteServiceProvider", "middleware" in approval-flow/ — no provider or middleware definitions exist here] -[NOT_FOUND: searched "redis", "pusher" in approval-flow/] -No real-time infrastructure configured. +[ASSUMED: components call Auth::user() throughout (e.g. app/Livewire/RequestList.php:35), so a standard Laravel auth guard is assumed to be configured by the host application] --- -## What This System Does NOT Have +## 8. Technology Stack Summary + +| Layer | Technology | Evidence | +|-------|------------|----------| +| Backend Framework | Laravel 10 | [VERIFIED: composer.json:6] | +| Frontend Framework | Livewire 3 | [VERIFIED: composer.json:7] | +| Primary Database | Driver-agnostic (migrations use the Schema builder) | [INFERRED: Schema::create in all four migrations; no DB driver named anywhere in the example] | +| Authentication | Not included in example (see §7 note) | `[ASSUMED]` above | +| External Services | None | [NOT_FOUND: searched "Http::", "guzzle", "api_key" in approval-flow/ — no external service integration] | + +--- -Based on searches finding no results: +## Why This Example is GOOD -1. **No Real-time Updates** - No Pusher/WebSockets -2. **No Service Layer** - Logic in models and components -3. **No File Attachments** - No upload functionality -4. **No Multi-level Approval** - Single reviewer only -5. **No Email/SMS Notifications** - Just logging -6. **No Alpine.js** - Pure Livewire -7. **No CSS Framework** - Plain CSS classes -8. **No API Endpoints** - Web routes only +1. **Every claim is cited or admitted.** Each factual statement carries a `[VERIFIED: path:line]` tag that resolves against `examples/livewire/approval-flow/`, or an explicit `[NOT_FOUND]` / `[ASSUMED]` / `[INFERRED]` admission. +2. **Quotes are exact copy-paste.** Every fenced block matches the cited line range character-for-character, so `verify.py` phase 2 passes. +3. **Absence is documented with real searches.** The `[NOT_FOUND]` items name the patterns searched (Pusher, services, attachments, Alpine.js...), so a reader can re-run them. +4. **Section 3 stays at discovery level.** Tables describe entry surfaces and what moves — no step-by-step traces, no arrow diagrams; detailed tracing is deferred to `02-code-flows.md`. +5. **The Frontend → Backend map uses complete, resolvable paths.** Blade sources are cited as `resources/views/livewire/...blade.php:line`, not bare filenames. +6. **Warts are surfaced, not hidden.** Duplicated business rules, the discarded rejection reason, and log-only notifications are documented with the exact comment lines. +7. **It is machine-checkable.** Running the command in the metadata block exits 0. diff --git a/examples/model-systems/bad-architecture-doc-example.md b/examples/model-systems/bad-architecture-doc-example.md new file mode 100644 index 0000000..f2ca59f --- /dev/null +++ b/examples/model-systems/bad-architecture-doc-example.md @@ -0,0 +1,179 @@ +# Whisper Architecture Overview + +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns specific to model-centric systems — above all, documenting a famous model from prior knowledge instead of from the repo. Each ❌ callout explains a failure. See good-architecture-doc-example.md for the correct approach. + +## Metadata + +| Field | Value | +|-------|-------| +| System | OpenAI Whisper | +| Documented | 2026-08-03 | +| Verification Status | Complete | + +> **❌ PROBLEMS:** No repository, no path, no commit hash — nothing pins these +> claims to a checkable snapshot. "Complete" is not a verification status; the +> methodology requires `Verified` backed by a passing `verify.py` run. The good +> example pins Repository, Path, Upstream commit, and a verify command. + +## 1. System Classification + +| Field | Value | +|-------|-------| +| Type | State-of-the-art ASR foundation model | +| Evidence | Whisper is one of the most widely used speech models in the world | +| Confidence | HIGH | + +> **❌ PROBLEMS:** Popularity is not evidence — the classification cites the +> model's reputation, not a single file. The real evidence is in the repo: +> `torch` in requirements.txt:3, `nn.Module` classes in whisper/model.py:252, +> and a checkpoint URL registry at whisper/__init__.py:17-32. `HIGH` is not a +> valid confidence tag, and the required Category / Overlay Loaded rows +> (Model-Centric, 01a overlay) are missing entirely. + +## 2. Model Architecture + +Whisper is a Transformer encoder–decoder trained on 680,000 hours of +multilingual and multitask supervised data collected from the web. The audio +encoder ingests 80-channel log-mel spectrograms through two convolutional +layers followed by a stack of Transformer blocks with pre-activation residual +connections; the text decoder uses learned positional embeddings and +cross-attends to the encoder output. The `large-v3` checkpoint has 32 encoder +and 32 decoder layers, a width of 1280, 20 attention heads, and roughly 1.55 +billion parameters, while `turbo` prunes the decoder to 4 layers for an 8x +speedup at nearly identical accuracy. [VERIFIED] + +On benchmarks, Whisper achieves a 2.7% word error rate on LibriSpeech +test-clean and approaches human-level robustness on out-of-distribution +audio, which is why no fine-tuning is usually necessary. + +> **❌ PROBLEMS:** Every number here comes from the Whisper *paper*, not from +> this repository — the documenter "knows" Whisper from training data and wrote +> it down as if it were verified. Layer counts, widths, and parameter counts +> appear nowhere in the code: `ModelDimensions` is an empty schema filled from +> `checkpoint["dims"]` at load time (whisper/model.py:25-36, +> whisper/__init__.py:154). "680,000 hours" appears nowhere in this repo; the +> README mentions training only as prose pointing at the paper (README.md:8, +> README.md:15). The WER claim is nowhere in this repo either — README.md:76 +> links WER figures *in the paper*. A bare `[VERIFIED]` with no file:line +> after a paragraph of paper facts is the signature failure mode for famous +> ML repos. Even "Transformer" is not stated in the model code — the good +> example marks the architecture type `[INFERRED]` from the attention-block +> classes (whisper/model.py:81, 142, 174, 207). + +## 3. Model Weights + +The pretrained weights for all model sizes ship in the repository under +`whisper/assets/`, so the package works offline out of the box. Loading a +model simply deserializes the bundled checkpoint: + +``` +load_model("turbo") + ↓ +read whisper/assets/turbo.pt + ↓ +model ready +``` + +> **❌ PROBLEMS:** False, and checkably so. No `.pt`, `.safetensors`, or +> `.onnx` file exists anywhere in this repo — `_MODELS` at +> whisper/__init__.py:17-32 is a table of **download URLs**, and +> whisper/__init__.py:136-137 fetches the checkpoint at runtime into +> `~/.cache/whisper` (whisper/__init__.py:132-134), verifying a SHA-256 from +> the URL path (whisper/__init__.py:57, 90-93). `whisper/assets/` holds the +> mel filterbank and tokenizer vocab — not weights (whisper/audio.py:105, +> whisper/tokenizer.py:332) — and in this vendored copy it is pruned anyway +> (VENDORED.md:13). The ASCII arrow diagram is also a banned form: Section 3 +> must use tables. + +## 4. Training Pipeline + +Training uses AdamW with a linear learning-rate warmup over the first 2048 +steps, gradient checkpointing, and BF16 mixed precision across 256 GPUs. The +dataset mixture is weighted toward English but includes 96 other languages, +with SpecAugment applied to the mel spectrograms. The training loop lives in +the repo and can be re-run to reproduce the released checkpoints, and +fine-tuning on custom data is supported through the same entry point. + +> **❌ PROBLEMS:** None of this exists — not a single line. Searching the +> package for "train", "optimizer", "loss", and "backward" returns zero +> matches; the only gradient-related code *disables* gradients for inference +> (whisper/decoding.py:18, whisper/decoding.py:792, whisper/timing.py:196). +> There is no training loop, no fine-tuning entry point, no SpecAugment, no +> dataset code — nowhere in this repo. The optimizer, warmup schedule, GPU +> count, and mixed-precision details are speculation dressed as documentation. +> The correct treatment is a Boundaries section with recorded searches: +> `[NOT_FOUND: searched "train", "optimizer", "loss", "backward" in whisper/]`. + +## 5. Decoding Internals + +Transcription decodes each 30-second window as follows: + +1. Start decoding at temperature 0.0 using beam search with 5 beams; each + step expands every beam by the top-k next tokens and keeps the 5 highest + cumulative log-probability sequences, with finished beams set aside once + they emit end-of-text. +2. Compute the gzip compression ratio of the decoded text; if it exceeds 2.4 + the output is judged repetitive and the result is discarded. +3. Check the average log probability; if it is below -1.0 the decode is + judged low-confidence and discarded. +4. On either failure, raise the temperature by 0.2 and switch from beam + search to multinomial sampling with best-of-5 reranking, because sampling + explores the distribution better than beams at high temperature. +5. Repeat up to temperature 1.0; if the no-speech probability exceeds 0.6 the + window is skipped as silence. + +## Boundaries + +This system does NOT: + +- Provide training pipelines [VERIFIED: no training code found] +- Support fine-tuning [VERIFIED/NOT_FOUND] +- Include evaluation scripts [VERIFIED/NOT_FOUND] + +> **❌ PROBLEMS (Section 5):** This is step-by-step algorithm tracing at a +> depth the architecture overview bans outright — the overlay's rule is +> discovery only: name where decoding lives, defer the how to +> 02-code-flows.md. The fallback logic does exist (whisper/transcribe.py:184-224, +> defaults at whisper/transcribe.py:43) and the strategy choice exists +> (whisper/decoding.py:508-551, GreedyDecoder at whisper/decoding.py:272, +> BeamSearchDecoder at whisper/decoding.py:301), but not one of the five steps +> carries a citation, and step 4's "because sampling explores the distribution +> better" is an invented rationale that appears nowhere in the code. +> +> **❌ PROBLEMS (Boundaries):** The right facts wearing illegal tags. +> `[VERIFIED: no training code found]` abuses VERIFIED for an *absence* — a +> verifier can only resolve VERIFIED to a file and line, so absence claims +> must use `[NOT_FOUND: searched ...]` with the search terms recorded. +> `[VERIFIED/NOT_FOUND]` is not a tag at all — it is an unfilled template +> placeholder left in the doc. Correct forms: +> `[NOT_FOUND: searched "train", "optimizer", "loss", "backward" in whisper/]`, +> `[NOT_FOUND: searched "fine-tune", "finetune", "lora", "adapter" in whisper/]`. + +## Why This Example is BAD + +1. **False:** "Trained on 680,000 hours; 32 layers; 1.55B parameters; 2.7% + WER." **Reality:** paper knowledge, nowhere in this repo. Dimensions come + from the downloaded checkpoint at runtime (whisper/model.py:25-36, + whisper/__init__.py:154); the README's training and WER mentions are prose + pointing at the paper (README.md:8, README.md:76). +2. **False:** "Weights ship in the repository under whisper/assets/." + **Reality:** weights are downloaded at runtime from a URL table + (whisper/__init__.py:17-32, 136-137); `assets/` holds mel filters and + tokenizer vocab (whisper/audio.py:105, whisper/tokenizer.py:332) and is + pruned in this vendored copy (VENDORED.md:13). +3. **False:** "The training loop lives in the repo; fine-tuning is supported." + **Reality:** zero matches for "train", "optimizer", "loss", "backward" + anywhere in whisper/ — this is an inference-only codebase. +4. **Banned depth:** five-step beam-search and temperature-fallback trace with + an invented rationale — architecture docs identify surfaces + (whisper/transcribe.py:184-224) and defer internals to 02-code-flows.md. +5. **Illegal tags:** bare `[VERIFIED]` after paper facts, + `[VERIFIED: no training code found]` for an absence, and the template + placeholder `[VERIFIED/NOT_FOUND]` — none of these can be checked by + `verify.py`; not one claim in the document carries a file:line citation. +6. **Headline lesson:** the model's fame is not evidence about this + repository. The more famous the model, the more the documenter already + "knows" — and the more ruthlessly every claim must be re-derived from the + files actually present, with `[NOT_FOUND]` for everything (training, data, + metrics, weights) that lives in the paper or the checkpoint instead of the + code. diff --git a/examples/model-systems/good-architecture-doc-example.md b/examples/model-systems/good-architecture-doc-example.md new file mode 100644 index 0000000..33fc3c9 --- /dev/null +++ b/examples/model-systems/good-architecture-doc-example.md @@ -0,0 +1,368 @@ +# Architecture Overview: Whisper (vendored slim copy) + +> **This is the canonical model-centric (ML/AI) example for the agent-system-mapper methodology.** +> +> It documents the vendored slim copy of `openai/whisper` under +> `examples/model-systems/whisper/` and demonstrates the +> `01a-overlay-model-systems.md` overlay applied on top of the standard +> `01-architecture-overview.md` format. Every citation resolves against the +> vendored source, so the doc is machine-verifiable. Pay particular attention +> to how it handles what is NOT here: model weights (downloaded at runtime), +> training code (never existed in this repo), and the pruned `whisper/assets/` +> directory. + +## Metadata + +| Field | Value | +|-------|-------| +| Repository | `agent-system-mapper` | +| Path | `examples/model-systems/whisper/` | +| Upstream | `openai/whisper, commit c0d2f62` | +| Commit | `213c7d4` | +| Documented | `2026-08-03` | +| Verification Status | `Verified` | + +**Verify with:** + +```bash +python3 verify.py examples/model-systems/good-architecture-doc-example.md --repo-root examples/model-systems/whisper +``` + +## Verification Summary + +- VERIFIED: 100 tags — 160 `file:line` citations, all resolved (0 informal) +- INFERRED: 1 tag +- NOT_FOUND: 13 tags (each records the searches performed) +- ASSUMED: 0 tags +- NEEDS_VERIFICATION: 1 tag +- Quoted code blocks: 3/3 exact copies of the cited lines + +## Example Reference + +| Field | Value | +|-------|-------| +| Methodology | `prompts/01-architecture-overview.md` + `prompts/01a-overlay-model-systems.md` (overlay) | +| Key Format Elements | Section 3 as tables (no arrows, no traces); Model Asset Inventory separating weights from code; `[NOT_FOUND]` with recorded searches for training code; overlay's discovery-only rule (no attention math, no sampling internals) | + +--- + +## 0. System Classification + +| Field | Value | +|-------|-------| +| Category | Model-Centric (ML/AI) | +| Type | Inference library + CLI for a pretrained speech-recognition model | +| Evidence | `torch` dependency [VERIFIED: requirements.txt:3]; `nn.Module` model classes [VERIFIED: whisper/model.py:252]; registry of pretrained checkpoint URLs [VERIFIED: whisper/__init__.py:17-32] | +| Overlay Loaded | Yes: `01a-overlay-model-systems.md` | +| Confidence | `[VERIFIED]` | + +### Model System Classification (Overlay Step 0) + +| Field | Value | +|-------|-------| +| Model Type | Inference-only | +| Has Training Code | No — [NOT_FOUND: searched "train", "optimizer", "loss", "backward" in whisper/ — zero matches; README.md:8 and README.md:15 mention training only as prose about the upstream project] | +| Has Inference Code | Yes [VERIFIED: whisper/decoding.py:792-798, whisper/transcribe.py:38-56] | +| Weight Format | `.pt` (PyTorch checkpoints, loaded with `torch.load`) [VERIFIED: whisper/__init__.py:18, whisper/__init__.py:151] | +| Weight Source | Downloaded at runtime from `openaipublic.azureedge.net`; NOT shipped in the repo [VERIFIED: whisper/__init__.py:17-32] [NOT_FOUND: searched for *.pt, *.safetensors, *.onnx, *.bin files under examples/model-systems/whisper/ — none present] | + +--- + +## 1. System Purpose + +Whisper is a speech-recognition package: it loads a pretrained encoder–decoder +checkpoint and turns audio files into transcribed (or English-translated) text, +with optional word-level timestamps and subtitle-format output. This vendored +copy contains the Python source only — the package code defines the model +*architecture* and the *inference pipeline*, while the model *behavior* lives in +weights that are downloaded on first use into `~/.cache/whisper` +[VERIFIED: whisper/__init__.py:132-137]. The upstream README describes the approach and +links the paper [VERIFIED: README.md:8], but nothing about training exists in +this codebase (see Section 12, Boundaries). + +Note on this snapshot: `whisper/assets/` (mel filterbank + tokenizer vocab +data) and `whisper/normalizers/english.json` were pruned when vendoring +[VERIFIED: VENDORED.md:13], so this copy documents structure faithfully but is +not runnable as-is (see Section 6). + +--- + +## 2. Component Map + +| Component | Location | Responsibility | Evidence | +|-----------|----------|----------------|----------| +| Package API + weight download | `whisper/__init__.py` | `_MODELS` URL registry, `_download` with SHA-256 check, `available_models`, `load_model` | [VERIFIED: whisper/__init__.py:17-32, 54-95, 98-100, 103-161] | +| CLI shim | `whisper/__main__.py` | `python -m whisper` delegates to `transcribe.cli` | [VERIFIED: whisper/__main__.py:1-3] | +| Audio front-end | `whisper/audio.py` | ffmpeg-based loading, pad/trim, mel filterbank, log-mel spectrogram | [VERIFIED: whisper/audio.py:25-62, 65-88, 91-107, 110-157] | +| Model definition | `whisper/model.py` | `ModelDimensions`, `AudioEncoder`, `TextDecoder`, `Whisper` module; binds `transcribe`/`decode`/`detect_language` as methods | [VERIFIED: whisper/model.py:25-36, 174, 207, 252, 343-345] | +| Decoding surface | `whisper/decoding.py` | `DecodingOptions`/`DecodingResult` dataclasses, `DecodingTask`, `decode`, `detect_language` | [VERIFIED: whisper/decoding.py:80-114, 117-127, 508, 792-798, 18-21] | +| Transcription orchestrator + CLI | `whisper/transcribe.py` | `transcribe` (windowed inference over long audio), `cli` (argparse surface) | [VERIFIED: whisper/transcribe.py:38-56, 517-528] | +| Word-timing | `whisper/timing.py` | DTW alignment for word-level timestamps (`add_word_timestamps`) | [VERIFIED: whisper/timing.py:141-151, 279] | +| CUDA kernels | `whisper/triton_ops.py` | Triton DTW/median kernels, imported lazily from `timing.dtw_cuda` | [VERIFIED: whisper/triton_ops.py:13-14, whisper/timing.py:109] | +| Tokenizer | `whisper/tokenizer.py` | `Tokenizer` wrapper over tiktoken encodings; special control tokens | [VERIFIED: whisper/tokenizer.py:131-132, 330-363, 366-395] | +| Output writers | `whisper/utils.py` | `ResultWriter` family (txt/vtt/srt/tsv/json), `get_writer` | [VERIFIED: whisper/utils.py:85, 109, 238, 251, 265, 287, 296-318] | +| Text normalizers | `whisper/normalizers/` | Basic + English text normalization | [VERIFIED: whisper/normalizers/__init__.py:1-2] | +| Version | `whisper/version.py` | Single `__version__` string | [VERIFIED: whisper/version.py:1] | + +--- + +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) + +### 3.1 Primary Execution Surfaces + +| Entry Surface | Type | Pipeline Stage | Primary Components Involved | Evidence | +|---------------|------|----------------|-----------------------------|----------| +| `python -m whisper AUDIO...` | CLI | Full pipeline | `__main__` → `transcribe.cli` → `load_model` → `transcribe` → writers | [VERIFIED: whisper/__main__.py:1-3, whisper/transcribe.py:517] | +| `whisper AUDIO...` (console script) | CLI | Full pipeline | packaging entry point `whisper.transcribe:cli` | [VERIFIED: pyproject.toml:35] | +| `whisper.load_model(name)` | Library API | Weight acquisition + model construction | `_download`, `ModelDimensions`, `Whisper` | [VERIFIED: whisper/__init__.py:103-161] | +| `model.transcribe(audio)` | Library API | Full pipeline (long audio, windowed) | `log_mel_spectrogram`, `decode`, `Tokenizer`, `add_word_timestamps` | [VERIFIED: whisper/transcribe.py:38-56, whisper/model.py:344] | +| `whisper.decode(model, mel, options)` | Library API | Single 30-second segment | `DecodingTask`, `TextDecoder` | [VERIFIED: whisper/decoding.py:792-798] | +| `whisper.detect_language(model, mel)` | Library API | Language ID only | encoder + single-token decoder pass | [VERIFIED: whisper/decoding.py:18-21] | + +The console script is the only installed executable, declared in one line +[VERIFIED: pyproject.toml:35]: + +```toml +scripts.whisper = "whisper.transcribe:cli" +``` + +The package re-exports the library surface (`load_audio`, +`log_mel_spectrogram`, `pad_or_trim`, `decode`, `detect_language`, +`load_model`, `transcribe`) from the top of the package +[VERIFIED: whisper/__init__.py:11-15]. + +[NOT_FOUND: searched "flask", "fastapi", "gradio", "streamlit", "socket" in whisper/ — no web demo or HTTP serving surface exists] + +### 3.2 High-Level Data Movement (Non-Procedural) + +What moves, not how it moves. Stage internals (windowing, fallback, search +strategies) are deliberately not described here — see `02-code-flows.md`. + +| Stage | Input | Output | Component | Evidence | +|-------|-------|--------|-----------|----------| +| Weight acquisition | Model name | Verified `.pt` checkpoint file (cached) | `_download` | [VERIFIED: whisper/__init__.py:54-95] | +| Model construction | Checkpoint file | `Whisper` module on device | `load_model`, `ModelDimensions` | [VERIFIED: whisper/__init__.py:147-161] | +| Audio decode | Audio file path | float32 mono waveform @ 16 kHz | `load_audio` (ffmpeg subprocess) | [VERIFIED: whisper/audio.py:25-62] | +| Feature extraction | Waveform | Log-mel spectrogram tensor | `log_mel_spectrogram`, `mel_filters` | [VERIFIED: whisper/audio.py:91-107, 110-157] | +| Encoding | Mel segment | Audio feature tensor | `AudioEncoder` (inside `Whisper`) | [VERIFIED: whisper/model.py:174-204] | +| Decoding | Audio features + prompt tokens | Token ids + per-segment metrics | `decode` / `DecodingTask`, `TextDecoder` | [VERIFIED: whisper/decoding.py:792-826, whisper/model.py:207-249] | +| Text assembly | Token ids across windows | `{text, segments, language}` dict | `transcribe`, `Tokenizer.decode` | [VERIFIED: whisper/transcribe.py:510-514] | +| Output writing | Result dict | `.txt`/`.vtt`/`.srt`/`.tsv`/`.json` files | `get_writer` + `ResultWriter` subclasses | [VERIFIED: whisper/utils.py:296-318] | + +### 3.3 Pointers to Code Flow Documentation + +Candidates for detailed flow tracing (see `02-code-flows.md`): + +- **CLI transcription end-to-end** — argument parsing through writer output [VERIFIED: whisper/transcribe.py:517, whisper/transcribe.py:613-619] +- **Weight download & cache validation** — cache hit, checksum mismatch, re-download [VERIFIED: whisper/__init__.py:54-95] +- **Windowed transcription with fallback** — the per-segment decode strategy exists at whisper/transcribe.py:184-224 but its internals belong in the flow doc, not here +- **Word-timestamp alignment** — DTW with CUDA/CPU selection [VERIFIED: whisper/timing.py:141-151] + +### Section 3 Self-Check + +- [x] No method bodies longer than 3 lines quoted +- [x] No loops or conditionals described +- [x] No sampling, search, or attention algorithms explained +- [x] All movements as conceptual stages in tables +- [x] Defers to `02-code-flows.md` + +--- + +## 3b. Frontend → Backend Interaction Map + +Not applicable — this system is a CLI tool and Python library. The only user +surfaces are the terminal and the importable API (Section 3.1). [NOT_FOUND: +searched "html", "template", "fetch(", "ajax" in whisper/ — no frontend] + +--- + +## 4. File/Folder Conventions + +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| `whisper/` flat module layout | One module per pipeline concern (audio, model, decoding, transcribe, timing, tokenizer, utils) | [VERIFIED: whisper/version.py:1 and the ten sibling modules listed in Section 2] | +| `whisper/normalizers/` | Only subpackage; text normalization for downstream comparison of transcripts | [VERIFIED: whisper/normalizers/__init__.py:1-2] | +| `whisper/assets/` | Expected data directory for mel filterbank + tokenizer vocab; referenced by code but pruned from this vendored copy | [VERIFIED: whisper/audio.py:105, whisper/tokenizer.py:332] and [VERIFIED: VENDORED.md:13] | +| Late imports to break cycles | `model.py` imports `decode`/`transcribe`/`detect_language` functions and binds them as `Whisper` methods | [VERIFIED: whisper/model.py:12-14, whisper/model.py:343-345] | +| `TYPE_CHECKING` guards | Modules type-hint `Whisper` without importing it at runtime | [VERIFIED: whisper/transcribe.py:34-35, whisper/decoding.py:14-15, whisper/timing.py:15-16] | + +--- + +## 5. External Dependencies + +| Dependency | Purpose | Evidence | +|------------|---------|----------| +| `torch` | Model definition, tensor ops, checkpoint loading | [VERIFIED: whisper/model.py:8-10, whisper/__init__.py:151, requirements.txt:3] | +| `tiktoken` | BPE encodings backing the `Tokenizer` | [VERIFIED: whisper/tokenizer.py:8, whisper/tokenizer.py:357-363] | +| `numpy` | Array handling across audio/decoding | [VERIFIED: whisper/audio.py:6, whisper/decoding.py:4] | +| `numba` | JIT-compiled CPU DTW for word timing | [VERIFIED: whisper/timing.py:57-58, 82-83] | +| `triton` (Linux x86_64 only) | CUDA DTW/median kernels | [VERIFIED: whisper/triton_ops.py:6-10, requirements.txt:7] | +| `tqdm` | Download and transcription progress bars | [VERIFIED: whisper/__init__.py:9, whisper/transcribe.py:264-266] | +| `more-itertools` | `windowed()` in the English normalizer | [VERIFIED: whisper/normalizers/english.py:7] | +| `ffmpeg` (system binary, not a Python package) | Audio decode/resample via subprocess; must be on `PATH` | [VERIFIED: whisper/audio.py:42-58] | +| `openaipublic.azureedge.net` (network service) | Hosts the pretrained checkpoints fetched at runtime | [VERIFIED: whisper/__init__.py:17-32, whisper/__init__.py:73] | + +--- + +## 6. Known Issues & Risks + +| Risk | Location | Notes | +|------|----------|-------| +| This vendored snapshot cannot run inference | `whisper/audio.py:105`, `whisper/tokenizer.py:332` | Both load files from `whisper/assets/`, which was pruned when vendoring [VERIFIED: VENDORED.md:13]. Documenting this honestly beats pretending the assets exist. | +| English normalizer data pruned | `whisper/normalizers/english.py:458` | `EnglishSpellingNormalizer` opens `english.json` next to the module; also pruned [VERIFIED: VENDORED.md:13] | +| Checkpoint download has no retry/resume | `whisper/__init__.py:90-93` | A SHA-256 mismatch after download raises and asks the user to retry manually | +| Control flow admitted to be obscure | `whisper/transcribe.py:268-271` | An in-code NOTE says the main loop "is obscurely flattened to make the diff readable" and should be simplified later | +| Import-time failure mode for CUDA timing | `whisper/triton_ops.py:6-10` | Raises `RuntimeError` if `triton` is missing; mitigated by lazy import + CPU fallback [VERIFIED: whisper/timing.py:109, whisper/timing.py:141-151] | +| Typo in user-facing warning | `whisper/transcribe.py:577-582` | Warning text says "receipted" where "received" is meant | + +--- + +## 7. Entry Points Summary + +| Entry Type | Count | Locations | +|------------|-------|-----------| +| CLI scripts | 2 | `python -m whisper` [VERIFIED: whisper/__main__.py:1-3]; `whisper` console script [VERIFIED: pyproject.toml:35] | +| Public library functions | 8 re-exported names | `load_audio`, `log_mel_spectrogram`, `pad_or_trim`, `decode`, `detect_language`, `load_model`, `transcribe`, `available_models` [VERIFIED: whisper/__init__.py:11-15, 98-100, 103] | +| HTTP routes | 0 | [NOT_FOUND: no server code; the only network use is outbound checkpoint download at whisper/__init__.py:73] | +| Out-of-process commands | 1 | `ffmpeg` subprocess for audio decoding [VERIFIED: whisper/audio.py:45-58] | +| Event listeners / webhooks | 0 | [NOT_FOUND: searched "listen", "webhook", "callback registration" in whisper/] | + +--- + +## 8. Technology Stack Summary + +| Layer | Technology | Evidence | +|-------|------------|----------| +| Language | Python >= 3.8 | [VERIFIED: pyproject.toml:13] | +| ML framework | PyTorch (`torch`) | [VERIFIED: requirements.txt:3, whisper/model.py:8-10] | +| Tokenization | tiktoken (BPE) | [VERIFIED: whisper/tokenizer.py:8] | +| Acceleration | numba (CPU JIT), triton (CUDA kernels) | [VERIFIED: whisper/timing.py:57, whisper/triton_ops.py:13-14] | +| Audio I/O | ffmpeg via subprocess | [VERIFIED: whisper/audio.py:42-58] | +| Packaging | setuptools via `pyproject.toml`; console script entry | [VERIFIED: pyproject.toml:2, pyproject.toml:35] | +| License | MIT (code and upstream model weights) | [VERIFIED: pyproject.toml:11, README.md:160] | +| External services | Azure CDN for checkpoint hosting (runtime only) | [VERIFIED: whisper/__init__.py:17-32] | + +--- + +## 9. Model Asset Inventory (Overlay) + +The single most important distinction in a model-centric system: **what ships +as code in this repo vs. what arrives as data at runtime**. + +| Asset | In this repo? | Runtime source | Evidence | +|-------|---------------|----------------|----------| +| Model weights (`{name}.pt`) | **No** | Downloaded on first `load_model` into `$XDG_CACHE_HOME/whisper` (default `~/.cache/whisper`) from the `_MODELS` URL table | [VERIFIED: whisper/__init__.py:17-32, whisper/__init__.py:132-137] [NOT_FOUND: searched for *.pt, *.safetensors, *.onnx, *.bin under examples/model-systems/whisper/ — none present; upstream does not ship them either, per VENDORED.md:15-16] | +| Weight integrity check | n/a (code only) | Expected SHA-256 is the second-to-last URL path segment; verified on cache hit and after download | [VERIFIED: whisper/__init__.py:57, whisper/__init__.py:63-71, whisper/__init__.py:90-93] | +| Mel filterbank (`assets/mel_filters.npz`) | **No — pruned in this vendored copy** | Code expects it inside the package | [VERIFIED: whisper/audio.py:105] code path; [VERIFIED: VENDORED.md:13] pruning | +| Tokenizer vocab (`assets/gpt2.tiktoken`, `assets/multilingual.tiktoken`) | **No — pruned in this vendored copy** | Code expects it inside the package | [VERIFIED: whisper/tokenizer.py:332]; [VERIFIED: VENDORED.md:13] | +| English spelling map (`normalizers/english.json`) | **No — pruned in this vendored copy** | Code expects it next to the module | [VERIFIED: whisper/normalizers/english.py:458]; [VERIFIED: VENDORED.md:13] | +| Cross-attention alignment heads | **Yes** — inline base85-encoded blobs in source | Decoded at load time for word timing | [VERIFIED: whisper/__init__.py:36-51, whisper/model.py:278-285] | + +The weight download is two lines of dispatch, cited exactly +[VERIFIED: whisper/__init__.py:136-137]: + +```python + if name in _MODELS: + checkpoint_file = _download(_MODELS[name], download_root, in_memory) +``` + +And the pruned-asset dependency is a single load path +[VERIFIED: whisper/audio.py:105]: + +```python + filters_path = os.path.join(os.path.dirname(__file__), "assets", "mel_filters.npz") +``` + +--- + +## 10. Model Architecture (Verified Only) (Overlay) + +Only what the code states. Layer counts, parameter counts, and training details +live in the downloaded checkpoint and the upstream paper — not in this repo. + +| Field | Value | Evidence | +|-------|-------|----------| +| Model Class | `Whisper(nn.Module)` composed of `AudioEncoder` + `TextDecoder` | [VERIFIED: whisper/model.py:252-269] | +| Dimensions schema | `ModelDimensions` — 10 integers (mels, ctx sizes, widths, heads, layers, vocab) populated from `checkpoint["dims"]` at load time | [VERIFIED: whisper/model.py:25-36, whisper/__init__.py:154] | +| Parameter Count | [NOT_FOUND: not stated anywhere in the code; determined by the downloaded checkpoint] | — | +| Layer counts / widths per model size | [NOT_FOUND: not hard-coded; `load_model` reads them from the checkpoint. The code names model sizes only as download keys — whisper/__init__.py:17-32] | — | +| Architecture Type | Encoder–decoder built from residual attention blocks | [INFERRED: `AudioEncoder` and `TextDecoder` are stacks of `ResidualAttentionBlock` containing `MultiHeadAttention` — whisper/model.py:81, 142, 174, 207. The README's "Transformer sequence-to-sequence model" phrasing (README.md:15) is upstream prose, not code evidence] | +| Multilingual detection | Derived from vocab size at runtime (`n_vocab >= 51865`) | [VERIFIED: whisper/model.py:302-308] | +| Weight File(s) | `tiny`/`base`/`small`/`medium`/`large-v1..v3`/`turbo` variants as `{name}.pt` | [VERIFIED: whisper/__init__.py:17-32] | +| Weight Source | `https://openaipublic.azureedge.net/main/whisper/models/...` | [VERIFIED: whisper/__init__.py:18] | +| Training data | [NOT_FOUND: no dataset handling, no data loaders, no references in whisper/; the famous "680,000 hours" figure is from the upstream paper and appears nowhere in this repo] | — | +| Evaluation / benchmarks | [NOT_FOUND: searched "evaluate", "benchmark", "dataset", "wer" in whisper/ — zero matches; README.md:76 links WER figures in the paper, which is prose, not repo evidence] | — | + +--- + +## 11. Configuration & Control Surface (Overlay) + +| Config | Location | Controls | Evidence | +|--------|----------|----------|----------| +| CLI flags (~30) | `whisper/transcribe.py` | model choice, task, language, temperature schedule, thresholds, word timestamps, output format/dir, threads | [VERIFIED: whisper/transcribe.py:528-567] | +| `DecodingOptions` dataclass | `whisper/decoding.py` | per-segment decoding controls (task, language, temperature, beam/best-of sizes, prompts, token suppression, timestamp rules, fp16) | [VERIFIED: whisper/decoding.py:80-114] | +| `transcribe()` keyword args | `whisper/transcribe.py` | temperature tuple with fallback thresholds, prompt carrying, clip timestamps, hallucination-silence threshold | [VERIFIED: whisper/transcribe.py:38-56] | +| `XDG_CACHE_HOME` env var | `whisper/__init__.py` | Overrides the weight cache directory | [VERIFIED: whisper/__init__.py:132-134] | +| Audio constants | `whisper/audio.py` | 16 kHz sample rate, 30-second chunks, hop/frame geometry — every downstream tensor shape depends on these | [VERIFIED: whisper/audio.py:12-22] | +| Special control tokens | `whisper/tokenizer.py` | Task/language/timestamp control codes appended to the vocabulary in a fixed order | [VERIFIED: whisper/tokenizer.py:340-355] | + +--- + +## 12. Boundaries & Non-Responsibilities (Overlay) + +This system does NOT: + +- Train models — [NOT_FOUND: searched "train", "optimizer", "loss", "backward" in whisper/ — zero matches. The only gradient-related code *disables* gradients for inference: whisper/decoding.py:18, whisper/decoding.py:792, whisper/timing.py:196] +- Fine-tune or adapt weights — [NOT_FOUND: searched "fine-tune", "finetune", "lora", "adapter", "save_pretrained" in whisper/ — zero matches] +- Evaluate or benchmark — [NOT_FOUND: searched "evaluate", "benchmark", "dataset" in whisper/ — zero matches; tests were also pruned from this vendored copy per VENDORED.md:13] +- Ship model weights or serve them — weights are fetched from a CDN at runtime [VERIFIED: whisper/__init__.py:73] +- Record or stream audio — input is a file path or an in-memory array only [VERIFIED: whisper/transcribe.py:40, whisper/audio.py:25] + +Out of scope by design: speaker diarization, real-time/streaming transcription, +model quantization, and serving infrastructure — none of these appear anywhere +in the source. +[NEEDS_VERIFICATION: whether the upstream tests (pruned from this snapshot) exercise any additional surface cannot be checked from this copy] + +--- + +## 13. Risk & Change Surface (Overlay) + +| File/Component | Risk Level | Why | +|----------------|------------|-----| +| `_MODELS` URL/checksum registry | Critical | Every checkpoint fetch depends on these URLs and their embedded SHA-256 path segments; a stale entry bricks `load_model` [VERIFIED: whisper/__init__.py:17-32, whisper/__init__.py:57] | +| `whisper/assets/` contents (absent here) | Critical | Tokenizer vocab and mel filterbank; inference cannot start without them [VERIFIED: whisper/audio.py:105, whisper/tokenizer.py:332] | +| Special-token list order | Critical | Token ids are assigned by position in the `specials` list; reordering silently changes every control token id [VERIFIED: whisper/tokenizer.py:340-355] | +| Audio geometry constants | High | `SAMPLE_RATE`, `N_FFT`, `HOP_LENGTH`, `CHUNK_LENGTH` fix all tensor shapes and timing math [VERIFIED: whisper/audio.py:12-22] | +| `ModelDimensions` field names | High | Must match `checkpoint["dims"]` keys exactly for every published checkpoint [VERIFIED: whisper/model.py:25-36, whisper/__init__.py:154] | +| Output writer formats | Medium | Five subtitle/data formats consumed by external tooling [VERIFIED: whisper/utils.py:296-318] | + +--- + +## Why This Example is GOOD + +1. **Fame is not evidence.** Whisper is one of the best-known models in the + world, and that is exactly why this doc cites `whisper/__init__.py:17-32` + for what the repo *does* contain and marks training data, parameter counts, + and WER numbers as `[NOT_FOUND]` — those facts live in the paper and the + checkpoint, not in this repository. A doc for an obscure repo and a doc for + a famous repo must be held to the identical evidence standard. +2. **Assets are not code.** The Model Asset Inventory (Section 9) separates + three different kinds of "not in the repo": weights that are downloaded at + runtime (true upstream behavior, cited to the download code), data files + pruned only from this vendored copy (cited to `VENDORED.md:13` instead of + pretending they exist), and inline data that genuinely lives in source + (`_ALIGNMENT_HEADS`). Each gets its own evidence trail. +3. **Absence is searched, not assumed.** Every "does not exist" claim records + the exact search terms used (`train`, `optimizer`, `loss`, `backward`, + `lora`, `evaluate`, ...), so a reader can re-run them. +4. **Discovery-only depth.** Section 3 names the temperature-fallback and + beam-search *locations* without explaining a single step of how they work — + no sampling algorithms, no attention math, no arrow diagrams. That depth is + deferred to `02-code-flows.md`, per the overlay's non-procedural rule. +5. **Inference-only classification is explicit.** The overlay's Step 0 table + states up front that this is an inference-only system with runtime-fetched + `.pt` weights, so no reader can come away believing the repo trains, + fine-tunes, or evaluates anything. +6. **Machine-verifiable.** Every `[VERIFIED: file:line]` tag resolves against + `examples/model-systems/whisper/`, and the three quoted blocks are exact + copies of their cited lines — `verify.py` exits 0 on this document. diff --git a/examples/model-systems/whisper/LICENSE b/examples/model-systems/whisper/LICENSE new file mode 100644 index 0000000..d255525 --- /dev/null +++ b/examples/model-systems/whisper/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2022 OpenAI + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/examples/model-systems/whisper/README.md b/examples/model-systems/whisper/README.md new file mode 100644 index 0000000..196b48f --- /dev/null +++ b/examples/model-systems/whisper/README.md @@ -0,0 +1,160 @@ +# Whisper + +[[Blog]](https://openai.com/blog/whisper) +[[Paper]](https://arxiv.org/abs/2212.04356) +[[Model card]](https://github.com/openai/whisper/blob/main/model-card.md) +[[Colab example]](https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb) + +Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. + + +## Approach + +![Approach](https://raw.githubusercontent.com/openai/whisper/main/approach.png) + +A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targets. + + +## Setup + +We used Python 3.9.9 and [PyTorch](https://pytorch.org/) 1.10.1 to train and test our models, but the codebase is expected to be compatible with Python 3.8-3.11 and recent PyTorch versions. The codebase also depends on a few Python packages, most notably [OpenAI's tiktoken](https://github.com/openai/tiktoken) for their fast tokenizer implementation. You can download and install (or update to) the latest release of Whisper with the following command: + + pip install -U openai-whisper + +Alternatively, the following command will pull and install the latest commit from this repository, along with its Python dependencies: + + pip install git+https://github.com/openai/whisper.git + +To update the package to the latest version of this repository, please run: + + pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git + +It also requires the command-line tool [`ffmpeg`](https://ffmpeg.org/) to be installed on your system, which is available from most package managers: + +```bash +# on Ubuntu or Debian +sudo apt update && sudo apt install ffmpeg + +# on Arch Linux +sudo pacman -S ffmpeg + +# on MacOS using Homebrew (https://brew.sh/) +brew install ffmpeg + +# on Windows using Chocolatey (https://chocolatey.org/) +choco install ffmpeg + +# on Windows using Scoop (https://scoop.sh/) +scoop install ffmpeg +``` + +You may need [`rust`](http://rust-lang.org) installed as well, in case [tiktoken](https://github.com/openai/tiktoken) does not provide a pre-built wheel for your platform. If you see installation errors during the `pip install` command above, please follow the [Getting started page](https://www.rust-lang.org/learn/get-started) to install Rust development environment. Additionally, you may need to configure the `PATH` environment variable, e.g. `export PATH="$HOME/.cargo/bin:$PATH"`. If the installation fails with `No module named 'setuptools_rust'`, you need to install `setuptools_rust`, e.g. by running: + +```bash +pip install setuptools-rust +``` + + +## Available models and languages + +There are six model sizes, four with English-only versions, offering speed and accuracy tradeoffs. +Below are the names of the available models and their approximate memory requirements and inference speed relative to the large model. +The relative speeds below are measured by transcribing English speech on a A100, and the real-world speed may vary significantly depending on many factors including the language, the speaking speed, and the available hardware. + +| Size | Parameters | English-only model | Multilingual model | Required VRAM | Relative speed | +|:------:|:----------:|:------------------:|:------------------:|:-------------:|:--------------:| +| tiny | 39 M | `tiny.en` | `tiny` | ~1 GB | ~10x | +| base | 74 M | `base.en` | `base` | ~1 GB | ~7x | +| small | 244 M | `small.en` | `small` | ~2 GB | ~4x | +| medium | 769 M | `medium.en` | `medium` | ~5 GB | ~2x | +| large | 1550 M | N/A | `large` | ~10 GB | 1x | +| turbo | 809 M | N/A | `turbo` | ~6 GB | ~8x | + +The `.en` models for English-only applications tend to perform better, especially for the `tiny.en` and `base.en` models. We observed that the difference becomes less significant for the `small.en` and `medium.en` models. +Additionally, the `turbo` model is an optimized version of `large-v3` that offers faster transcription speed with a minimal degradation in accuracy. + +Whisper's performance varies widely depending on the language. The figure below shows a performance breakdown of `large-v3` and `large-v2` models by language, using WERs (word error rates) or CER (character error rates, shown in *Italic*) evaluated on the Common Voice 15 and Fleurs datasets. Additional WER/CER metrics corresponding to the other models and datasets can be found in Appendix D.1, D.2, and D.4 of [the paper](https://arxiv.org/abs/2212.04356), as well as the BLEU (Bilingual Evaluation Understudy) scores for translation in Appendix D.3. + +![WER breakdown by language](https://github.com/openai/whisper/assets/266841/f4619d66-1058-4005-8f67-a9d811b77c62) + +## Command-line usage + +The following command will transcribe speech in audio files, using the `turbo` model: + +```bash +whisper audio.flac audio.mp3 audio.wav --model turbo +``` + +The default setting (which selects the `turbo` model) works well for transcribing English. However, **the `turbo` model is not trained for translation tasks**. If you need to **translate non-English speech into English**, use one of the **multilingual models** (`tiny`, `base`, `small`, `medium`, `large`) instead of `turbo`. + +For example, to transcribe an audio file containing non-English speech, you can specify the language: + +```bash +whisper japanese.wav --language Japanese +``` + +To **translate** speech into English, use: + +```bash +whisper japanese.wav --model medium --language Japanese --task translate +``` + +> **Note:** The `turbo` model will return the original language even if `--task translate` is specified. Use `medium` or `large` for the best translation results. + +Run the following to view all available options: + +```bash +whisper --help +``` + +See [tokenizer.py](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py) for the list of all available languages. + + +## Python usage + +Transcription can also be performed within Python: + +```python +import whisper + +model = whisper.load_model("turbo") +result = model.transcribe("audio.mp3") +print(result["text"]) +``` + +Internally, the `transcribe()` method reads the entire file and processes the audio with a sliding 30-second window, performing autoregressive sequence-to-sequence predictions on each window. + +Below is an example usage of `whisper.detect_language()` and `whisper.decode()` which provide lower-level access to the model. + +```python +import whisper + +model = whisper.load_model("turbo") + +# load audio and pad/trim it to fit 30 seconds +audio = whisper.load_audio("audio.mp3") +audio = whisper.pad_or_trim(audio) + +# make log-Mel spectrogram and move to the same device as the model +mel = whisper.log_mel_spectrogram(audio, n_mels=model.dims.n_mels).to(model.device) + +# detect the spoken language +_, probs = model.detect_language(mel) +print(f"Detected language: {max(probs, key=probs.get)}") + +# decode the audio +options = whisper.DecodingOptions() +result = whisper.decode(model, mel, options) + +# print the recognized text +print(result.text) +``` + +## More examples + +Please use the [🙌 Show and tell](https://github.com/openai/whisper/discussions/categories/show-and-tell) category in Discussions for sharing more example usages of Whisper and third-party extensions such as web demos, integrations with other tools, ports for different platforms, etc. + + +## License + +Whisper's code and model weights are released under the MIT License. See [LICENSE](https://github.com/openai/whisper/blob/main/LICENSE) for further details. diff --git a/examples/model-systems/whisper/VENDORED.md b/examples/model-systems/whisper/VENDORED.md new file mode 100644 index 0000000..0c9c771 --- /dev/null +++ b/examples/model-systems/whisper/VENDORED.md @@ -0,0 +1,20 @@ +# Vendored source: openai/whisper (slim) + +Slim, pinned copy of Whisper's Python source so the model-systems +good/bad architecture examples have resolvable citations. This is the +reference target for the `01a-overlay-model-systems.md` overlay. + +| Field | Value | +|-------|-------| +| Upstream | https://github.com/openai/whisper | +| Upstream commit | c0d2f62 | +| Vendored | 2026-08-03 | +| Contents | `whisper/*.py`, `whisper/normalizers/*.py`, README, packaging manifests, LICENSE | +| Pruned | `whisper/assets/` (mel filter + tokenizer binaries), `whisper/normalizers/english.json`, tests, notebooks, model weights | + +Model weights are NOT in the upstream repo either — they download at +runtime. Documentation of this system must reflect that (see the good +example's Model Asset Inventory). + +Do not edit these files — refresh by re-copying from upstream and +re-running the verifier on the example docs. diff --git a/examples/model-systems/whisper/pyproject.toml b/examples/model-systems/whisper/pyproject.toml new file mode 100644 index 0000000..21b90e7 --- /dev/null +++ b/examples/model-systems/whisper/pyproject.toml @@ -0,0 +1,54 @@ +[build-system] +build-backend = "setuptools.build_meta" + +requires = [ "setuptools>=61.2" ] + +[project] +name = "openai-whisper" +description = "Robust Speech Recognition via Large-Scale Weak Supervision" +readme.content-type = "text/markdown" +readme.file = "README.md" +license = { text = "MIT" } +authors = [ { name = "OpenAI" } ] +requires-python = ">=3.8" +classifiers = [ + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", +] +dynamic = [ "version" ] +dependencies = [ + "more-itertools", + "numba", + "numpy", + "tiktoken", + "torch", + "tqdm", + "triton>=2; (platform_machine=='x86_64' and sys_platform=='linux') or sys_platform=='linux2'", +] +optional-dependencies.dev = [ "black", "flake8", "isort", "pytest", "scipy" ] +urls = { Homepage = "https://github.com/openai/whisper" } +scripts.whisper = "whisper.transcribe:cli" + +[tool.setuptools] +py-modules = [ "whisper" ] +include-package-data = true + +[tool.setuptools.dynamic] +version = { attr = "whisper.version.__version__" } + +[tool.setuptools.packages.find] +exclude = [ "tests*" ] +namespaces = false + +[tool.black] + +[tool.isort] +profile = "black" +include_trailing_comma = true +line_length = 88 +multi_line_output = 3 diff --git a/examples/model-systems/whisper/requirements.txt b/examples/model-systems/whisper/requirements.txt new file mode 100644 index 0000000..8ee5920 --- /dev/null +++ b/examples/model-systems/whisper/requirements.txt @@ -0,0 +1,7 @@ +numba +numpy +torch +tqdm +more-itertools +tiktoken +triton>=2.0.0;platform_machine=="x86_64" and sys_platform=="linux" or sys_platform=="linux2" diff --git a/examples/model-systems/whisper/whisper/__init__.py b/examples/model-systems/whisper/whisper/__init__.py new file mode 100644 index 0000000..f284ec0 --- /dev/null +++ b/examples/model-systems/whisper/whisper/__init__.py @@ -0,0 +1,161 @@ +import hashlib +import io +import os +import urllib +import warnings +from typing import List, Optional, Union + +import torch +from tqdm import tqdm + +from .audio import load_audio, log_mel_spectrogram, pad_or_trim +from .decoding import DecodingOptions, DecodingResult, decode, detect_language +from .model import ModelDimensions, Whisper +from .transcribe import transcribe +from .version import __version__ + +_MODELS = { + "tiny.en": "https://openaipublic.azureedge.net/main/whisper/models/d3dd57d32accea0b295c96e26691aa14d8822fac7d9d27d5dc00b4ca2826dd03/tiny.en.pt", + "tiny": "https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt", + "base.en": "https://openaipublic.azureedge.net/main/whisper/models/25a8566e1d0c1e2231d1c762132cd20e0f96a85d16145c3a00adf5d1ac670ead/base.en.pt", + "base": "https://openaipublic.azureedge.net/main/whisper/models/ed3a0b6b1c0edf879ad9b11b1af5a0e6ab5db9205f891f668f8b0e6c6326e34e/base.pt", + "small.en": "https://openaipublic.azureedge.net/main/whisper/models/f953ad0fd29cacd07d5a9eda5624af0f6bcf2258be67c92b79389873d91e0872/small.en.pt", + "small": "https://openaipublic.azureedge.net/main/whisper/models/9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794/small.pt", + "medium.en": "https://openaipublic.azureedge.net/main/whisper/models/d7440d1dc186f76616474e0ff0b3b6b879abc9d1a4926b7adfa41db2d497ab4f/medium.en.pt", + "medium": "https://openaipublic.azureedge.net/main/whisper/models/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt", + "large-v1": "https://openaipublic.azureedge.net/main/whisper/models/e4b87e7e0bf463eb8e6956e646f1e277e901512310def2c24bf0e11bd3c28e9a/large-v1.pt", + "large-v2": "https://openaipublic.azureedge.net/main/whisper/models/81f7c96c852ee8fc832187b0132e569d6c3065a3252ed18e56effd0b6a73e524/large-v2.pt", + "large-v3": "https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt", + "large": "https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt", + "large-v3-turbo": "https://openaipublic.azureedge.net/main/whisper/models/aff26ae408abcba5fbf8813c21e62b0941638c5f6eebfb145be0c9839262a19a/large-v3-turbo.pt", + "turbo": "https://openaipublic.azureedge.net/main/whisper/models/aff26ae408abcba5fbf8813c21e62b0941638c5f6eebfb145be0c9839262a19a/large-v3-turbo.pt", +} + +# base85-encoded (n_layers, n_heads) boolean arrays indicating the cross-attention heads that are +# highly correlated to the word-level timing, i.e. the alignment between audio and text tokens. +_ALIGNMENT_HEADS = { + "tiny.en": b"ABzY8J1N>@0{>%R00Bk>$p{7v037`oCl~+#00", + "tiny": b"ABzY8bu8Lr0{>%RKn9Fp%m@SkK7Kt=7ytkO", + "base.en": b"ABzY8;40c<0{>%RzzG;p*o+Vo09|#PsxSZm00", + "base": b"ABzY8KQ!870{>%RzyTQH3`Q^yNP!>##QT-?_)10{>%RpeA61k&I|OI3I$65C{;;pbCHh0B{qLQ;+}v00", + "small": b"ABzY8DmU6=0{>%Rpa?J`kvJ6qF(V^F86#Xh7JUGMK}P%R7%R7}kK1fFL7w6%<-Pf*t^=N)Qr&0RR9", + "large-v1": b"ABzY8r9j$a0{>%R7#4sLmoOs{s)o3~84-RPdcFk!JR%R7=D0pU<_bnWW*tkYAhobTNnu$jnkEkXqp)j;w1Tzk)UH3X%SZd&fFZ2fC2yj", + "large-v3": b"ABzY8gWO1E0{>%R7(9S+Kn!D~%ngiGaR?*L!iJG9p-nab0JQ=-{D1-g00", + "large": b"ABzY8gWO1E0{>%R7(9S+Kn!D~%ngiGaR?*L!iJG9p-nab0JQ=-{D1-g00", + "large-v3-turbo": b"ABzY8j^C+e0{>%RARaKHP%t(lGR*)0g!tONPyhe`", + "turbo": b"ABzY8j^C+e0{>%RARaKHP%t(lGR*)0g!tONPyhe`", +} + + +def _download(url: str, root: str, in_memory: bool) -> Union[bytes, str]: + os.makedirs(root, exist_ok=True) + + expected_sha256 = url.split("/")[-2] + download_target = os.path.join(root, os.path.basename(url)) + + if os.path.exists(download_target) and not os.path.isfile(download_target): + raise RuntimeError(f"{download_target} exists and is not a regular file") + + if os.path.isfile(download_target): + with open(download_target, "rb") as f: + model_bytes = f.read() + if hashlib.sha256(model_bytes).hexdigest() == expected_sha256: + return model_bytes if in_memory else download_target + else: + warnings.warn( + f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file" + ) + + with urllib.request.urlopen(url) as source, open(download_target, "wb") as output: + with tqdm( + total=int(source.info().get("Content-Length")), + ncols=80, + unit="iB", + unit_scale=True, + unit_divisor=1024, + ) as loop: + while True: + buffer = source.read(8192) + if not buffer: + break + + output.write(buffer) + loop.update(len(buffer)) + + model_bytes = open(download_target, "rb").read() + if hashlib.sha256(model_bytes).hexdigest() != expected_sha256: + raise RuntimeError( + "Model has been downloaded but the SHA256 checksum does not not match. Please retry loading the model." + ) + + return model_bytes if in_memory else download_target + + +def available_models() -> List[str]: + """Returns the names of available models""" + return list(_MODELS.keys()) + + +def load_model( + name: str, + device: Optional[Union[str, torch.device]] = None, + download_root: str = None, + in_memory: bool = False, +) -> Whisper: + """ + Load a Whisper ASR model + + Parameters + ---------- + name : str + one of the official model names listed by `whisper.available_models()`, or + path to a model checkpoint containing the model dimensions and the model state_dict. + device : Union[str, torch.device] + the PyTorch device to put the model into + download_root: str + path to download the model files; by default, it uses "~/.cache/whisper" + in_memory: bool + whether to preload the model weights into host memory + + Returns + ------- + model : Whisper + The Whisper ASR model instance + """ + + if device is None: + device = "cuda" if torch.cuda.is_available() else "cpu" + if download_root is None: + default = os.path.join(os.path.expanduser("~"), ".cache") + download_root = os.path.join(os.getenv("XDG_CACHE_HOME", default), "whisper") + + if name in _MODELS: + checkpoint_file = _download(_MODELS[name], download_root, in_memory) + alignment_heads = _ALIGNMENT_HEADS[name] + elif os.path.isfile(name): + checkpoint_file = open(name, "rb").read() if in_memory else name + alignment_heads = None + else: + raise RuntimeError( + f"Model {name} not found; available models = {available_models()}" + ) + + with ( + io.BytesIO(checkpoint_file) if in_memory else open(checkpoint_file, "rb") + ) as fp: + kwargs = {"weights_only": True} if torch.__version__ >= "1.13" else {} + checkpoint = torch.load(fp, map_location=device, **kwargs) + del checkpoint_file + + dims = ModelDimensions(**checkpoint["dims"]) + model = Whisper(dims) + model.load_state_dict(checkpoint["model_state_dict"]) + + if alignment_heads is not None: + model.set_alignment_heads(alignment_heads) + + return model.to(device) diff --git a/examples/model-systems/whisper/whisper/__main__.py b/examples/model-systems/whisper/whisper/__main__.py new file mode 100644 index 0000000..d14f205 --- /dev/null +++ b/examples/model-systems/whisper/whisper/__main__.py @@ -0,0 +1,3 @@ +from .transcribe import cli + +cli() diff --git a/examples/model-systems/whisper/whisper/audio.py b/examples/model-systems/whisper/whisper/audio.py new file mode 100644 index 0000000..826250f --- /dev/null +++ b/examples/model-systems/whisper/whisper/audio.py @@ -0,0 +1,157 @@ +import os +from functools import lru_cache +from subprocess import CalledProcessError, run +from typing import Optional, Union + +import numpy as np +import torch +import torch.nn.functional as F + +from .utils import exact_div + +# hard-coded audio hyperparameters +SAMPLE_RATE = 16000 +N_FFT = 400 +HOP_LENGTH = 160 +CHUNK_LENGTH = 30 +N_SAMPLES = CHUNK_LENGTH * SAMPLE_RATE # 480000 samples in a 30-second chunk +N_FRAMES = exact_div(N_SAMPLES, HOP_LENGTH) # 3000 frames in a mel spectrogram input + +N_SAMPLES_PER_TOKEN = HOP_LENGTH * 2 # the initial convolutions has stride 2 +FRAMES_PER_SECOND = exact_div(SAMPLE_RATE, HOP_LENGTH) # 10ms per audio frame +TOKENS_PER_SECOND = exact_div(SAMPLE_RATE, N_SAMPLES_PER_TOKEN) # 20ms per audio token + + +def load_audio(file: str, sr: int = SAMPLE_RATE): + """ + Open an audio file and read as mono waveform, resampling as necessary + + Parameters + ---------- + file: str + The audio file to open + + sr: int + The sample rate to resample the audio if necessary + + Returns + ------- + A NumPy array containing the audio waveform, in float32 dtype. + """ + + # This launches a subprocess to decode audio while down-mixing + # and resampling as necessary. Requires the ffmpeg CLI in PATH. + # fmt: off + cmd = [ + "ffmpeg", + "-nostdin", + "-threads", "0", + "-i", file, + "-f", "s16le", + "-ac", "1", + "-acodec", "pcm_s16le", + "-ar", str(sr), + "-" + ] + # fmt: on + try: + out = run(cmd, capture_output=True, check=True).stdout + except CalledProcessError as e: + raise RuntimeError(f"Failed to load audio: {e.stderr.decode()}") from e + + return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0 + + +def pad_or_trim(array, length: int = N_SAMPLES, *, axis: int = -1): + """ + Pad or trim the audio array to N_SAMPLES, as expected by the encoder. + """ + if torch.is_tensor(array): + if array.shape[axis] > length: + array = array.index_select( + dim=axis, index=torch.arange(length, device=array.device) + ) + + if array.shape[axis] < length: + pad_widths = [(0, 0)] * array.ndim + pad_widths[axis] = (0, length - array.shape[axis]) + array = F.pad(array, [pad for sizes in pad_widths[::-1] for pad in sizes]) + else: + if array.shape[axis] > length: + array = array.take(indices=range(length), axis=axis) + + if array.shape[axis] < length: + pad_widths = [(0, 0)] * array.ndim + pad_widths[axis] = (0, length - array.shape[axis]) + array = np.pad(array, pad_widths) + + return array + + +@lru_cache(maxsize=None) +def mel_filters(device, n_mels: int) -> torch.Tensor: + """ + load the mel filterbank matrix for projecting STFT into a Mel spectrogram. + Allows decoupling librosa dependency; saved using: + + np.savez_compressed( + "mel_filters.npz", + mel_80=librosa.filters.mel(sr=16000, n_fft=400, n_mels=80), + mel_128=librosa.filters.mel(sr=16000, n_fft=400, n_mels=128), + ) + """ + assert n_mels in {80, 128}, f"Unsupported n_mels: {n_mels}" + + filters_path = os.path.join(os.path.dirname(__file__), "assets", "mel_filters.npz") + with np.load(filters_path, allow_pickle=False) as f: + return torch.from_numpy(f[f"mel_{n_mels}"]).to(device) + + +def log_mel_spectrogram( + audio: Union[str, np.ndarray, torch.Tensor], + n_mels: int = 80, + padding: int = 0, + device: Optional[Union[str, torch.device]] = None, +): + """ + Compute the log-Mel spectrogram of + + Parameters + ---------- + audio: Union[str, np.ndarray, torch.Tensor], shape = (*) + The path to audio or either a NumPy array or Tensor containing the audio waveform in 16 kHz + + n_mels: int + The number of Mel-frequency filters, only 80 and 128 are supported + + padding: int + Number of zero samples to pad to the right + + device: Optional[Union[str, torch.device]] + If given, the audio tensor is moved to this device before STFT + + Returns + ------- + torch.Tensor, shape = (n_mels, n_frames) + A Tensor that contains the Mel spectrogram + """ + if not torch.is_tensor(audio): + if isinstance(audio, str): + audio = load_audio(audio) + audio = torch.from_numpy(audio) + + if device is not None: + audio = audio.to(device) + if padding > 0: + audio = F.pad(audio, (0, padding)) + window = torch.hann_window(N_FFT).to(audio.device) + stft = torch.stft(audio, N_FFT, HOP_LENGTH, window=window, return_complex=True) + magnitudes = stft[..., :-1].abs() ** 2 + + filters = mel_filters(audio.device, n_mels) + mel_spec = filters @ magnitudes + + log_spec = torch.clamp(mel_spec, min=1e-10).log10() + log_spec = torch.maximum(log_spec, log_spec.max() - 8.0) + log_spec = (log_spec + 4.0) / 4.0 + return log_spec diff --git a/examples/model-systems/whisper/whisper/decoding.py b/examples/model-systems/whisper/whisper/decoding.py new file mode 100644 index 0000000..49485d0 --- /dev/null +++ b/examples/model-systems/whisper/whisper/decoding.py @@ -0,0 +1,826 @@ +from dataclasses import dataclass, field, replace +from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Sequence, Tuple, Union + +import numpy as np +import torch +import torch.nn.functional as F +from torch import Tensor +from torch.distributions import Categorical + +from .audio import CHUNK_LENGTH +from .tokenizer import Tokenizer, get_tokenizer +from .utils import compression_ratio + +if TYPE_CHECKING: + from .model import Whisper + + +@torch.no_grad() +def detect_language( + model: "Whisper", mel: Tensor, tokenizer: Tokenizer = None +) -> Tuple[Tensor, List[dict]]: + """ + Detect the spoken language in the audio, and return them as list of strings, along with the ids + of the most probable language tokens and the probability distribution over all language tokens. + This is performed outside the main decode loop in order to not interfere with kv-caching. + + Returns + ------- + language_tokens : Tensor, shape = (n_audio,) + ids of the most probable language tokens, which appears after the startoftranscript token. + language_probs : List[Dict[str, float]], length = n_audio + list of dictionaries containing the probability distribution over all languages. + """ + if tokenizer is None: + tokenizer = get_tokenizer( + model.is_multilingual, num_languages=model.num_languages + ) + if ( + tokenizer.language is None + or tokenizer.language_token not in tokenizer.sot_sequence + ): + raise ValueError( + "This model doesn't have language tokens so it can't perform lang id" + ) + + single = mel.ndim == 2 + if single: + mel = mel.unsqueeze(0) + + # skip encoder forward pass if already-encoded audio features were given + if mel.shape[-2:] != (model.dims.n_audio_ctx, model.dims.n_audio_state): + mel = model.encoder(mel) + + # forward pass using a single token, startoftranscript + n_audio = mel.shape[0] + x = torch.tensor([[tokenizer.sot]] * n_audio).to(mel.device) # [n_audio, 1] + logits = model.logits(x, mel)[:, 0] + + # collect detected languages; suppress all non-language tokens + mask = torch.ones(logits.shape[-1], dtype=torch.bool) + mask[list(tokenizer.all_language_tokens)] = False + logits[:, mask] = -np.inf + language_tokens = logits.argmax(dim=-1) + language_token_probs = logits.softmax(dim=-1).cpu() + language_probs = [ + { + c: language_token_probs[i, j].item() + for j, c in zip(tokenizer.all_language_tokens, tokenizer.all_language_codes) + } + for i in range(n_audio) + ] + + if single: + language_tokens = language_tokens[0] + language_probs = language_probs[0] + + return language_tokens, language_probs + + +@dataclass(frozen=True) +class DecodingOptions: + # whether to perform X->X "transcribe" or X->English "translate" + task: str = "transcribe" + + # language that the audio is in; uses detected language if None + language: Optional[str] = None + + # sampling-related options + temperature: float = 0.0 + sample_len: Optional[int] = None # maximum number of tokens to sample + best_of: Optional[int] = None # number of independent sample trajectories, if t > 0 + beam_size: Optional[int] = None # number of beams in beam search, if t == 0 + patience: Optional[float] = None # patience in beam search (arxiv:2204.05424) + + # "alpha" in Google NMT, or None for length norm, when ranking generations + # to select which to return among the beams or best-of-N samples + length_penalty: Optional[float] = None + + # text or tokens to feed as the prompt or the prefix; for more info: + # https://github.com/openai/whisper/discussions/117#discussioncomment-3727051 + prompt: Optional[Union[str, List[int]]] = None # for the previous context + prefix: Optional[Union[str, List[int]]] = None # to prefix the current context + + # list of tokens ids (or comma-separated token ids) to suppress + # "-1" will suppress a set of symbols as defined in `tokenizer.non_speech_tokens()` + suppress_tokens: Optional[Union[str, Iterable[int]]] = "-1" + suppress_blank: bool = True # this will suppress blank outputs + + # timestamp sampling options + without_timestamps: bool = False # use <|notimestamps|> to sample text tokens only + max_initial_timestamp: Optional[float] = 1.0 + + # implementation details + fp16: bool = True # use fp16 for most of the calculation + + +@dataclass(frozen=True) +class DecodingResult: + audio_features: Tensor + language: str + language_probs: Optional[Dict[str, float]] = None + tokens: List[int] = field(default_factory=list) + text: str = "" + avg_logprob: float = np.nan + no_speech_prob: float = np.nan + temperature: float = np.nan + compression_ratio: float = np.nan + + +class Inference: + def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor: + """Perform a forward pass on the decoder and return per-token logits""" + raise NotImplementedError + + def rearrange_kv_cache(self, source_indices) -> None: + """Update the key-value cache according to the updated beams""" + raise NotImplementedError + + def cleanup_caching(self) -> None: + """Clean up any resources or hooks after decoding is finished""" + pass + + +class PyTorchInference(Inference): + def __init__(self, model: "Whisper", initial_token_length: int): + self.model: "Whisper" = model + self.initial_token_length = initial_token_length + self.kv_cache = {} + self.hooks = [] + + key_modules = [block.attn.key for block in self.model.decoder.blocks] + value_modules = [block.attn.value for block in self.model.decoder.blocks] + self.kv_modules = key_modules + value_modules + + def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor: + if not self.kv_cache: + self.kv_cache, self.hooks = self.model.install_kv_cache_hooks() + + if tokens.shape[-1] > self.initial_token_length: + # only need to use the last token except in the first forward pass + tokens = tokens[:, -1:] + + return self.model.decoder(tokens, audio_features, kv_cache=self.kv_cache) + + def cleanup_caching(self): + for hook in self.hooks: + hook.remove() + + self.kv_cache = {} + self.hooks = [] + + def rearrange_kv_cache(self, source_indices): + if source_indices != list(range(len(source_indices))): + for module in self.kv_modules: + # update the key/value cache to contain the selected sequences + self.kv_cache[module] = self.kv_cache[module][source_indices].detach() + + +class SequenceRanker: + def rank( + self, tokens: List[List[Tensor]], sum_logprobs: List[List[float]] + ) -> List[int]: + """ + Given a list of groups of samples and their cumulative log probabilities, + return the indices of the samples in each group to select as the final result + """ + raise NotImplementedError + + +class MaximumLikelihoodRanker(SequenceRanker): + """ + Select the sample with the highest log probabilities, penalized using either + a simple length normalization or Google NMT paper's length penalty + """ + + def __init__(self, length_penalty: Optional[float]): + self.length_penalty = length_penalty + + def rank(self, tokens: List[List[Tensor]], sum_logprobs: List[List[float]]): + def scores(logprobs, lengths): + result = [] + for logprob, length in zip(logprobs, lengths): + if self.length_penalty is None: + penalty = length + else: + # from the Google NMT paper + penalty = ((5 + length) / 6) ** self.length_penalty + result.append(logprob / penalty) + return result + + # get the sequence with the highest score + lengths = [[len(t) for t in s] for s in tokens] + return [np.argmax(scores(p, l)) for p, l in zip(sum_logprobs, lengths)] + + +class TokenDecoder: + def reset(self): + """Initialize any stateful variables for decoding a new sequence""" + + def update( + self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor + ) -> Tuple[Tensor, bool]: + """Specify how to select the next token, based on the current trace and logits + + Parameters + ---------- + tokens : Tensor, shape = (n_batch, current_sequence_length) + all tokens in the context so far, including the prefix and sot_sequence tokens + + logits : Tensor, shape = (n_batch, vocab_size) + per-token logits of the probability distribution at the current step + + sum_logprobs : Tensor, shape = (n_batch) + cumulative log probabilities for each sequence + + Returns + ------- + tokens : Tensor, shape = (n_batch, current_sequence_length + 1) + the tokens, appended with the selected next token + + completed : bool + True if all sequences has reached the end of text + + """ + raise NotImplementedError + + def finalize( + self, tokens: Tensor, sum_logprobs: Tensor + ) -> Tuple[Sequence[Sequence[Tensor]], List[List[float]]]: + """Finalize search and return the final candidate sequences + + Parameters + ---------- + tokens : Tensor, shape = (n_audio, n_group, current_sequence_length) + all tokens in the context so far, including the prefix and sot_sequence + + sum_logprobs : Tensor, shape = (n_audio, n_group) + cumulative log probabilities for each sequence + + Returns + ------- + tokens : Sequence[Sequence[Tensor]], length = n_audio + sequence of Tensors containing candidate token sequences, for each audio input + + sum_logprobs : List[List[float]], length = n_audio + sequence of cumulative log probabilities corresponding to the above + + """ + raise NotImplementedError + + +class GreedyDecoder(TokenDecoder): + def __init__(self, temperature: float, eot: int): + self.temperature = temperature + self.eot = eot + + def update( + self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor + ) -> Tuple[Tensor, bool]: + if self.temperature == 0: + next_tokens = logits.argmax(dim=-1) + else: + next_tokens = Categorical(logits=logits / self.temperature).sample() + + logprobs = F.log_softmax(logits.float(), dim=-1) + current_logprobs = logprobs[torch.arange(logprobs.shape[0]), next_tokens] + sum_logprobs += current_logprobs * (tokens[:, -1] != self.eot) + + next_tokens[tokens[:, -1] == self.eot] = self.eot + tokens = torch.cat([tokens, next_tokens[:, None]], dim=-1) + + completed = (tokens[:, -1] == self.eot).all() + return tokens, completed + + def finalize(self, tokens: Tensor, sum_logprobs: Tensor): + # make sure each sequence has at least one EOT token at the end + tokens = F.pad(tokens, (0, 1), value=self.eot) + return tokens, sum_logprobs.tolist() + + +class BeamSearchDecoder(TokenDecoder): + def __init__( + self, + beam_size: int, + eot: int, + inference: Inference, + patience: Optional[float] = None, + ): + self.beam_size = beam_size + self.eot = eot + self.inference = inference + self.patience = patience or 1.0 + self.max_candidates: int = round(beam_size * self.patience) + self.finished_sequences = None + + assert ( + self.max_candidates > 0 + ), f"Invalid beam size ({beam_size}) or patience ({patience})" + + def reset(self): + self.finished_sequences = None + + def update( + self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor + ) -> Tuple[Tensor, bool]: + if tokens.shape[0] % self.beam_size != 0: + raise ValueError(f"{tokens.shape}[0] % {self.beam_size} != 0") + + n_audio = tokens.shape[0] // self.beam_size + if self.finished_sequences is None: # for the first update + self.finished_sequences = [{} for _ in range(n_audio)] + + logprobs = F.log_softmax(logits.float(), dim=-1) + next_tokens, source_indices, finished_sequences = [], [], [] + for i in range(n_audio): + scores, sources, finished = {}, {}, {} + + # STEP 1: calculate the cumulative log probabilities for possible candidates + for j in range(self.beam_size): + idx = i * self.beam_size + j + prefix = tokens[idx].tolist() + for logprob, token in zip(*logprobs[idx].topk(self.beam_size + 1)): + new_logprob = (sum_logprobs[idx] + logprob).item() + sequence = tuple(prefix + [token.item()]) + scores[sequence] = new_logprob + sources[sequence] = idx + + # STEP 2: rank the candidates and keep the top beam_size sequences for each audio + saved = 0 + for sequence in sorted(scores, key=scores.get, reverse=True): + if sequence[-1] == self.eot: + finished[sequence] = scores[sequence] + else: + sum_logprobs[len(next_tokens)] = scores[sequence] + next_tokens.append(sequence) + source_indices.append(sources[sequence]) + + saved += 1 + if saved == self.beam_size: + break + + finished_sequences.append(finished) + + tokens = torch.tensor(next_tokens, device=tokens.device) + self.inference.rearrange_kv_cache(source_indices) + + # add newly finished sequences to self.finished_sequences + assert len(self.finished_sequences) == len(finished_sequences) + for previously_finished, newly_finished in zip( + self.finished_sequences, finished_sequences + ): + for seq in sorted(newly_finished, key=newly_finished.get, reverse=True): + if len(previously_finished) >= self.max_candidates: + break # the candidate list is full + previously_finished[seq] = newly_finished[seq] + + # mark as completed if all audio has enough number of samples + completed = all( + len(sequences) >= self.max_candidates + for sequences in self.finished_sequences + ) + return tokens, completed + + def finalize(self, preceding_tokens: Tensor, sum_logprobs: Tensor): + # collect all finished sequences, including patience, and add unfinished ones if not enough + sum_logprobs = sum_logprobs.cpu() + for i, sequences in enumerate(self.finished_sequences): + if ( + len(sequences) < self.beam_size + ): # when not enough sequences are finished + for j in list(np.argsort(sum_logprobs[i]))[::-1]: + sequence = preceding_tokens[i, j].tolist() + [self.eot] + sequences[tuple(sequence)] = sum_logprobs[i][j].item() + if len(sequences) >= self.beam_size: + break + + tokens: List[List[Tensor]] = [ + [torch.tensor(seq) for seq in sequences.keys()] + for sequences in self.finished_sequences + ] + sum_logprobs: List[List[float]] = [ + list(sequences.values()) for sequences in self.finished_sequences + ] + return tokens, sum_logprobs + + +class LogitFilter: + def apply(self, logits: Tensor, tokens: Tensor) -> None: + """Apply any filtering or masking to logits in-place + + Parameters + ---------- + logits : Tensor, shape = (n_batch, vocab_size) + per-token logits of the probability distribution at the current step + + tokens : Tensor, shape = (n_batch, current_sequence_length) + all tokens in the context so far, including the prefix and sot_sequence tokens + + """ + raise NotImplementedError + + +class SuppressBlank(LogitFilter): + def __init__(self, tokenizer: Tokenizer, sample_begin: int): + self.tokenizer = tokenizer + self.sample_begin = sample_begin + + def apply(self, logits: Tensor, tokens: Tensor): + if tokens.shape[1] == self.sample_begin: + logits[:, self.tokenizer.encode(" ") + [self.tokenizer.eot]] = -np.inf + + +class SuppressTokens(LogitFilter): + def __init__(self, suppress_tokens: Sequence[int]): + self.suppress_tokens = list(suppress_tokens) + + def apply(self, logits: Tensor, tokens: Tensor): + logits[:, self.suppress_tokens] = -np.inf + + +class ApplyTimestampRules(LogitFilter): + def __init__( + self, + tokenizer: Tokenizer, + sample_begin: int, + max_initial_timestamp_index: Optional[int], + ): + self.tokenizer = tokenizer + self.sample_begin = sample_begin + self.max_initial_timestamp_index = max_initial_timestamp_index + + def apply(self, logits: Tensor, tokens: Tensor): + # suppress <|notimestamps|> which is handled by without_timestamps + if self.tokenizer.no_timestamps is not None: + logits[:, self.tokenizer.no_timestamps] = -np.inf + + # timestamps have to appear in pairs, except directly before EOT; mask logits accordingly + for k in range(tokens.shape[0]): + sampled_tokens = tokens[k, self.sample_begin :] + seq = [t for t in sampled_tokens.tolist()] + last_was_timestamp = ( + len(seq) >= 1 and seq[-1] >= self.tokenizer.timestamp_begin + ) + penultimate_was_timestamp = ( + len(seq) < 2 or seq[-2] >= self.tokenizer.timestamp_begin + ) + + if last_was_timestamp: + if penultimate_was_timestamp: # has to be non-timestamp + logits[k, self.tokenizer.timestamp_begin :] = -np.inf + else: # cannot be normal text tokens + logits[k, : self.tokenizer.eot] = -np.inf + + timestamps = sampled_tokens[ + sampled_tokens.ge(self.tokenizer.timestamp_begin) + ] + if timestamps.numel() > 0: + # timestamps shouldn't decrease; forbid timestamp tokens smaller than the last + # also force each segment to have a nonzero length, to prevent infinite looping + if last_was_timestamp and not penultimate_was_timestamp: + timestamp_last = timestamps[-1] + else: + timestamp_last = timestamps[-1] + 1 + logits[k, self.tokenizer.timestamp_begin : timestamp_last] = -np.inf + + if tokens.shape[1] == self.sample_begin: + # suppress generating non-timestamp tokens at the beginning + logits[:, : self.tokenizer.timestamp_begin] = -np.inf + + # apply the `max_initial_timestamp` option + if self.max_initial_timestamp_index is not None: + last_allowed = ( + self.tokenizer.timestamp_begin + self.max_initial_timestamp_index + ) + logits[:, last_allowed + 1 :] = -np.inf + + # if sum of probability over timestamps is above any other token, sample timestamp + logprobs = F.log_softmax(logits.float(), dim=-1) + for k in range(tokens.shape[0]): + timestamp_logprob = logprobs[k, self.tokenizer.timestamp_begin :].logsumexp( + dim=-1 + ) + max_text_token_logprob = logprobs[k, : self.tokenizer.timestamp_begin].max() + if timestamp_logprob > max_text_token_logprob: + logits[k, : self.tokenizer.timestamp_begin] = -np.inf + + +class DecodingTask: + inference: Inference + sequence_ranker: SequenceRanker + decoder: TokenDecoder + logit_filters: List[LogitFilter] + + def __init__(self, model: "Whisper", options: DecodingOptions): + self.model = model + + language = options.language or "en" + tokenizer = get_tokenizer( + model.is_multilingual, + num_languages=model.num_languages, + language=language, + task=options.task, + ) + self.tokenizer: Tokenizer = tokenizer + self.options: DecodingOptions = self._verify_options(options) + + self.n_group: int = options.beam_size or options.best_of or 1 + self.n_ctx: int = model.dims.n_text_ctx + self.sample_len: int = options.sample_len or model.dims.n_text_ctx // 2 + + self.sot_sequence: Tuple[int] = tokenizer.sot_sequence + if self.options.without_timestamps: + self.sot_sequence = tokenizer.sot_sequence_including_notimestamps + + self.initial_tokens: Tuple[int] = self._get_initial_tokens() + self.sample_begin: int = len(self.initial_tokens) + self.sot_index: int = self.initial_tokens.index(tokenizer.sot) + + # inference: implements the forward pass through the decoder, including kv caching + self.inference = PyTorchInference(model, len(self.initial_tokens)) + + # sequence ranker: implements how to rank a group of sampled sequences + self.sequence_ranker = MaximumLikelihoodRanker(options.length_penalty) + + # decoder: implements how to select the next tokens, given the autoregressive distribution + if options.beam_size is not None: + self.decoder = BeamSearchDecoder( + options.beam_size, tokenizer.eot, self.inference, options.patience + ) + else: + self.decoder = GreedyDecoder(options.temperature, tokenizer.eot) + + # logit filters: applies various rules to suppress or penalize certain tokens + self.logit_filters = [] + if self.options.suppress_blank: + self.logit_filters.append(SuppressBlank(self.tokenizer, self.sample_begin)) + if self.options.suppress_tokens: + self.logit_filters.append(SuppressTokens(self._get_suppress_tokens())) + if not options.without_timestamps: + precision = CHUNK_LENGTH / model.dims.n_audio_ctx # usually 0.02 seconds + max_initial_timestamp_index = None + if options.max_initial_timestamp: + max_initial_timestamp_index = round( + self.options.max_initial_timestamp / precision + ) + self.logit_filters.append( + ApplyTimestampRules( + tokenizer, self.sample_begin, max_initial_timestamp_index + ) + ) + + def _verify_options(self, options: DecodingOptions) -> DecodingOptions: + if options.beam_size is not None and options.best_of is not None: + raise ValueError("beam_size and best_of can't be given together") + if options.temperature == 0: + if options.best_of is not None: + raise ValueError("best_of with greedy sampling (T=0) is not compatible") + if options.patience is not None and options.beam_size is None: + raise ValueError("patience requires beam_size to be given") + if options.length_penalty is not None and not ( + 0 <= options.length_penalty <= 1 + ): + raise ValueError("length_penalty (alpha) should be a value between 0 and 1") + + return options + + def _get_initial_tokens(self) -> Tuple[int]: + tokens = list(self.sot_sequence) + + if prefix := self.options.prefix: + prefix_tokens = ( + self.tokenizer.encode(" " + prefix.strip()) + if isinstance(prefix, str) + else prefix + ) + if self.sample_len is not None: + max_prefix_len = self.n_ctx // 2 - self.sample_len + prefix_tokens = prefix_tokens[-max_prefix_len:] + tokens = tokens + prefix_tokens + + if prompt := self.options.prompt: + prompt_tokens = ( + self.tokenizer.encode(" " + prompt.strip()) + if isinstance(prompt, str) + else prompt + ) + tokens = ( + [self.tokenizer.sot_prev] + + prompt_tokens[-(self.n_ctx // 2 - 1) :] + + tokens + ) + + return tuple(tokens) + + def _get_suppress_tokens(self) -> Tuple[int]: + suppress_tokens = self.options.suppress_tokens + + if isinstance(suppress_tokens, str): + suppress_tokens = [int(t) for t in suppress_tokens.split(",")] + + if -1 in suppress_tokens: + suppress_tokens = [t for t in suppress_tokens if t >= 0] + suppress_tokens.extend(self.tokenizer.non_speech_tokens) + elif suppress_tokens is None or len(suppress_tokens) == 0: + suppress_tokens = [] # interpret empty string as an empty list + else: + assert isinstance(suppress_tokens, list), "suppress_tokens must be a list" + + suppress_tokens.extend( + [ + self.tokenizer.transcribe, + self.tokenizer.translate, + self.tokenizer.sot, + self.tokenizer.sot_prev, + self.tokenizer.sot_lm, + ] + ) + if self.tokenizer.no_speech is not None: + # no-speech probability is collected separately + suppress_tokens.append(self.tokenizer.no_speech) + + return tuple(sorted(set(suppress_tokens))) + + def _get_audio_features(self, mel: Tensor): + if self.options.fp16: + mel = mel.half() + + if mel.shape[-2:] == ( + self.model.dims.n_audio_ctx, + self.model.dims.n_audio_state, + ): + # encoded audio features are given; skip audio encoding + audio_features = mel + else: + audio_features = self.model.encoder(mel) + + if audio_features.dtype != ( + torch.float16 if self.options.fp16 else torch.float32 + ): + return TypeError( + f"audio_features has an incorrect dtype: {audio_features.dtype}" + ) + + return audio_features + + def _detect_language(self, audio_features: Tensor, tokens: Tensor): + languages = [self.options.language] * audio_features.shape[0] + lang_probs = None + + if self.options.language is None or self.options.task == "lang_id": + lang_tokens, lang_probs = self.model.detect_language( + audio_features, self.tokenizer + ) + languages = [max(probs, key=probs.get) for probs in lang_probs] + if self.options.language is None: + tokens[:, self.sot_index + 1] = lang_tokens # write language tokens + + return languages, lang_probs + + def _main_loop(self, audio_features: Tensor, tokens: Tensor): + n_batch = tokens.shape[0] + sum_logprobs: Tensor = torch.zeros(n_batch, device=audio_features.device) + no_speech_probs = [np.nan] * n_batch + + try: + for i in range(self.sample_len): + logits = self.inference.logits(tokens, audio_features) + + if ( + i == 0 and self.tokenizer.no_speech is not None + ): # save no_speech_probs + probs_at_sot = logits[:, self.sot_index].float().softmax(dim=-1) + no_speech_probs = probs_at_sot[:, self.tokenizer.no_speech].tolist() + + # now we need to consider the logits at the last token only + logits = logits[:, -1] + + # apply the logit filters, e.g. for suppressing or applying penalty to + for logit_filter in self.logit_filters: + logit_filter.apply(logits, tokens) + + # expand the tokens tensor with the selected next tokens + tokens, completed = self.decoder.update(tokens, logits, sum_logprobs) + + if completed or tokens.shape[-1] > self.n_ctx: + break + finally: + self.inference.cleanup_caching() + + return tokens, sum_logprobs, no_speech_probs + + @torch.no_grad() + def run(self, mel: Tensor) -> List[DecodingResult]: + self.decoder.reset() + tokenizer: Tokenizer = self.tokenizer + n_audio: int = mel.shape[0] + + audio_features: Tensor = self._get_audio_features(mel) # encoder forward pass + tokens: Tensor = torch.tensor([self.initial_tokens]).repeat(n_audio, 1) + + # detect language if requested, overwriting the language token + languages, language_probs = self._detect_language(audio_features, tokens) + if self.options.task == "lang_id": + return [ + DecodingResult( + audio_features=features, language=language, language_probs=probs + ) + for features, language, probs in zip( + audio_features, languages, language_probs + ) + ] + + # repeat text tensors by the group size, for beam search or best-of-n sampling + tokens = tokens.repeat_interleave(self.n_group, dim=0).to(audio_features.device) + + # call the main sampling loop + tokens, sum_logprobs, no_speech_probs = self._main_loop(audio_features, tokens) + + # reshape the tensors to have (n_audio, n_group) as the first two dimensions + audio_features = audio_features[:: self.n_group] + no_speech_probs = no_speech_probs[:: self.n_group] + assert audio_features.shape[0] == len(no_speech_probs) == n_audio + + tokens = tokens.reshape(n_audio, self.n_group, -1) + sum_logprobs = sum_logprobs.reshape(n_audio, self.n_group) + + # get the final candidates for each group, and slice between the first sampled token and EOT + tokens, sum_logprobs = self.decoder.finalize(tokens, sum_logprobs) + tokens: List[List[Tensor]] = [ + [t[self.sample_begin : (t == tokenizer.eot).nonzero()[0, 0]] for t in s] + for s in tokens + ] + + # select the top-ranked sample in each group + selected = self.sequence_ranker.rank(tokens, sum_logprobs) + tokens: List[List[int]] = [t[i].tolist() for i, t in zip(selected, tokens)] + texts: List[str] = [tokenizer.decode(t).strip() for t in tokens] + + sum_logprobs: List[float] = [lp[i] for i, lp in zip(selected, sum_logprobs)] + avg_logprobs: List[float] = [ + lp / (len(t) + 1) for t, lp in zip(tokens, sum_logprobs) + ] + + fields = ( + texts, + languages, + tokens, + audio_features, + avg_logprobs, + no_speech_probs, + ) + if len(set(map(len, fields))) != 1: + raise RuntimeError(f"inconsistent result lengths: {list(map(len, fields))}") + + return [ + DecodingResult( + audio_features=features, + language=language, + tokens=tokens, + text=text, + avg_logprob=avg_logprob, + no_speech_prob=no_speech_prob, + temperature=self.options.temperature, + compression_ratio=compression_ratio(text), + ) + for text, language, tokens, features, avg_logprob, no_speech_prob in zip( + *fields + ) + ] + + +@torch.no_grad() +def decode( + model: "Whisper", + mel: Tensor, + options: DecodingOptions = DecodingOptions(), + **kwargs, +) -> Union[DecodingResult, List[DecodingResult]]: + """ + Performs decoding of 30-second audio segment(s), provided as Mel spectrogram(s). + + Parameters + ---------- + model: Whisper + the Whisper model instance + + mel: torch.Tensor, shape = (80, 3000) or (*, 80, 3000) + A tensor containing the Mel spectrogram(s) + + options: DecodingOptions + A dataclass that contains all necessary options for decoding 30-second segments + + Returns + ------- + result: Union[DecodingResult, List[DecodingResult]] + The result(s) of decoding contained in `DecodingResult` dataclass instance(s) + """ + if single := mel.ndim == 2: + mel = mel.unsqueeze(0) + + if kwargs: + options = replace(options, **kwargs) + + result = DecodingTask(model, options).run(mel) + + return result[0] if single else result diff --git a/examples/model-systems/whisper/whisper/model.py b/examples/model-systems/whisper/whisper/model.py new file mode 100644 index 0000000..e537447 --- /dev/null +++ b/examples/model-systems/whisper/whisper/model.py @@ -0,0 +1,345 @@ +import base64 +import gzip +from contextlib import contextmanager +from dataclasses import dataclass +from typing import Dict, Iterable, Optional, Tuple + +import numpy as np +import torch +import torch.nn.functional as F +from torch import Tensor, nn + +from .decoding import decode as decode_function +from .decoding import detect_language as detect_language_function +from .transcribe import transcribe as transcribe_function + +try: + from torch.nn.functional import scaled_dot_product_attention + + SDPA_AVAILABLE = True +except (ImportError, RuntimeError, OSError): + scaled_dot_product_attention = None + SDPA_AVAILABLE = False + + +@dataclass +class ModelDimensions: + n_mels: int + n_audio_ctx: int + n_audio_state: int + n_audio_head: int + n_audio_layer: int + n_vocab: int + n_text_ctx: int + n_text_state: int + n_text_head: int + n_text_layer: int + + +class LayerNorm(nn.LayerNorm): + def forward(self, x: Tensor) -> Tensor: + return super().forward(x.float()).type(x.dtype) + + +class Linear(nn.Linear): + def forward(self, x: Tensor) -> Tensor: + return F.linear( + x, + self.weight.to(x.dtype), + None if self.bias is None else self.bias.to(x.dtype), + ) + + +class Conv1d(nn.Conv1d): + def _conv_forward( + self, x: Tensor, weight: Tensor, bias: Optional[Tensor] + ) -> Tensor: + return super()._conv_forward( + x, weight.to(x.dtype), None if bias is None else bias.to(x.dtype) + ) + + +def sinusoids(length, channels, max_timescale=10000): + """Returns sinusoids for positional embedding""" + assert channels % 2 == 0 + log_timescale_increment = np.log(max_timescale) / (channels // 2 - 1) + inv_timescales = torch.exp(-log_timescale_increment * torch.arange(channels // 2)) + scaled_time = torch.arange(length)[:, np.newaxis] * inv_timescales[np.newaxis, :] + return torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=1) + + +@contextmanager +def disable_sdpa(): + prev_state = MultiHeadAttention.use_sdpa + try: + MultiHeadAttention.use_sdpa = False + yield + finally: + MultiHeadAttention.use_sdpa = prev_state + + +class MultiHeadAttention(nn.Module): + use_sdpa = True + + def __init__(self, n_state: int, n_head: int): + super().__init__() + self.n_head = n_head + self.query = Linear(n_state, n_state) + self.key = Linear(n_state, n_state, bias=False) + self.value = Linear(n_state, n_state) + self.out = Linear(n_state, n_state) + + def forward( + self, + x: Tensor, + xa: Optional[Tensor] = None, + mask: Optional[Tensor] = None, + kv_cache: Optional[dict] = None, + ): + q = self.query(x) + + if kv_cache is None or xa is None or self.key not in kv_cache: + # hooks, if installed (i.e. kv_cache is not None), will prepend the cached kv tensors; + # otherwise, perform key/value projections for self- or cross-attention as usual. + k = self.key(x if xa is None else xa) + v = self.value(x if xa is None else xa) + else: + # for cross-attention, calculate keys and values once and reuse in subsequent calls. + k = kv_cache[self.key] + v = kv_cache[self.value] + + wv, qk = self.qkv_attention(q, k, v, mask) + return self.out(wv), qk + + def qkv_attention( + self, q: Tensor, k: Tensor, v: Tensor, mask: Optional[Tensor] = None + ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]: + n_batch, n_ctx, n_state = q.shape + scale = (n_state // self.n_head) ** -0.25 + q = q.view(*q.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) + k = k.view(*k.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) + v = v.view(*v.shape[:2], self.n_head, -1).permute(0, 2, 1, 3) + + if SDPA_AVAILABLE and MultiHeadAttention.use_sdpa: + a = scaled_dot_product_attention( + q, k, v, is_causal=mask is not None and n_ctx > 1 + ) + out = a.permute(0, 2, 1, 3).flatten(start_dim=2) + qk = None + else: + qk = (q * scale) @ (k * scale).transpose(-1, -2) + if mask is not None: + qk = qk + mask[:n_ctx, :n_ctx] + qk = qk.float() + + w = F.softmax(qk, dim=-1).to(q.dtype) + out = (w @ v).permute(0, 2, 1, 3).flatten(start_dim=2) + qk = qk.detach() + + return out, qk + + +class ResidualAttentionBlock(nn.Module): + def __init__(self, n_state: int, n_head: int, cross_attention: bool = False): + super().__init__() + + self.attn = MultiHeadAttention(n_state, n_head) + self.attn_ln = LayerNorm(n_state) + + self.cross_attn = ( + MultiHeadAttention(n_state, n_head) if cross_attention else None + ) + self.cross_attn_ln = LayerNorm(n_state) if cross_attention else None + + n_mlp = n_state * 4 + self.mlp = nn.Sequential( + Linear(n_state, n_mlp), nn.GELU(), Linear(n_mlp, n_state) + ) + self.mlp_ln = LayerNorm(n_state) + + def forward( + self, + x: Tensor, + xa: Optional[Tensor] = None, + mask: Optional[Tensor] = None, + kv_cache: Optional[dict] = None, + ): + x = x + self.attn(self.attn_ln(x), mask=mask, kv_cache=kv_cache)[0] + if self.cross_attn: + x = x + self.cross_attn(self.cross_attn_ln(x), xa, kv_cache=kv_cache)[0] + x = x + self.mlp(self.mlp_ln(x)) + return x + + +class AudioEncoder(nn.Module): + def __init__( + self, n_mels: int, n_ctx: int, n_state: int, n_head: int, n_layer: int + ): + super().__init__() + self.conv1 = Conv1d(n_mels, n_state, kernel_size=3, padding=1) + self.conv2 = Conv1d(n_state, n_state, kernel_size=3, stride=2, padding=1) + self.register_buffer("positional_embedding", sinusoids(n_ctx, n_state)) + + self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList( + [ResidualAttentionBlock(n_state, n_head) for _ in range(n_layer)] + ) + self.ln_post = LayerNorm(n_state) + + def forward(self, x: Tensor): + """ + x : torch.Tensor, shape = (batch_size, n_mels, n_ctx) + the mel spectrogram of the audio + """ + x = F.gelu(self.conv1(x)) + x = F.gelu(self.conv2(x)) + x = x.permute(0, 2, 1) + + assert x.shape[1:] == self.positional_embedding.shape, "incorrect audio shape" + x = (x + self.positional_embedding).to(x.dtype) + + for block in self.blocks: + x = block(x) + + x = self.ln_post(x) + return x + + +class TextDecoder(nn.Module): + def __init__( + self, n_vocab: int, n_ctx: int, n_state: int, n_head: int, n_layer: int + ): + super().__init__() + + self.token_embedding = nn.Embedding(n_vocab, n_state) + self.positional_embedding = nn.Parameter(torch.empty(n_ctx, n_state)) + + self.blocks: Iterable[ResidualAttentionBlock] = nn.ModuleList( + [ + ResidualAttentionBlock(n_state, n_head, cross_attention=True) + for _ in range(n_layer) + ] + ) + self.ln = LayerNorm(n_state) + + mask = torch.empty(n_ctx, n_ctx).fill_(-np.inf).triu_(1) + self.register_buffer("mask", mask, persistent=False) + + def forward(self, x: Tensor, xa: Tensor, kv_cache: Optional[dict] = None): + """ + x : torch.LongTensor, shape = (batch_size, <= n_ctx) + the text tokens + xa : torch.Tensor, shape = (batch_size, n_audio_ctx, n_audio_state) + the encoded audio features to be attended on + """ + offset = next(iter(kv_cache.values())).shape[1] if kv_cache else 0 + x = ( + self.token_embedding(x) + + self.positional_embedding[offset : offset + x.shape[-1]] + ) + x = x.to(xa.dtype) + + for block in self.blocks: + x = block(x, xa, mask=self.mask, kv_cache=kv_cache) + + x = self.ln(x) + logits = ( + x @ torch.transpose(self.token_embedding.weight.to(x.dtype), 0, 1) + ).float() + + return logits + + +class Whisper(nn.Module): + def __init__(self, dims: ModelDimensions): + super().__init__() + self.dims = dims + self.encoder = AudioEncoder( + self.dims.n_mels, + self.dims.n_audio_ctx, + self.dims.n_audio_state, + self.dims.n_audio_head, + self.dims.n_audio_layer, + ) + self.decoder = TextDecoder( + self.dims.n_vocab, + self.dims.n_text_ctx, + self.dims.n_text_state, + self.dims.n_text_head, + self.dims.n_text_layer, + ) + # use the last half among the decoder layers for time alignment by default; + # to use a specific set of heads, see `set_alignment_heads()` below. + all_heads = torch.zeros( + self.dims.n_text_layer, self.dims.n_text_head, dtype=torch.bool + ) + all_heads[self.dims.n_text_layer // 2 :] = True + self.register_buffer("alignment_heads", all_heads.to_sparse(), persistent=False) + + def set_alignment_heads(self, dump: bytes): + array = np.frombuffer( + gzip.decompress(base64.b85decode(dump)), dtype=bool + ).copy() + mask = torch.from_numpy(array).reshape( + self.dims.n_text_layer, self.dims.n_text_head + ) + self.register_buffer("alignment_heads", mask.to_sparse(), persistent=False) + + def embed_audio(self, mel: torch.Tensor): + return self.encoder(mel) + + def logits(self, tokens: torch.Tensor, audio_features: torch.Tensor): + return self.decoder(tokens, audio_features) + + def forward( + self, mel: torch.Tensor, tokens: torch.Tensor + ) -> Dict[str, torch.Tensor]: + return self.decoder(tokens, self.encoder(mel)) + + @property + def device(self): + return next(self.parameters()).device + + @property + def is_multilingual(self): + return self.dims.n_vocab >= 51865 + + @property + def num_languages(self): + return self.dims.n_vocab - 51765 - int(self.is_multilingual) + + def install_kv_cache_hooks(self, cache: Optional[dict] = None): + """ + The `MultiHeadAttention` module optionally accepts `kv_cache` which stores the key and value + tensors calculated for the previous positions. This method returns a dictionary that stores + all caches, and the necessary hooks for the key and value projection modules that save the + intermediate tensors to be reused during later calculations. + + Returns + ------- + cache : Dict[nn.Module, torch.Tensor] + A dictionary object mapping the key/value projection modules to its cache + hooks : List[RemovableHandle] + List of PyTorch RemovableHandle objects to stop the hooks to be called + """ + cache = {**cache} if cache is not None else {} + hooks = [] + + def save_to_cache(module, _, output): + if module not in cache or output.shape[1] > self.dims.n_text_ctx: + # save as-is, for the first token or cross attention + cache[module] = output + else: + cache[module] = torch.cat([cache[module], output], dim=1).detach() + return cache[module] + + def install_hooks(layer: nn.Module): + if isinstance(layer, MultiHeadAttention): + hooks.append(layer.key.register_forward_hook(save_to_cache)) + hooks.append(layer.value.register_forward_hook(save_to_cache)) + + self.decoder.apply(install_hooks) + return cache, hooks + + detect_language = detect_language_function + transcribe = transcribe_function + decode = decode_function diff --git a/examples/model-systems/whisper/whisper/normalizers/__init__.py b/examples/model-systems/whisper/whisper/normalizers/__init__.py new file mode 100644 index 0000000..896d5e3 --- /dev/null +++ b/examples/model-systems/whisper/whisper/normalizers/__init__.py @@ -0,0 +1,2 @@ +from .basic import BasicTextNormalizer as BasicTextNormalizer +from .english import EnglishTextNormalizer as EnglishTextNormalizer diff --git a/examples/model-systems/whisper/whisper/normalizers/basic.py b/examples/model-systems/whisper/whisper/normalizers/basic.py new file mode 100644 index 0000000..8690ae7 --- /dev/null +++ b/examples/model-systems/whisper/whisper/normalizers/basic.py @@ -0,0 +1,80 @@ +import re +import unicodedata + +import regex + +# non-ASCII letters that are not separated by "NFKD" normalization +ADDITIONAL_DIACRITICS = { + "œ": "oe", + "Œ": "OE", + "ø": "o", + "Ø": "O", + "æ": "ae", + "Æ": "AE", + "ß": "ss", + "ẞ": "SS", + "đ": "d", + "Đ": "D", + "ð": "d", + "Ð": "D", + "þ": "th", + "Þ": "th", + "ł": "l", + "Ł": "L", +} + + +def remove_symbols_and_diacritics(s: str, keep=""): + """ + Replace any other markers, symbols, and punctuations with a space, + and drop any diacritics (category 'Mn' and some manual mappings) + """ + return "".join( + ( + c + if c in keep + else ( + ADDITIONAL_DIACRITICS[c] + if c in ADDITIONAL_DIACRITICS + else ( + "" + if unicodedata.category(c) == "Mn" + else " " if unicodedata.category(c)[0] in "MSP" else c + ) + ) + ) + for c in unicodedata.normalize("NFKD", s) + ) + + +def remove_symbols(s: str): + """ + Replace any other markers, symbols, punctuations with a space, keeping diacritics + """ + return "".join( + " " if unicodedata.category(c)[0] in "MSP" else c + for c in unicodedata.normalize("NFKC", s) + ) + + +class BasicTextNormalizer: + def __init__(self, remove_diacritics: bool = False, split_letters: bool = False): + self.clean = ( + remove_symbols_and_diacritics if remove_diacritics else remove_symbols + ) + self.split_letters = split_letters + + def __call__(self, s: str): + s = s.lower() + s = re.sub(r"[<\[][^>\]]*[>\]]", "", s) # remove words between brackets + s = re.sub(r"\(([^)]+?)\)", "", s) # remove words between parenthesis + s = self.clean(s).lower() + + if self.split_letters: + s = " ".join(regex.findall(r"\X", s, regex.U)) + + s = re.sub( + r"\s+", " ", s + ) # replace any successive whitespace characters with a space + + return s diff --git a/examples/model-systems/whisper/whisper/normalizers/english.py b/examples/model-systems/whisper/whisper/normalizers/english.py new file mode 100644 index 0000000..4932042 --- /dev/null +++ b/examples/model-systems/whisper/whisper/normalizers/english.py @@ -0,0 +1,550 @@ +import json +import os +import re +from fractions import Fraction +from typing import Iterator, List, Match, Optional, Union + +from more_itertools import windowed + +from .basic import remove_symbols_and_diacritics + + +class EnglishNumberNormalizer: + """ + Convert any spelled-out numbers into arabic numbers, while handling: + + - remove any commas + - keep the suffixes such as: `1960s`, `274th`, `32nd`, etc. + - spell out currency symbols after the number. e.g. `$20 million` -> `20000000 dollars` + - spell out `one` and `ones` + - interpret successive single-digit numbers as nominal: `one oh one` -> `101` + """ + + def __init__(self): + super().__init__() + + self.zeros = {"o", "oh", "zero"} + self.ones = { + name: i + for i, name in enumerate( + [ + "one", + "two", + "three", + "four", + "five", + "six", + "seven", + "eight", + "nine", + "ten", + "eleven", + "twelve", + "thirteen", + "fourteen", + "fifteen", + "sixteen", + "seventeen", + "eighteen", + "nineteen", + ], + start=1, + ) + } + self.ones_plural = { + "sixes" if name == "six" else name + "s": (value, "s") + for name, value in self.ones.items() + } + self.ones_ordinal = { + "zeroth": (0, "th"), + "first": (1, "st"), + "second": (2, "nd"), + "third": (3, "rd"), + "fifth": (5, "th"), + "twelfth": (12, "th"), + **{ + name + ("h" if name.endswith("t") else "th"): (value, "th") + for name, value in self.ones.items() + if value > 3 and value != 5 and value != 12 + }, + } + self.ones_suffixed = {**self.ones_plural, **self.ones_ordinal} + + self.tens = { + "twenty": 20, + "thirty": 30, + "forty": 40, + "fifty": 50, + "sixty": 60, + "seventy": 70, + "eighty": 80, + "ninety": 90, + } + self.tens_plural = { + name.replace("y", "ies"): (value, "s") for name, value in self.tens.items() + } + self.tens_ordinal = { + name.replace("y", "ieth"): (value, "th") + for name, value in self.tens.items() + } + self.tens_suffixed = {**self.tens_plural, **self.tens_ordinal} + + self.multipliers = { + "hundred": 100, + "thousand": 1_000, + "million": 1_000_000, + "billion": 1_000_000_000, + "trillion": 1_000_000_000_000, + "quadrillion": 1_000_000_000_000_000, + "quintillion": 1_000_000_000_000_000_000, + "sextillion": 1_000_000_000_000_000_000_000, + "septillion": 1_000_000_000_000_000_000_000_000, + "octillion": 1_000_000_000_000_000_000_000_000_000, + "nonillion": 1_000_000_000_000_000_000_000_000_000_000, + "decillion": 1_000_000_000_000_000_000_000_000_000_000_000, + } + self.multipliers_plural = { + name + "s": (value, "s") for name, value in self.multipliers.items() + } + self.multipliers_ordinal = { + name + "th": (value, "th") for name, value in self.multipliers.items() + } + self.multipliers_suffixed = { + **self.multipliers_plural, + **self.multipliers_ordinal, + } + self.decimals = {*self.ones, *self.tens, *self.zeros} + + self.preceding_prefixers = { + "minus": "-", + "negative": "-", + "plus": "+", + "positive": "+", + } + self.following_prefixers = { + "pound": "£", + "pounds": "£", + "euro": "€", + "euros": "€", + "dollar": "$", + "dollars": "$", + "cent": "¢", + "cents": "¢", + } + self.prefixes = set( + list(self.preceding_prefixers.values()) + + list(self.following_prefixers.values()) + ) + self.suffixers = { + "per": {"cent": "%"}, + "percent": "%", + } + self.specials = {"and", "double", "triple", "point"} + + self.words = set( + [ + key + for mapping in [ + self.zeros, + self.ones, + self.ones_suffixed, + self.tens, + self.tens_suffixed, + self.multipliers, + self.multipliers_suffixed, + self.preceding_prefixers, + self.following_prefixers, + self.suffixers, + self.specials, + ] + for key in mapping + ] + ) + self.literal_words = {"one", "ones"} + + def process_words(self, words: List[str]) -> Iterator[str]: + prefix: Optional[str] = None + value: Optional[Union[str, int]] = None + skip = False + + def to_fraction(s: str): + try: + return Fraction(s) + except ValueError: + return None + + def output(result: Union[str, int]): + nonlocal prefix, value + result = str(result) + if prefix is not None: + result = prefix + result + value = None + prefix = None + return result + + if len(words) == 0: + return + + for prev, current, next in windowed([None] + words + [None], 3): + if skip: + skip = False + continue + + next_is_numeric = next is not None and re.match(r"^\d+(\.\d+)?$", next) + has_prefix = current[0] in self.prefixes + current_without_prefix = current[1:] if has_prefix else current + if re.match(r"^\d+(\.\d+)?$", current_without_prefix): + # arabic numbers (potentially with signs and fractions) + f = to_fraction(current_without_prefix) + assert f is not None + if value is not None: + if isinstance(value, str) and value.endswith("."): + # concatenate decimals / ip address components + value = str(value) + str(current) + continue + else: + yield output(value) + + prefix = current[0] if has_prefix else prefix + if f.denominator == 1: + value = f.numerator # store integers as int + else: + value = current_without_prefix + elif current not in self.words: + # non-numeric words + if value is not None: + yield output(value) + yield output(current) + elif current in self.zeros: + value = str(value or "") + "0" + elif current in self.ones: + ones = self.ones[current] + + if value is None: + value = ones + elif isinstance(value, str) or prev in self.ones: + if ( + prev in self.tens and ones < 10 + ): # replace the last zero with the digit + assert value[-1] == "0" + value = value[:-1] + str(ones) + else: + value = str(value) + str(ones) + elif ones < 10: + if value % 10 == 0: + value += ones + else: + value = str(value) + str(ones) + else: # eleven to nineteen + if value % 100 == 0: + value += ones + else: + value = str(value) + str(ones) + elif current in self.ones_suffixed: + # ordinal or cardinal; yield the number right away + ones, suffix = self.ones_suffixed[current] + if value is None: + yield output(str(ones) + suffix) + elif isinstance(value, str) or prev in self.ones: + if prev in self.tens and ones < 10: + assert value[-1] == "0" + yield output(value[:-1] + str(ones) + suffix) + else: + yield output(str(value) + str(ones) + suffix) + elif ones < 10: + if value % 10 == 0: + yield output(str(value + ones) + suffix) + else: + yield output(str(value) + str(ones) + suffix) + else: # eleven to nineteen + if value % 100 == 0: + yield output(str(value + ones) + suffix) + else: + yield output(str(value) + str(ones) + suffix) + value = None + elif current in self.tens: + tens = self.tens[current] + if value is None: + value = tens + elif isinstance(value, str): + value = str(value) + str(tens) + else: + if value % 100 == 0: + value += tens + else: + value = str(value) + str(tens) + elif current in self.tens_suffixed: + # ordinal or cardinal; yield the number right away + tens, suffix = self.tens_suffixed[current] + if value is None: + yield output(str(tens) + suffix) + elif isinstance(value, str): + yield output(str(value) + str(tens) + suffix) + else: + if value % 100 == 0: + yield output(str(value + tens) + suffix) + else: + yield output(str(value) + str(tens) + suffix) + elif current in self.multipliers: + multiplier = self.multipliers[current] + if value is None: + value = multiplier + elif isinstance(value, str) or value == 0: + f = to_fraction(value) + p = f * multiplier if f is not None else None + if f is not None and p.denominator == 1: + value = p.numerator + else: + yield output(value) + value = multiplier + else: + before = value // 1000 * 1000 + residual = value % 1000 + value = before + residual * multiplier + elif current in self.multipliers_suffixed: + multiplier, suffix = self.multipliers_suffixed[current] + if value is None: + yield output(str(multiplier) + suffix) + elif isinstance(value, str): + f = to_fraction(value) + p = f * multiplier if f is not None else None + if f is not None and p.denominator == 1: + yield output(str(p.numerator) + suffix) + else: + yield output(value) + yield output(str(multiplier) + suffix) + else: # int + before = value // 1000 * 1000 + residual = value % 1000 + value = before + residual * multiplier + yield output(str(value) + suffix) + value = None + elif current in self.preceding_prefixers: + # apply prefix (positive, minus, etc.) if it precedes a number + if value is not None: + yield output(value) + + if next in self.words or next_is_numeric: + prefix = self.preceding_prefixers[current] + else: + yield output(current) + elif current in self.following_prefixers: + # apply prefix (dollars, cents, etc.) only after a number + if value is not None: + prefix = self.following_prefixers[current] + yield output(value) + else: + yield output(current) + elif current in self.suffixers: + # apply suffix symbols (percent -> '%') + if value is not None: + suffix = self.suffixers[current] + if isinstance(suffix, dict): + if next in suffix: + yield output(str(value) + suffix[next]) + skip = True + else: + yield output(value) + yield output(current) + else: + yield output(str(value) + suffix) + else: + yield output(current) + elif current in self.specials: + if next not in self.words and not next_is_numeric: + # apply special handling only if the next word can be numeric + if value is not None: + yield output(value) + yield output(current) + elif current == "and": + # ignore "and" after hundreds, thousands, etc. + if prev not in self.multipliers: + if value is not None: + yield output(value) + yield output(current) + elif current == "double" or current == "triple": + if next in self.ones or next in self.zeros: + repeats = 2 if current == "double" else 3 + ones = self.ones.get(next, 0) + value = str(value or "") + str(ones) * repeats + skip = True + else: + if value is not None: + yield output(value) + yield output(current) + elif current == "point": + if next in self.decimals or next_is_numeric: + value = str(value or "") + "." + else: + # should all have been covered at this point + raise ValueError(f"Unexpected token: {current}") + else: + # all should have been covered at this point + raise ValueError(f"Unexpected token: {current}") + + if value is not None: + yield output(value) + + def preprocess(self, s: str): + # replace " and a half" with " point five" + results = [] + + segments = re.split(r"\band\s+a\s+half\b", s) + for i, segment in enumerate(segments): + if len(segment.strip()) == 0: + continue + if i == len(segments) - 1: + results.append(segment) + else: + results.append(segment) + last_word = segment.rsplit(maxsplit=2)[-1] + if last_word in self.decimals or last_word in self.multipliers: + results.append("point five") + else: + results.append("and a half") + + s = " ".join(results) + + # put a space at number/letter boundary + s = re.sub(r"([a-z])([0-9])", r"\1 \2", s) + s = re.sub(r"([0-9])([a-z])", r"\1 \2", s) + + # but remove spaces which could be a suffix + s = re.sub(r"([0-9])\s+(st|nd|rd|th|s)\b", r"\1\2", s) + + return s + + def postprocess(self, s: str): + def combine_cents(m: Match): + try: + currency = m.group(1) + integer = m.group(2) + cents = int(m.group(3)) + return f"{currency}{integer}.{cents:02d}" + except ValueError: + return m.string + + def extract_cents(m: Match): + try: + return f"¢{int(m.group(1))}" + except ValueError: + return m.string + + # apply currency postprocessing; "$2 and ¢7" -> "$2.07" + s = re.sub(r"([€£$])([0-9]+) (?:and )?¢([0-9]{1,2})\b", combine_cents, s) + s = re.sub(r"[€£$]0.([0-9]{1,2})\b", extract_cents, s) + + # write "one(s)" instead of "1(s)", just for the readability + s = re.sub(r"\b1(s?)\b", r"one\1", s) + + return s + + def __call__(self, s: str): + s = self.preprocess(s) + s = " ".join(word for word in self.process_words(s.split()) if word is not None) + s = self.postprocess(s) + + return s + + +class EnglishSpellingNormalizer: + """ + Applies British-American spelling mappings as listed in [1]. + + [1] https://www.tysto.com/uk-us-spelling-list.html + """ + + def __init__(self): + mapping_path = os.path.join(os.path.dirname(__file__), "english.json") + self.mapping = json.load(open(mapping_path)) + + def __call__(self, s: str): + return " ".join(self.mapping.get(word, word) for word in s.split()) + + +class EnglishTextNormalizer: + def __init__(self): + self.ignore_patterns = r"\b(hmm|mm|mhm|mmm|uh|um)\b" + self.replacers = { + # common contractions + r"\bwon't\b": "will not", + r"\bcan't\b": "can not", + r"\blet's\b": "let us", + r"\bain't\b": "aint", + r"\by'all\b": "you all", + r"\bwanna\b": "want to", + r"\bgotta\b": "got to", + r"\bgonna\b": "going to", + r"\bi'ma\b": "i am going to", + r"\bimma\b": "i am going to", + r"\bwoulda\b": "would have", + r"\bcoulda\b": "could have", + r"\bshoulda\b": "should have", + r"\bma'am\b": "madam", + # contractions in titles/prefixes + r"\bmr\b": "mister ", + r"\bmrs\b": "missus ", + r"\bst\b": "saint ", + r"\bdr\b": "doctor ", + r"\bprof\b": "professor ", + r"\bcapt\b": "captain ", + r"\bgov\b": "governor ", + r"\bald\b": "alderman ", + r"\bgen\b": "general ", + r"\bsen\b": "senator ", + r"\brep\b": "representative ", + r"\bpres\b": "president ", + r"\brev\b": "reverend ", + r"\bhon\b": "honorable ", + r"\basst\b": "assistant ", + r"\bassoc\b": "associate ", + r"\blt\b": "lieutenant ", + r"\bcol\b": "colonel ", + r"\bjr\b": "junior ", + r"\bsr\b": "senior ", + r"\besq\b": "esquire ", + # prefect tenses, ideally it should be any past participles, but it's harder.. + r"'d been\b": " had been", + r"'s been\b": " has been", + r"'d gone\b": " had gone", + r"'s gone\b": " has gone", + r"'d done\b": " had done", # "'s done" is ambiguous + r"'s got\b": " has got", + # general contractions + r"n't\b": " not", + r"'re\b": " are", + r"'s\b": " is", + r"'d\b": " would", + r"'ll\b": " will", + r"'t\b": " not", + r"'ve\b": " have", + r"'m\b": " am", + } + self.standardize_numbers = EnglishNumberNormalizer() + self.standardize_spellings = EnglishSpellingNormalizer() + + def __call__(self, s: str): + s = s.lower() + + s = re.sub(r"[<\[][^>\]]*[>\]]", "", s) # remove words between brackets + s = re.sub(r"\(([^)]+?)\)", "", s) # remove words between parenthesis + s = re.sub(self.ignore_patterns, "", s) + s = re.sub(r"\s+'", "'", s) # when there's a space before an apostrophe + + for pattern, replacement in self.replacers.items(): + s = re.sub(pattern, replacement, s) + + s = re.sub(r"(\d),(\d)", r"\1\2", s) # remove commas between digits + s = re.sub(r"\.([^0-9]|$)", r" \1", s) # remove periods not followed by numbers + s = remove_symbols_and_diacritics(s, keep=".%$¢€£") # keep numeric symbols + + s = self.standardize_numbers(s) + s = self.standardize_spellings(s) + + # now remove prefix/suffix symbols that are not preceded/followed by numbers + s = re.sub(r"[.$¢€£]([^0-9])", r" \1", s) + s = re.sub(r"([^0-9])%", r"\1 ", s) + + s = re.sub(r"\s+", " ", s) # replace any successive whitespaces with a space + + return s diff --git a/examples/model-systems/whisper/whisper/timing.py b/examples/model-systems/whisper/whisper/timing.py new file mode 100644 index 0000000..2340000 --- /dev/null +++ b/examples/model-systems/whisper/whisper/timing.py @@ -0,0 +1,388 @@ +import itertools +import subprocess +import warnings +from dataclasses import dataclass +from typing import TYPE_CHECKING, List + +import numba +import numpy as np +import torch +import torch.nn.functional as F + +from .audio import HOP_LENGTH, SAMPLE_RATE, TOKENS_PER_SECOND +from .tokenizer import Tokenizer + +if TYPE_CHECKING: + from .model import Whisper + + +def median_filter(x: torch.Tensor, filter_width: int): + """Apply a median filter of width `filter_width` along the last dimension of `x`""" + pad_width = filter_width // 2 + if x.shape[-1] <= pad_width: + # F.pad requires the padding width to be smaller than the input dimension + return x + + if (ndim := x.ndim) <= 2: + # `F.pad` does not support 1D or 2D inputs for reflect padding but supports 3D and 4D + x = x[None, None, :] + + assert ( + filter_width > 0 and filter_width % 2 == 1 + ), "`filter_width` should be an odd number" + + result = None + x = F.pad(x, (filter_width // 2, filter_width // 2, 0, 0), mode="reflect") + if x.is_cuda: + try: + from .triton_ops import median_filter_cuda + + result = median_filter_cuda(x, filter_width) + except (RuntimeError, subprocess.CalledProcessError): + warnings.warn( + "Failed to launch Triton kernels, likely due to missing CUDA toolkit; " + "falling back to a slower median kernel implementation..." + ) + + if result is None: + # sort() is faster than torch.median (https://github.com/pytorch/pytorch/issues/51450) + result = x.unfold(-1, filter_width, 1).sort()[0][..., filter_width // 2] + + if ndim <= 2: + result = result[0, 0] + + return result + + +@numba.jit(nopython=True) +def backtrace(trace: np.ndarray): + i = trace.shape[0] - 1 + j = trace.shape[1] - 1 + trace[0, :] = 2 + trace[:, 0] = 1 + + result = [] + while i > 0 or j > 0: + result.append((i - 1, j - 1)) + + if trace[i, j] == 0: + i -= 1 + j -= 1 + elif trace[i, j] == 1: + i -= 1 + elif trace[i, j] == 2: + j -= 1 + else: + raise ValueError("Unexpected trace[i, j]") + + result = np.array(result) + return result[::-1, :].T + + +@numba.jit(nopython=True, parallel=True) +def dtw_cpu(x: np.ndarray): + N, M = x.shape + cost = np.ones((N + 1, M + 1), dtype=np.float32) * np.inf + trace = -np.ones((N + 1, M + 1), dtype=np.float32) + + cost[0, 0] = 0 + for j in range(1, M + 1): + for i in range(1, N + 1): + c0 = cost[i - 1, j - 1] + c1 = cost[i - 1, j] + c2 = cost[i, j - 1] + + if c0 < c1 and c0 < c2: + c, t = c0, 0 + elif c1 < c0 and c1 < c2: + c, t = c1, 1 + else: + c, t = c2, 2 + + cost[i, j] = x[i - 1, j - 1] + c + trace[i, j] = t + + return backtrace(trace) + + +def dtw_cuda(x, BLOCK_SIZE=1024): + from .triton_ops import dtw_kernel + + M, N = x.shape + assert M < BLOCK_SIZE, f"M should be smaller than {BLOCK_SIZE=}" + + x_skew = ( + F.pad(x, (0, M + 1), value=np.inf).flatten()[: M * (N + M)].reshape(M, N + M) + ) + x_skew = x_skew.T.contiguous() + cost = torch.ones(N + M + 2, M + 2) * np.inf + cost[0, 0] = 0 + cost = cost.to(x.device) + trace = torch.zeros_like(cost, dtype=torch.int32) + + dtw_kernel[(1,)]( + cost, + trace, + x_skew, + x_skew.stride(0), + cost.stride(0), + trace.stride(0), + N, + M, + BLOCK_SIZE=BLOCK_SIZE, + ) + + trace = trace.T.flatten()[: (M + 1) * (M + N + 3)].reshape(M + 1, M + N + 3)[ + :, : N + 1 + ] + return backtrace(trace.cpu().numpy()) + + +def dtw(x: torch.Tensor) -> np.ndarray: + if x.is_cuda: + try: + return dtw_cuda(x) + except (RuntimeError, subprocess.CalledProcessError): + warnings.warn( + "Failed to launch Triton kernels, likely due to missing CUDA toolkit; " + "falling back to a slower DTW implementation..." + ) + + return dtw_cpu(x.double().cpu().numpy()) + + +@dataclass +class WordTiming: + word: str + tokens: List[int] + start: float + end: float + probability: float + + +def find_alignment( + model: "Whisper", + tokenizer: Tokenizer, + text_tokens: List[int], + mel: torch.Tensor, + num_frames: int, + *, + medfilt_width: int = 7, + qk_scale: float = 1.0, +) -> List[WordTiming]: + if len(text_tokens) == 0: + return [] + + tokens = torch.tensor( + [ + *tokenizer.sot_sequence, + tokenizer.no_timestamps, + *text_tokens, + tokenizer.eot, + ] + ).to(model.device) + + # install hooks on the cross attention layers to retrieve the attention weights + QKs = [None] * model.dims.n_text_layer + hooks = [ + block.cross_attn.register_forward_hook( + lambda _, ins, outs, index=i: QKs.__setitem__(index, outs[-1][0]) + ) + for i, block in enumerate(model.decoder.blocks) + ] + + from .model import disable_sdpa + + with torch.no_grad(), disable_sdpa(): + logits = model(mel.unsqueeze(0), tokens.unsqueeze(0))[0] + sampled_logits = logits[len(tokenizer.sot_sequence) :, : tokenizer.eot] + token_probs = sampled_logits.softmax(dim=-1) + text_token_probs = token_probs[np.arange(len(text_tokens)), text_tokens] + text_token_probs = text_token_probs.tolist() + + for hook in hooks: + hook.remove() + + # heads * tokens * frames + weights = torch.stack([QKs[_l][_h] for _l, _h in model.alignment_heads.indices().T]) + weights = weights[:, :, : num_frames // 2] + weights = (weights * qk_scale).softmax(dim=-1) + std, mean = torch.std_mean(weights, dim=-2, keepdim=True, unbiased=False) + weights = (weights - mean) / std + weights = median_filter(weights, medfilt_width) + + matrix = weights.mean(axis=0) + matrix = matrix[len(tokenizer.sot_sequence) : -1] + text_indices, time_indices = dtw(-matrix) + + words, word_tokens = tokenizer.split_to_word_tokens(text_tokens + [tokenizer.eot]) + if len(word_tokens) <= 1: + # return on eot only + # >>> np.pad([], (1, 0)) + # array([0.]) + # This results in crashes when we lookup jump_times with float, like + # IndexError: arrays used as indices must be of integer (or boolean) type + return [] + word_boundaries = np.pad(np.cumsum([len(t) for t in word_tokens[:-1]]), (1, 0)) + + jumps = np.pad(np.diff(text_indices), (1, 0), constant_values=1).astype(bool) + jump_times = time_indices[jumps] / TOKENS_PER_SECOND + start_times = jump_times[word_boundaries[:-1]] + end_times = jump_times[word_boundaries[1:]] + word_probabilities = [ + np.mean(text_token_probs[i:j]) + for i, j in zip(word_boundaries[:-1], word_boundaries[1:]) + ] + + return [ + WordTiming(word, tokens, start, end, probability) + for word, tokens, start, end, probability in zip( + words, word_tokens, start_times, end_times, word_probabilities + ) + ] + + +def merge_punctuations(alignment: List[WordTiming], prepended: str, appended: str): + # merge prepended punctuations + i = len(alignment) - 2 + j = len(alignment) - 1 + while i >= 0: + previous = alignment[i] + following = alignment[j] + if previous.word.startswith(" ") and previous.word.strip() in prepended: + # prepend it to the following word + following.word = previous.word + following.word + following.tokens = previous.tokens + following.tokens + previous.word = "" + previous.tokens = [] + else: + j = i + i -= 1 + + # merge appended punctuations + i = 0 + j = 1 + while j < len(alignment): + previous = alignment[i] + following = alignment[j] + if not previous.word.endswith(" ") and following.word in appended: + # append it to the previous word + previous.word = previous.word + following.word + previous.tokens = previous.tokens + following.tokens + following.word = "" + following.tokens = [] + else: + i = j + j += 1 + + +def add_word_timestamps( + *, + segments: List[dict], + model: "Whisper", + tokenizer: Tokenizer, + mel: torch.Tensor, + num_frames: int, + prepend_punctuations: str = "\"'“¿([{-", + append_punctuations: str = "\"'.。,，!！?？:：”)]}、", + last_speech_timestamp: float, + **kwargs, +): + if len(segments) == 0: + return + + text_tokens_per_segment = [ + [token for token in segment["tokens"] if token < tokenizer.eot] + for segment in segments + ] + + text_tokens = list(itertools.chain.from_iterable(text_tokens_per_segment)) + alignment = find_alignment(model, tokenizer, text_tokens, mel, num_frames, **kwargs) + word_durations = np.array([t.end - t.start for t in alignment]) + word_durations = word_durations[word_durations.nonzero()] + median_duration = np.median(word_durations) if len(word_durations) > 0 else 0.0 + median_duration = min(0.7, float(median_duration)) + max_duration = median_duration * 2 + + # hack: truncate long words at sentence boundaries. + # a better segmentation algorithm based on VAD should be able to replace this. + if len(word_durations) > 0: + sentence_end_marks = ".。!！?？" + # ensure words at sentence boundaries are not longer than twice the median word duration. + for i in range(1, len(alignment)): + if alignment[i].end - alignment[i].start > max_duration: + if alignment[i].word in sentence_end_marks: + alignment[i].end = alignment[i].start + max_duration + elif alignment[i - 1].word in sentence_end_marks: + alignment[i].start = alignment[i].end - max_duration + + merge_punctuations(alignment, prepend_punctuations, append_punctuations) + + time_offset = segments[0]["seek"] * HOP_LENGTH / SAMPLE_RATE + word_index = 0 + + for segment, text_tokens in zip(segments, text_tokens_per_segment): + saved_tokens = 0 + words = [] + + while word_index < len(alignment) and saved_tokens < len(text_tokens): + timing = alignment[word_index] + + if timing.word: + words.append( + dict( + word=timing.word, + start=round(time_offset + timing.start, 2), + end=round(time_offset + timing.end, 2), + probability=timing.probability, + ) + ) + + saved_tokens += len(timing.tokens) + word_index += 1 + + # hack: truncate long words at segment boundaries. + # a better segmentation algorithm based on VAD should be able to replace this. + if len(words) > 0: + # ensure the first and second word after a pause is not longer than + # twice the median word duration. + if words[0]["end"] - last_speech_timestamp > median_duration * 4 and ( + words[0]["end"] - words[0]["start"] > max_duration + or ( + len(words) > 1 + and words[1]["end"] - words[0]["start"] > max_duration * 2 + ) + ): + if ( + len(words) > 1 + and words[1]["end"] - words[1]["start"] > max_duration + ): + boundary = max(words[1]["end"] / 2, words[1]["end"] - max_duration) + words[0]["end"] = words[1]["start"] = boundary + words[0]["start"] = max(0, words[0]["end"] - max_duration) + + # prefer the segment-level start timestamp if the first word is too long. + if ( + segment["start"] < words[0]["end"] + and segment["start"] - 0.5 > words[0]["start"] + ): + words[0]["start"] = max( + 0, min(words[0]["end"] - median_duration, segment["start"]) + ) + else: + segment["start"] = words[0]["start"] + + # prefer the segment-level end timestamp if the last word is too long. + if ( + segment["end"] > words[-1]["start"] + and segment["end"] + 0.5 < words[-1]["end"] + ): + words[-1]["end"] = max( + words[-1]["start"] + median_duration, segment["end"] + ) + else: + segment["end"] = words[-1]["end"] + + last_speech_timestamp = segment["end"] + + segment["words"] = words diff --git a/examples/model-systems/whisper/whisper/tokenizer.py b/examples/model-systems/whisper/whisper/tokenizer.py new file mode 100644 index 0000000..2af8375 --- /dev/null +++ b/examples/model-systems/whisper/whisper/tokenizer.py @@ -0,0 +1,395 @@ +import base64 +import os +import string +from dataclasses import dataclass, field +from functools import cached_property, lru_cache +from typing import Dict, List, Optional, Tuple + +import tiktoken + +LANGUAGES = { + "en": "english", + "zh": "chinese", + "de": "german", + "es": "spanish", + "ru": "russian", + "ko": "korean", + "fr": "french", + "ja": "japanese", + "pt": "portuguese", + "tr": "turkish", + "pl": "polish", + "ca": "catalan", + "nl": "dutch", + "ar": "arabic", + "sv": "swedish", + "it": "italian", + "id": "indonesian", + "hi": "hindi", + "fi": "finnish", + "vi": "vietnamese", + "he": "hebrew", + "uk": "ukrainian", + "el": "greek", + "ms": "malay", + "cs": "czech", + "ro": "romanian", + "da": "danish", + "hu": "hungarian", + "ta": "tamil", + "no": "norwegian", + "th": "thai", + "ur": "urdu", + "hr": "croatian", + "bg": "bulgarian", + "lt": "lithuanian", + "la": "latin", + "mi": "maori", + "ml": "malayalam", + "cy": "welsh", + "sk": "slovak", + "te": "telugu", + "fa": "persian", + "lv": "latvian", + "bn": "bengali", + "sr": "serbian", + "az": "azerbaijani", + "sl": "slovenian", + "kn": "kannada", + "et": "estonian", + "mk": "macedonian", + "br": "breton", + "eu": "basque", + "is": "icelandic", + "hy": "armenian", + "ne": "nepali", + "mn": "mongolian", + "bs": "bosnian", + "kk": "kazakh", + "sq": "albanian", + "sw": "swahili", + "gl": "galician", + "mr": "marathi", + "pa": "punjabi", + "si": "sinhala", + "km": "khmer", + "sn": "shona", + "yo": "yoruba", + "so": "somali", + "af": "afrikaans", + "oc": "occitan", + "ka": "georgian", + "be": "belarusian", + "tg": "tajik", + "sd": "sindhi", + "gu": "gujarati", + "am": "amharic", + "yi": "yiddish", + "lo": "lao", + "uz": "uzbek", + "fo": "faroese", + "ht": "haitian creole", + "ps": "pashto", + "tk": "turkmen", + "nn": "nynorsk", + "mt": "maltese", + "sa": "sanskrit", + "lb": "luxembourgish", + "my": "myanmar", + "bo": "tibetan", + "tl": "tagalog", + "mg": "malagasy", + "as": "assamese", + "tt": "tatar", + "haw": "hawaiian", + "ln": "lingala", + "ha": "hausa", + "ba": "bashkir", + "jw": "javanese", + "su": "sundanese", + "yue": "cantonese", +} + +# language code lookup by name, with a few language aliases +TO_LANGUAGE_CODE = { + **{language: code for code, language in LANGUAGES.items()}, + "burmese": "my", + "valencian": "ca", + "flemish": "nl", + "haitian": "ht", + "letzeburgesch": "lb", + "pushto": "ps", + "panjabi": "pa", + "moldavian": "ro", + "moldovan": "ro", + "sinhalese": "si", + "castilian": "es", + "mandarin": "zh", +} + + +@dataclass +class Tokenizer: + """A thin wrapper around `tiktoken` providing quick access to special tokens""" + + encoding: tiktoken.Encoding + num_languages: int + language: Optional[str] = None + task: Optional[str] = None + sot_sequence: Tuple[int] = () + special_tokens: Dict[str, int] = field(default_factory=dict) + + def __post_init__(self): + for special in self.encoding.special_tokens_set: + special_token = self.encoding.encode_single_token(special) + self.special_tokens[special] = special_token + + sot: int = self.special_tokens["<|startoftranscript|>"] + translate: int = self.special_tokens["<|translate|>"] + transcribe: int = self.special_tokens["<|transcribe|>"] + + langs = tuple(LANGUAGES.keys())[: self.num_languages] + sot_sequence = [sot] + if self.language is not None: + sot_sequence.append(sot + 1 + langs.index(self.language)) + if self.task is not None: + task_token: int = transcribe if self.task == "transcribe" else translate + sot_sequence.append(task_token) + + self.sot_sequence = tuple(sot_sequence) + + def encode(self, text, **kwargs): + return self.encoding.encode(text, **kwargs) + + def decode(self, token_ids: List[int], **kwargs) -> str: + token_ids = [t for t in token_ids if t < self.timestamp_begin] + return self.encoding.decode(token_ids, **kwargs) + + def decode_with_timestamps(self, token_ids: List[int], **kwargs) -> str: + """ + Timestamp tokens are above other special tokens' id range and are ignored by `decode()`. + This method decodes given tokens with timestamps tokens annotated, e.g. "<|1.08|>". + """ + return self.encoding.decode(token_ids, **kwargs) + + @cached_property + def eot(self) -> int: + return self.encoding.eot_token + + @cached_property + def transcribe(self) -> int: + return self.special_tokens["<|transcribe|>"] + + @cached_property + def translate(self) -> int: + return self.special_tokens["<|translate|>"] + + @cached_property + def sot(self) -> int: + return self.special_tokens["<|startoftranscript|>"] + + @cached_property + def sot_lm(self) -> int: + return self.special_tokens["<|startoflm|>"] + + @cached_property + def sot_prev(self) -> int: + return self.special_tokens["<|startofprev|>"] + + @cached_property + def no_speech(self) -> int: + return self.special_tokens["<|nospeech|>"] + + @cached_property + def no_timestamps(self) -> int: + return self.special_tokens["<|notimestamps|>"] + + @cached_property + def timestamp_begin(self) -> int: + return self.special_tokens["<|0.00|>"] + + @cached_property + def language_token(self) -> int: + """Returns the token id corresponding to the value of the `language` field""" + if self.language is None: + raise ValueError("This tokenizer does not have language token configured") + + return self.to_language_token(self.language) + + def to_language_token(self, language): + if token := self.special_tokens.get(f"<|{language}|>", None): + return token + + raise KeyError(f"Language {language} not found in tokenizer.") + + @cached_property + def all_language_tokens(self) -> Tuple[int]: + result = [] + for token, token_id in self.special_tokens.items(): + if token.strip("<|>") in LANGUAGES: + result.append(token_id) + return tuple(result)[: self.num_languages] + + @cached_property + def all_language_codes(self) -> Tuple[str]: + return tuple(self.decode([_l]).strip("<|>") for _l in self.all_language_tokens) + + @cached_property + def sot_sequence_including_notimestamps(self) -> Tuple[int]: + return tuple(list(self.sot_sequence) + [self.no_timestamps]) + + @cached_property + def non_speech_tokens(self) -> Tuple[int]: + """ + Returns the list of tokens to suppress in order to avoid any speaker tags or non-speech + annotations, to prevent sampling texts that are not actually spoken in the audio, e.g. + + - ♪♪♪ + - ( SPEAKING FOREIGN LANGUAGE ) + - [DAVID] Hey there, + + keeping basic punctuations like commas, periods, question marks, exclamation points, etc. + """ + symbols = list('"#()*+/:;<=>@[\\]^_`{|}~「」『』') + symbols += ( + "<< >> <<< >>> -- --- -( -[ (' (\" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪".split() + ) + + # symbols that may be a single token or multiple tokens depending on the tokenizer. + # In case they're multiple tokens, suppress the first token, which is safe because: + # These are between U+2640 and U+267F miscellaneous symbols that are okay to suppress + # in generations, and in the 3-byte UTF-8 representation they share the first two bytes. + miscellaneous = set("♩♪♫♬♭♮♯") + assert all(0x2640 <= ord(c) <= 0x267F for c in miscellaneous) + + # allow hyphens "-" and single quotes "'" between words, but not at the beginning of a word + result = {self.encoding.encode(" -")[0], self.encoding.encode(" '")[0]} + for symbol in symbols + list(miscellaneous): + for tokens in [ + self.encoding.encode(symbol), + self.encoding.encode(" " + symbol), + ]: + if len(tokens) == 1 or symbol in miscellaneous: + result.add(tokens[0]) + + return tuple(sorted(result)) + + def split_to_word_tokens(self, tokens: List[int]): + if self.language in {"zh", "ja", "th", "lo", "my", "yue"}: + # These languages don't typically use spaces, so it is difficult to split words + # without morpheme analysis. Here, we instead split words at any + # position where the tokens are decoded as valid unicode points + return self.split_tokens_on_unicode(tokens) + + return self.split_tokens_on_spaces(tokens) + + def split_tokens_on_unicode(self, tokens: List[int]): + decoded_full = self.decode_with_timestamps(tokens) + replacement_char = "\ufffd" + + words = [] + word_tokens = [] + current_tokens = [] + unicode_offset = 0 + + for token in tokens: + current_tokens.append(token) + decoded = self.decode_with_timestamps(current_tokens) + + if ( + replacement_char not in decoded + or decoded_full[unicode_offset + decoded.index(replacement_char)] + == replacement_char + ): + words.append(decoded) + word_tokens.append(current_tokens) + current_tokens = [] + unicode_offset += len(decoded) + + return words, word_tokens + + def split_tokens_on_spaces(self, tokens: List[int]): + subwords, subword_tokens_list = self.split_tokens_on_unicode(tokens) + words = [] + word_tokens = [] + + for subword, subword_tokens in zip(subwords, subword_tokens_list): + special = subword_tokens[0] >= self.eot + with_space = subword.startswith(" ") + punctuation = subword.strip() in string.punctuation + if special or with_space or punctuation or len(words) == 0: + words.append(subword) + word_tokens.append(subword_tokens) + else: + words[-1] = words[-1] + subword + word_tokens[-1].extend(subword_tokens) + + return words, word_tokens + + +@lru_cache(maxsize=None) +def get_encoding(name: str = "gpt2", num_languages: int = 99): + vocab_path = os.path.join(os.path.dirname(__file__), "assets", f"{name}.tiktoken") + ranks = { + base64.b64decode(token): int(rank) + for token, rank in (line.split() for line in open(vocab_path) if line) + } + n_vocab = len(ranks) + special_tokens = {} + + specials = [ + "<|endoftext|>", + "<|startoftranscript|>", + *[f"<|{lang}|>" for lang in list(LANGUAGES.keys())[:num_languages]], + "<|translate|>", + "<|transcribe|>", + "<|startoflm|>", + "<|startofprev|>", + "<|nospeech|>", + "<|notimestamps|>", + *[f"<|{i * 0.02:.2f}|>" for i in range(1501)], + ] + + for token in specials: + special_tokens[token] = n_vocab + n_vocab += 1 + + return tiktoken.Encoding( + name=os.path.basename(vocab_path), + explicit_n_vocab=n_vocab, + pat_str=r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""", + mergeable_ranks=ranks, + special_tokens=special_tokens, + ) + + +@lru_cache(maxsize=None) +def get_tokenizer( + multilingual: bool, + *, + num_languages: int = 99, + language: Optional[str] = None, + task: Optional[str] = None, # Literal["transcribe", "translate", None] +) -> Tokenizer: + if language is not None: + language = language.lower() + if language not in LANGUAGES: + if language in TO_LANGUAGE_CODE: + language = TO_LANGUAGE_CODE[language] + else: + raise ValueError(f"Unsupported language: {language}") + + if multilingual: + encoding_name = "multilingual" + language = language or "en" + task = task or "transcribe" + else: + encoding_name = "gpt2" + language = None + task = None + + encoding = get_encoding(name=encoding_name, num_languages=num_languages) + + return Tokenizer( + encoding=encoding, num_languages=num_languages, language=language, task=task + ) diff --git a/examples/model-systems/whisper/whisper/transcribe.py b/examples/model-systems/whisper/whisper/transcribe.py new file mode 100644 index 0000000..0a4cc36 --- /dev/null +++ b/examples/model-systems/whisper/whisper/transcribe.py @@ -0,0 +1,623 @@ +import argparse +import os +import traceback +import warnings +from typing import TYPE_CHECKING, List, Optional, Tuple, Union + +import numpy as np +import torch +import tqdm + +from .audio import ( + FRAMES_PER_SECOND, + HOP_LENGTH, + N_FRAMES, + N_SAMPLES, + SAMPLE_RATE, + log_mel_spectrogram, + pad_or_trim, +) +from .decoding import DecodingOptions, DecodingResult +from .timing import add_word_timestamps +from .tokenizer import LANGUAGES, TO_LANGUAGE_CODE, get_tokenizer +from .utils import ( + exact_div, + format_timestamp, + get_end, + get_writer, + make_safe, + optional_float, + optional_int, + str2bool, +) + +if TYPE_CHECKING: + from .model import Whisper + + +def transcribe( + model: "Whisper", + audio: Union[str, np.ndarray, torch.Tensor], + *, + verbose: Optional[bool] = None, + temperature: Union[float, Tuple[float, ...]] = (0.0, 0.2, 0.4, 0.6, 0.8, 1.0), + compression_ratio_threshold: Optional[float] = 2.4, + logprob_threshold: Optional[float] = -1.0, + no_speech_threshold: Optional[float] = 0.6, + condition_on_previous_text: bool = True, + initial_prompt: Optional[str] = None, + carry_initial_prompt: bool = False, + word_timestamps: bool = False, + prepend_punctuations: str = "\"'“¿([{-", + append_punctuations: str = "\"'.。,，!！?？:：”)]}、", + clip_timestamps: Union[str, List[float]] = "0", + hallucination_silence_threshold: Optional[float] = None, + **decode_options, +): + """ + Transcribe an audio file using Whisper + + Parameters + ---------- + model: Whisper + The Whisper model instance + + audio: Union[str, np.ndarray, torch.Tensor] + The path to the audio file to open, or the audio waveform + + verbose: bool + Whether to display the text being decoded to the console. If True, displays all the details, + If False, displays minimal details. If None, does not display anything + + temperature: Union[float, Tuple[float, ...]] + Temperature for sampling. It can be a tuple of temperatures, which will be successively used + upon failures according to either `compression_ratio_threshold` or `logprob_threshold`. + + compression_ratio_threshold: float + If the gzip compression ratio is above this value, treat as failed + + logprob_threshold: float + If the average log probability over sampled tokens is below this value, treat as failed + + no_speech_threshold: float + If the no_speech probability is higher than this value AND the average log probability + over sampled tokens is below `logprob_threshold`, consider the segment as silent + + condition_on_previous_text: bool + if True, the previous output of the model is provided as a prompt for the next window; + disabling may make the text inconsistent across windows, but the model becomes less prone to + getting stuck in a failure loop, such as repetition looping or timestamps going out of sync. + + word_timestamps: bool + Extract word-level timestamps using the cross-attention pattern and dynamic time warping, + and include the timestamps for each word in each segment. + + prepend_punctuations: str + If word_timestamps is True, merge these punctuation symbols with the next word + + append_punctuations: str + If word_timestamps is True, merge these punctuation symbols with the previous word + + initial_prompt: Optional[str] + Optional text to provide as a prompt for the first window. This can be used to provide, or + "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns + to make it more likely to predict those word correctly. + + carry_initial_prompt: bool + If carry_initial_prompt is True, `initial_prompt` is prepended to the prompt of each internal + `decode()` call. If there is not enough context space at the start of the prompt, it is + left-sliced to make space. + + decode_options: dict + Keyword arguments to construct `DecodingOptions` instances + + clip_timestamps: Union[str, List[float]] + Comma-separated list start,end,start,end,... timestamps (in seconds) of clips to process. + The last end timestamp defaults to the end of the file. + + hallucination_silence_threshold: Optional[float] + When word_timestamps is True, skip silent periods longer than this threshold (in seconds) + when a possible hallucination is detected + + Returns + ------- + A dictionary containing the resulting text ("text") and segment-level details ("segments"), and + the spoken language ("language"), which is detected when `decode_options["language"]` is None. + """ + dtype = torch.float16 if decode_options.get("fp16", True) else torch.float32 + if model.device == torch.device("cpu"): + if torch.cuda.is_available(): + warnings.warn("Performing inference on CPU when CUDA is available") + if dtype == torch.float16: + warnings.warn("FP16 is not supported on CPU; using FP32 instead") + dtype = torch.float32 + + if dtype == torch.float32: + decode_options["fp16"] = False + + # Pad 30-seconds of silence to the input audio, for slicing + mel = log_mel_spectrogram(audio, model.dims.n_mels, padding=N_SAMPLES) + content_frames = mel.shape[-1] - N_FRAMES + content_duration = float(content_frames * HOP_LENGTH / SAMPLE_RATE) + + if decode_options.get("language", None) is None: + if not model.is_multilingual: + decode_options["language"] = "en" + else: + if verbose: + print( + "Detecting language using up to the first 30 seconds. Use `--language` to specify the language" + ) + mel_segment = pad_or_trim(mel, N_FRAMES).to(model.device).to(dtype) + _, probs = model.detect_language(mel_segment) + decode_options["language"] = max(probs, key=probs.get) + if verbose is not None: + print( + f"Detected language: {LANGUAGES[decode_options['language']].title()}" + ) + + language: str = decode_options["language"] + task: str = decode_options.get("task", "transcribe") + tokenizer = get_tokenizer( + model.is_multilingual, + num_languages=model.num_languages, + language=language, + task=task, + ) + + if isinstance(clip_timestamps, str): + clip_timestamps = [ + float(ts) for ts in (clip_timestamps.split(",") if clip_timestamps else []) + ] + seek_points: List[int] = [round(ts * FRAMES_PER_SECOND) for ts in clip_timestamps] + if len(seek_points) == 0: + seek_points.append(0) + if len(seek_points) % 2 == 1: + seek_points.append(content_frames) + seek_clips: List[Tuple[int, int]] = list(zip(seek_points[::2], seek_points[1::2])) + + punctuation = "\"'“¿([{-\"'.。,，!！?？:：”)]}、" + + if word_timestamps and task == "translate": + warnings.warn("Word-level timestamps on translations may not be reliable.") + + def decode_with_fallback(segment: torch.Tensor) -> DecodingResult: + temperatures = ( + [temperature] if isinstance(temperature, (int, float)) else temperature + ) + decode_result = None + + for t in temperatures: + kwargs = {**decode_options} + if t > 0: + # disable beam_size and patience when t > 0 + kwargs.pop("beam_size", None) + kwargs.pop("patience", None) + else: + # disable best_of when t == 0 + kwargs.pop("best_of", None) + + options = DecodingOptions(**kwargs, temperature=t) + decode_result = model.decode(segment, options) + + needs_fallback = False + if ( + compression_ratio_threshold is not None + and decode_result.compression_ratio > compression_ratio_threshold + ): + needs_fallback = True # too repetitive + if ( + logprob_threshold is not None + and decode_result.avg_logprob < logprob_threshold + ): + needs_fallback = True # average log probability is too low + if ( + no_speech_threshold is not None + and decode_result.no_speech_prob > no_speech_threshold + and logprob_threshold is not None + and decode_result.avg_logprob < logprob_threshold + ): + needs_fallback = False # silence + if not needs_fallback: + break + + return decode_result + + clip_idx = 0 + seek = seek_clips[clip_idx][0] + input_stride = exact_div( + N_FRAMES, model.dims.n_audio_ctx + ) # mel frames per output token: 2 + time_precision = ( + input_stride * HOP_LENGTH / SAMPLE_RATE + ) # time per output token: 0.02 (seconds) + all_tokens = [] + all_segments = [] + prompt_reset_since = 0 + + remaining_prompt_length = model.dims.n_text_ctx // 2 - 1 + if initial_prompt is not None: + initial_prompt_tokens = tokenizer.encode(" " + initial_prompt.strip()) + all_tokens.extend(initial_prompt_tokens) + remaining_prompt_length -= len(initial_prompt_tokens) + else: + initial_prompt_tokens = [] + + def new_segment( + *, start: float, end: float, tokens: torch.Tensor, result: DecodingResult + ): + tokens = tokens.tolist() + text_tokens = [token for token in tokens if token < tokenizer.eot] + return { + "seek": seek, + "start": start, + "end": end, + "text": tokenizer.decode(text_tokens), + "tokens": tokens, + "temperature": result.temperature, + "avg_logprob": result.avg_logprob, + "compression_ratio": result.compression_ratio, + "no_speech_prob": result.no_speech_prob, + } + + # show the progress bar when verbose is False (if True, transcribed text will be printed) + with tqdm.tqdm( + total=content_frames, unit="frames", disable=verbose is not False + ) as pbar: + last_speech_timestamp = 0.0 + # NOTE: This loop is obscurely flattened to make the diff readable. + # A later commit should turn this into a simpler nested loop. + # for seek_clip_start, seek_clip_end in seek_clips: + # while seek < seek_clip_end + while clip_idx < len(seek_clips): + seek_clip_start, seek_clip_end = seek_clips[clip_idx] + if seek < seek_clip_start: + seek = seek_clip_start + if seek >= seek_clip_end: + clip_idx += 1 + if clip_idx < len(seek_clips): + seek = seek_clips[clip_idx][0] + continue + time_offset = float(seek * HOP_LENGTH / SAMPLE_RATE) + window_end_time = float((seek + N_FRAMES) * HOP_LENGTH / SAMPLE_RATE) + segment_size = min(N_FRAMES, content_frames - seek, seek_clip_end - seek) + mel_segment = mel[:, seek : seek + segment_size] + segment_duration = segment_size * HOP_LENGTH / SAMPLE_RATE + mel_segment = pad_or_trim(mel_segment, N_FRAMES).to(model.device).to(dtype) + + if carry_initial_prompt: + nignored = max(len(initial_prompt_tokens), prompt_reset_since) + remaining_prompt = all_tokens[nignored:][-remaining_prompt_length:] + decode_options["prompt"] = initial_prompt_tokens + remaining_prompt + else: + decode_options["prompt"] = all_tokens[prompt_reset_since:] + + result: DecodingResult = decode_with_fallback(mel_segment) + tokens = torch.tensor(result.tokens) + + if no_speech_threshold is not None: + # no voice activity check + should_skip = result.no_speech_prob > no_speech_threshold + if ( + logprob_threshold is not None + and result.avg_logprob > logprob_threshold + ): + # don't skip if the logprob is high enough, despite the no_speech_prob + should_skip = False + + if should_skip: + seek += segment_size # fast-forward to the next segment boundary + continue + + previous_seek = seek + current_segments = [] + + # anomalous words are very long/short/improbable + def word_anomaly_score(word: dict) -> float: + probability = word.get("probability", 0.0) + duration = word["end"] - word["start"] + score = 0.0 + if probability < 0.15: + score += 1.0 + if duration < 0.133: + score += (0.133 - duration) * 15 + if duration > 2.0: + score += duration - 2.0 + return score + + def is_segment_anomaly(segment: Optional[dict]) -> bool: + if segment is None or not segment["words"]: + return False + words = [w for w in segment["words"] if w["word"] not in punctuation] + words = words[:8] + score = sum(word_anomaly_score(w) for w in words) + return score >= 3 or score + 0.01 >= len(words) + + def next_words_segment(segments: List[dict]) -> Optional[dict]: + return next((s for s in segments if s["words"]), None) + + timestamp_tokens: torch.Tensor = tokens.ge(tokenizer.timestamp_begin) + single_timestamp_ending = timestamp_tokens[-2:].tolist() == [False, True] + + consecutive = torch.where(timestamp_tokens[:-1] & timestamp_tokens[1:])[0] + consecutive.add_(1) + if len(consecutive) > 0: + # if the output contains two consecutive timestamp tokens + slices = consecutive.tolist() + if single_timestamp_ending: + slices.append(len(tokens)) + + last_slice = 0 + for current_slice in slices: + sliced_tokens = tokens[last_slice:current_slice] + start_timestamp_pos = ( + sliced_tokens[0].item() - tokenizer.timestamp_begin + ) + end_timestamp_pos = ( + sliced_tokens[-1].item() - tokenizer.timestamp_begin + ) + current_segments.append( + new_segment( + start=time_offset + start_timestamp_pos * time_precision, + end=time_offset + end_timestamp_pos * time_precision, + tokens=sliced_tokens, + result=result, + ) + ) + last_slice = current_slice + + if single_timestamp_ending: + # single timestamp at the end means no speech after the last timestamp. + seek += segment_size + else: + # otherwise, ignore the unfinished segment and seek to the last timestamp + last_timestamp_pos = ( + tokens[last_slice - 1].item() - tokenizer.timestamp_begin + ) + seek += last_timestamp_pos * input_stride + else: + duration = segment_duration + timestamps = tokens[timestamp_tokens.nonzero().flatten()] + if ( + len(timestamps) > 0 + and timestamps[-1].item() != tokenizer.timestamp_begin + ): + # no consecutive timestamps but it has a timestamp; use the last one. + last_timestamp_pos = ( + timestamps[-1].item() - tokenizer.timestamp_begin + ) + duration = last_timestamp_pos * time_precision + + current_segments.append( + new_segment( + start=time_offset, + end=time_offset + duration, + tokens=tokens, + result=result, + ) + ) + seek += segment_size + + if word_timestamps: + add_word_timestamps( + segments=current_segments, + model=model, + tokenizer=tokenizer, + mel=mel_segment, + num_frames=segment_size, + prepend_punctuations=prepend_punctuations, + append_punctuations=append_punctuations, + last_speech_timestamp=last_speech_timestamp, + ) + + if not single_timestamp_ending: + last_word_end = get_end(current_segments) + if last_word_end is not None and last_word_end > time_offset: + seek = round(last_word_end * FRAMES_PER_SECOND) + + # skip silence before possible hallucinations + if hallucination_silence_threshold is not None: + threshold = hallucination_silence_threshold + if not single_timestamp_ending: + last_word_end = get_end(current_segments) + if last_word_end is not None and last_word_end > time_offset: + remaining_duration = window_end_time - last_word_end + if remaining_duration > threshold: + seek = round(last_word_end * FRAMES_PER_SECOND) + else: + seek = previous_seek + segment_size + + # if first segment might be a hallucination, skip leading silence + first_segment = next_words_segment(current_segments) + if first_segment is not None and is_segment_anomaly(first_segment): + gap = first_segment["start"] - time_offset + if gap > threshold: + seek = previous_seek + round(gap * FRAMES_PER_SECOND) + continue + + # skip silence before any possible hallucination that is surrounded + # by silence or more hallucinations + hal_last_end = last_speech_timestamp + for si in range(len(current_segments)): + segment = current_segments[si] + if not segment["words"]: + continue + if is_segment_anomaly(segment): + next_segment = next_words_segment( + current_segments[si + 1 :] + ) + if next_segment is not None: + hal_next_start = next_segment["words"][0]["start"] + else: + hal_next_start = time_offset + segment_duration + silence_before = ( + segment["start"] - hal_last_end > threshold + or segment["start"] < threshold + or segment["start"] - time_offset < 2.0 + ) + silence_after = ( + hal_next_start - segment["end"] > threshold + or is_segment_anomaly(next_segment) + or window_end_time - segment["end"] < 2.0 + ) + if silence_before and silence_after: + seek = round( + max(time_offset + 1, segment["start"]) + * FRAMES_PER_SECOND + ) + if content_duration - segment["end"] < threshold: + seek = content_frames + current_segments[si:] = [] + break + hal_last_end = segment["end"] + + last_word_end = get_end(current_segments) + if last_word_end is not None: + last_speech_timestamp = last_word_end + + if verbose: + for segment in current_segments: + start, end, text = segment["start"], segment["end"], segment["text"] + line = f"[{format_timestamp(start)} --> {format_timestamp(end)}] {text}" + print(make_safe(line)) + + # if a segment is instantaneous or does not contain text, clear it + for i, segment in enumerate(current_segments): + if segment["start"] == segment["end"] or segment["text"].strip() == "": + segment["text"] = "" + segment["tokens"] = [] + segment["words"] = [] + + all_segments.extend( + [ + {"id": i, **segment} + for i, segment in enumerate( + current_segments, start=len(all_segments) + ) + ] + ) + all_tokens.extend( + [token for segment in current_segments for token in segment["tokens"]] + ) + + if not condition_on_previous_text or result.temperature > 0.5: + # do not feed the prompt tokens if a high temperature was used + prompt_reset_since = len(all_tokens) + + # update progress bar + pbar.update(min(content_frames, seek) - previous_seek) + + return dict( + text=tokenizer.decode(all_tokens[len(initial_prompt_tokens) :]), + segments=all_segments, + language=language, + ) + + +def cli(): + from . import available_models + + def valid_model_name(name): + if name in available_models() or os.path.exists(name): + return name + raise ValueError( + f"model should be one of {available_models()} or path to a model checkpoint" + ) + + # fmt: off + parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) + parser.add_argument("audio", nargs="+", type=str, help="audio file(s) to transcribe") + parser.add_argument("--model", default="turbo", type=valid_model_name, help="name of the Whisper model to use") + parser.add_argument("--model_dir", type=str, default=None, help="the path to save model files; uses ~/.cache/whisper by default") + parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu", help="device to use for PyTorch inference") + parser.add_argument("--output_dir", "-o", type=str, default=".", help="directory to save the outputs") + parser.add_argument("--output_format", "-f", type=str, default="all", choices=["txt", "vtt", "srt", "tsv", "json", "all"], help="format of the output file; if not specified, all available formats will be produced") + parser.add_argument("--verbose", type=str2bool, default=True, help="whether to print out the progress and debug messages") + + parser.add_argument("--task", type=str, default="transcribe", choices=["transcribe", "translate"], help="whether to perform X->X speech recognition ('transcribe') or X->English translation ('translate')") + parser.add_argument("--language", type=str, default=None, choices=sorted(LANGUAGES.keys()) + sorted([k.title() for k in TO_LANGUAGE_CODE.keys()]), help="language spoken in the audio, specify None to perform language detection") + + parser.add_argument("--temperature", type=float, default=0, help="temperature to use for sampling") + parser.add_argument("--best_of", type=optional_int, default=5, help="number of candidates when sampling with non-zero temperature") + parser.add_argument("--beam_size", type=optional_int, default=5, help="number of beams in beam search, only applicable when temperature is zero") + parser.add_argument("--patience", type=float, default=None, help="optional patience value to use in beam decoding, as in https://arxiv.org/abs/2204.05424, the default (1.0) is equivalent to conventional beam search") + parser.add_argument("--length_penalty", type=float, default=None, help="optional token length penalty coefficient (alpha) as in https://arxiv.org/abs/1609.08144, uses simple length normalization by default") + + parser.add_argument("--suppress_tokens", type=str, default="-1", help="comma-separated list of token ids to suppress during sampling; '-1' will suppress most special characters except common punctuations") + parser.add_argument("--initial_prompt", type=str, default=None, help="optional text to provide as a prompt for the first window.") + parser.add_argument("--carry_initial_prompt", type=str2bool, default=False, help="if True, prepend initial_prompt to every internal decode() call. May reduce the effectiveness of condition_on_previous_text") + + parser.add_argument("--condition_on_previous_text", type=str2bool, default=True, help="if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop") + parser.add_argument("--fp16", type=str2bool, default=True, help="whether to perform inference in fp16; True by default") + + parser.add_argument("--temperature_increment_on_fallback", type=optional_float, default=0.2, help="temperature to increase when falling back when the decoding fails to meet either of the thresholds below") + parser.add_argument("--compression_ratio_threshold", type=optional_float, default=2.4, help="if the gzip compression ratio is higher than this value, treat the decoding as failed") + parser.add_argument("--logprob_threshold", type=optional_float, default=-1.0, help="if the average log probability is lower than this value, treat the decoding as failed") + parser.add_argument("--no_speech_threshold", type=optional_float, default=0.6, help="if the probability of the <|nospeech|> token is higher than this value AND the decoding has failed due to `logprob_threshold`, consider the segment as silence") + parser.add_argument("--word_timestamps", type=str2bool, default=False, help="(experimental) extract word-level timestamps and refine the results based on them") + parser.add_argument("--prepend_punctuations", type=str, default="\"\'“¿([{-", help="if word_timestamps is True, merge these punctuation symbols with the next word") + parser.add_argument("--append_punctuations", type=str, default="\"\'.。,，!！?？:：”)]}、", help="if word_timestamps is True, merge these punctuation symbols with the previous word") + parser.add_argument("--highlight_words", type=str2bool, default=False, help="(requires --word_timestamps True) underline each word as it is spoken in srt and vtt") + parser.add_argument("--max_line_width", type=optional_int, default=None, help="(requires --word_timestamps True) the maximum number of characters in a line before breaking the line") + parser.add_argument("--max_line_count", type=optional_int, default=None, help="(requires --word_timestamps True) the maximum number of lines in a segment") + parser.add_argument("--max_words_per_line", type=optional_int, default=None, help="(requires --word_timestamps True, no effect with --max_line_width) the maximum number of words in a segment") + parser.add_argument("--threads", type=optional_int, default=0, help="number of threads used by torch for CPU inference; supercedes MKL_NUM_THREADS/OMP_NUM_THREADS") + parser.add_argument("--clip_timestamps", type=str, default="0", help="comma-separated list start,end,start,end,... timestamps (in seconds) of clips to process, where the last end timestamp defaults to the end of the file") + parser.add_argument("--hallucination_silence_threshold", type=optional_float, help="(requires --word_timestamps True) skip silent periods longer than this threshold (in seconds) when a possible hallucination is detected") + # fmt: on + + args = parser.parse_args().__dict__ + model_name: str = args.pop("model") + model_dir: str = args.pop("model_dir") + output_dir: str = args.pop("output_dir") + output_format: str = args.pop("output_format") + device: str = args.pop("device") + os.makedirs(output_dir, exist_ok=True) + + if model_name.endswith(".en") and args["language"] not in {"en", "English"}: + if args["language"] is not None: + warnings.warn( + f"{model_name} is an English-only model but receipted '{args['language']}'; using English instead." + ) + args["language"] = "en" + + temperature = args.pop("temperature") + if (increment := args.pop("temperature_increment_on_fallback")) is not None: + temperature = tuple(np.arange(temperature, 1.0 + 1e-6, increment)) + else: + temperature = [temperature] + + if (threads := args.pop("threads")) > 0: + torch.set_num_threads(threads) + + from . import load_model + + model = load_model(model_name, device=device, download_root=model_dir) + + writer = get_writer(output_format, output_dir) + word_options = [ + "highlight_words", + "max_line_count", + "max_line_width", + "max_words_per_line", + ] + if not args["word_timestamps"]: + for option in word_options: + if args[option]: + parser.error(f"--{option} requires --word_timestamps True") + if args["max_line_count"] and not args["max_line_width"]: + warnings.warn("--max_line_count has no effect without --max_line_width") + if args["max_words_per_line"] and args["max_line_width"]: + warnings.warn("--max_words_per_line has no effect with --max_line_width") + writer_args = {arg: args.pop(arg) for arg in word_options} + for audio_path in args.pop("audio"): + try: + result = transcribe(model, audio_path, temperature=temperature, **args) + writer(result, audio_path, **writer_args) + except Exception as e: + traceback.print_exc() + print(f"Skipping {audio_path} due to {type(e).__name__}: {str(e)}") + + +if __name__ == "__main__": + cli() diff --git a/examples/model-systems/whisper/whisper/triton_ops.py b/examples/model-systems/whisper/whisper/triton_ops.py new file mode 100644 index 0000000..13d417b --- /dev/null +++ b/examples/model-systems/whisper/whisper/triton_ops.py @@ -0,0 +1,117 @@ +from functools import lru_cache + +import numpy as np +import torch + +try: + import triton + import triton.language as tl +except ImportError: + raise RuntimeError("triton import failed; try `pip install --pre triton`") + + +@triton.jit +def dtw_kernel( + cost, trace, x, x_stride, cost_stride, trace_stride, N, M, BLOCK_SIZE: tl.constexpr +): + offsets = tl.arange(0, BLOCK_SIZE) + mask = offsets < M + + for k in range(1, N + M + 1): # k = i + j + tl.debug_barrier() + + p0 = cost + (k - 1) * cost_stride + p1 = cost + k * cost_stride + p2 = cost + k * cost_stride + 1 + + c0 = tl.load(p0 + offsets, mask=mask) + c1 = tl.load(p1 + offsets, mask=mask) + c2 = tl.load(p2 + offsets, mask=mask) + + x_row = tl.load(x + (k - 1) * x_stride + offsets, mask=mask, other=0) + cost_row = x_row + tl.minimum(tl.minimum(c0, c1), c2) + + cost_ptr = cost + (k + 1) * cost_stride + 1 + tl.store(cost_ptr + offsets, cost_row, mask=mask) + + trace_ptr = trace + (k + 1) * trace_stride + 1 + tl.store(trace_ptr + offsets, 2, mask=mask & (c2 <= c0) & (c2 <= c1)) + tl.store(trace_ptr + offsets, 1, mask=mask & (c1 <= c0) & (c1 <= c2)) + tl.store(trace_ptr + offsets, 0, mask=mask & (c0 <= c1) & (c0 <= c2)) + + +@lru_cache(maxsize=None) +def median_kernel(filter_width: int): + @triton.jit + def kernel( + y, x, x_stride, y_stride, BLOCK_SIZE: tl.constexpr + ): # x.shape[-1] == filter_width + row_idx = tl.program_id(0) + offsets = tl.arange(0, BLOCK_SIZE) + mask = offsets < y_stride + + x_ptr = x + row_idx * x_stride # noqa: F841 + y_ptr = y + row_idx * y_stride + + LOAD_ALL_ROWS_HERE # noqa: F821 + + BUBBLESORT_HERE # noqa: F821 + + tl.store(y_ptr + offsets, MIDDLE_ROW_HERE, mask=mask) # noqa: F821 + + kernel = triton.JITFunction(kernel.fn) + new_kernel = kernel.src.replace( + " LOAD_ALL_ROWS_HERE", + "\n".join( + [ + f" row{i} = tl.load(x_ptr + offsets + {i}, mask=mask)" + for i in range(filter_width) + ] + ), + ) + + new_kernel = new_kernel.replace( + " BUBBLESORT_HERE", + "\n\n".join( + [ + "\n\n".join( + [ + "\n".join( + [ + f" smaller = tl.where(row{j} < row{j + 1}, row{j}, row{j + 1})", + f" larger = tl.where(row{j} > row{j + 1}, row{j}, row{j + 1})", + f" row{j} = smaller", + f" row{j + 1} = larger", + ] + ) + for j in range(filter_width - i - 1) + ] + ) + for i in range(filter_width // 2 + 1) + ] + ), + ) + + new_kernel = new_kernel.replace("MIDDLE_ROW_HERE", f"row{filter_width // 2}") + + if hasattr(kernel, "_unsafe_update_src") is True: + kernel._unsafe_update_src(new_kernel) + kernel.hash = None + else: + kernel.src = new_kernel + + return kernel + + +def median_filter_cuda(x: torch.Tensor, filter_width: int): + """Apply a median filter of given width along the last dimension of x""" + slices = x.contiguous().unfold(-1, filter_width, 1) + grid = np.prod(slices.shape[:-2]) + + kernel = median_kernel(filter_width) + y = torch.empty_like(slices[..., 0]) + + BLOCK_SIZE = 1 << (y.stride(-2) - 1).bit_length() + kernel[(grid,)](y, x, x.stride(-2), y.stride(-2), BLOCK_SIZE=BLOCK_SIZE) + + return y diff --git a/examples/model-systems/whisper/whisper/utils.py b/examples/model-systems/whisper/whisper/utils.py new file mode 100644 index 0000000..13792f7 --- /dev/null +++ b/examples/model-systems/whisper/whisper/utils.py @@ -0,0 +1,318 @@ +import json +import os +import re +import sys +import zlib +from typing import Callable, List, Optional, TextIO + +system_encoding = sys.getdefaultencoding() + +if system_encoding != "utf-8": + + def make_safe(string): + # replaces any character not representable using the system default encoding with an '?', + # avoiding UnicodeEncodeError (https://github.com/openai/whisper/discussions/729). + return string.encode(system_encoding, errors="replace").decode(system_encoding) + +else: + + def make_safe(string): + # utf-8 can encode any Unicode code point, so no need to do the round-trip encoding + return string + + +def exact_div(x, y): + assert x % y == 0 + return x // y + + +def str2bool(string): + str2val = {"True": True, "False": False} + if string in str2val: + return str2val[string] + else: + raise ValueError(f"Expected one of {set(str2val.keys())}, got {string}") + + +def optional_int(string): + return None if string == "None" else int(string) + + +def optional_float(string): + return None if string == "None" else float(string) + + +def compression_ratio(text) -> float: + text_bytes = text.encode("utf-8") + return len(text_bytes) / len(zlib.compress(text_bytes)) + + +def format_timestamp( + seconds: float, always_include_hours: bool = False, decimal_marker: str = "." +): + assert seconds >= 0, "non-negative timestamp expected" + milliseconds = round(seconds * 1000.0) + + hours = milliseconds // 3_600_000 + milliseconds -= hours * 3_600_000 + + minutes = milliseconds // 60_000 + milliseconds -= minutes * 60_000 + + seconds = milliseconds // 1_000 + milliseconds -= seconds * 1_000 + + hours_marker = f"{hours:02d}:" if always_include_hours or hours > 0 else "" + return ( + f"{hours_marker}{minutes:02d}:{seconds:02d}{decimal_marker}{milliseconds:03d}" + ) + + +def get_start(segments: List[dict]) -> Optional[float]: + return next( + (w["start"] for s in segments for w in s["words"]), + segments[0]["start"] if segments else None, + ) + + +def get_end(segments: List[dict]) -> Optional[float]: + return next( + (w["end"] for s in reversed(segments) for w in reversed(s["words"])), + segments[-1]["end"] if segments else None, + ) + + +class ResultWriter: + extension: str + + def __init__(self, output_dir: str): + self.output_dir = output_dir + + def __call__( + self, result: dict, audio_path: str, options: Optional[dict] = None, **kwargs + ): + audio_basename = os.path.basename(audio_path) + audio_basename = os.path.splitext(audio_basename)[0] + output_path = os.path.join( + self.output_dir, audio_basename + "." + self.extension + ) + + with open(output_path, "w", encoding="utf-8") as f: + self.write_result(result, file=f, options=options, **kwargs) + + def write_result( + self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + raise NotImplementedError + + +class WriteTXT(ResultWriter): + extension: str = "txt" + + def write_result( + self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + for segment in result["segments"]: + print(segment["text"].strip(), file=file, flush=True) + + +class SubtitlesWriter(ResultWriter): + always_include_hours: bool + decimal_marker: str + + def iterate_result( + self, + result: dict, + options: Optional[dict] = None, + *, + max_line_width: Optional[int] = None, + max_line_count: Optional[int] = None, + highlight_words: bool = False, + max_words_per_line: Optional[int] = None, + ): + options = options or {} + max_line_width = max_line_width or options.get("max_line_width") + max_line_count = max_line_count or options.get("max_line_count") + highlight_words = highlight_words or options.get("highlight_words", False) + max_words_per_line = max_words_per_line or options.get("max_words_per_line") + preserve_segments = max_line_count is None or max_line_width is None + max_line_width = max_line_width or 1000 + max_words_per_line = max_words_per_line or 1000 + + def iterate_subtitles(): + line_len = 0 + line_count = 1 + # the next subtitle to yield (a list of word timings with whitespace) + subtitle: List[dict] = [] + last: float = get_start(result["segments"]) or 0.0 + for segment in result["segments"]: + chunk_index = 0 + words_count = max_words_per_line + while chunk_index < len(segment["words"]): + remaining_words = len(segment["words"]) - chunk_index + if max_words_per_line > len(segment["words"]) - chunk_index: + words_count = remaining_words + for i, original_timing in enumerate( + segment["words"][chunk_index : chunk_index + words_count] + ): + timing = original_timing.copy() + long_pause = ( + not preserve_segments and timing["start"] - last > 3.0 + ) + has_room = line_len + len(timing["word"]) <= max_line_width + seg_break = i == 0 and len(subtitle) > 0 and preserve_segments + if ( + line_len > 0 + and has_room + and not long_pause + and not seg_break + ): + # line continuation + line_len += len(timing["word"]) + else: + # new line + timing["word"] = timing["word"].strip() + if ( + len(subtitle) > 0 + and max_line_count is not None + and (long_pause or line_count >= max_line_count) + or seg_break + ): + # subtitle break + yield subtitle + subtitle = [] + line_count = 1 + elif line_len > 0: + # line break + line_count += 1 + timing["word"] = "\n" + timing["word"] + line_len = len(timing["word"].strip()) + subtitle.append(timing) + last = timing["start"] + chunk_index += max_words_per_line + if len(subtitle) > 0: + yield subtitle + + if len(result["segments"]) > 0 and "words" in result["segments"][0]: + for subtitle in iterate_subtitles(): + subtitle_start = self.format_timestamp(subtitle[0]["start"]) + subtitle_end = self.format_timestamp(subtitle[-1]["end"]) + subtitle_text = "".join([word["word"] for word in subtitle]) + if highlight_words: + last = subtitle_start + all_words = [timing["word"] for timing in subtitle] + for i, this_word in enumerate(subtitle): + start = self.format_timestamp(this_word["start"]) + end = self.format_timestamp(this_word["end"]) + if last != start: + yield last, start, subtitle_text + + yield start, end, "".join( + [ + ( + re.sub(r"^(\s*)(.*)$", r"\1\2", word) + if j == i + else word + ) + for j, word in enumerate(all_words) + ] + ) + last = end + else: + yield subtitle_start, subtitle_end, subtitle_text + else: + for segment in result["segments"]: + segment_start = self.format_timestamp(segment["start"]) + segment_end = self.format_timestamp(segment["end"]) + segment_text = segment["text"].strip().replace("-->", "->") + yield segment_start, segment_end, segment_text + + def format_timestamp(self, seconds: float): + return format_timestamp( + seconds=seconds, + always_include_hours=self.always_include_hours, + decimal_marker=self.decimal_marker, + ) + + +class WriteVTT(SubtitlesWriter): + extension: str = "vtt" + always_include_hours: bool = False + decimal_marker: str = "." + + def write_result( + self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + print("WEBVTT\n", file=file) + for start, end, text in self.iterate_result(result, options, **kwargs): + print(f"{start} --> {end}\n{text}\n", file=file, flush=True) + + +class WriteSRT(SubtitlesWriter): + extension: str = "srt" + always_include_hours: bool = True + decimal_marker: str = "," + + def write_result( + self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + for i, (start, end, text) in enumerate( + self.iterate_result(result, options, **kwargs), start=1 + ): + print(f"{i}\n{start} --> {end}\n{text}\n", file=file, flush=True) + + +class WriteTSV(ResultWriter): + """ + Write a transcript to a file in TSV (tab-separated values) format containing lines like: + \t\t + + Using integer milliseconds as start and end times means there's no chance of interference from + an environment setting a language encoding that causes the decimal in a floating point number + to appear as a comma; also is faster and more efficient to parse & store, e.g., in C++. + """ + + extension: str = "tsv" + + def write_result( + self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + print("start", "end", "text", sep="\t", file=file) + for segment in result["segments"]: + print(round(1000 * segment["start"]), file=file, end="\t") + print(round(1000 * segment["end"]), file=file, end="\t") + print(segment["text"].strip().replace("\t", " "), file=file, flush=True) + + +class WriteJSON(ResultWriter): + extension: str = "json" + + def write_result( + self, result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + json.dump(result, file) + + +def get_writer( + output_format: str, output_dir: str +) -> Callable[[dict, TextIO, dict], None]: + writers = { + "txt": WriteTXT, + "vtt": WriteVTT, + "srt": WriteSRT, + "tsv": WriteTSV, + "json": WriteJSON, + } + + if output_format == "all": + all_writers = [writer(output_dir) for writer in writers.values()] + + def write_all( + result: dict, file: TextIO, options: Optional[dict] = None, **kwargs + ): + for writer in all_writers: + writer(result, file, options, **kwargs) + + return write_all + + return writers[output_format](output_dir) diff --git a/examples/model-systems/whisper/whisper/version.py b/examples/model-systems/whisper/whisper/version.py new file mode 100644 index 0000000..67426aa --- /dev/null +++ b/examples/model-systems/whisper/whisper/version.py @@ -0,0 +1 @@ +__version__ = "20250625" diff --git a/examples/nextjs/bad-architecture-doc-example.md b/examples/nextjs/bad-architecture-doc-example.md new file mode 100644 index 0000000..1d781ee --- /dev/null +++ b/examples/nextjs/bad-architecture-doc-example.md @@ -0,0 +1,73 @@ +# LinkBoard Architecture Overview + +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns — for Next.js, above all App-Router/Pages-Router conflation. Each ❌ callout explains a failure. See good-architecture-doc-example.md for the correct approach. + +## What This App Does + +LinkBoard is a production-grade, real-time link-sharing platform for distributed teams. Members authenticate with their workspace account, share links, vote in real time, and receive notifications when their submissions trend. The platform is built on Next.js with a PostgreSQL database and scales horizontally across regions. + +> **❌ PROBLEMS:** Inflated purpose with zero citations. There is no authentication, no real-time anything, no notifications, and no database — the whole data layer is one in-memory `Map` seeded with three links (`lib/store.js:14`). "Scales horizontally" is the opposite of the truth: the store's own header comment warns that instances silently diverge (`lib/store.js:4-6`). The good doc states what the app is in one cited paragraph. + +## Technology Stack + +- Next.js 14 using the Pages Router for maximum stability +- React 18 with SWR for client-side data fetching +- Prisma ORM over PostgreSQL +- NextAuth.js for session management +- TailwindCSS for styling + +> **❌ PROBLEMS:** Four of five bullets are invented. The repo uses the **App Router** — there is no `pages/` directory at all; routing lives in `app/` (`app/page.jsx:5`, `app/layout.jsx:8`). `package.json` lists exactly three dependencies: `next`, `react`, `react-dom` (`package.json:10-14`) — no SWR, no Prisma, no NextAuth, no Tailwind. Data fetching on pages is not client-side at all: the board page is a server component that calls the store in-process (`app/page.jsx:1-6`). This is the classic Pages-Router conflation: the author documented the Next.js they remember, not the Next.js in the repo. + +## Routing & Data Fetching + +Routes are defined as files under `pages/`. The home page at `pages/index.js` fetches the link list in `getServerSideProps`, which runs on every request and passes props to the page component. The detail page at `pages/links/[id].js` uses `getServerSideProps` with `context.params.id`, and `pages/submit.js` renders the submission form. Static marketing pages use `getStaticProps` with incremental static regeneration. + +> **❌ PROBLEMS:** Every path in this section is fabricated. There is no `pages/` directory and the strings `getServerSideProps`, `getStaticProps`, and `getInitialProps` appear nowhere in the repo. The real routes are App Router files: `app/page.jsx` (board, `app/page.jsx:5`), `app/links/[id]/page.jsx` (detail, line 6), and `app/submit/page.jsx` (submit, line 8). Server components make `getServerSideProps` unnecessary — the page function itself runs on the server and reads the store directly (`app/page.jsx:6`). Citing files that do not exist is the single fastest way to destroy a reader's trust. + +## API Layer + +The API lives in `pages/api/links.js`, which exports a default handler that switches on `req.method` to implement GET, POST, PUT, and DELETE. Voting is handled by `pages/api/links/[id]/vote.js`. All handlers use the shared `withAuth` wrapper to reject unauthenticated requests before touching the database. + +> **❌ PROBLEMS:** Wrong router, wrong file shape, wrong handler signature, invented wrapper. The real API routes are App Router **route handlers**: `app/api/links/route.js` exports named `GET` and `POST` functions (`app/api/links/route.js:7`, `app/api/links/route.js:11`) — there is no default export and no `req.method` switch. The vote endpoint is `PATCH` in `app/api/links/[id]/route.js:4`, not a `vote.js` file, and no PUT or DELETE exists anywhere. `withAuth` is pure invention — the handlers call the store and return `NextResponse.json` with nothing in between (`app/api/links/[id]/route.js:5-9`). + +## Authentication & Middleware + +`middleware.js` at the project root intercepts every request, validates the NextAuth JWT from the session cookie, and redirects unauthenticated users to `/login`. The `/submit` page and all mutating API routes are protected; the board is public. Role claims in the token distinguish admins, who may delete links, from members. + +> **❌ PROBLEMS:** This entire section describes a file that does not exist. There is no `middleware.js` or `middleware.ts` anywhere in the repo, no login page, no cookie handling, no JWT, no roles, and no delete capability. Every page and API route is open — the POST handler validates `title` and `url` but never asks who is calling (`app/api/links/route.js:11-25`). An honest doc records this as a verified absence, the way the good doc's Entry Points section does with its middleware `[NOT_FOUND]` entry. Inventing a security layer is the most dangerous hallucination in this document: a reader could ship the app believing it is protected. + +## Data Layer + +`lib/store.js` is a thin repository wrapper around Prisma. It opens a pooled PostgreSQL connection, and each helper (`listLinks`, `getLink`, `createLink`, `voteLink`) delegates to `prisma.link` queries. Votes use an atomic `increment` update to stay consistent under concurrent load. + +> **❌ PROBLEMS:** The function names are real — the author clearly skimmed the exports (`lib/store.js:41`, `lib/store.js:45`, `lib/store.js:49`, `lib/store.js:63`) — but everything about their implementation is fiction. `lib/store.js` contains a module-level `new Map(...)` with three hard-coded seed links (`lib/store.js:14-39`); the words Prisma, PostgreSQL, and pool appear nowhere in the repo. Far from "consistent under concurrent load", the store's own comment warns that data is wiped on restart and diverges across serverless instances (`lib/store.js:4-6`). Wrapping real symbol names around an imagined implementation is exactly the failure mode citation-plus-quote verification exists to catch. + +## Hydration & Data Flow + +1. Browser requests `/` and receives the server-rendered HTML shell +2. Next.js sends the JavaScript bundle for the whole page +3. React hydrates every component on the page, attaching event listeners +4. SWR takes over data fetching and revalidates the link list every 30 seconds +5. When the user votes, the mutation is sent and SWR optimistically updates the cache +6. The reconciled list re-renders with fresh vote counts + +> **❌ PROBLEMS:** Two failures at once. First, the content is wrong: in the App Router only the two `'use client'` components are hydrated — `SubmitForm` (`components/SubmitForm.jsx:1`) and `VoteButton` (`components/VoteButton.jsx:1`); server components ship as rendered output, not as hydratable bundle code. There is no SWR, no polling, no revalidation interval, and no optimistic cache — voting is a bare `fetch` PATCH followed by `router.refresh()` (`components/VoteButton.jsx:14-16`). Second, the format is wrong: a numbered step-by-step trace of runtime behavior does not belong in an architecture overview at all — the methodology defers execution tracing to `02-code-flows.md`, and nothing here could be verified by reading source anyway. + +## Database Schema + +| Table | Columns | +|-------|---------| +| users | id, email, password_hash, workspace_id | +| links | id, title, url, tag, votes, user_id, created_at | +| votes | id, link_id, user_id, created_at | + +> **❌ PROBLEMS:** There is no database, so there is no schema. No `users` table (no users exist at all), no `votes` table (votes are a plain integer field incremented in place, `lib/store.js:66`), no `user_id` anywhere. The real record shape is the object literal in the seed data: `id`, `title`, `url`, `tag`, `votes`, `createdAt` (`lib/store.js:15-22`) — note the doc even gets the casing wrong (`created_at` vs `createdAt`). A schema table with made-up tables reads as authoritative and is pure fiction. + +## Why This Example is BAD + +1. **Claims the Pages Router in an App Router repo.** `pages/index.js`, `getServerSideProps`, `pages/api/links.js` — none exist; routing is `app/page.jsx:5`, `app/links/[id]/page.jsx:6`, `app/submit/page.jsx:8`, and the API is named `GET`/`POST` exports in `app/api/links/route.js:7` and `app/api/links/route.js:11`. This conflation is *the* signature Next.js hallucination. +2. **Invents middleware-based authentication.** No `middleware.js`, no NextAuth, no `withAuth`, no login route exists; every surface is open (`app/api/links/route.js:11-25` checks fields, never identity). A fabricated security layer can cause real-world harm. +3. **Claims Prisma/PostgreSQL behind `lib/store.js`.** The file is an in-memory `Map` with seed data (`lib/store.js:14-39`) whose own comment documents data loss on restart (`lib/store.js:4-6`). +4. **Describes a hydration flow step by step.** Wrong content (only `components/SubmitForm.jsx:1` and `components/VoteButton.jsx:1` are client components; no SWR — voting is `fetch` plus `router.refresh()`, `components/VoteButton.jsx:14-16`) and wrong genre — execution traces belong in `02-code-flows.md`, not an architecture overview. +5. **Presents an invented database schema.** No `users` or `votes` tables; the real shape is the seed object literal (`lib/store.js:15-22`), including `createdAt`, not `created_at`. +6. **Zero citations and zero verification tags.** Nothing is marked `[VERIFIED]`, `[INFERRED]`, or `[NOT_FOUND]`, so a reader cannot distinguish the real function names (there are a few) from the fiction wrapped around them — and `verify.py` has nothing to check. Unverifiable confidence is worse than admitted uncertainty. diff --git a/examples/nextjs/good-architecture-doc-example.md b/examples/nextjs/good-architecture-doc-example.md new file mode 100644 index 0000000..b7c871c --- /dev/null +++ b/examples/nextjs/good-architecture-doc-example.md @@ -0,0 +1,308 @@ +# LinkBoard Architecture Overview + +## Metadata +| Field | Value | +|-------|-------| +| Repository | `agent-system-mapper` | +| Path | `examples/nextjs/linkboard/` | +| Commit | `213c7d4` | +| Documented | `2026-08-03` | +| Verification Status | `Verified` | + +Verify with: + +```bash +python3 verify.py examples/nextjs/good-architecture-doc-example.md --repo-root examples/nextjs/linkboard +``` + +## Verification Summary +- `[VERIFIED]`: 71 tags — 63 carrying 85 machine-checkable `path:line` citations (100% resolved), 8 informal (dynamic-segment paths; see the note in "Why This Example is GOOD") +- `[INFERRED]`: 2 claims +- `[NOT_FOUND]`: 10 items (middleware, Pages Router, database, auth, styling, tests, env config, error/loading conventions, lockfile, external services) +- `[ASSUMED]`: 1 item (npm as package manager) +- `[NEEDS_VERIFICATION]`: 1 item (per-platform process model for the in-memory store) + +--- + +## 0. System Classification + +| Field | Value | +|-------|-------| +| Category | Traditional Code | +| Type | Full-stack Next.js **App Router** application: server-rendered pages plus co-located JSON API routes in one deployable | +| Evidence | `app/` directory with `layout.jsx`/`page.jsx` files [VERIFIED: app/layout.jsx:8, app/page.jsx:5]; HTTP handlers under `app/api/` [VERIFIED: app/api/links/route.js:7, 11]; `next` dependency [VERIFIED: package.json:11] | +| Overlay Loaded | No | +| Confidence | `[VERIFIED]` | + +The dependency block that anchors the classification, quoted: + +[VERIFIED: package.json:10-14] +```json + "dependencies": { + "next": "~14.2.3", + "react": "^18.2.0", + "react-dom": "^18.2.0" + } +``` + +**Why "Frontend SPA" would be the wrong classification:** the pages are server components that read data in-process (no client-side data fetching on render) [VERIFIED: app/page.jsx:1-6], and the repo ships its own HTTP API surface under `app/api/` [VERIFIED: app/api/links/route.js:7-9]. A SPA classification would miss the entire backend half of the system. + +--- + +## 1. System Purpose + +LinkBoard is a small team link-sharing board. Teammates browse a vote-ranked list of shared links on the board page, open a detail page per link, submit new links through a form, and upvote links. Reads happen server-side directly against an in-memory store; writes go through JSON API routes called from two client components. The store seeds three example links and holds everything in process memory [VERIFIED: lib/store.js:14, 41-43]. + +--- + +## 2. Component Map + +| Component | Location | Responsibility | Evidence | +|-----------|----------|----------------|----------| +| Root layout | `app/layout.jsx` | Wraps every route with header and nav | [VERIFIED: app/layout.jsx:8-19] | +| Board page | `app/page.jsx` | Server component; lists links sorted by votes | [VERIFIED: app/page.jsx:5-6] | +| Link detail page | `app/links/[id]/page.jsx` | Server component for the dynamic `/links/:id` route; 404s on unknown id | [VERIFIED: app/links/[id]/page.jsx:6-10] [VERIFIED: lib/store.js:45-47] | +| Submit page | `app/submit/page.jsx` | Server component shell that renders the client form | [VERIFIED: app/submit/page.jsx:8, 12] | +| SubmitForm | `components/SubmitForm.jsx` | Client component; controlled form that POSTs to the API | [VERIFIED: components/SubmitForm.jsx:1, 16-20] | +| VoteButton | `components/VoteButton.jsx` | Client component; PATCHes a vote, then refreshes | [VERIFIED: components/VoteButton.jsx:1, 14] | +| LinkCard | `components/LinkCard.jsx` | Server component; renders one board row | [VERIFIED: components/LinkCard.jsx:5-6] | +| Collection API route | `app/api/links/route.js` | `GET` list and `POST` create handlers | [VERIFIED: app/api/links/route.js:7, 11] | +| Item API route | `app/api/links/[id]/route.js` | `PATCH` vote handler for one link | [VERIFIED: app/api/links/[id]/route.js:4] [VERIFIED: lib/store.js:63-68] | +| Store | `lib/store.js` | In-memory `Map` with seed data; all reads and writes | [VERIFIED: lib/store.js:14, 41, 49, 63] | + +[INFERRED: `LinkCard`, the three pages, and the layout are React **server** components — none of them opens with a `'use client'` directive, and App Router files default to server components. Only `SubmitForm` and `VoteButton` carry the directive.] + +--- + +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) + +Page routes and API routes are distinct execution surfaces: page requests render React server components to HTML, while `/api/*` requests run plain HTTP handlers that return JSON. Detailed tracing of any surface belongs in `02-code-flows.md`. + +### 3.1 Primary Execution Surfaces + +| Entry Surface | Type | Primary Components Involved | Evidence | +|---------------|------|-----------------------------|----------| +| `GET /` | Page (server-rendered) | BoardPage, `listLinks`, LinkCard, VoteButton | [VERIFIED: app/page.jsx:5-6] | +| `GET /links/:id` | Page (server-rendered, dynamic segment) | LinkDetailPage, `getLink`, VoteButton | [VERIFIED: app/links/[id]/page.jsx:6-7] [VERIFIED: lib/store.js:45-47] | +| `GET /submit` | Page (server-rendered) | SubmitPage, SubmitForm | [VERIFIED: app/submit/page.jsx:8, 12] | +| `GET /api/links` | API (JSON) | `GET` handler, `listLinks` | [VERIFIED: app/api/links/route.js:7-9] | +| `POST /api/links` | API (JSON) | `POST` handler, `createLink` | [VERIFIED: app/api/links/route.js:11, lib/store.js:49] | +| `PATCH /api/links/:id` | API (JSON, dynamic segment) | `PATCH` handler, `voteLink` | [VERIFIED: app/api/links/[id]/route.js:4-5] [VERIFIED: lib/store.js:63-68] | + +### 3.2 High-Level Data Movement (Non-Procedural) + +| Stage | Input Type | Output Type | Participating Components | +|-------|------------|-------------|--------------------------| +| Page render | HTTP GET for a page path | HTML (server-rendered React tree) | layout, page components, `lib/store.js` reads | +| Link listing | Module state (`Map`) | Vote-sorted array of link objects | `listLinks` [VERIFIED: lib/store.js:41-43] | +| Link creation | JSON body (`title`, `url`, `tag`) | 201 JSON link object, or 400 JSON error | `POST` handler, `createLink` [VERIFIED: app/api/links/route.js:11-30] | +| Vote increment | URL id segment | JSON link object with incremented `votes`, or 404 | `PATCH` handler, `voteLink` [VERIFIED: lib/store.js:63-68] | +| Client refresh | Completed mutation | Re-rendered server components | `router.refresh()` [VERIFIED: components/VoteButton.jsx:16, components/SubmitForm.jsx:27] | + +### 3.3 Pointers to Code Flow Documentation + +Candidates for detailed flow tracing (see `02-code-flows.md`): + +- **Submit-a-link flow** — SubmitForm POST through validation to store insert and redirect +- **Vote flow** — VoteButton PATCH through the dynamic API route to `voteLink` and refresh +- **Board render** — server-side read path from request to sorted HTML list + +### Section 3 Self-Check +- [x] No method bodies longer than 3 lines quoted in this section +- [x] No loops or conditionals explained +- [x] All movements described as conceptual stages, not steps +- [x] Detailed tracing deferred to `02-code-flows.md` + +--- + +## 3b. Frontend → Backend Interaction Map + +Every client-initiated backend call in the system, one row per distinct interaction. Both sides are cited: the frontend trigger and the backend handler. + +| Frontend Source | Trigger Type | Backend Target | Handler / Method | Evidence | +|-----------------|--------------|----------------|------------------|----------| +| `components/SubmitForm.jsx` | fetch (POST, form submit) | `app/api/links/route.js` | `POST` | [VERIFIED: components/SubmitForm.jsx:16-20] [VERIFIED: app/api/links/route.js:11] | +| `components/VoteButton.jsx` | fetch (PATCH, button click) | `app/api/links/[id]/route.js` | `PATCH` | [VERIFIED: components/VoteButton.jsx:14] [VERIFIED: app/api/links/[id]/route.js:4] | + +The client sides of both rows are quoted below; the fetch targets are relative paths, so both calls are same-origin. + +[VERIFIED: components/SubmitForm.jsx:16-20] +```jsx + const res = await fetch('/api/links', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ title, url, tag }), + }) +``` + +[VERIFIED: components/VoteButton.jsx:14] +```jsx + await fetch(`/api/links/${id}`, { method: 'PATCH' }) +``` + +[INFERRED: these are the only frontend-to-backend interactions — `fetch(` appears nowhere else in the repo, and no form uses a native `action=` attribute.] + +--- + +## 4. File/Folder Conventions + +App Router conventions in use — the file's **name and location** determine its role: + +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| `app/layout.jsx` | Root layout; wraps every route's page in shared HTML | [VERIFIED: app/layout.jsx:8-19] | +| `app/**/page.jsx` | Routable page at the folder's URL path (`/`, `/links/:id`, `/submit`) | [VERIFIED: app/page.jsx:5, app/submit/page.jsx:8] | +| `app/api/**/route.js` | HTTP endpoint; exports functions named after HTTP verbs (`GET`, `POST`, `PATCH`) | [VERIFIED: app/api/links/route.js:7, 11] | +| `[id]` folder name | Dynamic URL segment, delivered to the handler via `params` | [VERIFIED: app/links/[id]/page.jsx:6-7] | +| `components/` | Shared React components (mixed server and client) | [VERIFIED: components/LinkCard.jsx:5, components/VoteButton.jsx:6] | +| `lib/` | Non-React server-side modules (the store) | [VERIFIED: lib/store.js:41-68] | + +**Server-vs-client component split.** Exactly two files opt into the client runtime with the `'use client'` directive as their first line [VERIFIED: components/SubmitForm.jsx:1, components/VoteButton.jsx:1]. One of the two directives, quoted: + +[VERIFIED: components/VoteButton.jsx:1] +```jsx +'use client' +``` + +Everything else under `app/` and `components/` has no directive and therefore runs as server components. The board page demonstrates the payoff — it imports the store and reads it in-process, with no HTTP hop: + +[VERIFIED: app/page.jsx:1-6] +```jsx +import { listLinks } from '../lib/store' +import LinkCard from '../components/LinkCard' + +// Server component: reads the store directly, no fetch involved. +export default function BoardPage() { + const links = listLinks() +``` + +[NOT_FOUND: no loading.jsx, error.jsx, or not-found.jsx files anywhere under app/ — searched for all three names. The app defines no custom loading, error, or 404 UI; `notFound()` in the detail page falls through to the framework default.] + +--- + +## 5. External Dependencies + +| Dependency | Purpose | Evidence | +|------------|---------|----------| +| `next` `~14.2.3` | Framework: routing, server components, API routes | [VERIFIED: package.json:11] | +| `react` / `react-dom` `^18.2.0` | Component runtime | [VERIFIED: package.json:12-13] | +| `next/link` | Client-side navigation in layout and cards | [VERIFIED: app/layout.jsx:1, components/LinkCard.jsx:1] | +| `next/navigation` | `useRouter` in client components; `notFound` in the detail page | [VERIFIED: components/SubmitForm.jsx:4, components/VoteButton.jsx:4, app/links/[id]/page.jsx:1] | +| `next/server` | `NextResponse` JSON helpers in API routes | [VERIFIED: app/api/links/route.js:1, app/api/links/[id]/route.js:1] | + +[NOT_FOUND: no database and no ORM — searched "prisma", "postgres", "mysql", "sqlite", "mongo", "drizzle", zero matches. The only persistence is the in-memory Map in lib/store.js.] + +[NOT_FOUND: no external services — every `fetch(` call in the repo targets a relative `/api/...` path; no third-party host, SDK, webhook, or analytics call exists.] + +[NOT_FOUND: no environment configuration — searched "process.env" across the repo, zero matches; no `.env*` files are present (the `.gitignore` merely excludes them).] + +[NOT_FOUND: no lockfile — package-lock.json, yarn.lock, and pnpm-lock.yaml are all absent.] [ASSUMED: npm is the package manager, based on convention only — nothing in the repo pins one.] + +--- + +## 6. Known Issues & Risks + +### 6.1 In-memory store loses data and breaks on serverless + +[VERIFIED: lib/store.js:1-7] +```js +/** + * In-memory link store shared by server components and API routes. + * + * Wart: Module-level Map — every server restart wipes the data, and the + * store is NOT shared across serverless instances. Two lambdas each get + * their own copy, so votes and submissions silently diverge in production. + */ +``` + +All submissions and votes live in a module-level `Map` [VERIFIED: lib/store.js:14]. A restart resets to the three seed links, and on multi-instance or serverless deployments each instance holds an independent copy. [NEEDS_VERIFICATION: how quickly instances diverge in practice depends on the deployment platform's process model — cannot be confirmed from source.] + +### 6.2 POST handler skips validation on `tag` + +`title` and `url` are validated, but `tag` is stored untouched: + +[VERIFIED: app/api/links/route.js:26-29] +```js + // Wart: tag is passed straight through with no validation — whatever + // value (or type) the client sends lands in the store as-is. + const link = createLink({ title: body.title, url: body.url, tag: body.tag }) + return NextResponse.json(link, { status: 201 }) +``` + +### 6.3 `MAX_TITLE_LENGTH` duplicated in two files + +[VERIFIED: lib/store.js:9-10] +```js +// Wart: duplicated in app/api/links/route.js instead of being imported there. +export const MAX_TITLE_LENGTH = 80 +``` + +[VERIFIED: app/api/links/route.js:4-5] +```js +// Wart: duplicated from lib/store.js — the two copies can silently drift. +const MAX_TITLE_LENGTH = 80 +``` + +The store exports the constant, but the API route re-declares it instead of importing. If one copy changes, validation (route) and truncation (store) disagree silently. + +### 6.4 VoteButton has no error handling on its fetch + +[VERIFIED: components/VoteButton.jsx:10-17] +```jsx + // Wart: no error handling — a failed PATCH is silently swallowed and + // the on-screen count goes stale with no feedback to the user. + async function vote() { + setPending(true) + await fetch(`/api/links/${id}`, { method: 'PATCH' }) + setPending(false) + router.refresh() + } +``` + +No `.ok` check, no `try/catch` — contrast with SubmitForm, which surfaces API errors to the user [VERIFIED: components/SubmitForm.jsx:21-25]. + +--- + +## 7. Entry Points Summary + +| Route/Entry | Method | Handler | Middleware | Verified | +|-------------|--------|---------|------------|----------| +| `/` | GET | `BoardPage` in `app/page.jsx` | none | [VERIFIED: app/page.jsx:5] | +| `/links/:id` | GET | `LinkDetailPage` in `app/links/[id]/page.jsx` | none | [VERIFIED: app/links/[id]/page.jsx:6] [VERIFIED: lib/store.js:45] | +| `/submit` | GET | `SubmitPage` in `app/submit/page.jsx` | none | [VERIFIED: app/submit/page.jsx:8] | +| `/api/links` | GET | `GET` in `app/api/links/route.js` | none | [VERIFIED: app/api/links/route.js:7] | +| `/api/links` | POST | `POST` in `app/api/links/route.js` | none | [VERIFIED: app/api/links/route.js:11] | +| `/api/links/:id` | PATCH | `PATCH` in `app/api/links/[id]/route.js` | none | [VERIFIED: app/api/links/[id]/route.js:4] [VERIFIED: lib/store.js:63] | + +[NOT_FOUND: no middleware.js or middleware.ts anywhere in the repo — searched "middleware", zero matches. The "Middleware: none" column above is a verified absence, not an omission.] + +[NOT_FOUND: no pages/ directory and no Pages Router data hooks — searched "getServerSideProps", "getStaticProps", "getInitialProps", zero matches. Routing is App Router only; there is no `pages/api/` either.] + +[NOT_FOUND: no authentication — searched "auth", "session", "jwt", "cookie", "next-auth", zero matches. Every page and API route is open.] + +--- + +## 8. Technology Stack + +| Layer | Technology | Evidence | +|-------|------------|----------| +| Framework | Next.js `~14.2.3`, App Router | [VERIFIED: package.json:11, app/layout.jsx:8] | +| UI runtime | React 18 (server components by default, two client components) | [VERIFIED: package.json:12, components/SubmitForm.jsx:1] | +| API layer | Route handlers under `app/api/` returning `NextResponse.json` | [VERIFIED: app/api/links/route.js:1, 8] | +| Persistence | In-memory `Map` in `lib/store.js` (no database) | [VERIFIED: lib/store.js:14] | +| Build config | `next.config.mjs` with `reactStrictMode` only | [VERIFIED: next.config.mjs:2-4] | +| Styling | None | [NOT_FOUND: no .css files and no styling framework — searched "*.css", "tailwind", "styled", zero matches; `className` attributes exist but nothing styles them] | +| Tests | None | [NOT_FOUND: no *.test.* or *.spec.* files, no jest/vitest config anywhere in the repo] | + +--- + +## Why This Example is GOOD + +1. **Correct system classification.** It resists the reflex to call anything with React a "Frontend SPA". The evidence (server components reading a store in-process, plus an `app/api/` HTTP surface) drives the classification to full-stack App Router app. +2. **Page routes and API routes are documented as distinct surfaces.** Section 3.1 separates server-rendered page entries from JSON API entries instead of flattening them into one "routes" list. +3. **Every claim is cited or admitted.** Positive claims carry `[VERIFIED: path:line]` citations that the verifier resolves against the real tree; absences are explicit `[NOT_FOUND: ...]` entries that name the searches performed. +4. **The frontend-to-backend map cites both sides.** Each row in section 3b points at the client `fetch` call *and* the handler that receives it — a reader can open both files and see the contract. +5. **Honest negatives for things people assume Next.js apps have.** No middleware, no Pages Router (`getServerSideProps` / `pages/api` do not exist here), no database behind the store, no auth, no lockfile — each recorded as a `[NOT_FOUND]` with the search that proved it. +6. **Quotes are exact.** Every fenced block after a citation is a copy-paste of the cited lines, so the verifier's quote-matching phase passes; nothing is paraphrased into fiction. +7. **No invented execution narratives.** Data movement is expressed as tables of stages, not step-by-step arrow traces; detailed tracing is deferred to `02-code-flows.md`. +8. **Known limitation, stated instead of hidden:** citations into the two dynamic-segment files (paths containing `[id]`) cannot be machine-parsed, because the verifier's tag grammar ends a tag at the first `]`. Those citations appear in full for human readers but count as "informal" in the verifier report; every claim about those files is therefore paired with a machine-checkable citation into a caller or into `lib/store.js`. diff --git a/examples/nextjs/linkboard/.gitignore b/examples/nextjs/linkboard/.gitignore new file mode 100644 index 0000000..04569cc --- /dev/null +++ b/examples/nextjs/linkboard/.gitignore @@ -0,0 +1,5 @@ +node_modules/ +.next/ +out/ +.env*.local +npm-debug.log* diff --git a/examples/nextjs/linkboard/app/api/links/[id]/route.js b/examples/nextjs/linkboard/app/api/links/[id]/route.js new file mode 100644 index 0000000..306dd04 --- /dev/null +++ b/examples/nextjs/linkboard/app/api/links/[id]/route.js @@ -0,0 +1,10 @@ +import { NextResponse } from 'next/server' +import { voteLink } from '../../../../lib/store' + +export async function PATCH(request, { params }) { + const link = voteLink(params.id) + if (!link) { + return NextResponse.json({ error: 'link not found' }, { status: 404 }) + } + return NextResponse.json(link) +} diff --git a/examples/nextjs/linkboard/app/api/links/route.js b/examples/nextjs/linkboard/app/api/links/route.js new file mode 100644 index 0000000..c274d6a --- /dev/null +++ b/examples/nextjs/linkboard/app/api/links/route.js @@ -0,0 +1,30 @@ +import { NextResponse } from 'next/server' +import { listLinks, createLink } from '../../../lib/store' + +// Wart: duplicated from lib/store.js — the two copies can silently drift. +const MAX_TITLE_LENGTH = 80 + +export async function GET() { + return NextResponse.json(listLinks()) +} + +export async function POST(request) { + const body = await request.json() + + if (!body.title || body.title.length > MAX_TITLE_LENGTH) { + return NextResponse.json( + { error: `title is required (max ${MAX_TITLE_LENGTH} chars)` }, + { status: 400 }, + ) + } + if (!body.url || !body.url.startsWith('http')) { + return NextResponse.json( + { error: 'url must start with http' }, + { status: 400 }, + ) + } + // Wart: tag is passed straight through with no validation — whatever + // value (or type) the client sends lands in the store as-is. + const link = createLink({ title: body.title, url: body.url, tag: body.tag }) + return NextResponse.json(link, { status: 201 }) +} diff --git a/examples/nextjs/linkboard/app/layout.jsx b/examples/nextjs/linkboard/app/layout.jsx new file mode 100644 index 0000000..99c2106 --- /dev/null +++ b/examples/nextjs/linkboard/app/layout.jsx @@ -0,0 +1,23 @@ +import Link from 'next/link' + +export const metadata = { + title: 'LinkBoard', + description: 'Team link-sharing board', +} + +export default function RootLayout({ children }) { + return ( + + +
+

LinkBoard

+ +
+
{children}
+ + + ) +} diff --git a/examples/nextjs/linkboard/app/links/[id]/page.jsx b/examples/nextjs/linkboard/app/links/[id]/page.jsx new file mode 100644 index 0000000..248aa6c --- /dev/null +++ b/examples/nextjs/linkboard/app/links/[id]/page.jsx @@ -0,0 +1,22 @@ +import { notFound } from 'next/navigation' +import { getLink } from '../../../lib/store' +import VoteButton from '../../../components/VoteButton' + +// Server component for the /links/[id] dynamic route. +export default function LinkDetailPage({ params }) { + const link = getLink(params.id) + if (!link) { + notFound() + } + return ( +
+

{link.title}

+

+ {link.url} +

+

Tag: {link.tag}

+

Shared: {link.createdAt}

+ +
+ ) +} diff --git a/examples/nextjs/linkboard/app/page.jsx b/examples/nextjs/linkboard/app/page.jsx new file mode 100644 index 0000000..f59bb01 --- /dev/null +++ b/examples/nextjs/linkboard/app/page.jsx @@ -0,0 +1,20 @@ +import { listLinks } from '../lib/store' +import LinkCard from '../components/LinkCard' + +// Server component: reads the store directly, no fetch involved. +export default function BoardPage() { + const links = listLinks() + return ( +
+

Top links

+ {links.length === 0 &&

No links yet. Submit the first one!

} +
    + {links.map((link) => ( +
  • + +
    • + ))} +
+
+ ) +} diff --git a/examples/nextjs/linkboard/app/submit/page.jsx b/examples/nextjs/linkboard/app/submit/page.jsx new file mode 100644 index 0000000..ddad6dd --- /dev/null +++ b/examples/nextjs/linkboard/app/submit/page.jsx @@ -0,0 +1,15 @@ +import SubmitForm from '../../components/SubmitForm' + +export const metadata = { + title: 'Submit a link — LinkBoard', +} + +// Server component shell; the interactive form is a client component. +export default function SubmitPage() { + return ( +
+

Submit a link

+ +
+ ) +} diff --git a/examples/nextjs/linkboard/components/LinkCard.jsx b/examples/nextjs/linkboard/components/LinkCard.jsx new file mode 100644 index 0000000..5c60bcb --- /dev/null +++ b/examples/nextjs/linkboard/components/LinkCard.jsx @@ -0,0 +1,15 @@ +import Link from 'next/link' +import VoteButton from './VoteButton' + +// Server component: pure rendering, no state and no event handlers. +export default function LinkCard({ link }) { + const host = new URL(link.url).host + return ( +
+ {link.title} + ({host}) + {link.tag} + +
+ ) +} diff --git a/examples/nextjs/linkboard/components/SubmitForm.jsx b/examples/nextjs/linkboard/components/SubmitForm.jsx new file mode 100644 index 0000000..f233fef --- /dev/null +++ b/examples/nextjs/linkboard/components/SubmitForm.jsx @@ -0,0 +1,48 @@ +'use client' + +import { useState } from 'react' +import { useRouter } from 'next/navigation' + +export default function SubmitForm() { + const router = useRouter() + const [title, setTitle] = useState('') + const [url, setUrl] = useState('') + const [tag, setTag] = useState('') + const [error, setError] = useState(null) + + async function handleSubmit(event) { + event.preventDefault() + setError(null) + const res = await fetch('/api/links', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ title, url, tag }), + }) + if (!res.ok) { + const body = await res.json() + setError(body.error ?? 'Something went wrong') + return + } + router.push('/') + router.refresh() + } + + return ( +
+ + + + {error &&

{error}

} + +
+ ) +} diff --git a/examples/nextjs/linkboard/components/VoteButton.jsx b/examples/nextjs/linkboard/components/VoteButton.jsx new file mode 100644 index 0000000..35f5efa --- /dev/null +++ b/examples/nextjs/linkboard/components/VoteButton.jsx @@ -0,0 +1,24 @@ +'use client' + +import { useState } from 'react' +import { useRouter } from 'next/navigation' + +export default function VoteButton({ id, votes }) { + const router = useRouter() + const [pending, setPending] = useState(false) + + // Wart: no error handling — a failed PATCH is silently swallowed and + // the on-screen count goes stale with no feedback to the user. + async function vote() { + setPending(true) + await fetch(`/api/links/${id}`, { method: 'PATCH' }) + setPending(false) + router.refresh() + } + + return ( + + ) +} diff --git a/examples/nextjs/linkboard/lib/store.js b/examples/nextjs/linkboard/lib/store.js new file mode 100644 index 0000000..5b584f7 --- /dev/null +++ b/examples/nextjs/linkboard/lib/store.js @@ -0,0 +1,68 @@ +/** + * In-memory link store shared by server components and API routes. + * + * Wart: Module-level Map — every server restart wipes the data, and the + * store is NOT shared across serverless instances. Two lambdas each get + * their own copy, so votes and submissions silently diverge in production. + */ + +// Wart: duplicated in app/api/links/route.js instead of being imported there. +export const MAX_TITLE_LENGTH = 80 + +let nextId = 4 + +const links = new Map([ + ['1', { + id: '1', + title: 'Next.js App Router docs', + url: 'https://nextjs.org/docs/app', + tag: 'docs', + votes: 5, + createdAt: '2026-07-01T09:00:00Z', + }], + ['2', { + id: '2', + title: 'React Server Components explainer', + url: 'https://react.dev/reference/rsc/server-components', + tag: 'reading', + votes: 3, + createdAt: '2026-07-02T14:30:00Z', + }], + ['3', { + id: '3', + title: 'Team retro board', + url: 'https://example.com/retro', + tag: 'internal', + votes: 1, + createdAt: '2026-07-03T08:15:00Z', + }], +]) + +export function listLinks() { + return [...links.values()].sort((a, b) => b.votes - a.votes) +} + +export function getLink(id) { + return links.get(id) ?? null +} + +export function createLink({ title, url, tag }) { + const id = String(nextId++) + const link = { + id, + title: title.slice(0, MAX_TITLE_LENGTH), + url, + tag, + votes: 0, + createdAt: new Date().toISOString(), + } + links.set(id, link) + return link +} + +export function voteLink(id) { + const link = links.get(id) + if (!link) return null + link.votes += 1 + return link +} diff --git a/examples/nextjs/linkboard/next.config.mjs b/examples/nextjs/linkboard/next.config.mjs new file mode 100644 index 0000000..94be31c --- /dev/null +++ b/examples/nextjs/linkboard/next.config.mjs @@ -0,0 +1,6 @@ +/** @type {import('next').NextConfig} */ +const nextConfig = { + reactStrictMode: true, +} + +export default nextConfig diff --git a/examples/nextjs/linkboard/package.json b/examples/nextjs/linkboard/package.json new file mode 100644 index 0000000..7956c0a --- /dev/null +++ b/examples/nextjs/linkboard/package.json @@ -0,0 +1,15 @@ +{ + "name": "linkboard", + "version": "0.1.0", + "private": true, + "scripts": { + "dev": "next dev", + "build": "next build", + "start": "next start" + }, + "dependencies": { + "next": "~14.2.3", + "react": "^18.2.0", + "react-dom": "^18.2.0" + } +} diff --git a/examples/packages/requests/bad-architecture-doc-example.md b/examples/packages/requests/bad-architecture-doc-example.md index 776cab2..b954854 100644 --- a/examples/packages/requests/bad-architecture-doc-example.md +++ b/examples/packages/requests/bad-architecture-doc-example.md @@ -11,6 +11,11 @@ | Evidence | It's the most popular Python HTTP library | | Confidence | `[HIGH]` | +> **❌ PROBLEMS:** +> - `[HIGH]` is not a methodology tag — only `[VERIFIED]`, `[INFERRED]`, `[NOT_FOUND]`, `[ASSUMED]`, `[NEEDS_VERIFICATION]` exist +> - "It's the most popular Python HTTP library" is popularity, not evidence — real classification evidence is in the code, e.g. the public-surface re-export at `requests/__init__.py:164` +> - Missing Category and Overlay Loaded rows required by the 01 output format + ## 2. Component Map ### Core Components @@ -39,6 +44,12 @@ Response is parsed and returned Connection is returned to pool for reuse ``` +> **❌ PROBLEMS:** +> - No file paths or citations for any "layer" — every bullet needs a location and a verification tag +> - Arrow diagram traces execution step-by-step — that belongs in 02-code-flows, not the architecture overview +> - "Connection is returned to pool for reuse" is wrong for the flow shown: `requests.get()` runs the session in a with-block that closes it on return (`requests/api.py:58-59`), discarding the pool — reuse only happens when the caller holds a `Session` +> - "Utility Layer - Helper functions and authentication" conflates two unrelated modules: helpers live in `requests/utils.py:1-7`, auth handlers in `requests/auth.py:1-6` + ## 3. Key Data Flows ### GET Request Flow @@ -78,6 +89,13 @@ The library supports multiple authentication methods: 4. Kerberos - Enterprise SSO integration 5. NTLM - Windows domain authentication +> **❌ PROBLEMS:** +> - `[sessions.py:~400]`-style approximations are unverifiable — and wrong: `Session.__enter__` is at `requests/sessions.py:451`, `Session.send` at `requests/sessions.py:673`, `HTTPAdapter.send` at `requests/adapters.py:590` (the `~500` guess for `Session.request` at `requests/sessions.py:500` is luck, not verification) +> - Bare filenames like `[api.py:62]` resolve against nothing — citations must be relative to the source root (`requests/api.py:62`) +> - Step-by-step GET tracing with internal bullets ("Merges session settings", "Handles redirects") belongs in 02-code-flows +> - OAuth, Kerberos, and NTLM are hallucinated: `requests/auth.py` ships only `AuthBase` (:69), `HTTPBasicAuth` (:76), `HTTPProxyAuth` (:99), `HTTPDigestAuth` (:107); the sole "OAuth" occurrence in the whole package is a comment at `requests/models.py:374` about third-party auth hooks +> - "MD5 hashing" describes algorithm internals — too detailed for an architecture overview + ## 4. External Dependencies - **urllib3** - Connection pooling and HTTP handling @@ -87,6 +105,12 @@ The library supports multiple authentication methods: - **cryptography** - TLS and encryption support - **pyOpenSSL** - Additional SSL features +> **❌ PROBLEMS:** +> - No Evidence column, no citations — a dependency list must point at the import that proves it +> - chardet is listed as required, but `requests/compat.py:33-39` tries chardet *or* charset_normalizer, and both are optional (either satisfies detection) +> - cryptography and pyOpenSSL are not dependencies — they are a conditional fallback imported only when stdlib `ssl` lacks SNI (`requests/__init__.py:130-138`) +> - idna is real but uncited — the actual use is a lazy import during URL preparation at `requests/models.py:400-401` + ## 5. Performance Characteristics - Connection pooling reduces latency by 40-60% @@ -95,10 +119,20 @@ The library supports multiple authentication methods: - Memory-efficient streaming for large files - Async support via requests-async extension +> **❌ PROBLEMS:** +> - "40-60%" and "100+ requests" are fabricated numbers with no source — performance claims need `[NEEDS_VERIFICATION]` at minimum +> - "Automatic retry logic with exponential backoff" is false: `DEFAULT_RETRIES = 0` at `requests/adapters.py:72`, and the `HTTPAdapter` docstring says "By default, Requests does not retry failed connections" (`requests/adapters.py:153-157`) — retries are opt-in +> - "Async support" is false: zero `async`/`await` anywhere in `requests/*.py`; requests-async is a third-party package, not this library's architecture +> - Connection pooling is implemented by urllib3's `PoolManager` (`requests/adapters.py:26`), not by requests — attributing it here misleads readers about where the behavior lives + ## 6. Architecture Summary The requests library follows a clean layered architecture with excellent separation of concerns. The API layer provides a simple interface, while the session and adapter layers handle complexity. This design makes it easy to extend and customize behavior. +> **❌ PROBLEMS:** +> - Pure editorializing — "clean", "excellent", "easy to extend" are opinions with no citations and no falsifiable content +> - Says nothing a reader could check: no boundaries, no `[NOT_FOUND]` scoping, no pointer to the actual extension point (`Session.mount` at `requests/sessions.py:799`) + --- **Why this is BAD:** diff --git a/examples/packages/requests/good-architecture-doc-example.md b/examples/packages/requests/good-architecture-doc-example.md index d63fa5e..3d917f9 100644 --- a/examples/packages/requests/good-architecture-doc-example.md +++ b/examples/packages/requests/good-architecture-doc-example.md @@ -1,68 +1,158 @@ # Architecture Overview: Requests Library > **This is an example of GOOD documentation following the 01-architecture-overview methodology.** +> +> Every citation below resolves against the pinned copy of the `requests` source +> vendored at `examples/packages/requests/source/` (see `source/VENDORED.md` for +> the upstream pin), so this example is self-verifiable. ## Metadata | Field | Value | |-------|-------| -| Source Commit | `v2.32.3` | -| Generated | `2025-01-15` | -| Primary Language | Python | +| Repository | `agent-system-mapper` | +| Path | `examples/packages/requests/source/` | +| Upstream | `psf/requests 2.32.5, commit 7029833` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | +| Verification Status | `Verified` | -## 1. System Classification +Verify with: + +```bash +python3 verify.py examples/packages/requests/good-architecture-doc-example.md --repo-root examples/packages/requests/source +``` + +## Verification Summary +- VERIFIED: 70 tags (125 file:line citations, 100% resolving; 1 structural claim without a line citation; 6 quoted blocks matching at 100% similarity) +- INFERRED: 1 claim +- NOT_FOUND: 9 items (each with the search documented) +- ASSUMED: 0 items +- NEEDS_VERIFICATION: 0 items + +--- + +## 0. System Classification | Field | Value | |-------|-------| | Category | Traditional Code | | Type | Library/Package | -| Evidence | Exports functions/classes via `__init__.py`; no routes, no CLI entry point | +| Evidence | The package re-exports its whole public surface from `__init__.py` [VERIFIED: requests/__init__.py:164, 177, 178, 179]; no web routes or console_scripts exist in the vendored tree [NOT_FOUND: searched "route", "argparse", "click" in requests/ — zero matches] | | Overlay Loaded | No | -| Confidence | `[VERIFIED: src/requests/__init__.py:1-100]` | +| Confidence | `[VERIFIED]` | -## 2. Component Map +The public functional API is re-exported at import time [VERIFIED: requests/__init__.py:164] -### Core Components +```python +from .api import delete, get, head, options, patch, post, put, request +``` + +--- + +## 1. System Purpose + +Requests is a synchronous HTTP client library for Python — "Python HTTP for +Humans" per its own metadata. It wraps urllib3 behind a small, human-friendly +API: module-level verb functions for one-shot calls, and a `Session` object for +cookie persistence, configuration, and connection pooling across calls +[VERIFIED: requests/__init__.py:10, requests/sessions.py:5-6]. + +Package identity is defined in one place [VERIFIED: requests/__version__.py:5-8] + +```python +__title__ = "requests" +__description__ = "Python HTTP for Humans." +__url__ = "https://requests.readthedocs.io" +__version__ = "2.32.5" +``` + +--- + +## 2. Component Map | Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| Public API | `src/requests/api.py` | Module-level HTTP methods (get, post, put, etc.) | [VERIFIED: api.py:14-157] | -| Session | `src/requests/sessions.py` | Connection persistence, settings management | [VERIFIED: sessions.py:1-7] | -| Request/Response | `src/requests/models.py` | Data structures for HTTP requests and responses | [VERIFIED: models.py, 35510 bytes] | -| HTTPAdapter | `src/requests/adapters.py` | Transport layer, urllib3 integration | [VERIFIED: adapters.py, 26285 bytes] | -| Auth Handlers | `src/requests/auth.py` | HTTPBasicAuth, HTTPDigestAuth | [VERIFIED: auth.py, 10186 bytes] | -| Cookies | `src/requests/cookies.py` | RequestsCookieJar, cookie handling | [VERIFIED: cookies.py, 18590 bytes] | -| Exceptions | `src/requests/exceptions.py` | Error classes | [VERIFIED: exceptions.py, 4260 bytes] | -| Utilities | `src/requests/utils.py` | Helper functions | [VERIFIED: utils.py, 33213 bytes] | +| Module-level API | `requests/api.py` | Verb functions (`request`, `get`, `options`, `head`, `post`, `put`, `patch`, `delete`) that delegate to a throwaway `Session` | [VERIFIED: requests/api.py:14, 62, 76, 88, 103, 118, 133, 148] | +| `Session` | `requests/sessions.py` | Cookie persistence, setting merges, redirect handling, adapter mounting | [VERIFIED: requests/sessions.py:356] | +| `SessionRedirectMixin` | `requests/sessions.py` | Redirect resolution, auth/proxy rebuilding across hops | [VERIFIED: requests/sessions.py:106, 282, 302] | +| `Request` / `PreparedRequest` / `Response` | `requests/models.py` | The three primary data objects of the library | [VERIFIED: requests/models.py:230, 313, 640] | +| `BaseAdapter` / `HTTPAdapter` | `requests/adapters.py` | Transport layer; bridges `PreparedRequest` to urllib3 and builds `Response` | [VERIFIED: requests/adapters.py:113, 143, 336] | +| Auth handlers | `requests/auth.py` | `AuthBase`, `HTTPBasicAuth`, `HTTPProxyAuth`, `HTTPDigestAuth` | [VERIFIED: requests/auth.py:69, 76, 99, 107] | +| Cookie machinery | `requests/cookies.py` | `RequestsCookieJar` plus jar/dict conversion and merge helpers | [VERIFIED: requests/cookies.py:176, 521, 542] | +| Exception tree | `requests/exceptions.py` | `RequestException(IOError)` root with specific subclasses | [VERIFIED: requests/exceptions.py:12] | +| Hooks system | `requests/hooks.py` | Single `response` hook event plus dispatcher | [VERIFIED: requests/hooks.py:12, 15, 22] | +| Status codes | `requests/status_codes.py` | `codes` LookupDict mapping names to numeric statuses | [VERIFIED: requests/status_codes.py:106] | +| Data structures | `requests/structures.py` | `CaseInsensitiveDict`, `LookupDict` | [VERIFIED: requests/structures.py:13, 83] | +| Utilities | `requests/utils.py` | Header/proxy/encoding helpers, "also useful for external consumption" | [VERIFIED: requests/utils.py:1-7] | +| Internal utilities | `requests/_internal_utils.py` | Native-string coercion and header validators, internal-only | [VERIFIED: requests/_internal_utils.py:1-7] | +| Compat shims | `requests/compat.py` | Legacy Python-2/3 layer and character-detection resolution | [VERIFIED: requests/compat.py:1-8] | +| CA bundle indirection | `requests/certs.py` | Re-exports certifi's `where()` as the default CA bundle source | [VERIFIED: requests/certs.py:14] | +| Diagnostics | `requests/help.py` | Bug-report environment dump (`info()`, `main()`) | [VERIFIED: requests/help.py:69, 128] | +| Legacy namespace | `requests/packages.py` | sys.modules aliasing so `requests.packages.urllib3` still resolves | [VERIFIED: requests/packages.py:8-9] | +| Version metadata | `requests/__version__.py` | Title, version, license constants | [VERIFIED: requests/__version__.py:5, 8, 12] | + +The central class documents its own responsibilities [VERIFIED: requests/sessions.py:356-359] + +```python +class Session(SessionRedirectMixin): + """A Requests session. + + Provides cookie persistence, connection-pooling, and configuration. +``` -[NOT_FOUND: No CLI entry point (`__main__.py` or console_scripts)] -[NOT_FOUND: No web routes or server components] +[NOT_FOUND: no subpackages — `requests/` is a flat set of 18 modules, no nested directories] -## 3. Execution Surfaces & High-Level Data Movement +--- + +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) + +For a library, the execution surfaces are its **public API entry points** — the +importable names through which caller code enters the package. -### 3.1 Primary Execution Surfaces +### 3.1 Primary Execution Surfaces (Public API Entry Points) | Entry Surface | Type | Primary Components Involved | Evidence | -|--------------|------|-----------------------------|----------| -| `requests.get(url)` | Library API | api.request → Session → HTTPAdapter | [VERIFIED: api.py:62-73] | -| `requests.post(url)` | Library API | api.request → Session → HTTPAdapter | [VERIFIED: api.py:103-115] | -| `requests.request(method, url)` | Library API | Session.request → PreparedRequest → Response | [VERIFIED: api.py:14-59] | -| `Session()` context manager | Library API | Session.__enter__, Session.__exit__ | [VERIFIED: sessions.py, class Session] | - -### 3.2 High-Level Data Movement - -| Stage | Input | Output | Components | -|-------|-------|--------|------------| -| API call | URL, params, kwargs | Response object | api.py functions | -| Session management | Request kwargs | Merged settings | Session class | -| Request preparation | Raw params | PreparedRequest | Request, PreparedRequest in models.py | -| Transport | PreparedRequest | urllib3 Response | HTTPAdapter | -| Response build | urllib3 Response | requests.Response | Response in models.py | +|---------------|------|-----------------------------|----------| +| `requests.request(method, url, **kwargs)` | Library API | api.request → throwaway `Session` → `Session.request` | [VERIFIED: requests/api.py:14] | +| `requests.get(url, params=None)` | Library API | Delegates to `api.request("get", ...)` | [VERIFIED: requests/api.py:62] | +| `requests.options(url)` / `requests.head(url)` | Library API | Delegate to `api.request` (head disables redirects by default) | [VERIFIED: requests/api.py:76, 88, 99] | +| `requests.post(url, data=None, json=None)` | Library API | Delegates with body kwargs | [VERIFIED: requests/api.py:103] | +| `requests.put(url)` / `requests.patch(url)` / `requests.delete(url)` | Library API | Delegate to `api.request` | [VERIFIED: requests/api.py:118, 133, 148] | +| `Session()` — direct use or context manager | Library API | `Session.__init__` mounts default adapters; `__enter__`/`__exit__` manage close | [VERIFIED: requests/sessions.py:390, 451-455] | +| `Session.request/get/options/head/post/put/patch/delete/send` | Library API | Full per-session verb surface plus low-level `send` | [VERIFIED: requests/sessions.py:500, 593, 604, 615, 626, 639, 651, 663, 673] | +| `Response` consumption — `iter_content`, `content`, `text`, `json`, `raise_for_status` | Library API | Response object accessors | [VERIFIED: requests/models.py:799, 891, 910, 947, 999] | +| `HTTPAdapter(max_retries=...)` + `Session.mount(prefix, adapter)` | Extension point | Custom transport configuration | [VERIFIED: requests/adapters.py:178, requests/sessions.py:799] | +| `python -m requests.help` | Diagnostic CLI | `info()` JSON dump via `main()` | [VERIFIED: requests/help.py:128, 133-134] | +| `python -m requests.certs` | Diagnostic CLI | Prints the certifi CA bundle path | [VERIFIED: requests/certs.py:16-17] | + +[NOT_FOUND: no `__main__.py` and no console_scripts — the only runnable modules are the two diagnostics above] + +### 3.2 High-Level Data Movement (Non-Procedural) + +| Stage | Input | Output | Participating Components | Evidence | +|-------|-------|--------|--------------------------|----------| +| Entry | method, URL, kwargs | Delegated session call | `api.request` | [VERIFIED: requests/api.py:58-59] | +| Settings merge | Request kwargs + Session defaults | Merged settings/hooks | `merge_setting`, `merge_hooks` | [VERIFIED: requests/sessions.py:61, 91] | +| Preparation | `Request` | `PreparedRequest` | `Session.prepare_request`, `PreparedRequest` | [VERIFIED: requests/sessions.py:457, requests/models.py:313] | +| Transport | `PreparedRequest` | urllib3 response | `Session.send` → `HTTPAdapter.send` | [VERIFIED: requests/sessions.py:673, requests/adapters.py:590-591] | +| Response build | urllib3 response | `requests.Response` | `HTTPAdapter.build_response` | [VERIFIED: requests/adapters.py:336] | +| Hook dispatch | `Response` | Possibly-replaced `Response` | `dispatch_hook` | [VERIFIED: requests/hooks.py:22] | + +The module-level surface hands every call to a short-lived session +[VERIFIED: requests/api.py:58-59] + +```python + with sessions.Session() as session: + return session.request(method=method, url=url, **kwargs) +``` ### 3.3 Pointers to Code Flow Documentation -- **Simple GET request** - see 02-code-flows.md -- **Session-based requests** - see 02-code-flows.md -- **Authentication flow** - see 02-code-flows.md +Candidates for detailed flow tracing (see 02-code-flows.md): + +- **Module-level GET** — `requests.get` through session creation, preparation, transport, and teardown +- **Session-persistent request with redirects** — `Session.request` → `resolve_redirects` +- **Digest auth challenge/response** — `HTTPDigestAuth` handler lifecycle ### Section 3 Self-Check - [x] No method bodies longer than 3 lines quoted @@ -70,60 +160,105 @@ - [x] All movements as conceptual stages - [x] Defers to 02-code-flows.md -## 4. External Dependencies +--- -| Dependency | Purpose | Evidence | -|------------|---------|----------| -| urllib3 | HTTP connection handling | [VERIFIED: sessions.py:15 `from .adapters import HTTPAdapter`; adapters.py imports urllib3] | -| charset_normalizer OR chardet | Character encoding | [VERIFIED: __init__.py:47-52, try/except import] | -| certifi | CA certificates | [VERIFIED: certs.py imports certifi] | -| idna | International domain names | [VERIFIED: models.py likely uses for URL encoding] | +## Boundaries & Non-Responsibilities -[NOT_FOUND: cryptography, pyOpenSSL - not in core dependencies] +Explicitly **NOT** in this library: -## 5. Boundaries & Non-Responsibilities +- **Async/await API** [NOT_FOUND: searched "async def" and "await " across requests/*.py — zero matches; the API is fully synchronous] +- **WebSocket support** [NOT_FOUND: case-insensitive search for "websocket" across requests/*.py — zero matches] +- **OAuth, Kerberos, or NTLM authentication** — shipped handlers are Basic, Proxy-Basic, and Digest only [VERIFIED: requests/auth.py:69, 76, 99, 107] [NOT_FOUND: case-insensitive search for "oauth", "kerberos", "ntlm" — the only hit is a comment at requests/models.py line 374 noting that auth *hooks* allow third-party schemes] +- **Automatic retries** — retries default to zero and are opt-in per adapter [VERIFIED: requests/adapters.py:153-157] +- **HTTP/2** [NOT_FOUND: searched "http2" and "HTTP/2" across requests/*.py — zero matches; transport is urllib3's HTTP/1.1 stack] +- **Connection-pool implementation** — pooling is delegated to urllib3's `PoolManager`, not implemented here [VERIFIED: requests/adapters.py:26] +- **CA certificate store** — trust roots come from certifi, not from this package [VERIFIED: requests/certs.py:14] -Explicitly **NOT** in this library: -- Async/await support [VERIFIED: no async keywords in source] -- WebSocket support [NOT_FOUND: no websocket imports] -- OAuth implementation [NOT_FOUND: no oauth in auth.py] -- HTTP/2 support [NOT_FOUND: urllib3 handles, not requests] +Transport defaults make the no-retry boundary concrete +[VERIFIED: requests/adapters.py:70-73] -## 6. Entry Points Summary +```python +DEFAULT_POOLBLOCK = False +DEFAULT_POOLSIZE = 10 +DEFAULT_RETRIES = 0 +DEFAULT_POOL_TIMEOUT = None +``` -| Entry Type | Count | Locations | -|------------|-------|-----------| -| Public API functions | 7 | api.py: request, get, post, put, patch, delete, head, options | -| Classes | 4 | Session, Request, PreparedRequest, Response | -| CLI Commands | 0 | [NOT_FOUND] | -| Web Routes | 0 | [NOT_FOUND] | +--- -## 7. Technology Stack Summary +## 4. File/Folder Conventions -| Layer | Technology | Evidence | -|-------|------------|----------| -| Language | Python 3.8+ | [VERIFIED: pyproject.toml or setup.py] | -| HTTP Backend | urllib3 | [VERIFIED: adapters.py imports] | -| Encoding | charset_normalizer | [VERIFIED: __init__.py:47] | -| Certificates | certifi | [VERIFIED: certs.py:5] | +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| Flat single package | All 18 modules sit directly in `requests/` with no nested directories | [VERIFIED: directory listing of requests/ shows 18 .py files and no subdirectories] | +| `requests.` docstring headers | Each module opens with a docstring naming itself and stating its role | [VERIFIED: requests/utils.py:1-7, requests/sessions.py:1-7] | +| `_`-prefix marks internals | `_internal_utils.py` documents itself as consumed internally only | [VERIFIED: requests/_internal_utils.py:1-7] | +| `compat.py` as legacy shim | Self-described as remaining "for backwards compatibility until the next major version" | [VERIFIED: requests/compat.py:5-7] | +| `packages.py` as namespace alias | Keeps `requests.packages.urllib3` importable via sys.modules aliasing | [VERIFIED: requests/packages.py:8-14] | +| `__version__.py` as metadata home | Single source for title/version/license constants | [VERIFIED: requests/__version__.py:5-14] | + +--- + +## 5. External Dependencies -## 8. Verification Summary +| Dependency | Required? | Purpose | Evidence | +|------------|-----------|---------|----------| +| urllib3 | Required | Transport: `PoolManager`, `Retry`, `Timeout`, proxy support | [VERIFIED: requests/adapters.py:26-29] | +| certifi | Required | Default CA bundle (`where()`), consumed as `DEFAULT_CA_BUNDLE_PATH` | [VERIFIED: requests/certs.py:14, requests/utils.py:64] | +| idna | Required | IDNA host encoding during URL preparation (lazy import) | [VERIFIED: requests/models.py:400-401] | +| charset_normalizer / chardet | Optional (either) | Response character-set detection; first importable of the two wins | [VERIFIED: requests/compat.py:33-39] | +| simplejson | Optional | Used as the `json` implementation when importable | [VERIFIED: requests/compat.py:58-64] | +| pyOpenSSL + cryptography | Optional fallback | Injected into urllib3 only when stdlib `ssl` lacks SNI support | [VERIFIED: requests/__init__.py:130-138] | -| Status | Count | -|--------|-------| -| VERIFIED | 18 | -| NOT_FOUND | 6 | -| INFERRED | 0 | -| ASSUMED | 0 | +Character detection is genuinely either/or, not a hard chardet dependency +[VERIFIED: requests/compat.py:33-39] + +```python + for lib in ("chardet", "charset_normalizer"): + if chardet is None: + try: + chardet = importlib.import_module(lib) + except ImportError: + pass + return chardet +``` + +[NOT_FOUND: no dependency manifest in the vendored copy — packaging metadata (pyproject/setup) was pruned per source/VENDORED.md, so version pins cannot be cited here] + +--- + +## 6. Known Issues & Risks + +| Risk | Location | Notes | +|------|----------|-------| +| Module-level API builds and closes a fresh `Session` per call | `requests/api.py:55-59` | [VERIFIED: requests/api.py:58-59] the with-block closes the session on return; [INFERRED] therefore module-level calls get no cross-call connection reuse — callers wanting pooling must hold a `Session` themselves | +| `verify=False` disables TLS verification entirely | `requests/sessions.py:416-424` | [VERIFIED: requests/sessions.py:419-422] the attribute docstring itself warns this makes applications "vulnerable to man-in-the-middle (MitM) attacks" | +| Deprecated `session()` factory still exported | `requests/sessions.py:819-831` | [VERIFIED: requests/sessions.py:823, requests/__init__.py:178] deprecated since 1.0.0 yet still part of the top-level import surface | +| Legacy sys.modules aliasing | `requests/packages.py:5-6` | [VERIFIED: requests/packages.py:5-6] the module's own comment concedes this exists "for backwards compatibility reasons" | +| Non-string basic-auth credentials only warn | `requests/auth.py:35-53` | [VERIFIED: requests/auth.py:35-43] coercion with `DeprecationWarning`, removal deferred to 3.0.0 per in-code comment | + +--- + +## 8. Technology Stack Summary + +| Layer | Technology | Evidence | +|-------|------------|----------| +| Language | Python 3 (`is_py2`/`is_py3` kept only as legacy compat constants) | [VERIFIED: requests/compat.py:52-55] | +| Supported Python range | Not citable from the vendored copy | [NOT_FOUND: python_requires lives in packaging metadata, which was pruned from source/ per VENDORED.md] | +| HTTP transport | urllib3 | [VERIFIED: requests/adapters.py:26-29] | +| TLS trust | certifi via `requests/certs.py` | [VERIFIED: requests/certs.py:14] | +| Character detection | charset_normalizer or chardet (optional) | [VERIFIED: requests/compat.py:42] | +| IDN handling | idna | [VERIFIED: requests/models.py:401] | +| License | Apache-2.0 | [VERIFIED: requests/__version__.py:12] | --- -**Why this is GOOD:** +## Why This Example is GOOD -- **Every claim verified** - `[VERIFIED: file:line]` or `[NOT_FOUND]` for everything -- **Tables not arrows** - Execution surfaces use table format, not step-by-step diagrams -- **Describes WHAT not HOW** - "Session management" stage, not implementation details -- **Explicit boundaries** - States what's NOT in the library to prevent hallucination -- **Accurate dependencies** - Only lists actual imports, marks cryptography as NOT_FOUND -- **Defers to 02-code-flows** - Doesn't trace execution, just identifies surfaces -- **Self-check completed** - Validates Section 3 rules before submission +- **Every citation resolves** — all `[VERIFIED: path:line]` tags use paths relative to the vendored source root (`requests/api.py:14`, not bare `api.py:14`), so the Verify-with command exits 0. +- **No hedges or disjunctions inside tags** — guesses like `[VERIFIED: pyproject.toml or setup.py]` and `[VERIFIED: models.py likely uses ...]` became exact citations or honest `[NOT_FOUND]` entries with the search documented. +- **No byte counts as evidence** — a file's size proves it exists, not what it does; every component row now cites the line that carries the claim. +- **Quotes are exact copy-paste** — each fenced block matches the cited slice of the file, so the verifier's phase-2 quote check passes at 100% similarity. +- **Explicit boundaries** — the Boundaries & Non-Responsibilities section (required for packages) states what the library does NOT do, each negative claim backed by a documented search or a citation to the default that proves it. +- **Tables, not arrows** — Section 3 lists entry surfaces and conceptual stages, deferring all step-by-step tracing to 02-code-flows.md. +- **Accurate self-accounting** — the Verification Summary counts match the verifier's own tag census for this document. diff --git a/examples/packages/requests/source/LICENSE b/examples/packages/requests/source/LICENSE new file mode 100644 index 0000000..67db858 --- /dev/null +++ b/examples/packages/requests/source/LICENSE @@ -0,0 +1,175 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. diff --git a/examples/packages/requests/source/VENDORED.md b/examples/packages/requests/source/VENDORED.md new file mode 100644 index 0000000..725638c --- /dev/null +++ b/examples/packages/requests/source/VENDORED.md @@ -0,0 +1,16 @@ +# Vendored source: requests + +Slim, pinned copy of the `requests` package source so the good/bad +architecture examples in the parent directory have resolvable citations. + +| Field | Value | +|-------|-------| +| Upstream | https://github.com/psf/requests | +| Version | 2.32.5 | +| Upstream commit | 7029833 | +| Vendored | 2026-08-03 | +| Contents | `src/requests/*.py` only (as `requests/`), plus LICENSE | +| Pruned | tests, docs, packaging metadata, ext assets | + +Do not edit these files — refresh by re-copying from upstream and +re-running the verifier on the example docs. diff --git a/examples/packages/requests/source/requests/__init__.py b/examples/packages/requests/source/requests/__init__.py new file mode 100644 index 0000000..051cda1 --- /dev/null +++ b/examples/packages/requests/source/requests/__init__.py @@ -0,0 +1,184 @@ +# __ +# /__) _ _ _ _ _/ _ +# / ( (- (/ (/ (- _) / _) +# / + +""" +Requests HTTP Library +~~~~~~~~~~~~~~~~~~~~~ + +Requests is an HTTP library, written in Python, for human beings. +Basic GET usage: + + >>> import requests + >>> r = requests.get('https://www.python.org') + >>> r.status_code + 200 + >>> b'Python is a programming language' in r.content + True + +... or POST: + + >>> payload = dict(key1='value1', key2='value2') + >>> r = requests.post('https://httpbin.org/post', data=payload) + >>> print(r.text) + { + ... + "form": { + "key1": "value1", + "key2": "value2" + }, + ... + } + +The other HTTP methods are supported - see `requests.api`. Full documentation +is at . + +:copyright: (c) 2017 by Kenneth Reitz. +:license: Apache 2.0, see LICENSE for more details. +""" + +import warnings + +import urllib3 + +from .exceptions import RequestsDependencyWarning + +try: + from charset_normalizer import __version__ as charset_normalizer_version +except ImportError: + charset_normalizer_version = None + +try: + from chardet import __version__ as chardet_version +except ImportError: + chardet_version = None + + +def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version): + urllib3_version = urllib3_version.split(".") + assert urllib3_version != ["dev"] # Verify urllib3 isn't installed from git. + + # Sometimes, urllib3 only reports its version as 16.1. + if len(urllib3_version) == 2: + urllib3_version.append("0") + + # Check urllib3 for compatibility. + major, minor, patch = urllib3_version # noqa: F811 + major, minor, patch = int(major), int(minor), int(patch) + # urllib3 >= 1.21.1 + assert major >= 1 + if major == 1: + assert minor >= 21 + + # Check charset_normalizer for compatibility. + if chardet_version: + major, minor, patch = chardet_version.split(".")[:3] + major, minor, patch = int(major), int(minor), int(patch) + # chardet_version >= 3.0.2, < 6.0.0 + assert (3, 0, 2) <= (major, minor, patch) < (6, 0, 0) + elif charset_normalizer_version: + major, minor, patch = charset_normalizer_version.split(".")[:3] + major, minor, patch = int(major), int(minor), int(patch) + # charset_normalizer >= 2.0.0 < 4.0.0 + assert (2, 0, 0) <= (major, minor, patch) < (4, 0, 0) + else: + warnings.warn( + "Unable to find acceptable character detection dependency " + "(chardet or charset_normalizer).", + RequestsDependencyWarning, + ) + + +def _check_cryptography(cryptography_version): + # cryptography < 1.3.4 + try: + cryptography_version = list(map(int, cryptography_version.split("."))) + except ValueError: + return + + if cryptography_version < [1, 3, 4]: + warning = "Old version of cryptography ({}) may cause slowdown.".format( + cryptography_version + ) + warnings.warn(warning, RequestsDependencyWarning) + + +# Check imported dependencies for compatibility. +try: + check_compatibility( + urllib3.__version__, chardet_version, charset_normalizer_version + ) +except (AssertionError, ValueError): + warnings.warn( + "urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported " + "version!".format( + urllib3.__version__, chardet_version, charset_normalizer_version + ), + RequestsDependencyWarning, + ) + +# Attempt to enable urllib3's fallback for SNI support +# if the standard library doesn't support SNI or the +# 'ssl' library isn't available. +try: + try: + import ssl + except ImportError: + ssl = None + + if not getattr(ssl, "HAS_SNI", False): + from urllib3.contrib import pyopenssl + + pyopenssl.inject_into_urllib3() + + # Check cryptography version + from cryptography import __version__ as cryptography_version + + _check_cryptography(cryptography_version) +except ImportError: + pass + +# urllib3's DependencyWarnings should be silenced. +from urllib3.exceptions import DependencyWarning + +warnings.simplefilter("ignore", DependencyWarning) + +# Set default logging handler to avoid "No handler found" warnings. +import logging +from logging import NullHandler + +from . import packages, utils +from .__version__ import ( + __author__, + __author_email__, + __build__, + __cake__, + __copyright__, + __description__, + __license__, + __title__, + __url__, + __version__, +) +from .api import delete, get, head, options, patch, post, put, request +from .exceptions import ( + ConnectionError, + ConnectTimeout, + FileModeWarning, + HTTPError, + JSONDecodeError, + ReadTimeout, + RequestException, + Timeout, + TooManyRedirects, + URLRequired, +) +from .models import PreparedRequest, Request, Response +from .sessions import Session, session +from .status_codes import codes + +logging.getLogger(__name__).addHandler(NullHandler()) + +# FileModeWarnings go off per the default. +warnings.simplefilter("default", FileModeWarning, append=True) diff --git a/examples/packages/requests/source/requests/__version__.py b/examples/packages/requests/source/requests/__version__.py new file mode 100644 index 0000000..effdd98 --- /dev/null +++ b/examples/packages/requests/source/requests/__version__.py @@ -0,0 +1,14 @@ +# .-. .-. .-. . . .-. .-. .-. .-. +# |( |- |.| | | |- `-. | `-. +# ' ' `-' `-`.`-' `-' `-' ' `-' + +__title__ = "requests" +__description__ = "Python HTTP for Humans." +__url__ = "https://requests.readthedocs.io" +__version__ = "2.32.5" +__build__ = 0x023205 +__author__ = "Kenneth Reitz" +__author_email__ = "me@kennethreitz.org" +__license__ = "Apache-2.0" +__copyright__ = "Copyright Kenneth Reitz" +__cake__ = "\u2728 \U0001f370 \u2728" diff --git a/examples/packages/requests/source/requests/_internal_utils.py b/examples/packages/requests/source/requests/_internal_utils.py new file mode 100644 index 0000000..f2cf635 --- /dev/null +++ b/examples/packages/requests/source/requests/_internal_utils.py @@ -0,0 +1,50 @@ +""" +requests._internal_utils +~~~~~~~~~~~~~~ + +Provides utility functions that are consumed internally by Requests +which depend on extremely few external helpers (such as compat) +""" +import re + +from .compat import builtin_str + +_VALID_HEADER_NAME_RE_BYTE = re.compile(rb"^[^:\s][^:\r\n]*$") +_VALID_HEADER_NAME_RE_STR = re.compile(r"^[^:\s][^:\r\n]*$") +_VALID_HEADER_VALUE_RE_BYTE = re.compile(rb"^\S[^\r\n]*$|^$") +_VALID_HEADER_VALUE_RE_STR = re.compile(r"^\S[^\r\n]*$|^$") + +_HEADER_VALIDATORS_STR = (_VALID_HEADER_NAME_RE_STR, _VALID_HEADER_VALUE_RE_STR) +_HEADER_VALIDATORS_BYTE = (_VALID_HEADER_NAME_RE_BYTE, _VALID_HEADER_VALUE_RE_BYTE) +HEADER_VALIDATORS = { + bytes: _HEADER_VALIDATORS_BYTE, + str: _HEADER_VALIDATORS_STR, +} + + +def to_native_string(string, encoding="ascii"): + """Given a string object, regardless of type, returns a representation of + that string in the native string type, encoding and decoding where + necessary. This assumes ASCII unless told otherwise. + """ + if isinstance(string, builtin_str): + out = string + else: + out = string.decode(encoding) + + return out + + +def unicode_is_ascii(u_string): + """Determine if unicode string only contains ASCII characters. + + :param str u_string: unicode string to check. Must be unicode + and not Python 2 `str`. + :rtype: bool + """ + assert isinstance(u_string, str) + try: + u_string.encode("ascii") + return True + except UnicodeEncodeError: + return False diff --git a/examples/packages/requests/source/requests/adapters.py b/examples/packages/requests/source/requests/adapters.py new file mode 100644 index 0000000..670c927 --- /dev/null +++ b/examples/packages/requests/source/requests/adapters.py @@ -0,0 +1,696 @@ +""" +requests.adapters +~~~~~~~~~~~~~~~~~ + +This module contains the transport adapters that Requests uses to define +and maintain connections. +""" + +import os.path +import socket # noqa: F401 +import typing +import warnings + +from urllib3.exceptions import ClosedPoolError, ConnectTimeoutError +from urllib3.exceptions import HTTPError as _HTTPError +from urllib3.exceptions import InvalidHeader as _InvalidHeader +from urllib3.exceptions import ( + LocationValueError, + MaxRetryError, + NewConnectionError, + ProtocolError, +) +from urllib3.exceptions import ProxyError as _ProxyError +from urllib3.exceptions import ReadTimeoutError, ResponseError +from urllib3.exceptions import SSLError as _SSLError +from urllib3.poolmanager import PoolManager, proxy_from_url +from urllib3.util import Timeout as TimeoutSauce +from urllib3.util import parse_url +from urllib3.util.retry import Retry + +from .auth import _basic_auth_str +from .compat import basestring, urlparse +from .cookies import extract_cookies_to_jar +from .exceptions import ( + ConnectionError, + ConnectTimeout, + InvalidHeader, + InvalidProxyURL, + InvalidSchema, + InvalidURL, + ProxyError, + ReadTimeout, + RetryError, + SSLError, +) +from .models import Response +from .structures import CaseInsensitiveDict +from .utils import ( + DEFAULT_CA_BUNDLE_PATH, + extract_zipped_paths, + get_auth_from_url, + get_encoding_from_headers, + prepend_scheme_if_needed, + select_proxy, + urldefragauth, +) + +try: + from urllib3.contrib.socks import SOCKSProxyManager +except ImportError: + + def SOCKSProxyManager(*args, **kwargs): + raise InvalidSchema("Missing dependencies for SOCKS support.") + + +if typing.TYPE_CHECKING: + from .models import PreparedRequest + + +DEFAULT_POOLBLOCK = False +DEFAULT_POOLSIZE = 10 +DEFAULT_RETRIES = 0 +DEFAULT_POOL_TIMEOUT = None + + +def _urllib3_request_context( + request: "PreparedRequest", + verify: "bool | str | None", + client_cert: "typing.Tuple[str, str] | str | None", + poolmanager: "PoolManager", +) -> "(typing.Dict[str, typing.Any], typing.Dict[str, typing.Any])": + host_params = {} + pool_kwargs = {} + parsed_request_url = urlparse(request.url) + scheme = parsed_request_url.scheme.lower() + port = parsed_request_url.port + + cert_reqs = "CERT_REQUIRED" + if verify is False: + cert_reqs = "CERT_NONE" + elif isinstance(verify, str): + if not os.path.isdir(verify): + pool_kwargs["ca_certs"] = verify + else: + pool_kwargs["ca_cert_dir"] = verify + pool_kwargs["cert_reqs"] = cert_reqs + if client_cert is not None: + if isinstance(client_cert, tuple) and len(client_cert) == 2: + pool_kwargs["cert_file"] = client_cert[0] + pool_kwargs["key_file"] = client_cert[1] + else: + # According to our docs, we allow users to specify just the client + # cert path + pool_kwargs["cert_file"] = client_cert + host_params = { + "scheme": scheme, + "host": parsed_request_url.hostname, + "port": port, + } + return host_params, pool_kwargs + + +class BaseAdapter: + """The Base Transport Adapter""" + + def __init__(self): + super().__init__() + + def send( + self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None + ): + """Sends PreparedRequest object. Returns Response object. + + :param request: The :class:`PreparedRequest ` being sent. + :param stream: (optional) Whether to stream the request content. + :param timeout: (optional) How long to wait for the server to send + data before giving up, as a float, or a :ref:`(connect timeout, + read timeout) ` tuple. + :type timeout: float or tuple + :param verify: (optional) Either a boolean, in which case it controls whether we verify + the server's TLS certificate, or a string, in which case it must be a path + to a CA bundle to use + :param cert: (optional) Any user-provided SSL certificate to be trusted. + :param proxies: (optional) The proxies dictionary to apply to the request. + """ + raise NotImplementedError + + def close(self): + """Cleans up adapter specific items.""" + raise NotImplementedError + + +class HTTPAdapter(BaseAdapter): + """The built-in HTTP Adapter for urllib3. + + Provides a general-case interface for Requests sessions to contact HTTP and + HTTPS urls by implementing the Transport Adapter interface. This class will + usually be created by the :class:`Session ` class under the + covers. + + :param pool_connections: The number of urllib3 connection pools to cache. + :param pool_maxsize: The maximum number of connections to save in the pool. + :param max_retries: The maximum number of retries each connection + should attempt. Note, this applies only to failed DNS lookups, socket + connections and connection timeouts, never to requests where data has + made it to the server. By default, Requests does not retry failed + connections. If you need granular control over the conditions under + which we retry a request, import urllib3's ``Retry`` class and pass + that instead. + :param pool_block: Whether the connection pool should block for connections. + + Usage:: + + >>> import requests + >>> s = requests.Session() + >>> a = requests.adapters.HTTPAdapter(max_retries=3) + >>> s.mount('http://', a) + """ + + __attrs__ = [ + "max_retries", + "config", + "_pool_connections", + "_pool_maxsize", + "_pool_block", + ] + + def __init__( + self, + pool_connections=DEFAULT_POOLSIZE, + pool_maxsize=DEFAULT_POOLSIZE, + max_retries=DEFAULT_RETRIES, + pool_block=DEFAULT_POOLBLOCK, + ): + if max_retries == DEFAULT_RETRIES: + self.max_retries = Retry(0, read=False) + else: + self.max_retries = Retry.from_int(max_retries) + self.config = {} + self.proxy_manager = {} + + super().__init__() + + self._pool_connections = pool_connections + self._pool_maxsize = pool_maxsize + self._pool_block = pool_block + + self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block) + + def __getstate__(self): + return {attr: getattr(self, attr, None) for attr in self.__attrs__} + + def __setstate__(self, state): + # Can't handle by adding 'proxy_manager' to self.__attrs__ because + # self.poolmanager uses a lambda function, which isn't pickleable. + self.proxy_manager = {} + self.config = {} + + for attr, value in state.items(): + setattr(self, attr, value) + + self.init_poolmanager( + self._pool_connections, self._pool_maxsize, block=self._pool_block + ) + + def init_poolmanager( + self, connections, maxsize, block=DEFAULT_POOLBLOCK, **pool_kwargs + ): + """Initializes a urllib3 PoolManager. + + This method should not be called from user code, and is only + exposed for use when subclassing the + :class:`HTTPAdapter `. + + :param connections: The number of urllib3 connection pools to cache. + :param maxsize: The maximum number of connections to save in the pool. + :param block: Block when no free connections are available. + :param pool_kwargs: Extra keyword arguments used to initialize the Pool Manager. + """ + # save these values for pickling + self._pool_connections = connections + self._pool_maxsize = maxsize + self._pool_block = block + + self.poolmanager = PoolManager( + num_pools=connections, + maxsize=maxsize, + block=block, + **pool_kwargs, + ) + + def proxy_manager_for(self, proxy, **proxy_kwargs): + """Return urllib3 ProxyManager for the given proxy. + + This method should not be called from user code, and is only + exposed for use when subclassing the + :class:`HTTPAdapter `. + + :param proxy: The proxy to return a urllib3 ProxyManager for. + :param proxy_kwargs: Extra keyword arguments used to configure the Proxy Manager. + :returns: ProxyManager + :rtype: urllib3.ProxyManager + """ + if proxy in self.proxy_manager: + manager = self.proxy_manager[proxy] + elif proxy.lower().startswith("socks"): + username, password = get_auth_from_url(proxy) + manager = self.proxy_manager[proxy] = SOCKSProxyManager( + proxy, + username=username, + password=password, + num_pools=self._pool_connections, + maxsize=self._pool_maxsize, + block=self._pool_block, + **proxy_kwargs, + ) + else: + proxy_headers = self.proxy_headers(proxy) + manager = self.proxy_manager[proxy] = proxy_from_url( + proxy, + proxy_headers=proxy_headers, + num_pools=self._pool_connections, + maxsize=self._pool_maxsize, + block=self._pool_block, + **proxy_kwargs, + ) + + return manager + + def cert_verify(self, conn, url, verify, cert): + """Verify a SSL certificate. This method should not be called from user + code, and is only exposed for use when subclassing the + :class:`HTTPAdapter `. + + :param conn: The urllib3 connection object associated with the cert. + :param url: The requested URL. + :param verify: Either a boolean, in which case it controls whether we verify + the server's TLS certificate, or a string, in which case it must be a path + to a CA bundle to use + :param cert: The SSL certificate to verify. + """ + if url.lower().startswith("https") and verify: + cert_loc = None + + # Allow self-specified cert location. + if verify is not True: + cert_loc = verify + + if not cert_loc: + cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH) + + if not cert_loc or not os.path.exists(cert_loc): + raise OSError( + f"Could not find a suitable TLS CA certificate bundle, " + f"invalid path: {cert_loc}" + ) + + conn.cert_reqs = "CERT_REQUIRED" + + if not os.path.isdir(cert_loc): + conn.ca_certs = cert_loc + else: + conn.ca_cert_dir = cert_loc + else: + conn.cert_reqs = "CERT_NONE" + conn.ca_certs = None + conn.ca_cert_dir = None + + if cert: + if not isinstance(cert, basestring): + conn.cert_file = cert[0] + conn.key_file = cert[1] + else: + conn.cert_file = cert + conn.key_file = None + if conn.cert_file and not os.path.exists(conn.cert_file): + raise OSError( + f"Could not find the TLS certificate file, " + f"invalid path: {conn.cert_file}" + ) + if conn.key_file and not os.path.exists(conn.key_file): + raise OSError( + f"Could not find the TLS key file, invalid path: {conn.key_file}" + ) + + def build_response(self, req, resp): + """Builds a :class:`Response ` object from a urllib3 + response. This should not be called from user code, and is only exposed + for use when subclassing the + :class:`HTTPAdapter ` + + :param req: The :class:`PreparedRequest ` used to generate the response. + :param resp: The urllib3 response object. + :rtype: requests.Response + """ + response = Response() + + # Fallback to None if there's no status_code, for whatever reason. + response.status_code = getattr(resp, "status", None) + + # Make headers case-insensitive. + response.headers = CaseInsensitiveDict(getattr(resp, "headers", {})) + + # Set encoding. + response.encoding = get_encoding_from_headers(response.headers) + response.raw = resp + response.reason = response.raw.reason + + if isinstance(req.url, bytes): + response.url = req.url.decode("utf-8") + else: + response.url = req.url + + # Add new cookies from the server. + extract_cookies_to_jar(response.cookies, req, resp) + + # Give the Response some context. + response.request = req + response.connection = self + + return response + + def build_connection_pool_key_attributes(self, request, verify, cert=None): + """Build the PoolKey attributes used by urllib3 to return a connection. + + This looks at the PreparedRequest, the user-specified verify value, + and the value of the cert parameter to determine what PoolKey values + to use to select a connection from a given urllib3 Connection Pool. + + The SSL related pool key arguments are not consistently set. As of + this writing, use the following to determine what keys may be in that + dictionary: + + * If ``verify`` is ``True``, ``"ssl_context"`` will be set and will be the + default Requests SSL Context + * If ``verify`` is ``False``, ``"ssl_context"`` will not be set but + ``"cert_reqs"`` will be set + * If ``verify`` is a string, (i.e., it is a user-specified trust bundle) + ``"ca_certs"`` will be set if the string is not a directory recognized + by :py:func:`os.path.isdir`, otherwise ``"ca_cert_dir"`` will be + set. + * If ``"cert"`` is specified, ``"cert_file"`` will always be set. If + ``"cert"`` is a tuple with a second item, ``"key_file"`` will also + be present + + To override these settings, one may subclass this class, call this + method and use the above logic to change parameters as desired. For + example, if one wishes to use a custom :py:class:`ssl.SSLContext` one + must both set ``"ssl_context"`` and based on what else they require, + alter the other keys to ensure the desired behaviour. + + :param request: + The PreparedReqest being sent over the connection. + :type request: + :class:`~requests.models.PreparedRequest` + :param verify: + Either a boolean, in which case it controls whether + we verify the server's TLS certificate, or a string, in which case it + must be a path to a CA bundle to use. + :param cert: + (optional) Any user-provided SSL certificate for client + authentication (a.k.a., mTLS). This may be a string (i.e., just + the path to a file which holds both certificate and key) or a + tuple of length 2 with the certificate file path and key file + path. + :returns: + A tuple of two dictionaries. The first is the "host parameters" + portion of the Pool Key including scheme, hostname, and port. The + second is a dictionary of SSLContext related parameters. + """ + return _urllib3_request_context(request, verify, cert, self.poolmanager) + + def get_connection_with_tls_context(self, request, verify, proxies=None, cert=None): + """Returns a urllib3 connection for the given request and TLS settings. + This should not be called from user code, and is only exposed for use + when subclassing the :class:`HTTPAdapter `. + + :param request: + The :class:`PreparedRequest ` object to be sent + over the connection. + :param verify: + Either a boolean, in which case it controls whether we verify the + server's TLS certificate, or a string, in which case it must be a + path to a CA bundle to use. + :param proxies: + (optional) The proxies dictionary to apply to the request. + :param cert: + (optional) Any user-provided SSL certificate to be used for client + authentication (a.k.a., mTLS). + :rtype: + urllib3.ConnectionPool + """ + proxy = select_proxy(request.url, proxies) + try: + host_params, pool_kwargs = self.build_connection_pool_key_attributes( + request, + verify, + cert, + ) + except ValueError as e: + raise InvalidURL(e, request=request) + if proxy: + proxy = prepend_scheme_if_needed(proxy, "http") + proxy_url = parse_url(proxy) + if not proxy_url.host: + raise InvalidProxyURL( + "Please check proxy URL. It is malformed " + "and could be missing the host." + ) + proxy_manager = self.proxy_manager_for(proxy) + conn = proxy_manager.connection_from_host( + **host_params, pool_kwargs=pool_kwargs + ) + else: + # Only scheme should be lower case + conn = self.poolmanager.connection_from_host( + **host_params, pool_kwargs=pool_kwargs + ) + + return conn + + def get_connection(self, url, proxies=None): + """DEPRECATED: Users should move to `get_connection_with_tls_context` + for all subclasses of HTTPAdapter using Requests>=2.32.2. + + Returns a urllib3 connection for the given URL. This should not be + called from user code, and is only exposed for use when subclassing the + :class:`HTTPAdapter `. + + :param url: The URL to connect to. + :param proxies: (optional) A Requests-style dictionary of proxies used on this request. + :rtype: urllib3.ConnectionPool + """ + warnings.warn( + ( + "`get_connection` has been deprecated in favor of " + "`get_connection_with_tls_context`. Custom HTTPAdapter subclasses " + "will need to migrate for Requests>=2.32.2. Please see " + "https://github.com/psf/requests/pull/6710 for more details." + ), + DeprecationWarning, + ) + proxy = select_proxy(url, proxies) + + if proxy: + proxy = prepend_scheme_if_needed(proxy, "http") + proxy_url = parse_url(proxy) + if not proxy_url.host: + raise InvalidProxyURL( + "Please check proxy URL. It is malformed " + "and could be missing the host." + ) + proxy_manager = self.proxy_manager_for(proxy) + conn = proxy_manager.connection_from_url(url) + else: + # Only scheme should be lower case + parsed = urlparse(url) + url = parsed.geturl() + conn = self.poolmanager.connection_from_url(url) + + return conn + + def close(self): + """Disposes of any internal state. + + Currently, this closes the PoolManager and any active ProxyManager, + which closes any pooled connections. + """ + self.poolmanager.clear() + for proxy in self.proxy_manager.values(): + proxy.clear() + + def request_url(self, request, proxies): + """Obtain the url to use when making the final request. + + If the message is being sent through a HTTP proxy, the full URL has to + be used. Otherwise, we should only use the path portion of the URL. + + This should not be called from user code, and is only exposed for use + when subclassing the + :class:`HTTPAdapter `. + + :param request: The :class:`PreparedRequest ` being sent. + :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs. + :rtype: str + """ + proxy = select_proxy(request.url, proxies) + scheme = urlparse(request.url).scheme + + is_proxied_http_request = proxy and scheme != "https" + using_socks_proxy = False + if proxy: + proxy_scheme = urlparse(proxy).scheme.lower() + using_socks_proxy = proxy_scheme.startswith("socks") + + url = request.path_url + if url.startswith("//"): # Don't confuse urllib3 + url = f"/{url.lstrip('/')}" + + if is_proxied_http_request and not using_socks_proxy: + url = urldefragauth(request.url) + + return url + + def add_headers(self, request, **kwargs): + """Add any headers needed by the connection. As of v2.0 this does + nothing by default, but is left for overriding by users that subclass + the :class:`HTTPAdapter `. + + This should not be called from user code, and is only exposed for use + when subclassing the + :class:`HTTPAdapter `. + + :param request: The :class:`PreparedRequest ` to add headers to. + :param kwargs: The keyword arguments from the call to send(). + """ + pass + + def proxy_headers(self, proxy): + """Returns a dictionary of the headers to add to any request sent + through a proxy. This works with urllib3 magic to ensure that they are + correctly sent to the proxy, rather than in a tunnelled request if + CONNECT is being used. + + This should not be called from user code, and is only exposed for use + when subclassing the + :class:`HTTPAdapter `. + + :param proxy: The url of the proxy being used for this request. + :rtype: dict + """ + headers = {} + username, password = get_auth_from_url(proxy) + + if username: + headers["Proxy-Authorization"] = _basic_auth_str(username, password) + + return headers + + def send( + self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None + ): + """Sends PreparedRequest object. Returns Response object. + + :param request: The :class:`PreparedRequest ` being sent. + :param stream: (optional) Whether to stream the request content. + :param timeout: (optional) How long to wait for the server to send + data before giving up, as a float, or a :ref:`(connect timeout, + read timeout) ` tuple. + :type timeout: float or tuple or urllib3 Timeout object + :param verify: (optional) Either a boolean, in which case it controls whether + we verify the server's TLS certificate, or a string, in which case it + must be a path to a CA bundle to use + :param cert: (optional) Any user-provided SSL certificate to be trusted. + :param proxies: (optional) The proxies dictionary to apply to the request. + :rtype: requests.Response + """ + + try: + conn = self.get_connection_with_tls_context( + request, verify, proxies=proxies, cert=cert + ) + except LocationValueError as e: + raise InvalidURL(e, request=request) + + self.cert_verify(conn, request.url, verify, cert) + url = self.request_url(request, proxies) + self.add_headers( + request, + stream=stream, + timeout=timeout, + verify=verify, + cert=cert, + proxies=proxies, + ) + + chunked = not (request.body is None or "Content-Length" in request.headers) + + if isinstance(timeout, tuple): + try: + connect, read = timeout + timeout = TimeoutSauce(connect=connect, read=read) + except ValueError: + raise ValueError( + f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " + f"or a single float to set both timeouts to the same value." + ) + elif isinstance(timeout, TimeoutSauce): + pass + else: + timeout = TimeoutSauce(connect=timeout, read=timeout) + + try: + resp = conn.urlopen( + method=request.method, + url=url, + body=request.body, + headers=request.headers, + redirect=False, + assert_same_host=False, + preload_content=False, + decode_content=False, + retries=self.max_retries, + timeout=timeout, + chunked=chunked, + ) + + except (ProtocolError, OSError) as err: + raise ConnectionError(err, request=request) + + except MaxRetryError as e: + if isinstance(e.reason, ConnectTimeoutError): + # TODO: Remove this in 3.0.0: see #2811 + if not isinstance(e.reason, NewConnectionError): + raise ConnectTimeout(e, request=request) + + if isinstance(e.reason, ResponseError): + raise RetryError(e, request=request) + + if isinstance(e.reason, _ProxyError): + raise ProxyError(e, request=request) + + if isinstance(e.reason, _SSLError): + # This branch is for urllib3 v1.22 and later. + raise SSLError(e, request=request) + + raise ConnectionError(e, request=request) + + except ClosedPoolError as e: + raise ConnectionError(e, request=request) + + except _ProxyError as e: + raise ProxyError(e) + + except (_SSLError, _HTTPError) as e: + if isinstance(e, _SSLError): + # This branch is for urllib3 versions earlier than v1.22 + raise SSLError(e, request=request) + elif isinstance(e, ReadTimeoutError): + raise ReadTimeout(e, request=request) + elif isinstance(e, _InvalidHeader): + raise InvalidHeader(e, request=request) + else: + raise + + return self.build_response(request, resp) diff --git a/examples/packages/requests/source/requests/api.py b/examples/packages/requests/source/requests/api.py new file mode 100644 index 0000000..5960744 --- /dev/null +++ b/examples/packages/requests/source/requests/api.py @@ -0,0 +1,157 @@ +""" +requests.api +~~~~~~~~~~~~ + +This module implements the Requests API. + +:copyright: (c) 2012 by Kenneth Reitz. +:license: Apache2, see LICENSE for more details. +""" + +from . import sessions + + +def request(method, url, **kwargs): + """Constructs and sends a :class:`Request `. + + :param method: method for the new :class:`Request` object: ``GET``, ``OPTIONS``, ``HEAD``, ``POST``, ``PUT``, ``PATCH``, or ``DELETE``. + :param url: URL for the new :class:`Request` object. + :param params: (optional) Dictionary, list of tuples or bytes to send + in the query string for the :class:`Request`. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. + :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. + :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. + :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload. + ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')`` + or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content_type'`` is a string + defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers + to add for the file. + :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. + :param timeout: (optional) How many seconds to wait for the server to send data + before giving up, as a float, or a :ref:`(connect timeout, read + timeout) ` tuple. + :type timeout: float or tuple + :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``. + :type allow_redirects: bool + :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. + :param verify: (optional) Either a boolean, in which case it controls whether we verify + the server's TLS certificate, or a string, in which case it must be a path + to a CA bundle to use. Defaults to ``True``. + :param stream: (optional) if ``False``, the response content will be immediately downloaded. + :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair. + :return: :class:`Response ` object + :rtype: requests.Response + + Usage:: + + >>> import requests + >>> req = requests.request('GET', 'https://httpbin.org/get') + >>> req + + """ + + # By using the 'with' statement we are sure the session is closed, thus we + # avoid leaving sockets open which can trigger a ResourceWarning in some + # cases, and look like a memory leak in others. + with sessions.Session() as session: + return session.request(method=method, url=url, **kwargs) + + +def get(url, params=None, **kwargs): + r"""Sends a GET request. + + :param url: URL for the new :class:`Request` object. + :param params: (optional) Dictionary, list of tuples or bytes to send + in the query string for the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :return: :class:`Response ` object + :rtype: requests.Response + """ + + return request("get", url, params=params, **kwargs) + + +def options(url, **kwargs): + r"""Sends an OPTIONS request. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :return: :class:`Response ` object + :rtype: requests.Response + """ + + return request("options", url, **kwargs) + + +def head(url, **kwargs): + r"""Sends a HEAD request. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. If + `allow_redirects` is not provided, it will be set to `False` (as + opposed to the default :meth:`request` behavior). + :return: :class:`Response ` object + :rtype: requests.Response + """ + + kwargs.setdefault("allow_redirects", False) + return request("head", url, **kwargs) + + +def post(url, data=None, json=None, **kwargs): + r"""Sends a POST request. + + :param url: URL for the new :class:`Request` object. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :return: :class:`Response ` object + :rtype: requests.Response + """ + + return request("post", url, data=data, json=json, **kwargs) + + +def put(url, data=None, **kwargs): + r"""Sends a PUT request. + + :param url: URL for the new :class:`Request` object. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :return: :class:`Response ` object + :rtype: requests.Response + """ + + return request("put", url, data=data, **kwargs) + + +def patch(url, data=None, **kwargs): + r"""Sends a PATCH request. + + :param url: URL for the new :class:`Request` object. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :return: :class:`Response ` object + :rtype: requests.Response + """ + + return request("patch", url, data=data, **kwargs) + + +def delete(url, **kwargs): + r"""Sends a DELETE request. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :return: :class:`Response ` object + :rtype: requests.Response + """ + + return request("delete", url, **kwargs) diff --git a/examples/packages/requests/source/requests/auth.py b/examples/packages/requests/source/requests/auth.py new file mode 100644 index 0000000..4a7ce6d --- /dev/null +++ b/examples/packages/requests/source/requests/auth.py @@ -0,0 +1,314 @@ +""" +requests.auth +~~~~~~~~~~~~~ + +This module contains the authentication handlers for Requests. +""" + +import hashlib +import os +import re +import threading +import time +import warnings +from base64 import b64encode + +from ._internal_utils import to_native_string +from .compat import basestring, str, urlparse +from .cookies import extract_cookies_to_jar +from .utils import parse_dict_header + +CONTENT_TYPE_FORM_URLENCODED = "application/x-www-form-urlencoded" +CONTENT_TYPE_MULTI_PART = "multipart/form-data" + + +def _basic_auth_str(username, password): + """Returns a Basic Auth string.""" + + # "I want us to put a big-ol' comment on top of it that + # says that this behaviour is dumb but we need to preserve + # it because people are relying on it." + # - Lukasa + # + # These are here solely to maintain backwards compatibility + # for things like ints. This will be removed in 3.0.0. + if not isinstance(username, basestring): + warnings.warn( + "Non-string usernames will no longer be supported in Requests " + "3.0.0. Please convert the object you've passed in ({!r}) to " + "a string or bytes object in the near future to avoid " + "problems.".format(username), + category=DeprecationWarning, + ) + username = str(username) + + if not isinstance(password, basestring): + warnings.warn( + "Non-string passwords will no longer be supported in Requests " + "3.0.0. Please convert the object you've passed in ({!r}) to " + "a string or bytes object in the near future to avoid " + "problems.".format(type(password)), + category=DeprecationWarning, + ) + password = str(password) + # -- End Removal -- + + if isinstance(username, str): + username = username.encode("latin1") + + if isinstance(password, str): + password = password.encode("latin1") + + authstr = "Basic " + to_native_string( + b64encode(b":".join((username, password))).strip() + ) + + return authstr + + +class AuthBase: + """Base class that all auth implementations derive from""" + + def __call__(self, r): + raise NotImplementedError("Auth hooks must be callable.") + + +class HTTPBasicAuth(AuthBase): + """Attaches HTTP Basic Authentication to the given Request object.""" + + def __init__(self, username, password): + self.username = username + self.password = password + + def __eq__(self, other): + return all( + [ + self.username == getattr(other, "username", None), + self.password == getattr(other, "password", None), + ] + ) + + def __ne__(self, other): + return not self == other + + def __call__(self, r): + r.headers["Authorization"] = _basic_auth_str(self.username, self.password) + return r + + +class HTTPProxyAuth(HTTPBasicAuth): + """Attaches HTTP Proxy Authentication to a given Request object.""" + + def __call__(self, r): + r.headers["Proxy-Authorization"] = _basic_auth_str(self.username, self.password) + return r + + +class HTTPDigestAuth(AuthBase): + """Attaches HTTP Digest Authentication to the given Request object.""" + + def __init__(self, username, password): + self.username = username + self.password = password + # Keep state in per-thread local storage + self._thread_local = threading.local() + + def init_per_thread_state(self): + # Ensure state is initialized just once per-thread + if not hasattr(self._thread_local, "init"): + self._thread_local.init = True + self._thread_local.last_nonce = "" + self._thread_local.nonce_count = 0 + self._thread_local.chal = {} + self._thread_local.pos = None + self._thread_local.num_401_calls = None + + def build_digest_header(self, method, url): + """ + :rtype: str + """ + + realm = self._thread_local.chal["realm"] + nonce = self._thread_local.chal["nonce"] + qop = self._thread_local.chal.get("qop") + algorithm = self._thread_local.chal.get("algorithm") + opaque = self._thread_local.chal.get("opaque") + hash_utf8 = None + + if algorithm is None: + _algorithm = "MD5" + else: + _algorithm = algorithm.upper() + # lambdas assume digest modules are imported at the top level + if _algorithm == "MD5" or _algorithm == "MD5-SESS": + + def md5_utf8(x): + if isinstance(x, str): + x = x.encode("utf-8") + return hashlib.md5(x).hexdigest() + + hash_utf8 = md5_utf8 + elif _algorithm == "SHA": + + def sha_utf8(x): + if isinstance(x, str): + x = x.encode("utf-8") + return hashlib.sha1(x).hexdigest() + + hash_utf8 = sha_utf8 + elif _algorithm == "SHA-256": + + def sha256_utf8(x): + if isinstance(x, str): + x = x.encode("utf-8") + return hashlib.sha256(x).hexdigest() + + hash_utf8 = sha256_utf8 + elif _algorithm == "SHA-512": + + def sha512_utf8(x): + if isinstance(x, str): + x = x.encode("utf-8") + return hashlib.sha512(x).hexdigest() + + hash_utf8 = sha512_utf8 + + KD = lambda s, d: hash_utf8(f"{s}:{d}") # noqa:E731 + + if hash_utf8 is None: + return None + + # XXX not implemented yet + entdig = None + p_parsed = urlparse(url) + #: path is request-uri defined in RFC 2616 which should not be empty + path = p_parsed.path or "/" + if p_parsed.query: + path += f"?{p_parsed.query}" + + A1 = f"{self.username}:{realm}:{self.password}" + A2 = f"{method}:{path}" + + HA1 = hash_utf8(A1) + HA2 = hash_utf8(A2) + + if nonce == self._thread_local.last_nonce: + self._thread_local.nonce_count += 1 + else: + self._thread_local.nonce_count = 1 + ncvalue = f"{self._thread_local.nonce_count:08x}" + s = str(self._thread_local.nonce_count).encode("utf-8") + s += nonce.encode("utf-8") + s += time.ctime().encode("utf-8") + s += os.urandom(8) + + cnonce = hashlib.sha1(s).hexdigest()[:16] + if _algorithm == "MD5-SESS": + HA1 = hash_utf8(f"{HA1}:{nonce}:{cnonce}") + + if not qop: + respdig = KD(HA1, f"{nonce}:{HA2}") + elif qop == "auth" or "auth" in qop.split(","): + noncebit = f"{nonce}:{ncvalue}:{cnonce}:auth:{HA2}" + respdig = KD(HA1, noncebit) + else: + # XXX handle auth-int. + return None + + self._thread_local.last_nonce = nonce + + # XXX should the partial digests be encoded too? + base = ( + f'username="{self.username}", realm="{realm}", nonce="{nonce}", ' + f'uri="{path}", response="{respdig}"' + ) + if opaque: + base += f', opaque="{opaque}"' + if algorithm: + base += f', algorithm="{algorithm}"' + if entdig: + base += f', digest="{entdig}"' + if qop: + base += f', qop="auth", nc={ncvalue}, cnonce="{cnonce}"' + + return f"Digest {base}" + + def handle_redirect(self, r, **kwargs): + """Reset num_401_calls counter on redirects.""" + if r.is_redirect: + self._thread_local.num_401_calls = 1 + + def handle_401(self, r, **kwargs): + """ + Takes the given response and tries digest-auth, if needed. + + :rtype: requests.Response + """ + + # If response is not 4xx, do not auth + # See https://github.com/psf/requests/issues/3772 + if not 400 <= r.status_code < 500: + self._thread_local.num_401_calls = 1 + return r + + if self._thread_local.pos is not None: + # Rewind the file position indicator of the body to where + # it was to resend the request. + r.request.body.seek(self._thread_local.pos) + s_auth = r.headers.get("www-authenticate", "") + + if "digest" in s_auth.lower() and self._thread_local.num_401_calls < 2: + self._thread_local.num_401_calls += 1 + pat = re.compile(r"digest ", flags=re.IGNORECASE) + self._thread_local.chal = parse_dict_header(pat.sub("", s_auth, count=1)) + + # Consume content and release the original connection + # to allow our new request to reuse the same one. + r.content + r.close() + prep = r.request.copy() + extract_cookies_to_jar(prep._cookies, r.request, r.raw) + prep.prepare_cookies(prep._cookies) + + prep.headers["Authorization"] = self.build_digest_header( + prep.method, prep.url + ) + _r = r.connection.send(prep, **kwargs) + _r.history.append(r) + _r.request = prep + + return _r + + self._thread_local.num_401_calls = 1 + return r + + def __call__(self, r): + # Initialize per-thread state, if needed + self.init_per_thread_state() + # If we have a saved nonce, skip the 401 + if self._thread_local.last_nonce: + r.headers["Authorization"] = self.build_digest_header(r.method, r.url) + try: + self._thread_local.pos = r.body.tell() + except AttributeError: + # In the case of HTTPDigestAuth being reused and the body of + # the previous request was a file-like object, pos has the + # file position of the previous body. Ensure it's set to + # None. + self._thread_local.pos = None + r.register_hook("response", self.handle_401) + r.register_hook("response", self.handle_redirect) + self._thread_local.num_401_calls = 1 + + return r + + def __eq__(self, other): + return all( + [ + self.username == getattr(other, "username", None), + self.password == getattr(other, "password", None), + ] + ) + + def __ne__(self, other): + return not self == other diff --git a/examples/packages/requests/source/requests/certs.py b/examples/packages/requests/source/requests/certs.py new file mode 100644 index 0000000..be422c3 --- /dev/null +++ b/examples/packages/requests/source/requests/certs.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python + +""" +requests.certs +~~~~~~~~~~~~~~ + +This module returns the preferred default CA certificate bundle. There is +only one — the one from the certifi package. + +If you are packaging Requests, e.g., for a Linux distribution or a managed +environment, you can change the definition of where() to return a separately +packaged CA bundle. +""" +from certifi import where + +if __name__ == "__main__": + print(where()) diff --git a/examples/packages/requests/source/requests/compat.py b/examples/packages/requests/source/requests/compat.py new file mode 100644 index 0000000..7f9d754 --- /dev/null +++ b/examples/packages/requests/source/requests/compat.py @@ -0,0 +1,106 @@ +""" +requests.compat +~~~~~~~~~~~~~~~ + +This module previously handled import compatibility issues +between Python 2 and Python 3. It remains for backwards +compatibility until the next major version. +""" + +import importlib +import sys + +# ------- +# urllib3 +# ------- +from urllib3 import __version__ as urllib3_version + +# Detect which major version of urllib3 is being used. +try: + is_urllib3_1 = int(urllib3_version.split(".")[0]) == 1 +except (TypeError, AttributeError): + # If we can't discern a version, prefer old functionality. + is_urllib3_1 = True + +# ------------------- +# Character Detection +# ------------------- + + +def _resolve_char_detection(): + """Find supported character detection libraries.""" + chardet = None + for lib in ("chardet", "charset_normalizer"): + if chardet is None: + try: + chardet = importlib.import_module(lib) + except ImportError: + pass + return chardet + + +chardet = _resolve_char_detection() + +# ------- +# Pythons +# ------- + +# Syntax sugar. +_ver = sys.version_info + +#: Python 2.x? +is_py2 = _ver[0] == 2 + +#: Python 3.x? +is_py3 = _ver[0] == 3 + +# json/simplejson module import resolution +has_simplejson = False +try: + import simplejson as json + + has_simplejson = True +except ImportError: + import json + +if has_simplejson: + from simplejson import JSONDecodeError +else: + from json import JSONDecodeError + +# Keep OrderedDict for backwards compatibility. +from collections import OrderedDict +from collections.abc import Callable, Mapping, MutableMapping +from http import cookiejar as cookielib +from http.cookies import Morsel +from io import StringIO + +# -------------- +# Legacy Imports +# -------------- +from urllib.parse import ( + quote, + quote_plus, + unquote, + unquote_plus, + urldefrag, + urlencode, + urljoin, + urlparse, + urlsplit, + urlunparse, +) +from urllib.request import ( + getproxies, + getproxies_environment, + parse_http_list, + proxy_bypass, + proxy_bypass_environment, +) + +builtin_str = str +str = str +bytes = bytes +basestring = (str, bytes) +numeric_types = (int, float) +integer_types = (int,) diff --git a/examples/packages/requests/source/requests/cookies.py b/examples/packages/requests/source/requests/cookies.py new file mode 100644 index 0000000..f69d0cd --- /dev/null +++ b/examples/packages/requests/source/requests/cookies.py @@ -0,0 +1,561 @@ +""" +requests.cookies +~~~~~~~~~~~~~~~~ + +Compatibility code to be able to use `http.cookiejar.CookieJar` with requests. + +requests.utils imports from here, so be careful with imports. +""" + +import calendar +import copy +import time + +from ._internal_utils import to_native_string +from .compat import Morsel, MutableMapping, cookielib, urlparse, urlunparse + +try: + import threading +except ImportError: + import dummy_threading as threading + + +class MockRequest: + """Wraps a `requests.Request` to mimic a `urllib2.Request`. + + The code in `http.cookiejar.CookieJar` expects this interface in order to correctly + manage cookie policies, i.e., determine whether a cookie can be set, given the + domains of the request and the cookie. + + The original request object is read-only. The client is responsible for collecting + the new headers via `get_new_headers()` and interpreting them appropriately. You + probably want `get_cookie_header`, defined below. + """ + + def __init__(self, request): + self._r = request + self._new_headers = {} + self.type = urlparse(self._r.url).scheme + + def get_type(self): + return self.type + + def get_host(self): + return urlparse(self._r.url).netloc + + def get_origin_req_host(self): + return self.get_host() + + def get_full_url(self): + # Only return the response's URL if the user hadn't set the Host + # header + if not self._r.headers.get("Host"): + return self._r.url + # If they did set it, retrieve it and reconstruct the expected domain + host = to_native_string(self._r.headers["Host"], encoding="utf-8") + parsed = urlparse(self._r.url) + # Reconstruct the URL as we expect it + return urlunparse( + [ + parsed.scheme, + host, + parsed.path, + parsed.params, + parsed.query, + parsed.fragment, + ] + ) + + def is_unverifiable(self): + return True + + def has_header(self, name): + return name in self._r.headers or name in self._new_headers + + def get_header(self, name, default=None): + return self._r.headers.get(name, self._new_headers.get(name, default)) + + def add_header(self, key, val): + """cookiejar has no legitimate use for this method; add it back if you find one.""" + raise NotImplementedError( + "Cookie headers should be added with add_unredirected_header()" + ) + + def add_unredirected_header(self, name, value): + self._new_headers[name] = value + + def get_new_headers(self): + return self._new_headers + + @property + def unverifiable(self): + return self.is_unverifiable() + + @property + def origin_req_host(self): + return self.get_origin_req_host() + + @property + def host(self): + return self.get_host() + + +class MockResponse: + """Wraps a `httplib.HTTPMessage` to mimic a `urllib.addinfourl`. + + ...what? Basically, expose the parsed HTTP headers from the server response + the way `http.cookiejar` expects to see them. + """ + + def __init__(self, headers): + """Make a MockResponse for `cookiejar` to read. + + :param headers: a httplib.HTTPMessage or analogous carrying the headers + """ + self._headers = headers + + def info(self): + return self._headers + + def getheaders(self, name): + self._headers.getheaders(name) + + +def extract_cookies_to_jar(jar, request, response): + """Extract the cookies from the response into a CookieJar. + + :param jar: http.cookiejar.CookieJar (not necessarily a RequestsCookieJar) + :param request: our own requests.Request object + :param response: urllib3.HTTPResponse object + """ + if not (hasattr(response, "_original_response") and response._original_response): + return + # the _original_response field is the wrapped httplib.HTTPResponse object, + req = MockRequest(request) + # pull out the HTTPMessage with the headers and put it in the mock: + res = MockResponse(response._original_response.msg) + jar.extract_cookies(res, req) + + +def get_cookie_header(jar, request): + """ + Produce an appropriate Cookie header string to be sent with `request`, or None. + + :rtype: str + """ + r = MockRequest(request) + jar.add_cookie_header(r) + return r.get_new_headers().get("Cookie") + + +def remove_cookie_by_name(cookiejar, name, domain=None, path=None): + """Unsets a cookie by name, by default over all domains and paths. + + Wraps CookieJar.clear(), is O(n). + """ + clearables = [] + for cookie in cookiejar: + if cookie.name != name: + continue + if domain is not None and domain != cookie.domain: + continue + if path is not None and path != cookie.path: + continue + clearables.append((cookie.domain, cookie.path, cookie.name)) + + for domain, path, name in clearables: + cookiejar.clear(domain, path, name) + + +class CookieConflictError(RuntimeError): + """There are two cookies that meet the criteria specified in the cookie jar. + Use .get and .set and include domain and path args in order to be more specific. + """ + + +class RequestsCookieJar(cookielib.CookieJar, MutableMapping): + """Compatibility class; is a http.cookiejar.CookieJar, but exposes a dict + interface. + + This is the CookieJar we create by default for requests and sessions that + don't specify one, since some clients may expect response.cookies and + session.cookies to support dict operations. + + Requests does not use the dict interface internally; it's just for + compatibility with external client code. All requests code should work + out of the box with externally provided instances of ``CookieJar``, e.g. + ``LWPCookieJar`` and ``FileCookieJar``. + + Unlike a regular CookieJar, this class is pickleable. + + .. warning:: dictionary operations that are normally O(1) may be O(n). + """ + + def get(self, name, default=None, domain=None, path=None): + """Dict-like get() that also supports optional domain and path args in + order to resolve naming collisions from using one cookie jar over + multiple domains. + + .. warning:: operation is O(n), not O(1). + """ + try: + return self._find_no_duplicates(name, domain, path) + except KeyError: + return default + + def set(self, name, value, **kwargs): + """Dict-like set() that also supports optional domain and path args in + order to resolve naming collisions from using one cookie jar over + multiple domains. + """ + # support client code that unsets cookies by assignment of a None value: + if value is None: + remove_cookie_by_name( + self, name, domain=kwargs.get("domain"), path=kwargs.get("path") + ) + return + + if isinstance(value, Morsel): + c = morsel_to_cookie(value) + else: + c = create_cookie(name, value, **kwargs) + self.set_cookie(c) + return c + + def iterkeys(self): + """Dict-like iterkeys() that returns an iterator of names of cookies + from the jar. + + .. seealso:: itervalues() and iteritems(). + """ + for cookie in iter(self): + yield cookie.name + + def keys(self): + """Dict-like keys() that returns a list of names of cookies from the + jar. + + .. seealso:: values() and items(). + """ + return list(self.iterkeys()) + + def itervalues(self): + """Dict-like itervalues() that returns an iterator of values of cookies + from the jar. + + .. seealso:: iterkeys() and iteritems(). + """ + for cookie in iter(self): + yield cookie.value + + def values(self): + """Dict-like values() that returns a list of values of cookies from the + jar. + + .. seealso:: keys() and items(). + """ + return list(self.itervalues()) + + def iteritems(self): + """Dict-like iteritems() that returns an iterator of name-value tuples + from the jar. + + .. seealso:: iterkeys() and itervalues(). + """ + for cookie in iter(self): + yield cookie.name, cookie.value + + def items(self): + """Dict-like items() that returns a list of name-value tuples from the + jar. Allows client-code to call ``dict(RequestsCookieJar)`` and get a + vanilla python dict of key value pairs. + + .. seealso:: keys() and values(). + """ + return list(self.iteritems()) + + def list_domains(self): + """Utility method to list all the domains in the jar.""" + domains = [] + for cookie in iter(self): + if cookie.domain not in domains: + domains.append(cookie.domain) + return domains + + def list_paths(self): + """Utility method to list all the paths in the jar.""" + paths = [] + for cookie in iter(self): + if cookie.path not in paths: + paths.append(cookie.path) + return paths + + def multiple_domains(self): + """Returns True if there are multiple domains in the jar. + Returns False otherwise. + + :rtype: bool + """ + domains = [] + for cookie in iter(self): + if cookie.domain is not None and cookie.domain in domains: + return True + domains.append(cookie.domain) + return False # there is only one domain in jar + + def get_dict(self, domain=None, path=None): + """Takes as an argument an optional domain and path and returns a plain + old Python dict of name-value pairs of cookies that meet the + requirements. + + :rtype: dict + """ + dictionary = {} + for cookie in iter(self): + if (domain is None or cookie.domain == domain) and ( + path is None or cookie.path == path + ): + dictionary[cookie.name] = cookie.value + return dictionary + + def __contains__(self, name): + try: + return super().__contains__(name) + except CookieConflictError: + return True + + def __getitem__(self, name): + """Dict-like __getitem__() for compatibility with client code. Throws + exception if there are more than one cookie with name. In that case, + use the more explicit get() method instead. + + .. warning:: operation is O(n), not O(1). + """ + return self._find_no_duplicates(name) + + def __setitem__(self, name, value): + """Dict-like __setitem__ for compatibility with client code. Throws + exception if there is already a cookie of that name in the jar. In that + case, use the more explicit set() method instead. + """ + self.set(name, value) + + def __delitem__(self, name): + """Deletes a cookie given a name. Wraps ``http.cookiejar.CookieJar``'s + ``remove_cookie_by_name()``. + """ + remove_cookie_by_name(self, name) + + def set_cookie(self, cookie, *args, **kwargs): + if ( + hasattr(cookie.value, "startswith") + and cookie.value.startswith('"') + and cookie.value.endswith('"') + ): + cookie.value = cookie.value.replace('\\"', "") + return super().set_cookie(cookie, *args, **kwargs) + + def update(self, other): + """Updates this jar with cookies from another CookieJar or dict-like""" + if isinstance(other, cookielib.CookieJar): + for cookie in other: + self.set_cookie(copy.copy(cookie)) + else: + super().update(other) + + def _find(self, name, domain=None, path=None): + """Requests uses this method internally to get cookie values. + + If there are conflicting cookies, _find arbitrarily chooses one. + See _find_no_duplicates if you want an exception thrown if there are + conflicting cookies. + + :param name: a string containing name of cookie + :param domain: (optional) string containing domain of cookie + :param path: (optional) string containing path of cookie + :return: cookie.value + """ + for cookie in iter(self): + if cookie.name == name: + if domain is None or cookie.domain == domain: + if path is None or cookie.path == path: + return cookie.value + + raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") + + def _find_no_duplicates(self, name, domain=None, path=None): + """Both ``__get_item__`` and ``get`` call this function: it's never + used elsewhere in Requests. + + :param name: a string containing name of cookie + :param domain: (optional) string containing domain of cookie + :param path: (optional) string containing path of cookie + :raises KeyError: if cookie is not found + :raises CookieConflictError: if there are multiple cookies + that match name and optionally domain and path + :return: cookie.value + """ + toReturn = None + for cookie in iter(self): + if cookie.name == name: + if domain is None or cookie.domain == domain: + if path is None or cookie.path == path: + if toReturn is not None: + # if there are multiple cookies that meet passed in criteria + raise CookieConflictError( + f"There are multiple cookies with name, {name!r}" + ) + # we will eventually return this as long as no cookie conflict + toReturn = cookie.value + + if toReturn: + return toReturn + raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") + + def __getstate__(self): + """Unlike a normal CookieJar, this class is pickleable.""" + state = self.__dict__.copy() + # remove the unpickleable RLock object + state.pop("_cookies_lock") + return state + + def __setstate__(self, state): + """Unlike a normal CookieJar, this class is pickleable.""" + self.__dict__.update(state) + if "_cookies_lock" not in self.__dict__: + self._cookies_lock = threading.RLock() + + def copy(self): + """Return a copy of this RequestsCookieJar.""" + new_cj = RequestsCookieJar() + new_cj.set_policy(self.get_policy()) + new_cj.update(self) + return new_cj + + def get_policy(self): + """Return the CookiePolicy instance used.""" + return self._policy + + +def _copy_cookie_jar(jar): + if jar is None: + return None + + if hasattr(jar, "copy"): + # We're dealing with an instance of RequestsCookieJar + return jar.copy() + # We're dealing with a generic CookieJar instance + new_jar = copy.copy(jar) + new_jar.clear() + for cookie in jar: + new_jar.set_cookie(copy.copy(cookie)) + return new_jar + + +def create_cookie(name, value, **kwargs): + """Make a cookie from underspecified parameters. + + By default, the pair of `name` and `value` will be set for the domain '' + and sent on every request (this is sometimes called a "supercookie"). + """ + result = { + "version": 0, + "name": name, + "value": value, + "port": None, + "domain": "", + "path": "/", + "secure": False, + "expires": None, + "discard": True, + "comment": None, + "comment_url": None, + "rest": {"HttpOnly": None}, + "rfc2109": False, + } + + badargs = set(kwargs) - set(result) + if badargs: + raise TypeError( + f"create_cookie() got unexpected keyword arguments: {list(badargs)}" + ) + + result.update(kwargs) + result["port_specified"] = bool(result["port"]) + result["domain_specified"] = bool(result["domain"]) + result["domain_initial_dot"] = result["domain"].startswith(".") + result["path_specified"] = bool(result["path"]) + + return cookielib.Cookie(**result) + + +def morsel_to_cookie(morsel): + """Convert a Morsel object into a Cookie containing the one k/v pair.""" + + expires = None + if morsel["max-age"]: + try: + expires = int(time.time() + int(morsel["max-age"])) + except ValueError: + raise TypeError(f"max-age: {morsel['max-age']} must be integer") + elif morsel["expires"]: + time_template = "%a, %d-%b-%Y %H:%M:%S GMT" + expires = calendar.timegm(time.strptime(morsel["expires"], time_template)) + return create_cookie( + comment=morsel["comment"], + comment_url=bool(morsel["comment"]), + discard=False, + domain=morsel["domain"], + expires=expires, + name=morsel.key, + path=morsel["path"], + port=None, + rest={"HttpOnly": morsel["httponly"]}, + rfc2109=False, + secure=bool(morsel["secure"]), + value=morsel.value, + version=morsel["version"] or 0, + ) + + +def cookiejar_from_dict(cookie_dict, cookiejar=None, overwrite=True): + """Returns a CookieJar from a key/value dictionary. + + :param cookie_dict: Dict of key/values to insert into CookieJar. + :param cookiejar: (optional) A cookiejar to add the cookies to. + :param overwrite: (optional) If False, will not replace cookies + already in the jar with new ones. + :rtype: CookieJar + """ + if cookiejar is None: + cookiejar = RequestsCookieJar() + + if cookie_dict is not None: + names_from_jar = [cookie.name for cookie in cookiejar] + for name in cookie_dict: + if overwrite or (name not in names_from_jar): + cookiejar.set_cookie(create_cookie(name, cookie_dict[name])) + + return cookiejar + + +def merge_cookies(cookiejar, cookies): + """Add cookies to cookiejar and returns a merged CookieJar. + + :param cookiejar: CookieJar object to add the cookies to. + :param cookies: Dictionary or CookieJar object to be added. + :rtype: CookieJar + """ + if not isinstance(cookiejar, cookielib.CookieJar): + raise ValueError("You can only merge into CookieJar") + + if isinstance(cookies, dict): + cookiejar = cookiejar_from_dict(cookies, cookiejar=cookiejar, overwrite=False) + elif isinstance(cookies, cookielib.CookieJar): + try: + cookiejar.update(cookies) + except AttributeError: + for cookie_in_jar in cookies: + cookiejar.set_cookie(cookie_in_jar) + + return cookiejar diff --git a/examples/packages/requests/source/requests/exceptions.py b/examples/packages/requests/source/requests/exceptions.py new file mode 100644 index 0000000..83986b4 --- /dev/null +++ b/examples/packages/requests/source/requests/exceptions.py @@ -0,0 +1,151 @@ +""" +requests.exceptions +~~~~~~~~~~~~~~~~~~~ + +This module contains the set of Requests' exceptions. +""" +from urllib3.exceptions import HTTPError as BaseHTTPError + +from .compat import JSONDecodeError as CompatJSONDecodeError + + +class RequestException(IOError): + """There was an ambiguous exception that occurred while handling your + request. + """ + + def __init__(self, *args, **kwargs): + """Initialize RequestException with `request` and `response` objects.""" + response = kwargs.pop("response", None) + self.response = response + self.request = kwargs.pop("request", None) + if response is not None and not self.request and hasattr(response, "request"): + self.request = self.response.request + super().__init__(*args, **kwargs) + + +class InvalidJSONError(RequestException): + """A JSON error occurred.""" + + +class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError): + """Couldn't decode the text into json""" + + def __init__(self, *args, **kwargs): + """ + Construct the JSONDecodeError instance first with all + args. Then use it's args to construct the IOError so that + the json specific args aren't used as IOError specific args + and the error message from JSONDecodeError is preserved. + """ + CompatJSONDecodeError.__init__(self, *args) + InvalidJSONError.__init__(self, *self.args, **kwargs) + + def __reduce__(self): + """ + The __reduce__ method called when pickling the object must + be the one from the JSONDecodeError (be it json/simplejson) + as it expects all the arguments for instantiation, not just + one like the IOError, and the MRO would by default call the + __reduce__ method from the IOError due to the inheritance order. + """ + return CompatJSONDecodeError.__reduce__(self) + + +class HTTPError(RequestException): + """An HTTP error occurred.""" + + +class ConnectionError(RequestException): + """A Connection error occurred.""" + + +class ProxyError(ConnectionError): + """A proxy error occurred.""" + + +class SSLError(ConnectionError): + """An SSL error occurred.""" + + +class Timeout(RequestException): + """The request timed out. + + Catching this error will catch both + :exc:`~requests.exceptions.ConnectTimeout` and + :exc:`~requests.exceptions.ReadTimeout` errors. + """ + + +class ConnectTimeout(ConnectionError, Timeout): + """The request timed out while trying to connect to the remote server. + + Requests that produced this error are safe to retry. + """ + + +class ReadTimeout(Timeout): + """The server did not send any data in the allotted amount of time.""" + + +class URLRequired(RequestException): + """A valid URL is required to make a request.""" + + +class TooManyRedirects(RequestException): + """Too many redirects.""" + + +class MissingSchema(RequestException, ValueError): + """The URL scheme (e.g. http or https) is missing.""" + + +class InvalidSchema(RequestException, ValueError): + """The URL scheme provided is either invalid or unsupported.""" + + +class InvalidURL(RequestException, ValueError): + """The URL provided was somehow invalid.""" + + +class InvalidHeader(RequestException, ValueError): + """The header value provided was somehow invalid.""" + + +class InvalidProxyURL(InvalidURL): + """The proxy URL provided is invalid.""" + + +class ChunkedEncodingError(RequestException): + """The server declared chunked encoding but sent an invalid chunk.""" + + +class ContentDecodingError(RequestException, BaseHTTPError): + """Failed to decode response content.""" + + +class StreamConsumedError(RequestException, TypeError): + """The content for this response was already consumed.""" + + +class RetryError(RequestException): + """Custom retries logic failed""" + + +class UnrewindableBodyError(RequestException): + """Requests encountered an error when trying to rewind a body.""" + + +# Warnings + + +class RequestsWarning(Warning): + """Base warning for Requests.""" + + +class FileModeWarning(RequestsWarning, DeprecationWarning): + """A file was opened in text mode, but Requests determined its binary length.""" + + +class RequestsDependencyWarning(RequestsWarning): + """An imported dependency doesn't match the expected version range.""" diff --git a/examples/packages/requests/source/requests/help.py b/examples/packages/requests/source/requests/help.py new file mode 100644 index 0000000..8fbcd65 --- /dev/null +++ b/examples/packages/requests/source/requests/help.py @@ -0,0 +1,134 @@ +"""Module containing bug report helper(s).""" + +import json +import platform +import ssl +import sys + +import idna +import urllib3 + +from . import __version__ as requests_version + +try: + import charset_normalizer +except ImportError: + charset_normalizer = None + +try: + import chardet +except ImportError: + chardet = None + +try: + from urllib3.contrib import pyopenssl +except ImportError: + pyopenssl = None + OpenSSL = None + cryptography = None +else: + import cryptography + import OpenSSL + + +def _implementation(): + """Return a dict with the Python implementation and version. + + Provide both the name and the version of the Python implementation + currently running. For example, on CPython 3.10.3 it will return + {'name': 'CPython', 'version': '3.10.3'}. + + This function works best on CPython and PyPy: in particular, it probably + doesn't work for Jython or IronPython. Future investigation should be done + to work out the correct shape of the code for those platforms. + """ + implementation = platform.python_implementation() + + if implementation == "CPython": + implementation_version = platform.python_version() + elif implementation == "PyPy": + implementation_version = "{}.{}.{}".format( + sys.pypy_version_info.major, + sys.pypy_version_info.minor, + sys.pypy_version_info.micro, + ) + if sys.pypy_version_info.releaselevel != "final": + implementation_version = "".join( + [implementation_version, sys.pypy_version_info.releaselevel] + ) + elif implementation == "Jython": + implementation_version = platform.python_version() # Complete Guess + elif implementation == "IronPython": + implementation_version = platform.python_version() # Complete Guess + else: + implementation_version = "Unknown" + + return {"name": implementation, "version": implementation_version} + + +def info(): + """Generate information for a bug report.""" + try: + platform_info = { + "system": platform.system(), + "release": platform.release(), + } + except OSError: + platform_info = { + "system": "Unknown", + "release": "Unknown", + } + + implementation_info = _implementation() + urllib3_info = {"version": urllib3.__version__} + charset_normalizer_info = {"version": None} + chardet_info = {"version": None} + if charset_normalizer: + charset_normalizer_info = {"version": charset_normalizer.__version__} + if chardet: + chardet_info = {"version": chardet.__version__} + + pyopenssl_info = { + "version": None, + "openssl_version": "", + } + if OpenSSL: + pyopenssl_info = { + "version": OpenSSL.__version__, + "openssl_version": f"{OpenSSL.SSL.OPENSSL_VERSION_NUMBER:x}", + } + cryptography_info = { + "version": getattr(cryptography, "__version__", ""), + } + idna_info = { + "version": getattr(idna, "__version__", ""), + } + + system_ssl = ssl.OPENSSL_VERSION_NUMBER + system_ssl_info = {"version": f"{system_ssl:x}" if system_ssl is not None else ""} + + return { + "platform": platform_info, + "implementation": implementation_info, + "system_ssl": system_ssl_info, + "using_pyopenssl": pyopenssl is not None, + "using_charset_normalizer": chardet is None, + "pyOpenSSL": pyopenssl_info, + "urllib3": urllib3_info, + "chardet": chardet_info, + "charset_normalizer": charset_normalizer_info, + "cryptography": cryptography_info, + "idna": idna_info, + "requests": { + "version": requests_version, + }, + } + + +def main(): + """Pretty-print the bug information as JSON.""" + print(json.dumps(info(), sort_keys=True, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/examples/packages/requests/source/requests/hooks.py b/examples/packages/requests/source/requests/hooks.py new file mode 100644 index 0000000..d181ba2 --- /dev/null +++ b/examples/packages/requests/source/requests/hooks.py @@ -0,0 +1,33 @@ +""" +requests.hooks +~~~~~~~~~~~~~~ + +This module provides the capabilities for the Requests hooks system. + +Available hooks: + +``response``: + The response generated from a Request. +""" +HOOKS = ["response"] + + +def default_hooks(): + return {event: [] for event in HOOKS} + + +# TODO: response is the only one + + +def dispatch_hook(key, hooks, hook_data, **kwargs): + """Dispatches a hook dictionary on a given piece of data.""" + hooks = hooks or {} + hooks = hooks.get(key) + if hooks: + if hasattr(hooks, "__call__"): + hooks = [hooks] + for hook in hooks: + _hook_data = hook(hook_data, **kwargs) + if _hook_data is not None: + hook_data = _hook_data + return hook_data diff --git a/examples/packages/requests/source/requests/models.py b/examples/packages/requests/source/requests/models.py new file mode 100644 index 0000000..c4b25fa --- /dev/null +++ b/examples/packages/requests/source/requests/models.py @@ -0,0 +1,1039 @@ +""" +requests.models +~~~~~~~~~~~~~~~ + +This module contains the primary objects that power Requests. +""" + +import datetime + +# Import encoding now, to avoid implicit import later. +# Implicit import within threads may cause LookupError when standard library is in a ZIP, +# such as in Embedded Python. See https://github.com/psf/requests/issues/3578. +import encodings.idna # noqa: F401 +from io import UnsupportedOperation + +from urllib3.exceptions import ( + DecodeError, + LocationParseError, + ProtocolError, + ReadTimeoutError, + SSLError, +) +from urllib3.fields import RequestField +from urllib3.filepost import encode_multipart_formdata +from urllib3.util import parse_url + +from ._internal_utils import to_native_string, unicode_is_ascii +from .auth import HTTPBasicAuth +from .compat import ( + Callable, + JSONDecodeError, + Mapping, + basestring, + builtin_str, + chardet, + cookielib, +) +from .compat import json as complexjson +from .compat import urlencode, urlsplit, urlunparse +from .cookies import _copy_cookie_jar, cookiejar_from_dict, get_cookie_header +from .exceptions import ( + ChunkedEncodingError, + ConnectionError, + ContentDecodingError, + HTTPError, + InvalidJSONError, + InvalidURL, +) +from .exceptions import JSONDecodeError as RequestsJSONDecodeError +from .exceptions import MissingSchema +from .exceptions import SSLError as RequestsSSLError +from .exceptions import StreamConsumedError +from .hooks import default_hooks +from .status_codes import codes +from .structures import CaseInsensitiveDict +from .utils import ( + check_header_validity, + get_auth_from_url, + guess_filename, + guess_json_utf, + iter_slices, + parse_header_links, + requote_uri, + stream_decode_response_unicode, + super_len, + to_key_val_list, +) + +#: The set of HTTP status codes that indicate an automatically +#: processable redirect. +REDIRECT_STATI = ( + codes.moved, # 301 + codes.found, # 302 + codes.other, # 303 + codes.temporary_redirect, # 307 + codes.permanent_redirect, # 308 +) + +DEFAULT_REDIRECT_LIMIT = 30 +CONTENT_CHUNK_SIZE = 10 * 1024 +ITER_CHUNK_SIZE = 512 + + +class RequestEncodingMixin: + @property + def path_url(self): + """Build the path URL to use.""" + + url = [] + + p = urlsplit(self.url) + + path = p.path + if not path: + path = "/" + + url.append(path) + + query = p.query + if query: + url.append("?") + url.append(query) + + return "".join(url) + + @staticmethod + def _encode_params(data): + """Encode parameters in a piece of data. + + Will successfully encode parameters when passed as a dict or a list of + 2-tuples. Order is retained if data is a list of 2-tuples but arbitrary + if parameters are supplied as a dict. + """ + + if isinstance(data, (str, bytes)): + return data + elif hasattr(data, "read"): + return data + elif hasattr(data, "__iter__"): + result = [] + for k, vs in to_key_val_list(data): + if isinstance(vs, basestring) or not hasattr(vs, "__iter__"): + vs = [vs] + for v in vs: + if v is not None: + result.append( + ( + k.encode("utf-8") if isinstance(k, str) else k, + v.encode("utf-8") if isinstance(v, str) else v, + ) + ) + return urlencode(result, doseq=True) + else: + return data + + @staticmethod + def _encode_files(files, data): + """Build the body for a multipart/form-data request. + + Will successfully encode files when passed as a dict or a list of + tuples. Order is retained if data is a list of tuples but arbitrary + if parameters are supplied as a dict. + The tuples may be 2-tuples (filename, fileobj), 3-tuples (filename, fileobj, contentype) + or 4-tuples (filename, fileobj, contentype, custom_headers). + """ + if not files: + raise ValueError("Files must be provided.") + elif isinstance(data, basestring): + raise ValueError("Data must not be a string.") + + new_fields = [] + fields = to_key_val_list(data or {}) + files = to_key_val_list(files or {}) + + for field, val in fields: + if isinstance(val, basestring) or not hasattr(val, "__iter__"): + val = [val] + for v in val: + if v is not None: + # Don't call str() on bytestrings: in Py3 it all goes wrong. + if not isinstance(v, bytes): + v = str(v) + + new_fields.append( + ( + field.decode("utf-8") + if isinstance(field, bytes) + else field, + v.encode("utf-8") if isinstance(v, str) else v, + ) + ) + + for k, v in files: + # support for explicit filename + ft = None + fh = None + if isinstance(v, (tuple, list)): + if len(v) == 2: + fn, fp = v + elif len(v) == 3: + fn, fp, ft = v + else: + fn, fp, ft, fh = v + else: + fn = guess_filename(v) or k + fp = v + + if isinstance(fp, (str, bytes, bytearray)): + fdata = fp + elif hasattr(fp, "read"): + fdata = fp.read() + elif fp is None: + continue + else: + fdata = fp + + rf = RequestField(name=k, data=fdata, filename=fn, headers=fh) + rf.make_multipart(content_type=ft) + new_fields.append(rf) + + body, content_type = encode_multipart_formdata(new_fields) + + return body, content_type + + +class RequestHooksMixin: + def register_hook(self, event, hook): + """Properly register a hook.""" + + if event not in self.hooks: + raise ValueError(f'Unsupported event specified, with event name "{event}"') + + if isinstance(hook, Callable): + self.hooks[event].append(hook) + elif hasattr(hook, "__iter__"): + self.hooks[event].extend(h for h in hook if isinstance(h, Callable)) + + def deregister_hook(self, event, hook): + """Deregister a previously registered hook. + Returns True if the hook existed, False if not. + """ + + try: + self.hooks[event].remove(hook) + return True + except ValueError: + return False + + +class Request(RequestHooksMixin): + """A user-created :class:`Request ` object. + + Used to prepare a :class:`PreparedRequest `, which is sent to the server. + + :param method: HTTP method to use. + :param url: URL to send. + :param headers: dictionary of headers to send. + :param files: dictionary of {filename: fileobject} files to multipart upload. + :param data: the body to attach to the request. If a dictionary or + list of tuples ``[(key, value)]`` is provided, form-encoding will + take place. + :param json: json for the body to attach to the request (if files or data is not specified). + :param params: URL parameters to append to the URL. If a dictionary or + list of tuples ``[(key, value)]`` is provided, form-encoding will + take place. + :param auth: Auth handler or (user, pass) tuple. + :param cookies: dictionary or CookieJar of cookies to attach to this request. + :param hooks: dictionary of callback hooks, for internal usage. + + Usage:: + + >>> import requests + >>> req = requests.Request('GET', 'https://httpbin.org/get') + >>> req.prepare() + + """ + + def __init__( + self, + method=None, + url=None, + headers=None, + files=None, + data=None, + params=None, + auth=None, + cookies=None, + hooks=None, + json=None, + ): + # Default empty dicts for dict params. + data = [] if data is None else data + files = [] if files is None else files + headers = {} if headers is None else headers + params = {} if params is None else params + hooks = {} if hooks is None else hooks + + self.hooks = default_hooks() + for k, v in list(hooks.items()): + self.register_hook(event=k, hook=v) + + self.method = method + self.url = url + self.headers = headers + self.files = files + self.data = data + self.json = json + self.params = params + self.auth = auth + self.cookies = cookies + + def __repr__(self): + return f"" + + def prepare(self): + """Constructs a :class:`PreparedRequest ` for transmission and returns it.""" + p = PreparedRequest() + p.prepare( + method=self.method, + url=self.url, + headers=self.headers, + files=self.files, + data=self.data, + json=self.json, + params=self.params, + auth=self.auth, + cookies=self.cookies, + hooks=self.hooks, + ) + return p + + +class PreparedRequest(RequestEncodingMixin, RequestHooksMixin): + """The fully mutable :class:`PreparedRequest ` object, + containing the exact bytes that will be sent to the server. + + Instances are generated from a :class:`Request ` object, and + should not be instantiated manually; doing so may produce undesirable + effects. + + Usage:: + + >>> import requests + >>> req = requests.Request('GET', 'https://httpbin.org/get') + >>> r = req.prepare() + >>> r + + + >>> s = requests.Session() + >>> s.send(r) + + """ + + def __init__(self): + #: HTTP verb to send to the server. + self.method = None + #: HTTP URL to send the request to. + self.url = None + #: dictionary of HTTP headers. + self.headers = None + # The `CookieJar` used to create the Cookie header will be stored here + # after prepare_cookies is called + self._cookies = None + #: request body to send to the server. + self.body = None + #: dictionary of callback hooks, for internal usage. + self.hooks = default_hooks() + #: integer denoting starting position of a readable file-like body. + self._body_position = None + + def prepare( + self, + method=None, + url=None, + headers=None, + files=None, + data=None, + params=None, + auth=None, + cookies=None, + hooks=None, + json=None, + ): + """Prepares the entire request with the given parameters.""" + + self.prepare_method(method) + self.prepare_url(url, params) + self.prepare_headers(headers) + self.prepare_cookies(cookies) + self.prepare_body(data, files, json) + self.prepare_auth(auth, url) + + # Note that prepare_auth must be last to enable authentication schemes + # such as OAuth to work on a fully prepared request. + + # This MUST go after prepare_auth. Authenticators could add a hook + self.prepare_hooks(hooks) + + def __repr__(self): + return f"" + + def copy(self): + p = PreparedRequest() + p.method = self.method + p.url = self.url + p.headers = self.headers.copy() if self.headers is not None else None + p._cookies = _copy_cookie_jar(self._cookies) + p.body = self.body + p.hooks = self.hooks + p._body_position = self._body_position + return p + + def prepare_method(self, method): + """Prepares the given HTTP method.""" + self.method = method + if self.method is not None: + self.method = to_native_string(self.method.upper()) + + @staticmethod + def _get_idna_encoded_host(host): + import idna + + try: + host = idna.encode(host, uts46=True).decode("utf-8") + except idna.IDNAError: + raise UnicodeError + return host + + def prepare_url(self, url, params): + """Prepares the given HTTP URL.""" + #: Accept objects that have string representations. + #: We're unable to blindly call unicode/str functions + #: as this will include the bytestring indicator (b'') + #: on python 3.x. + #: https://github.com/psf/requests/pull/2238 + if isinstance(url, bytes): + url = url.decode("utf8") + else: + url = str(url) + + # Remove leading whitespaces from url + url = url.lstrip() + + # Don't do any URL preparation for non-HTTP schemes like `mailto`, + # `data` etc to work around exceptions from `url_parse`, which + # handles RFC 3986 only. + if ":" in url and not url.lower().startswith("http"): + self.url = url + return + + # Support for unicode domain names and paths. + try: + scheme, auth, host, port, path, query, fragment = parse_url(url) + except LocationParseError as e: + raise InvalidURL(*e.args) + + if not scheme: + raise MissingSchema( + f"Invalid URL {url!r}: No scheme supplied. " + f"Perhaps you meant https://{url}?" + ) + + if not host: + raise InvalidURL(f"Invalid URL {url!r}: No host supplied") + + # In general, we want to try IDNA encoding the hostname if the string contains + # non-ASCII characters. This allows users to automatically get the correct IDNA + # behaviour. For strings containing only ASCII characters, we need to also verify + # it doesn't start with a wildcard (*), before allowing the unencoded hostname. + if not unicode_is_ascii(host): + try: + host = self._get_idna_encoded_host(host) + except UnicodeError: + raise InvalidURL("URL has an invalid label.") + elif host.startswith(("*", ".")): + raise InvalidURL("URL has an invalid label.") + + # Carefully reconstruct the network location + netloc = auth or "" + if netloc: + netloc += "@" + netloc += host + if port: + netloc += f":{port}" + + # Bare domains aren't valid URLs. + if not path: + path = "/" + + if isinstance(params, (str, bytes)): + params = to_native_string(params) + + enc_params = self._encode_params(params) + if enc_params: + if query: + query = f"{query}&{enc_params}" + else: + query = enc_params + + url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment])) + self.url = url + + def prepare_headers(self, headers): + """Prepares the given HTTP headers.""" + + self.headers = CaseInsensitiveDict() + if headers: + for header in headers.items(): + # Raise exception on invalid header value. + check_header_validity(header) + name, value = header + self.headers[to_native_string(name)] = value + + def prepare_body(self, data, files, json=None): + """Prepares the given HTTP body data.""" + + # Check if file, fo, generator, iterator. + # If not, run through normal process. + + # Nottin' on you. + body = None + content_type = None + + if not data and json is not None: + # urllib3 requires a bytes-like body. Python 2's json.dumps + # provides this natively, but Python 3 gives a Unicode string. + content_type = "application/json" + + try: + body = complexjson.dumps(json, allow_nan=False) + except ValueError as ve: + raise InvalidJSONError(ve, request=self) + + if not isinstance(body, bytes): + body = body.encode("utf-8") + + is_stream = all( + [ + hasattr(data, "__iter__"), + not isinstance(data, (basestring, list, tuple, Mapping)), + ] + ) + + if is_stream: + try: + length = super_len(data) + except (TypeError, AttributeError, UnsupportedOperation): + length = None + + body = data + + if getattr(body, "tell", None) is not None: + # Record the current file position before reading. + # This will allow us to rewind a file in the event + # of a redirect. + try: + self._body_position = body.tell() + except OSError: + # This differentiates from None, allowing us to catch + # a failed `tell()` later when trying to rewind the body + self._body_position = object() + + if files: + raise NotImplementedError( + "Streamed bodies and files are mutually exclusive." + ) + + if length: + self.headers["Content-Length"] = builtin_str(length) + else: + self.headers["Transfer-Encoding"] = "chunked" + else: + # Multi-part file uploads. + if files: + (body, content_type) = self._encode_files(files, data) + else: + if data: + body = self._encode_params(data) + if isinstance(data, basestring) or hasattr(data, "read"): + content_type = None + else: + content_type = "application/x-www-form-urlencoded" + + self.prepare_content_length(body) + + # Add content-type if it wasn't explicitly provided. + if content_type and ("content-type" not in self.headers): + self.headers["Content-Type"] = content_type + + self.body = body + + def prepare_content_length(self, body): + """Prepare Content-Length header based on request method and body""" + if body is not None: + length = super_len(body) + if length: + # If length exists, set it. Otherwise, we fallback + # to Transfer-Encoding: chunked. + self.headers["Content-Length"] = builtin_str(length) + elif ( + self.method not in ("GET", "HEAD") + and self.headers.get("Content-Length") is None + ): + # Set Content-Length to 0 for methods that can have a body + # but don't provide one. (i.e. not GET or HEAD) + self.headers["Content-Length"] = "0" + + def prepare_auth(self, auth, url=""): + """Prepares the given HTTP auth data.""" + + # If no Auth is explicitly provided, extract it from the URL first. + if auth is None: + url_auth = get_auth_from_url(self.url) + auth = url_auth if any(url_auth) else None + + if auth: + if isinstance(auth, tuple) and len(auth) == 2: + # special-case basic HTTP auth + auth = HTTPBasicAuth(*auth) + + # Allow auth to make its changes. + r = auth(self) + + # Update self to reflect the auth changes. + self.__dict__.update(r.__dict__) + + # Recompute Content-Length + self.prepare_content_length(self.body) + + def prepare_cookies(self, cookies): + """Prepares the given HTTP cookie data. + + This function eventually generates a ``Cookie`` header from the + given cookies using cookielib. Due to cookielib's design, the header + will not be regenerated if it already exists, meaning this function + can only be called once for the life of the + :class:`PreparedRequest ` object. Any subsequent calls + to ``prepare_cookies`` will have no actual effect, unless the "Cookie" + header is removed beforehand. + """ + if isinstance(cookies, cookielib.CookieJar): + self._cookies = cookies + else: + self._cookies = cookiejar_from_dict(cookies) + + cookie_header = get_cookie_header(self._cookies, self) + if cookie_header is not None: + self.headers["Cookie"] = cookie_header + + def prepare_hooks(self, hooks): + """Prepares the given hooks.""" + # hooks can be passed as None to the prepare method and to this + # method. To prevent iterating over None, simply use an empty list + # if hooks is False-y + hooks = hooks or [] + for event in hooks: + self.register_hook(event, hooks[event]) + + +class Response: + """The :class:`Response ` object, which contains a + server's response to an HTTP request. + """ + + __attrs__ = [ + "_content", + "status_code", + "headers", + "url", + "history", + "encoding", + "reason", + "cookies", + "elapsed", + "request", + ] + + def __init__(self): + self._content = False + self._content_consumed = False + self._next = None + + #: Integer Code of responded HTTP Status, e.g. 404 or 200. + self.status_code = None + + #: Case-insensitive Dictionary of Response Headers. + #: For example, ``headers['content-encoding']`` will return the + #: value of a ``'Content-Encoding'`` response header. + self.headers = CaseInsensitiveDict() + + #: File-like object representation of response (for advanced usage). + #: Use of ``raw`` requires that ``stream=True`` be set on the request. + #: This requirement does not apply for use internally to Requests. + self.raw = None + + #: Final URL location of Response. + self.url = None + + #: Encoding to decode with when accessing r.text. + self.encoding = None + + #: A list of :class:`Response ` objects from + #: the history of the Request. Any redirect responses will end + #: up here. The list is sorted from the oldest to the most recent request. + self.history = [] + + #: Textual reason of responded HTTP Status, e.g. "Not Found" or "OK". + self.reason = None + + #: A CookieJar of Cookies the server sent back. + self.cookies = cookiejar_from_dict({}) + + #: The amount of time elapsed between sending the request + #: and the arrival of the response (as a timedelta). + #: This property specifically measures the time taken between sending + #: the first byte of the request and finishing parsing the headers. It + #: is therefore unaffected by consuming the response content or the + #: value of the ``stream`` keyword argument. + self.elapsed = datetime.timedelta(0) + + #: The :class:`PreparedRequest ` object to which this + #: is a response. + self.request = None + + def __enter__(self): + return self + + def __exit__(self, *args): + self.close() + + def __getstate__(self): + # Consume everything; accessing the content attribute makes + # sure the content has been fully read. + if not self._content_consumed: + self.content + + return {attr: getattr(self, attr, None) for attr in self.__attrs__} + + def __setstate__(self, state): + for name, value in state.items(): + setattr(self, name, value) + + # pickled objects do not have .raw + setattr(self, "_content_consumed", True) + setattr(self, "raw", None) + + def __repr__(self): + return f"" + + def __bool__(self): + """Returns True if :attr:`status_code` is less than 400. + + This attribute checks if the status code of the response is between + 400 and 600 to see if there was a client error or a server error. If + the status code, is between 200 and 400, this will return True. This + is **not** a check to see if the response code is ``200 OK``. + """ + return self.ok + + def __nonzero__(self): + """Returns True if :attr:`status_code` is less than 400. + + This attribute checks if the status code of the response is between + 400 and 600 to see if there was a client error or a server error. If + the status code, is between 200 and 400, this will return True. This + is **not** a check to see if the response code is ``200 OK``. + """ + return self.ok + + def __iter__(self): + """Allows you to use a response as an iterator.""" + return self.iter_content(128) + + @property + def ok(self): + """Returns True if :attr:`status_code` is less than 400, False if not. + + This attribute checks if the status code of the response is between + 400 and 600 to see if there was a client error or a server error. If + the status code is between 200 and 400, this will return True. This + is **not** a check to see if the response code is ``200 OK``. + """ + try: + self.raise_for_status() + except HTTPError: + return False + return True + + @property + def is_redirect(self): + """True if this Response is a well-formed HTTP redirect that could have + been processed automatically (by :meth:`Session.resolve_redirects`). + """ + return "location" in self.headers and self.status_code in REDIRECT_STATI + + @property + def is_permanent_redirect(self): + """True if this Response one of the permanent versions of redirect.""" + return "location" in self.headers and self.status_code in ( + codes.moved_permanently, + codes.permanent_redirect, + ) + + @property + def next(self): + """Returns a PreparedRequest for the next request in a redirect chain, if there is one.""" + return self._next + + @property + def apparent_encoding(self): + """The apparent encoding, provided by the charset_normalizer or chardet libraries.""" + if chardet is not None: + return chardet.detect(self.content)["encoding"] + else: + # If no character detection library is available, we'll fall back + # to a standard Python utf-8 str. + return "utf-8" + + def iter_content(self, chunk_size=1, decode_unicode=False): + """Iterates over the response data. When stream=True is set on the + request, this avoids reading the content at once into memory for + large responses. The chunk size is the number of bytes it should + read into memory. This is not necessarily the length of each item + returned as decoding can take place. + + chunk_size must be of type int or None. A value of None will + function differently depending on the value of `stream`. + stream=True will read data as it arrives in whatever size the + chunks are received. If stream=False, data is returned as + a single chunk. + + If decode_unicode is True, content will be decoded using the best + available encoding based on the response. + """ + + def generate(): + # Special case for urllib3. + if hasattr(self.raw, "stream"): + try: + yield from self.raw.stream(chunk_size, decode_content=True) + except ProtocolError as e: + raise ChunkedEncodingError(e) + except DecodeError as e: + raise ContentDecodingError(e) + except ReadTimeoutError as e: + raise ConnectionError(e) + except SSLError as e: + raise RequestsSSLError(e) + else: + # Standard file-like object. + while True: + chunk = self.raw.read(chunk_size) + if not chunk: + break + yield chunk + + self._content_consumed = True + + if self._content_consumed and isinstance(self._content, bool): + raise StreamConsumedError() + elif chunk_size is not None and not isinstance(chunk_size, int): + raise TypeError( + f"chunk_size must be an int, it is instead a {type(chunk_size)}." + ) + # simulate reading small chunks of the content + reused_chunks = iter_slices(self._content, chunk_size) + + stream_chunks = generate() + + chunks = reused_chunks if self._content_consumed else stream_chunks + + if decode_unicode: + chunks = stream_decode_response_unicode(chunks, self) + + return chunks + + def iter_lines( + self, chunk_size=ITER_CHUNK_SIZE, decode_unicode=False, delimiter=None + ): + """Iterates over the response data, one line at a time. When + stream=True is set on the request, this avoids reading the + content at once into memory for large responses. + + .. note:: This method is not reentrant safe. + """ + + pending = None + + for chunk in self.iter_content( + chunk_size=chunk_size, decode_unicode=decode_unicode + ): + if pending is not None: + chunk = pending + chunk + + if delimiter: + lines = chunk.split(delimiter) + else: + lines = chunk.splitlines() + + if lines and lines[-1] and chunk and lines[-1][-1] == chunk[-1]: + pending = lines.pop() + else: + pending = None + + yield from lines + + if pending is not None: + yield pending + + @property + def content(self): + """Content of the response, in bytes.""" + + if self._content is False: + # Read the contents. + if self._content_consumed: + raise RuntimeError("The content for this response was already consumed") + + if self.status_code == 0 or self.raw is None: + self._content = None + else: + self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b"" + + self._content_consumed = True + # don't need to release the connection; that's been handled by urllib3 + # since we exhausted the data. + return self._content + + @property + def text(self): + """Content of the response, in unicode. + + If Response.encoding is None, encoding will be guessed using + ``charset_normalizer`` or ``chardet``. + + The encoding of the response content is determined based solely on HTTP + headers, following RFC 2616 to the letter. If you can take advantage of + non-HTTP knowledge to make a better guess at the encoding, you should + set ``r.encoding`` appropriately before accessing this property. + """ + + # Try charset from content-type + content = None + encoding = self.encoding + + if not self.content: + return "" + + # Fallback to auto-detected encoding. + if self.encoding is None: + encoding = self.apparent_encoding + + # Decode unicode from given encoding. + try: + content = str(self.content, encoding, errors="replace") + except (LookupError, TypeError): + # A LookupError is raised if the encoding was not found which could + # indicate a misspelling or similar mistake. + # + # A TypeError can be raised if encoding is None + # + # So we try blindly encoding. + content = str(self.content, errors="replace") + + return content + + def json(self, **kwargs): + r"""Decodes the JSON response body (if any) as a Python object. + + This may return a dictionary, list, etc. depending on what is in the response. + + :param \*\*kwargs: Optional arguments that ``json.loads`` takes. + :raises requests.exceptions.JSONDecodeError: If the response body does not + contain valid json. + """ + + if not self.encoding and self.content and len(self.content) > 3: + # No encoding set. JSON RFC 4627 section 3 states we should expect + # UTF-8, -16 or -32. Detect which one to use; If the detection or + # decoding fails, fall back to `self.text` (using charset_normalizer to make + # a best guess). + encoding = guess_json_utf(self.content) + if encoding is not None: + try: + return complexjson.loads(self.content.decode(encoding), **kwargs) + except UnicodeDecodeError: + # Wrong UTF codec detected; usually because it's not UTF-8 + # but some other 8-bit codec. This is an RFC violation, + # and the server didn't bother to tell us what codec *was* + # used. + pass + except JSONDecodeError as e: + raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) + + try: + return complexjson.loads(self.text, **kwargs) + except JSONDecodeError as e: + # Catch JSON-related errors and raise as requests.JSONDecodeError + # This aliases json.JSONDecodeError and simplejson.JSONDecodeError + raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) + + @property + def links(self): + """Returns the parsed header links of the response, if any.""" + + header = self.headers.get("link") + + resolved_links = {} + + if header: + links = parse_header_links(header) + + for link in links: + key = link.get("rel") or link.get("url") + resolved_links[key] = link + + return resolved_links + + def raise_for_status(self): + """Raises :class:`HTTPError`, if one occurred.""" + + http_error_msg = "" + if isinstance(self.reason, bytes): + # We attempt to decode utf-8 first because some servers + # choose to localize their reason strings. If the string + # isn't utf-8, we fall back to iso-8859-1 for all other + # encodings. (See PR #3538) + try: + reason = self.reason.decode("utf-8") + except UnicodeDecodeError: + reason = self.reason.decode("iso-8859-1") + else: + reason = self.reason + + if 400 <= self.status_code < 500: + http_error_msg = ( + f"{self.status_code} Client Error: {reason} for url: {self.url}" + ) + + elif 500 <= self.status_code < 600: + http_error_msg = ( + f"{self.status_code} Server Error: {reason} for url: {self.url}" + ) + + if http_error_msg: + raise HTTPError(http_error_msg, response=self) + + def close(self): + """Releases the connection back to the pool. Once this method has been + called the underlying ``raw`` object must not be accessed again. + + *Note: Should not normally need to be called explicitly.* + """ + if not self._content_consumed: + self.raw.close() + + release_conn = getattr(self.raw, "release_conn", None) + if release_conn is not None: + release_conn() diff --git a/examples/packages/requests/source/requests/packages.py b/examples/packages/requests/source/requests/packages.py new file mode 100644 index 0000000..5ab3d8e --- /dev/null +++ b/examples/packages/requests/source/requests/packages.py @@ -0,0 +1,23 @@ +import sys + +from .compat import chardet + +# This code exists for backwards compatibility reasons. +# I don't like it either. Just look the other way. :) + +for package in ("urllib3", "idna"): + locals()[package] = __import__(package) + # This traversal is apparently necessary such that the identities are + # preserved (requests.packages.urllib3.* is urllib3.*) + for mod in list(sys.modules): + if mod == package or mod.startswith(f"{package}."): + sys.modules[f"requests.packages.{mod}"] = sys.modules[mod] + +if chardet is not None: + target = chardet.__name__ + for mod in list(sys.modules): + if mod == target or mod.startswith(f"{target}."): + imported_mod = sys.modules[mod] + sys.modules[f"requests.packages.{mod}"] = imported_mod + mod = mod.replace(target, "chardet") + sys.modules[f"requests.packages.{mod}"] = imported_mod diff --git a/examples/packages/requests/source/requests/sessions.py b/examples/packages/requests/source/requests/sessions.py new file mode 100644 index 0000000..731550d --- /dev/null +++ b/examples/packages/requests/source/requests/sessions.py @@ -0,0 +1,831 @@ +""" +requests.sessions +~~~~~~~~~~~~~~~~~ + +This module provides a Session object to manage and persist settings across +requests (cookies, auth, proxies). +""" +import os +import sys +import time +from collections import OrderedDict +from datetime import timedelta + +from ._internal_utils import to_native_string +from .adapters import HTTPAdapter +from .auth import _basic_auth_str +from .compat import Mapping, cookielib, urljoin, urlparse +from .cookies import ( + RequestsCookieJar, + cookiejar_from_dict, + extract_cookies_to_jar, + merge_cookies, +) +from .exceptions import ( + ChunkedEncodingError, + ContentDecodingError, + InvalidSchema, + TooManyRedirects, +) +from .hooks import default_hooks, dispatch_hook + +# formerly defined here, reexposed here for backward compatibility +from .models import ( # noqa: F401 + DEFAULT_REDIRECT_LIMIT, + REDIRECT_STATI, + PreparedRequest, + Request, +) +from .status_codes import codes +from .structures import CaseInsensitiveDict +from .utils import ( # noqa: F401 + DEFAULT_PORTS, + default_headers, + get_auth_from_url, + get_environ_proxies, + get_netrc_auth, + requote_uri, + resolve_proxies, + rewind_body, + should_bypass_proxies, + to_key_val_list, +) + +# Preferred clock, based on which one is more accurate on a given system. +if sys.platform == "win32": + preferred_clock = time.perf_counter +else: + preferred_clock = time.time + + +def merge_setting(request_setting, session_setting, dict_class=OrderedDict): + """Determines appropriate setting for a given request, taking into account + the explicit setting on that request, and the setting in the session. If a + setting is a dictionary, they will be merged together using `dict_class` + """ + + if session_setting is None: + return request_setting + + if request_setting is None: + return session_setting + + # Bypass if not a dictionary (e.g. verify) + if not ( + isinstance(session_setting, Mapping) and isinstance(request_setting, Mapping) + ): + return request_setting + + merged_setting = dict_class(to_key_val_list(session_setting)) + merged_setting.update(to_key_val_list(request_setting)) + + # Remove keys that are set to None. Extract keys first to avoid altering + # the dictionary during iteration. + none_keys = [k for (k, v) in merged_setting.items() if v is None] + for key in none_keys: + del merged_setting[key] + + return merged_setting + + +def merge_hooks(request_hooks, session_hooks, dict_class=OrderedDict): + """Properly merges both requests and session hooks. + + This is necessary because when request_hooks == {'response': []}, the + merge breaks Session hooks entirely. + """ + if session_hooks is None or session_hooks.get("response") == []: + return request_hooks + + if request_hooks is None or request_hooks.get("response") == []: + return session_hooks + + return merge_setting(request_hooks, session_hooks, dict_class) + + +class SessionRedirectMixin: + def get_redirect_target(self, resp): + """Receives a Response. Returns a redirect URI or ``None``""" + # Due to the nature of how requests processes redirects this method will + # be called at least once upon the original response and at least twice + # on each subsequent redirect response (if any). + # If a custom mixin is used to handle this logic, it may be advantageous + # to cache the redirect location onto the response object as a private + # attribute. + if resp.is_redirect: + location = resp.headers["location"] + # Currently the underlying http module on py3 decode headers + # in latin1, but empirical evidence suggests that latin1 is very + # rarely used with non-ASCII characters in HTTP headers. + # It is more likely to get UTF8 header rather than latin1. + # This causes incorrect handling of UTF8 encoded location headers. + # To solve this, we re-encode the location in latin1. + location = location.encode("latin1") + return to_native_string(location, "utf8") + return None + + def should_strip_auth(self, old_url, new_url): + """Decide whether Authorization header should be removed when redirecting""" + old_parsed = urlparse(old_url) + new_parsed = urlparse(new_url) + if old_parsed.hostname != new_parsed.hostname: + return True + # Special case: allow http -> https redirect when using the standard + # ports. This isn't specified by RFC 7235, but is kept to avoid + # breaking backwards compatibility with older versions of requests + # that allowed any redirects on the same host. + if ( + old_parsed.scheme == "http" + and old_parsed.port in (80, None) + and new_parsed.scheme == "https" + and new_parsed.port in (443, None) + ): + return False + + # Handle default port usage corresponding to scheme. + changed_port = old_parsed.port != new_parsed.port + changed_scheme = old_parsed.scheme != new_parsed.scheme + default_port = (DEFAULT_PORTS.get(old_parsed.scheme, None), None) + if ( + not changed_scheme + and old_parsed.port in default_port + and new_parsed.port in default_port + ): + return False + + # Standard case: root URI must match + return changed_port or changed_scheme + + def resolve_redirects( + self, + resp, + req, + stream=False, + timeout=None, + verify=True, + cert=None, + proxies=None, + yield_requests=False, + **adapter_kwargs, + ): + """Receives a Response. Returns a generator of Responses or Requests.""" + + hist = [] # keep track of history + + url = self.get_redirect_target(resp) + previous_fragment = urlparse(req.url).fragment + while url: + prepared_request = req.copy() + + # Update history and keep track of redirects. + # resp.history must ignore the original request in this loop + hist.append(resp) + resp.history = hist[1:] + + try: + resp.content # Consume socket so it can be released + except (ChunkedEncodingError, ContentDecodingError, RuntimeError): + resp.raw.read(decode_content=False) + + if len(resp.history) >= self.max_redirects: + raise TooManyRedirects( + f"Exceeded {self.max_redirects} redirects.", response=resp + ) + + # Release the connection back into the pool. + resp.close() + + # Handle redirection without scheme (see: RFC 1808 Section 4) + if url.startswith("//"): + parsed_rurl = urlparse(resp.url) + url = ":".join([to_native_string(parsed_rurl.scheme), url]) + + # Normalize url case and attach previous fragment if needed (RFC 7231 7.1.2) + parsed = urlparse(url) + if parsed.fragment == "" and previous_fragment: + parsed = parsed._replace(fragment=previous_fragment) + elif parsed.fragment: + previous_fragment = parsed.fragment + url = parsed.geturl() + + # Facilitate relative 'location' headers, as allowed by RFC 7231. + # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource') + # Compliant with RFC3986, we percent encode the url. + if not parsed.netloc: + url = urljoin(resp.url, requote_uri(url)) + else: + url = requote_uri(url) + + prepared_request.url = to_native_string(url) + + self.rebuild_method(prepared_request, resp) + + # https://github.com/psf/requests/issues/1084 + if resp.status_code not in ( + codes.temporary_redirect, + codes.permanent_redirect, + ): + # https://github.com/psf/requests/issues/3490 + purged_headers = ("Content-Length", "Content-Type", "Transfer-Encoding") + for header in purged_headers: + prepared_request.headers.pop(header, None) + prepared_request.body = None + + headers = prepared_request.headers + headers.pop("Cookie", None) + + # Extract any cookies sent on the response to the cookiejar + # in the new request. Because we've mutated our copied prepared + # request, use the old one that we haven't yet touched. + extract_cookies_to_jar(prepared_request._cookies, req, resp.raw) + merge_cookies(prepared_request._cookies, self.cookies) + prepared_request.prepare_cookies(prepared_request._cookies) + + # Rebuild auth and proxy information. + proxies = self.rebuild_proxies(prepared_request, proxies) + self.rebuild_auth(prepared_request, resp) + + # A failed tell() sets `_body_position` to `object()`. This non-None + # value ensures `rewindable` will be True, allowing us to raise an + # UnrewindableBodyError, instead of hanging the connection. + rewindable = prepared_request._body_position is not None and ( + "Content-Length" in headers or "Transfer-Encoding" in headers + ) + + # Attempt to rewind consumed file-like object. + if rewindable: + rewind_body(prepared_request) + + # Override the original request. + req = prepared_request + + if yield_requests: + yield req + else: + resp = self.send( + req, + stream=stream, + timeout=timeout, + verify=verify, + cert=cert, + proxies=proxies, + allow_redirects=False, + **adapter_kwargs, + ) + + extract_cookies_to_jar(self.cookies, prepared_request, resp.raw) + + # extract redirect url, if any, for the next loop + url = self.get_redirect_target(resp) + yield resp + + def rebuild_auth(self, prepared_request, response): + """When being redirected we may want to strip authentication from the + request to avoid leaking credentials. This method intelligently removes + and reapplies authentication where possible to avoid credential loss. + """ + headers = prepared_request.headers + url = prepared_request.url + + if "Authorization" in headers and self.should_strip_auth( + response.request.url, url + ): + # If we get redirected to a new host, we should strip out any + # authentication headers. + del headers["Authorization"] + + # .netrc might have more auth for us on our new host. + new_auth = get_netrc_auth(url) if self.trust_env else None + if new_auth is not None: + prepared_request.prepare_auth(new_auth) + + def rebuild_proxies(self, prepared_request, proxies): + """This method re-evaluates the proxy configuration by considering the + environment variables. If we are redirected to a URL covered by + NO_PROXY, we strip the proxy configuration. Otherwise, we set missing + proxy keys for this URL (in case they were stripped by a previous + redirect). + + This method also replaces the Proxy-Authorization header where + necessary. + + :rtype: dict + """ + headers = prepared_request.headers + scheme = urlparse(prepared_request.url).scheme + new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env) + + if "Proxy-Authorization" in headers: + del headers["Proxy-Authorization"] + + try: + username, password = get_auth_from_url(new_proxies[scheme]) + except KeyError: + username, password = None, None + + # urllib3 handles proxy authorization for us in the standard adapter. + # Avoid appending this to TLS tunneled requests where it may be leaked. + if not scheme.startswith("https") and username and password: + headers["Proxy-Authorization"] = _basic_auth_str(username, password) + + return new_proxies + + def rebuild_method(self, prepared_request, response): + """When being redirected we may want to change the method of the request + based on certain specs or browser behavior. + """ + method = prepared_request.method + + # https://tools.ietf.org/html/rfc7231#section-6.4.4 + if response.status_code == codes.see_other and method != "HEAD": + method = "GET" + + # Do what the browsers do, despite standards... + # First, turn 302s into GETs. + if response.status_code == codes.found and method != "HEAD": + method = "GET" + + # Second, if a POST is responded to with a 301, turn it into a GET. + # This bizarre behaviour is explained in Issue 1704. + if response.status_code == codes.moved and method == "POST": + method = "GET" + + prepared_request.method = method + + +class Session(SessionRedirectMixin): + """A Requests session. + + Provides cookie persistence, connection-pooling, and configuration. + + Basic Usage:: + + >>> import requests + >>> s = requests.Session() + >>> s.get('https://httpbin.org/get') + + + Or as a context manager:: + + >>> with requests.Session() as s: + ... s.get('https://httpbin.org/get') + + """ + + __attrs__ = [ + "headers", + "cookies", + "auth", + "proxies", + "hooks", + "params", + "verify", + "cert", + "adapters", + "stream", + "trust_env", + "max_redirects", + ] + + def __init__(self): + #: A case-insensitive dictionary of headers to be sent on each + #: :class:`Request ` sent from this + #: :class:`Session `. + self.headers = default_headers() + + #: Default Authentication tuple or object to attach to + #: :class:`Request `. + self.auth = None + + #: Dictionary mapping protocol or protocol and host to the URL of the proxy + #: (e.g. {'http': 'foo.bar:3128', 'http://host.name': 'foo.bar:4012'}) to + #: be used on each :class:`Request `. + self.proxies = {} + + #: Event-handling hooks. + self.hooks = default_hooks() + + #: Dictionary of querystring data to attach to each + #: :class:`Request `. The dictionary values may be lists for + #: representing multivalued query parameters. + self.params = {} + + #: Stream response content default. + self.stream = False + + #: SSL Verification default. + #: Defaults to `True`, requiring requests to verify the TLS certificate at the + #: remote end. + #: If verify is set to `False`, requests will accept any TLS certificate + #: presented by the server, and will ignore hostname mismatches and/or + #: expired certificates, which will make your application vulnerable to + #: man-in-the-middle (MitM) attacks. + #: Only set this to `False` for testing. + self.verify = True + + #: SSL client certificate default, if String, path to ssl client + #: cert file (.pem). If Tuple, ('cert', 'key') pair. + self.cert = None + + #: Maximum number of redirects allowed. If the request exceeds this + #: limit, a :class:`TooManyRedirects` exception is raised. + #: This defaults to requests.models.DEFAULT_REDIRECT_LIMIT, which is + #: 30. + self.max_redirects = DEFAULT_REDIRECT_LIMIT + + #: Trust environment settings for proxy configuration, default + #: authentication and similar. + self.trust_env = True + + #: A CookieJar containing all currently outstanding cookies set on this + #: session. By default it is a + #: :class:`RequestsCookieJar `, but + #: may be any other ``cookielib.CookieJar`` compatible object. + self.cookies = cookiejar_from_dict({}) + + # Default connection adapters. + self.adapters = OrderedDict() + self.mount("https://", HTTPAdapter()) + self.mount("http://", HTTPAdapter()) + + def __enter__(self): + return self + + def __exit__(self, *args): + self.close() + + def prepare_request(self, request): + """Constructs a :class:`PreparedRequest ` for + transmission and returns it. The :class:`PreparedRequest` has settings + merged from the :class:`Request ` instance and those of the + :class:`Session`. + + :param request: :class:`Request` instance to prepare with this + session's settings. + :rtype: requests.PreparedRequest + """ + cookies = request.cookies or {} + + # Bootstrap CookieJar. + if not isinstance(cookies, cookielib.CookieJar): + cookies = cookiejar_from_dict(cookies) + + # Merge with session cookies + merged_cookies = merge_cookies( + merge_cookies(RequestsCookieJar(), self.cookies), cookies + ) + + # Set environment's basic authentication if not explicitly set. + auth = request.auth + if self.trust_env and not auth and not self.auth: + auth = get_netrc_auth(request.url) + + p = PreparedRequest() + p.prepare( + method=request.method.upper(), + url=request.url, + files=request.files, + data=request.data, + json=request.json, + headers=merge_setting( + request.headers, self.headers, dict_class=CaseInsensitiveDict + ), + params=merge_setting(request.params, self.params), + auth=merge_setting(auth, self.auth), + cookies=merged_cookies, + hooks=merge_hooks(request.hooks, self.hooks), + ) + return p + + def request( + self, + method, + url, + params=None, + data=None, + headers=None, + cookies=None, + files=None, + auth=None, + timeout=None, + allow_redirects=True, + proxies=None, + hooks=None, + stream=None, + verify=None, + cert=None, + json=None, + ): + """Constructs a :class:`Request `, prepares it and sends it. + Returns :class:`Response ` object. + + :param method: method for the new :class:`Request` object. + :param url: URL for the new :class:`Request` object. + :param params: (optional) Dictionary or bytes to be sent in the query + string for the :class:`Request`. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param json: (optional) json to send in the body of the + :class:`Request`. + :param headers: (optional) Dictionary of HTTP Headers to send with the + :class:`Request`. + :param cookies: (optional) Dict or CookieJar object to send with the + :class:`Request`. + :param files: (optional) Dictionary of ``'filename': file-like-objects`` + for multipart encoding upload. + :param auth: (optional) Auth tuple or callable to enable + Basic/Digest/Custom HTTP Auth. + :param timeout: (optional) How many seconds to wait for the server to send + data before giving up, as a float, or a :ref:`(connect timeout, + read timeout) ` tuple. + :type timeout: float or tuple + :param allow_redirects: (optional) Set to True by default. + :type allow_redirects: bool + :param proxies: (optional) Dictionary mapping protocol or protocol and + hostname to the URL of the proxy. + :param hooks: (optional) Dictionary mapping hook name to one event or + list of events, event must be callable. + :param stream: (optional) whether to immediately download the response + content. Defaults to ``False``. + :param verify: (optional) Either a boolean, in which case it controls whether we verify + the server's TLS certificate, or a string, in which case it must be a path + to a CA bundle to use. Defaults to ``True``. When set to + ``False``, requests will accept any TLS certificate presented by + the server, and will ignore hostname mismatches and/or expired + certificates, which will make your application vulnerable to + man-in-the-middle (MitM) attacks. Setting verify to ``False`` + may be useful during local development or testing. + :param cert: (optional) if String, path to ssl client cert file (.pem). + If Tuple, ('cert', 'key') pair. + :rtype: requests.Response + """ + # Create the Request. + req = Request( + method=method.upper(), + url=url, + headers=headers, + files=files, + data=data or {}, + json=json, + params=params or {}, + auth=auth, + cookies=cookies, + hooks=hooks, + ) + prep = self.prepare_request(req) + + proxies = proxies or {} + + settings = self.merge_environment_settings( + prep.url, proxies, stream, verify, cert + ) + + # Send the request. + send_kwargs = { + "timeout": timeout, + "allow_redirects": allow_redirects, + } + send_kwargs.update(settings) + resp = self.send(prep, **send_kwargs) + + return resp + + def get(self, url, **kwargs): + r"""Sends a GET request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + kwargs.setdefault("allow_redirects", True) + return self.request("GET", url, **kwargs) + + def options(self, url, **kwargs): + r"""Sends a OPTIONS request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + kwargs.setdefault("allow_redirects", True) + return self.request("OPTIONS", url, **kwargs) + + def head(self, url, **kwargs): + r"""Sends a HEAD request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + kwargs.setdefault("allow_redirects", False) + return self.request("HEAD", url, **kwargs) + + def post(self, url, data=None, json=None, **kwargs): + r"""Sends a POST request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param json: (optional) json to send in the body of the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + return self.request("POST", url, data=data, json=json, **kwargs) + + def put(self, url, data=None, **kwargs): + r"""Sends a PUT request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + return self.request("PUT", url, data=data, **kwargs) + + def patch(self, url, data=None, **kwargs): + r"""Sends a PATCH request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param data: (optional) Dictionary, list of tuples, bytes, or file-like + object to send in the body of the :class:`Request`. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + return self.request("PATCH", url, data=data, **kwargs) + + def delete(self, url, **kwargs): + r"""Sends a DELETE request. Returns :class:`Response` object. + + :param url: URL for the new :class:`Request` object. + :param \*\*kwargs: Optional arguments that ``request`` takes. + :rtype: requests.Response + """ + + return self.request("DELETE", url, **kwargs) + + def send(self, request, **kwargs): + """Send a given PreparedRequest. + + :rtype: requests.Response + """ + # Set defaults that the hooks can utilize to ensure they always have + # the correct parameters to reproduce the previous request. + kwargs.setdefault("stream", self.stream) + kwargs.setdefault("verify", self.verify) + kwargs.setdefault("cert", self.cert) + if "proxies" not in kwargs: + kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env) + + # It's possible that users might accidentally send a Request object. + # Guard against that specific failure case. + if isinstance(request, Request): + raise ValueError("You can only send PreparedRequests.") + + # Set up variables needed for resolve_redirects and dispatching of hooks + allow_redirects = kwargs.pop("allow_redirects", True) + stream = kwargs.get("stream") + hooks = request.hooks + + # Get the appropriate adapter to use + adapter = self.get_adapter(url=request.url) + + # Start time (approximately) of the request + start = preferred_clock() + + # Send the request + r = adapter.send(request, **kwargs) + + # Total elapsed time of the request (approximately) + elapsed = preferred_clock() - start + r.elapsed = timedelta(seconds=elapsed) + + # Response manipulation hooks + r = dispatch_hook("response", hooks, r, **kwargs) + + # Persist cookies + if r.history: + # If the hooks create history then we want those cookies too + for resp in r.history: + extract_cookies_to_jar(self.cookies, resp.request, resp.raw) + + extract_cookies_to_jar(self.cookies, request, r.raw) + + # Resolve redirects if allowed. + if allow_redirects: + # Redirect resolving generator. + gen = self.resolve_redirects(r, request, **kwargs) + history = [resp for resp in gen] + else: + history = [] + + # Shuffle things around if there's history. + if history: + # Insert the first (original) request at the start + history.insert(0, r) + # Get the last request made + r = history.pop() + r.history = history + + # If redirects aren't being followed, store the response on the Request for Response.next(). + if not allow_redirects: + try: + r._next = next( + self.resolve_redirects(r, request, yield_requests=True, **kwargs) + ) + except StopIteration: + pass + + if not stream: + r.content + + return r + + def merge_environment_settings(self, url, proxies, stream, verify, cert): + """ + Check the environment and merge it with some settings. + + :rtype: dict + """ + # Gather clues from the surrounding environment. + if self.trust_env: + # Set environment's proxies. + no_proxy = proxies.get("no_proxy") if proxies is not None else None + env_proxies = get_environ_proxies(url, no_proxy=no_proxy) + for k, v in env_proxies.items(): + proxies.setdefault(k, v) + + # Look for requests environment configuration + # and be compatible with cURL. + if verify is True or verify is None: + verify = ( + os.environ.get("REQUESTS_CA_BUNDLE") + or os.environ.get("CURL_CA_BUNDLE") + or verify + ) + + # Merge all the kwargs. + proxies = merge_setting(proxies, self.proxies) + stream = merge_setting(stream, self.stream) + verify = merge_setting(verify, self.verify) + cert = merge_setting(cert, self.cert) + + return {"proxies": proxies, "stream": stream, "verify": verify, "cert": cert} + + def get_adapter(self, url): + """ + Returns the appropriate connection adapter for the given URL. + + :rtype: requests.adapters.BaseAdapter + """ + for prefix, adapter in self.adapters.items(): + if url.lower().startswith(prefix.lower()): + return adapter + + # Nothing matches :-/ + raise InvalidSchema(f"No connection adapters were found for {url!r}") + + def close(self): + """Closes all adapters and as such the session""" + for v in self.adapters.values(): + v.close() + + def mount(self, prefix, adapter): + """Registers a connection adapter to a prefix. + + Adapters are sorted in descending order by prefix length. + """ + self.adapters[prefix] = adapter + keys_to_move = [k for k in self.adapters if len(k) < len(prefix)] + + for key in keys_to_move: + self.adapters[key] = self.adapters.pop(key) + + def __getstate__(self): + state = {attr: getattr(self, attr, None) for attr in self.__attrs__} + return state + + def __setstate__(self, state): + for attr, value in state.items(): + setattr(self, attr, value) + + +def session(): + """ + Returns a :class:`Session` for context-management. + + .. deprecated:: 1.0.0 + + This method has been deprecated since version 1.0.0 and is only kept for + backwards compatibility. New code should use :class:`~requests.sessions.Session` + to create a session. This may be removed at a future date. + + :rtype: Session + """ + return Session() diff --git a/examples/packages/requests/source/requests/status_codes.py b/examples/packages/requests/source/requests/status_codes.py new file mode 100644 index 0000000..c7945a2 --- /dev/null +++ b/examples/packages/requests/source/requests/status_codes.py @@ -0,0 +1,128 @@ +r""" +The ``codes`` object defines a mapping from common names for HTTP statuses +to their numerical codes, accessible either as attributes or as dictionary +items. + +Example:: + + >>> import requests + >>> requests.codes['temporary_redirect'] + 307 + >>> requests.codes.teapot + 418 + >>> requests.codes['\o/'] + 200 + +Some codes have multiple names, and both upper- and lower-case versions of +the names are allowed. For example, ``codes.ok``, ``codes.OK``, and +``codes.okay`` all correspond to the HTTP status code 200. +""" + +from .structures import LookupDict + +_codes = { + # Informational. + 100: ("continue",), + 101: ("switching_protocols",), + 102: ("processing", "early-hints"), + 103: ("checkpoint",), + 122: ("uri_too_long", "request_uri_too_long"), + 200: ("ok", "okay", "all_ok", "all_okay", "all_good", "\\o/", "✓"), + 201: ("created",), + 202: ("accepted",), + 203: ("non_authoritative_info", "non_authoritative_information"), + 204: ("no_content",), + 205: ("reset_content", "reset"), + 206: ("partial_content", "partial"), + 207: ("multi_status", "multiple_status", "multi_stati", "multiple_stati"), + 208: ("already_reported",), + 226: ("im_used",), + # Redirection. + 300: ("multiple_choices",), + 301: ("moved_permanently", "moved", "\\o-"), + 302: ("found",), + 303: ("see_other", "other"), + 304: ("not_modified",), + 305: ("use_proxy",), + 306: ("switch_proxy",), + 307: ("temporary_redirect", "temporary_moved", "temporary"), + 308: ( + "permanent_redirect", + "resume_incomplete", + "resume", + ), # "resume" and "resume_incomplete" to be removed in 3.0 + # Client Error. + 400: ("bad_request", "bad"), + 401: ("unauthorized",), + 402: ("payment_required", "payment"), + 403: ("forbidden",), + 404: ("not_found", "-o-"), + 405: ("method_not_allowed", "not_allowed"), + 406: ("not_acceptable",), + 407: ("proxy_authentication_required", "proxy_auth", "proxy_authentication"), + 408: ("request_timeout", "timeout"), + 409: ("conflict",), + 410: ("gone",), + 411: ("length_required",), + 412: ("precondition_failed", "precondition"), + 413: ("request_entity_too_large", "content_too_large"), + 414: ("request_uri_too_large", "uri_too_long"), + 415: ("unsupported_media_type", "unsupported_media", "media_type"), + 416: ( + "requested_range_not_satisfiable", + "requested_range", + "range_not_satisfiable", + ), + 417: ("expectation_failed",), + 418: ("im_a_teapot", "teapot", "i_am_a_teapot"), + 421: ("misdirected_request",), + 422: ("unprocessable_entity", "unprocessable", "unprocessable_content"), + 423: ("locked",), + 424: ("failed_dependency", "dependency"), + 425: ("unordered_collection", "unordered", "too_early"), + 426: ("upgrade_required", "upgrade"), + 428: ("precondition_required", "precondition"), + 429: ("too_many_requests", "too_many"), + 431: ("header_fields_too_large", "fields_too_large"), + 444: ("no_response", "none"), + 449: ("retry_with", "retry"), + 450: ("blocked_by_windows_parental_controls", "parental_controls"), + 451: ("unavailable_for_legal_reasons", "legal_reasons"), + 499: ("client_closed_request",), + # Server Error. + 500: ("internal_server_error", "server_error", "/o\\", "✗"), + 501: ("not_implemented",), + 502: ("bad_gateway",), + 503: ("service_unavailable", "unavailable"), + 504: ("gateway_timeout",), + 505: ("http_version_not_supported", "http_version"), + 506: ("variant_also_negotiates",), + 507: ("insufficient_storage",), + 509: ("bandwidth_limit_exceeded", "bandwidth"), + 510: ("not_extended",), + 511: ("network_authentication_required", "network_auth", "network_authentication"), +} + +codes = LookupDict(name="status_codes") + + +def _init(): + for code, titles in _codes.items(): + for title in titles: + setattr(codes, title, code) + if not title.startswith(("\\", "/")): + setattr(codes, title.upper(), code) + + def doc(code): + names = ", ".join(f"``{n}``" for n in _codes[code]) + return "* %d: %s" % (code, names) + + global __doc__ + __doc__ = ( + __doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes)) + if __doc__ is not None + else None + ) + + +_init() diff --git a/examples/packages/requests/source/requests/structures.py b/examples/packages/requests/source/requests/structures.py new file mode 100644 index 0000000..188e13e --- /dev/null +++ b/examples/packages/requests/source/requests/structures.py @@ -0,0 +1,99 @@ +""" +requests.structures +~~~~~~~~~~~~~~~~~~~ + +Data structures that power Requests. +""" + +from collections import OrderedDict + +from .compat import Mapping, MutableMapping + + +class CaseInsensitiveDict(MutableMapping): + """A case-insensitive ``dict``-like object. + + Implements all methods and operations of + ``MutableMapping`` as well as dict's ``copy``. Also + provides ``lower_items``. + + All keys are expected to be strings. The structure remembers the + case of the last key to be set, and ``iter(instance)``, + ``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()`` + will contain case-sensitive keys. However, querying and contains + testing is case insensitive:: + + cid = CaseInsensitiveDict() + cid['Accept'] = 'application/json' + cid['aCCEPT'] == 'application/json' # True + list(cid) == ['Accept'] # True + + For example, ``headers['content-encoding']`` will return the + value of a ``'Content-Encoding'`` response header, regardless + of how the header name was originally stored. + + If the constructor, ``.update``, or equality comparison + operations are given keys that have equal ``.lower()``s, the + behavior is undefined. + """ + + def __init__(self, data=None, **kwargs): + self._store = OrderedDict() + if data is None: + data = {} + self.update(data, **kwargs) + + def __setitem__(self, key, value): + # Use the lowercased key for lookups, but store the actual + # key alongside the value. + self._store[key.lower()] = (key, value) + + def __getitem__(self, key): + return self._store[key.lower()][1] + + def __delitem__(self, key): + del self._store[key.lower()] + + def __iter__(self): + return (casedkey for casedkey, mappedvalue in self._store.values()) + + def __len__(self): + return len(self._store) + + def lower_items(self): + """Like iteritems(), but with all lowercase keys.""" + return ((lowerkey, keyval[1]) for (lowerkey, keyval) in self._store.items()) + + def __eq__(self, other): + if isinstance(other, Mapping): + other = CaseInsensitiveDict(other) + else: + return NotImplemented + # Compare insensitively + return dict(self.lower_items()) == dict(other.lower_items()) + + # Copy is required + def copy(self): + return CaseInsensitiveDict(self._store.values()) + + def __repr__(self): + return str(dict(self.items())) + + +class LookupDict(dict): + """Dictionary lookup object.""" + + def __init__(self, name=None): + self.name = name + super().__init__() + + def __repr__(self): + return f"" + + def __getitem__(self, key): + # We allow fall-through here, so values default to None + + return self.__dict__.get(key, None) + + def get(self, key, default=None): + return self.__dict__.get(key, default) diff --git a/examples/packages/requests/source/requests/utils.py b/examples/packages/requests/source/requests/utils.py new file mode 100644 index 0000000..8ab5585 --- /dev/null +++ b/examples/packages/requests/source/requests/utils.py @@ -0,0 +1,1086 @@ +""" +requests.utils +~~~~~~~~~~~~~~ + +This module provides utility functions that are used within Requests +that are also useful for external consumption. +""" + +import codecs +import contextlib +import io +import os +import re +import socket +import struct +import sys +import tempfile +import warnings +import zipfile +from collections import OrderedDict + +from urllib3.util import make_headers, parse_url + +from . import certs +from .__version__ import __version__ + +# to_native_string is unused here, but imported here for backwards compatibility +from ._internal_utils import ( # noqa: F401 + _HEADER_VALIDATORS_BYTE, + _HEADER_VALIDATORS_STR, + HEADER_VALIDATORS, + to_native_string, +) +from .compat import ( + Mapping, + basestring, + bytes, + getproxies, + getproxies_environment, + integer_types, + is_urllib3_1, +) +from .compat import parse_http_list as _parse_list_header +from .compat import ( + proxy_bypass, + proxy_bypass_environment, + quote, + str, + unquote, + urlparse, + urlunparse, +) +from .cookies import cookiejar_from_dict +from .exceptions import ( + FileModeWarning, + InvalidHeader, + InvalidURL, + UnrewindableBodyError, +) +from .structures import CaseInsensitiveDict + +NETRC_FILES = (".netrc", "_netrc") + +DEFAULT_CA_BUNDLE_PATH = certs.where() + +DEFAULT_PORTS = {"http": 80, "https": 443} + +# Ensure that ', ' is used to preserve previous delimiter behavior. +DEFAULT_ACCEPT_ENCODING = ", ".join( + re.split(r",\s*", make_headers(accept_encoding=True)["accept-encoding"]) +) + + +if sys.platform == "win32": + # provide a proxy_bypass version on Windows without DNS lookups + + def proxy_bypass_registry(host): + try: + import winreg + except ImportError: + return False + + try: + internetSettings = winreg.OpenKey( + winreg.HKEY_CURRENT_USER, + r"Software\Microsoft\Windows\CurrentVersion\Internet Settings", + ) + # ProxyEnable could be REG_SZ or REG_DWORD, normalizing it + proxyEnable = int(winreg.QueryValueEx(internetSettings, "ProxyEnable")[0]) + # ProxyOverride is almost always a string + proxyOverride = winreg.QueryValueEx(internetSettings, "ProxyOverride")[0] + except (OSError, ValueError): + return False + if not proxyEnable or not proxyOverride: + return False + + # make a check value list from the registry entry: replace the + # '' string by the localhost entry and the corresponding + # canonical entry. + proxyOverride = proxyOverride.split(";") + # filter out empty strings to avoid re.match return true in the following code. + proxyOverride = filter(None, proxyOverride) + # now check if we match one of the registry values. + for test in proxyOverride: + if test == "": + if "." not in host: + return True + test = test.replace(".", r"\.") # mask dots + test = test.replace("*", r".*") # change glob sequence + test = test.replace("?", r".") # change glob char + if re.match(test, host, re.I): + return True + return False + + def proxy_bypass(host): # noqa + """Return True, if the host should be bypassed. + + Checks proxy settings gathered from the environment, if specified, + or the registry. + """ + if getproxies_environment(): + return proxy_bypass_environment(host) + else: + return proxy_bypass_registry(host) + + +def dict_to_sequence(d): + """Returns an internal sequence dictionary update.""" + + if hasattr(d, "items"): + d = d.items() + + return d + + +def super_len(o): + total_length = None + current_position = 0 + + if not is_urllib3_1 and isinstance(o, str): + # urllib3 2.x+ treats all strings as utf-8 instead + # of latin-1 (iso-8859-1) like http.client. + o = o.encode("utf-8") + + if hasattr(o, "__len__"): + total_length = len(o) + + elif hasattr(o, "len"): + total_length = o.len + + elif hasattr(o, "fileno"): + try: + fileno = o.fileno() + except (io.UnsupportedOperation, AttributeError): + # AttributeError is a surprising exception, seeing as how we've just checked + # that `hasattr(o, 'fileno')`. It happens for objects obtained via + # `Tarfile.extractfile()`, per issue 5229. + pass + else: + total_length = os.fstat(fileno).st_size + + # Having used fstat to determine the file length, we need to + # confirm that this file was opened up in binary mode. + if "b" not in o.mode: + warnings.warn( + ( + "Requests has determined the content-length for this " + "request using the binary size of the file: however, the " + "file has been opened in text mode (i.e. without the 'b' " + "flag in the mode). This may lead to an incorrect " + "content-length. In Requests 3.0, support will be removed " + "for files in text mode." + ), + FileModeWarning, + ) + + if hasattr(o, "tell"): + try: + current_position = o.tell() + except OSError: + # This can happen in some weird situations, such as when the file + # is actually a special file descriptor like stdin. In this + # instance, we don't know what the length is, so set it to zero and + # let requests chunk it instead. + if total_length is not None: + current_position = total_length + else: + if hasattr(o, "seek") and total_length is None: + # StringIO and BytesIO have seek but no usable fileno + try: + # seek to end of file + o.seek(0, 2) + total_length = o.tell() + + # seek back to current position to support + # partially read file-like objects + o.seek(current_position or 0) + except OSError: + total_length = 0 + + if total_length is None: + total_length = 0 + + return max(0, total_length - current_position) + + +def get_netrc_auth(url, raise_errors=False): + """Returns the Requests tuple auth for a given url from netrc.""" + + netrc_file = os.environ.get("NETRC") + if netrc_file is not None: + netrc_locations = (netrc_file,) + else: + netrc_locations = (f"~/{f}" for f in NETRC_FILES) + + try: + from netrc import NetrcParseError, netrc + + netrc_path = None + + for f in netrc_locations: + loc = os.path.expanduser(f) + if os.path.exists(loc): + netrc_path = loc + break + + # Abort early if there isn't one. + if netrc_path is None: + return + + ri = urlparse(url) + host = ri.hostname + + try: + _netrc = netrc(netrc_path).authenticators(host) + if _netrc: + # Return with login / password + login_i = 0 if _netrc[0] else 1 + return (_netrc[login_i], _netrc[2]) + except (NetrcParseError, OSError): + # If there was a parsing error or a permissions issue reading the file, + # we'll just skip netrc auth unless explicitly asked to raise errors. + if raise_errors: + raise + + # App Engine hackiness. + except (ImportError, AttributeError): + pass + + +def guess_filename(obj): + """Tries to guess the filename of the given object.""" + name = getattr(obj, "name", None) + if name and isinstance(name, basestring) and name[0] != "<" and name[-1] != ">": + return os.path.basename(name) + + +def extract_zipped_paths(path): + """Replace nonexistent paths that look like they refer to a member of a zip + archive with the location of an extracted copy of the target, or else + just return the provided path unchanged. + """ + if os.path.exists(path): + # this is already a valid path, no need to do anything further + return path + + # find the first valid part of the provided path and treat that as a zip archive + # assume the rest of the path is the name of a member in the archive + archive, member = os.path.split(path) + while archive and not os.path.exists(archive): + archive, prefix = os.path.split(archive) + if not prefix: + # If we don't check for an empty prefix after the split (in other words, archive remains unchanged after the split), + # we _can_ end up in an infinite loop on a rare corner case affecting a small number of users + break + member = "/".join([prefix, member]) + + if not zipfile.is_zipfile(archive): + return path + + zip_file = zipfile.ZipFile(archive) + if member not in zip_file.namelist(): + return path + + # we have a valid zip archive and a valid member of that archive + tmp = tempfile.gettempdir() + extracted_path = os.path.join(tmp, member.split("/")[-1]) + if not os.path.exists(extracted_path): + # use read + write to avoid the creating nested folders, we only want the file, avoids mkdir racing condition + with atomic_open(extracted_path) as file_handler: + file_handler.write(zip_file.read(member)) + return extracted_path + + +@contextlib.contextmanager +def atomic_open(filename): + """Write a file to the disk in an atomic fashion""" + tmp_descriptor, tmp_name = tempfile.mkstemp(dir=os.path.dirname(filename)) + try: + with os.fdopen(tmp_descriptor, "wb") as tmp_handler: + yield tmp_handler + os.replace(tmp_name, filename) + except BaseException: + os.remove(tmp_name) + raise + + +def from_key_val_list(value): + """Take an object and test to see if it can be represented as a + dictionary. Unless it can not be represented as such, return an + OrderedDict, e.g., + + :: + + >>> from_key_val_list([('key', 'val')]) + OrderedDict([('key', 'val')]) + >>> from_key_val_list('string') + Traceback (most recent call last): + ... + ValueError: cannot encode objects that are not 2-tuples + >>> from_key_val_list({'key': 'val'}) + OrderedDict([('key', 'val')]) + + :rtype: OrderedDict + """ + if value is None: + return None + + if isinstance(value, (str, bytes, bool, int)): + raise ValueError("cannot encode objects that are not 2-tuples") + + return OrderedDict(value) + + +def to_key_val_list(value): + """Take an object and test to see if it can be represented as a + dictionary. If it can be, return a list of tuples, e.g., + + :: + + >>> to_key_val_list([('key', 'val')]) + [('key', 'val')] + >>> to_key_val_list({'key': 'val'}) + [('key', 'val')] + >>> to_key_val_list('string') + Traceback (most recent call last): + ... + ValueError: cannot encode objects that are not 2-tuples + + :rtype: list + """ + if value is None: + return None + + if isinstance(value, (str, bytes, bool, int)): + raise ValueError("cannot encode objects that are not 2-tuples") + + if isinstance(value, Mapping): + value = value.items() + + return list(value) + + +# From mitsuhiko/werkzeug (used with permission). +def parse_list_header(value): + """Parse lists as described by RFC 2068 Section 2. + + In particular, parse comma-separated lists where the elements of + the list may include quoted-strings. A quoted-string could + contain a comma. A non-quoted string could have quotes in the + middle. Quotes are removed automatically after parsing. + + It basically works like :func:`parse_set_header` just that items + may appear multiple times and case sensitivity is preserved. + + The return value is a standard :class:`list`: + + >>> parse_list_header('token, "quoted value"') + ['token', 'quoted value'] + + To create a header from the :class:`list` again, use the + :func:`dump_header` function. + + :param value: a string with a list header. + :return: :class:`list` + :rtype: list + """ + result = [] + for item in _parse_list_header(value): + if item[:1] == item[-1:] == '"': + item = unquote_header_value(item[1:-1]) + result.append(item) + return result + + +# From mitsuhiko/werkzeug (used with permission). +def parse_dict_header(value): + """Parse lists of key, value pairs as described by RFC 2068 Section 2 and + convert them into a python dict: + + >>> d = parse_dict_header('foo="is a fish", bar="as well"') + >>> type(d) is dict + True + >>> sorted(d.items()) + [('bar', 'as well'), ('foo', 'is a fish')] + + If there is no value for a key it will be `None`: + + >>> parse_dict_header('key_without_value') + {'key_without_value': None} + + To create a header from the :class:`dict` again, use the + :func:`dump_header` function. + + :param value: a string with a dict header. + :return: :class:`dict` + :rtype: dict + """ + result = {} + for item in _parse_list_header(value): + if "=" not in item: + result[item] = None + continue + name, value = item.split("=", 1) + if value[:1] == value[-1:] == '"': + value = unquote_header_value(value[1:-1]) + result[name] = value + return result + + +# From mitsuhiko/werkzeug (used with permission). +def unquote_header_value(value, is_filename=False): + r"""Unquotes a header value. (Reversal of :func:`quote_header_value`). + This does not use the real unquoting but what browsers are actually + using for quoting. + + :param value: the header value to unquote. + :rtype: str + """ + if value and value[0] == value[-1] == '"': + # this is not the real unquoting, but fixing this so that the + # RFC is met will result in bugs with internet explorer and + # probably some other browsers as well. IE for example is + # uploading files with "C:\foo\bar.txt" as filename + value = value[1:-1] + + # if this is a filename and the starting characters look like + # a UNC path, then just return the value without quotes. Using the + # replace sequence below on a UNC path has the effect of turning + # the leading double slash into a single slash and then + # _fix_ie_filename() doesn't work correctly. See #458. + if not is_filename or value[:2] != "\\\\": + return value.replace("\\\\", "\\").replace('\\"', '"') + return value + + +def dict_from_cookiejar(cj): + """Returns a key/value dictionary from a CookieJar. + + :param cj: CookieJar object to extract cookies from. + :rtype: dict + """ + + cookie_dict = {cookie.name: cookie.value for cookie in cj} + return cookie_dict + + +def add_dict_to_cookiejar(cj, cookie_dict): + """Returns a CookieJar from a key/value dictionary. + + :param cj: CookieJar to insert cookies into. + :param cookie_dict: Dict of key/values to insert into CookieJar. + :rtype: CookieJar + """ + + return cookiejar_from_dict(cookie_dict, cj) + + +def get_encodings_from_content(content): + """Returns encodings from given content string. + + :param content: bytestring to extract encodings from. + """ + warnings.warn( + ( + "In requests 3.0, get_encodings_from_content will be removed. For " + "more information, please see the discussion on issue #2266. (This" + " warning should only appear once.)" + ), + DeprecationWarning, + ) + + charset_re = re.compile(r']', flags=re.I) + pragma_re = re.compile(r']', flags=re.I) + xml_re = re.compile(r'^<\?xml.*?encoding=["\']*(.+?)["\'>]') + + return ( + charset_re.findall(content) + + pragma_re.findall(content) + + xml_re.findall(content) + ) + + +def _parse_content_type_header(header): + """Returns content type and parameters from given header + + :param header: string + :return: tuple containing content type and dictionary of + parameters + """ + + tokens = header.split(";") + content_type, params = tokens[0].strip(), tokens[1:] + params_dict = {} + items_to_strip = "\"' " + + for param in params: + param = param.strip() + if param: + key, value = param, True + index_of_equals = param.find("=") + if index_of_equals != -1: + key = param[:index_of_equals].strip(items_to_strip) + value = param[index_of_equals + 1 :].strip(items_to_strip) + params_dict[key.lower()] = value + return content_type, params_dict + + +def get_encoding_from_headers(headers): + """Returns encodings from given HTTP Header Dict. + + :param headers: dictionary to extract encoding from. + :rtype: str + """ + + content_type = headers.get("content-type") + + if not content_type: + return None + + content_type, params = _parse_content_type_header(content_type) + + if "charset" in params: + return params["charset"].strip("'\"") + + if "text" in content_type: + return "ISO-8859-1" + + if "application/json" in content_type: + # Assume UTF-8 based on RFC 4627: https://www.ietf.org/rfc/rfc4627.txt since the charset was unset + return "utf-8" + + +def stream_decode_response_unicode(iterator, r): + """Stream decodes an iterator.""" + + if r.encoding is None: + yield from iterator + return + + decoder = codecs.getincrementaldecoder(r.encoding)(errors="replace") + for chunk in iterator: + rv = decoder.decode(chunk) + if rv: + yield rv + rv = decoder.decode(b"", final=True) + if rv: + yield rv + + +def iter_slices(string, slice_length): + """Iterate over slices of a string.""" + pos = 0 + if slice_length is None or slice_length <= 0: + slice_length = len(string) + while pos < len(string): + yield string[pos : pos + slice_length] + pos += slice_length + + +def get_unicode_from_response(r): + """Returns the requested content back in unicode. + + :param r: Response object to get unicode content from. + + Tried: + + 1. charset from content-type + 2. fall back and replace all unicode characters + + :rtype: str + """ + warnings.warn( + ( + "In requests 3.0, get_unicode_from_response will be removed. For " + "more information, please see the discussion on issue #2266. (This" + " warning should only appear once.)" + ), + DeprecationWarning, + ) + + tried_encodings = [] + + # Try charset from content-type + encoding = get_encoding_from_headers(r.headers) + + if encoding: + try: + return str(r.content, encoding) + except UnicodeError: + tried_encodings.append(encoding) + + # Fall back: + try: + return str(r.content, encoding, errors="replace") + except TypeError: + return r.content + + +# The unreserved URI characters (RFC 3986) +UNRESERVED_SET = frozenset( + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" + "0123456789-._~" +) + + +def unquote_unreserved(uri): + """Un-escape any percent-escape sequences in a URI that are unreserved + characters. This leaves all reserved, illegal and non-ASCII bytes encoded. + + :rtype: str + """ + parts = uri.split("%") + for i in range(1, len(parts)): + h = parts[i][0:2] + if len(h) == 2 and h.isalnum(): + try: + c = chr(int(h, 16)) + except ValueError: + raise InvalidURL(f"Invalid percent-escape sequence: '{h}'") + + if c in UNRESERVED_SET: + parts[i] = c + parts[i][2:] + else: + parts[i] = f"%{parts[i]}" + else: + parts[i] = f"%{parts[i]}" + return "".join(parts) + + +def requote_uri(uri): + """Re-quote the given URI. + + This function passes the given URI through an unquote/quote cycle to + ensure that it is fully and consistently quoted. + + :rtype: str + """ + safe_with_percent = "!#$%&'()*+,/:;=?@[]~" + safe_without_percent = "!#$&'()*+,/:;=?@[]~" + try: + # Unquote only the unreserved characters + # Then quote only illegal characters (do not quote reserved, + # unreserved, or '%') + return quote(unquote_unreserved(uri), safe=safe_with_percent) + except InvalidURL: + # We couldn't unquote the given URI, so let's try quoting it, but + # there may be unquoted '%'s in the URI. We need to make sure they're + # properly quoted so they do not cause issues elsewhere. + return quote(uri, safe=safe_without_percent) + + +def address_in_network(ip, net): + """This function allows you to check if an IP belongs to a network subnet + + Example: returns True if ip = 192.168.1.1 and net = 192.168.1.0/24 + returns False if ip = 192.168.1.1 and net = 192.168.100.0/24 + + :rtype: bool + """ + ipaddr = struct.unpack("=L", socket.inet_aton(ip))[0] + netaddr, bits = net.split("/") + netmask = struct.unpack("=L", socket.inet_aton(dotted_netmask(int(bits))))[0] + network = struct.unpack("=L", socket.inet_aton(netaddr))[0] & netmask + return (ipaddr & netmask) == (network & netmask) + + +def dotted_netmask(mask): + """Converts mask from /xx format to xxx.xxx.xxx.xxx + + Example: if mask is 24 function returns 255.255.255.0 + + :rtype: str + """ + bits = 0xFFFFFFFF ^ (1 << 32 - mask) - 1 + return socket.inet_ntoa(struct.pack(">I", bits)) + + +def is_ipv4_address(string_ip): + """ + :rtype: bool + """ + try: + socket.inet_aton(string_ip) + except OSError: + return False + return True + + +def is_valid_cidr(string_network): + """ + Very simple check of the cidr format in no_proxy variable. + + :rtype: bool + """ + if string_network.count("/") == 1: + try: + mask = int(string_network.split("/")[1]) + except ValueError: + return False + + if mask < 1 or mask > 32: + return False + + try: + socket.inet_aton(string_network.split("/")[0]) + except OSError: + return False + else: + return False + return True + + +@contextlib.contextmanager +def set_environ(env_name, value): + """Set the environment variable 'env_name' to 'value' + + Save previous value, yield, and then restore the previous value stored in + the environment variable 'env_name'. + + If 'value' is None, do nothing""" + value_changed = value is not None + if value_changed: + old_value = os.environ.get(env_name) + os.environ[env_name] = value + try: + yield + finally: + if value_changed: + if old_value is None: + del os.environ[env_name] + else: + os.environ[env_name] = old_value + + +def should_bypass_proxies(url, no_proxy): + """ + Returns whether we should bypass proxies or not. + + :rtype: bool + """ + + # Prioritize lowercase environment variables over uppercase + # to keep a consistent behaviour with other http projects (curl, wget). + def get_proxy(key): + return os.environ.get(key) or os.environ.get(key.upper()) + + # First check whether no_proxy is defined. If it is, check that the URL + # we're getting isn't in the no_proxy list. + no_proxy_arg = no_proxy + if no_proxy is None: + no_proxy = get_proxy("no_proxy") + parsed = urlparse(url) + + if parsed.hostname is None: + # URLs don't always have hostnames, e.g. file:/// urls. + return True + + if no_proxy: + # We need to check whether we match here. We need to see if we match + # the end of the hostname, both with and without the port. + no_proxy = (host for host in no_proxy.replace(" ", "").split(",") if host) + + if is_ipv4_address(parsed.hostname): + for proxy_ip in no_proxy: + if is_valid_cidr(proxy_ip): + if address_in_network(parsed.hostname, proxy_ip): + return True + elif parsed.hostname == proxy_ip: + # If no_proxy ip was defined in plain IP notation instead of cidr notation & + # matches the IP of the index + return True + else: + host_with_port = parsed.hostname + if parsed.port: + host_with_port += f":{parsed.port}" + + for host in no_proxy: + if parsed.hostname.endswith(host) or host_with_port.endswith(host): + # The URL does match something in no_proxy, so we don't want + # to apply the proxies on this URL. + return True + + with set_environ("no_proxy", no_proxy_arg): + # parsed.hostname can be `None` in cases such as a file URI. + try: + bypass = proxy_bypass(parsed.hostname) + except (TypeError, socket.gaierror): + bypass = False + + if bypass: + return True + + return False + + +def get_environ_proxies(url, no_proxy=None): + """ + Return a dict of environment proxies. + + :rtype: dict + """ + if should_bypass_proxies(url, no_proxy=no_proxy): + return {} + else: + return getproxies() + + +def select_proxy(url, proxies): + """Select a proxy for the url, if applicable. + + :param url: The url being for the request + :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs + """ + proxies = proxies or {} + urlparts = urlparse(url) + if urlparts.hostname is None: + return proxies.get(urlparts.scheme, proxies.get("all")) + + proxy_keys = [ + urlparts.scheme + "://" + urlparts.hostname, + urlparts.scheme, + "all://" + urlparts.hostname, + "all", + ] + proxy = None + for proxy_key in proxy_keys: + if proxy_key in proxies: + proxy = proxies[proxy_key] + break + + return proxy + + +def resolve_proxies(request, proxies, trust_env=True): + """This method takes proxy information from a request and configuration + input to resolve a mapping of target proxies. This will consider settings + such as NO_PROXY to strip proxy configurations. + + :param request: Request or PreparedRequest + :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs + :param trust_env: Boolean declaring whether to trust environment configs + + :rtype: dict + """ + proxies = proxies if proxies is not None else {} + url = request.url + scheme = urlparse(url).scheme + no_proxy = proxies.get("no_proxy") + new_proxies = proxies.copy() + + if trust_env and not should_bypass_proxies(url, no_proxy=no_proxy): + environ_proxies = get_environ_proxies(url, no_proxy=no_proxy) + + proxy = environ_proxies.get(scheme, environ_proxies.get("all")) + + if proxy: + new_proxies.setdefault(scheme, proxy) + return new_proxies + + +def default_user_agent(name="python-requests"): + """ + Return a string representing the default user agent. + + :rtype: str + """ + return f"{name}/{__version__}" + + +def default_headers(): + """ + :rtype: requests.structures.CaseInsensitiveDict + """ + return CaseInsensitiveDict( + { + "User-Agent": default_user_agent(), + "Accept-Encoding": DEFAULT_ACCEPT_ENCODING, + "Accept": "*/*", + "Connection": "keep-alive", + } + ) + + +def parse_header_links(value): + """Return a list of parsed link headers proxies. + + i.e. Link: ; rel=front; type="image/jpeg",; rel=back;type="image/jpeg" + + :rtype: list + """ + + links = [] + + replace_chars = " '\"" + + value = value.strip(replace_chars) + if not value: + return links + + for val in re.split(", *<", value): + try: + url, params = val.split(";", 1) + except ValueError: + url, params = val, "" + + link = {"url": url.strip("<> '\"")} + + for param in params.split(";"): + try: + key, value = param.split("=") + except ValueError: + break + + link[key.strip(replace_chars)] = value.strip(replace_chars) + + links.append(link) + + return links + + +# Null bytes; no need to recreate these on each call to guess_json_utf +_null = "\x00".encode("ascii") # encoding to ASCII for Python 3 +_null2 = _null * 2 +_null3 = _null * 3 + + +def guess_json_utf(data): + """ + :rtype: str + """ + # JSON always starts with two ASCII characters, so detection is as + # easy as counting the nulls and from their location and count + # determine the encoding. Also detect a BOM, if present. + sample = data[:4] + if sample in (codecs.BOM_UTF32_LE, codecs.BOM_UTF32_BE): + return "utf-32" # BOM included + if sample[:3] == codecs.BOM_UTF8: + return "utf-8-sig" # BOM included, MS style (discouraged) + if sample[:2] in (codecs.BOM_UTF16_LE, codecs.BOM_UTF16_BE): + return "utf-16" # BOM included + nullcount = sample.count(_null) + if nullcount == 0: + return "utf-8" + if nullcount == 2: + if sample[::2] == _null2: # 1st and 3rd are null + return "utf-16-be" + if sample[1::2] == _null2: # 2nd and 4th are null + return "utf-16-le" + # Did not detect 2 valid UTF-16 ascii-range characters + if nullcount == 3: + if sample[:3] == _null3: + return "utf-32-be" + if sample[1:] == _null3: + return "utf-32-le" + # Did not detect a valid UTF-32 ascii-range character + return None + + +def prepend_scheme_if_needed(url, new_scheme): + """Given a URL that may or may not have a scheme, prepend the given scheme. + Does not replace a present scheme with the one provided as an argument. + + :rtype: str + """ + parsed = parse_url(url) + scheme, auth, host, port, path, query, fragment = parsed + + # A defect in urlparse determines that there isn't a netloc present in some + # urls. We previously assumed parsing was overly cautious, and swapped the + # netloc and path. Due to a lack of tests on the original defect, this is + # maintained with parse_url for backwards compatibility. + netloc = parsed.netloc + if not netloc: + netloc, path = path, netloc + + if auth: + # parse_url doesn't provide the netloc with auth + # so we'll add it ourselves. + netloc = "@".join([auth, netloc]) + if scheme is None: + scheme = new_scheme + if path is None: + path = "" + + return urlunparse((scheme, netloc, path, "", query, fragment)) + + +def get_auth_from_url(url): + """Given a url with authentication components, extract them into a tuple of + username,password. + + :rtype: (str,str) + """ + parsed = urlparse(url) + + try: + auth = (unquote(parsed.username), unquote(parsed.password)) + except (AttributeError, TypeError): + auth = ("", "") + + return auth + + +def check_header_validity(header): + """Verifies that header parts don't contain leading whitespace + reserved characters, or return characters. + + :param header: tuple, in the format (name, value). + """ + name, value = header + _validate_header_part(header, name, 0) + _validate_header_part(header, value, 1) + + +def _validate_header_part(header, header_part, header_validator_index): + if isinstance(header_part, str): + validator = _HEADER_VALIDATORS_STR[header_validator_index] + elif isinstance(header_part, bytes): + validator = _HEADER_VALIDATORS_BYTE[header_validator_index] + else: + raise InvalidHeader( + f"Header part ({header_part!r}) from {header} " + f"must be of type str or bytes, not {type(header_part)}" + ) + + if not validator.match(header_part): + header_kind = "name" if header_validator_index == 0 else "value" + raise InvalidHeader( + f"Invalid leading whitespace, reserved character(s), or return " + f"character(s) in header {header_kind}: {header_part!r}" + ) + + +def urldefragauth(url): + """ + Given a url remove the fragment and the authentication part. + + :rtype: str + """ + scheme, netloc, path, params, query, fragment = urlparse(url) + + # see func:`prepend_scheme_if_needed` + if not netloc: + netloc, path = path, netloc + + netloc = netloc.rsplit("@", 1)[-1] + + return urlunparse((scheme, netloc, path, params, query, "")) + + +def rewind_body(prepared_request): + """Move file pointer back to its recorded starting position + so it can be read again on redirect. + """ + body_seek = getattr(prepared_request.body, "seek", None) + if body_seek is not None and isinstance( + prepared_request._body_position, integer_types + ): + try: + body_seek(prepared_request._body_position) + except OSError: + raise UnrewindableBodyError( + "An error occurred when rewinding request body for redirect." + ) + else: + raise UnrewindableBodyError("Unable to rewind request body for redirect.") diff --git a/examples/react/bad-architecture-doc-example.md b/examples/react/bad-architecture-doc-example.md index c859c4b..33807d7 100644 --- a/examples/react/bad-architecture-doc-example.md +++ b/examples/react/bad-architecture-doc-example.md @@ -1,9 +1,13 @@ # ExpenseTracker Architecture Overview +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns. Each ❌ callout explains a failure. See good-architecture-doc-example.md for the correct approach. + ## What This App Does ExpenseTracker is a comprehensive expense tracking application built with React. It allows users to track their daily expenses, categorize spending, view reports, and sync data across devices. +> **❌ PROBLEMS:** No citations, and two of the four capabilities are invented. "Sync data across devices" is impossible — all persistence is browser localStorage (`src/services/expenseService.js:6`, and the file's own comment "Currently uses localStorage, not a real API" at `src/services/expenseService.js:3`). "View reports" doesn't exist either — the only routes are `/`, `/add`, and `/expense/:id` (`src/App.jsx:15-19`); there is no reports page. + ## Technology Stack - React 18 with hooks @@ -13,6 +17,8 @@ ExpenseTracker is a comprehensive expense tracking application built with React. - TailwindCSS for styling - Jest and React Testing Library for tests +> **❌ PROBLEMS:** Half the stack is hallucinated. `package.json:5-13` lists exactly three dependencies (`react`, `react-dom`, `react-router-dom`) plus two dev dependencies (`vite`, `@vitejs/plugin-react`). There is no Redux, no Axios, no TailwindCSS, no Jest/RTL — and no `test` script (`package.json:14-18`). Styling is plain CSS (`src/index.css:1-17`) plus inline style objects (`src/components/Header.jsx:19-43`). A good doc would record each as `[NOT_FOUND: searched ...]`. + ## Components The app uses a component-based architecture: @@ -24,17 +30,23 @@ The app uses a component-based architecture: - ExpenseForm - Form for adding/editing expenses - Dashboard - Main overview page +> **❌ PROBLEMS:** The named components do exist, but with zero `file:line` citations the reader cannot tell verified claims from guesses — and one detail is wrong: ExpenseForm only *adds* expenses; editing is unimplemented (`src/services/expenseService.js:87-89` throws `'Not implemented'`, and the context has no update function, `src/context/ExpenseContext.jsx:91-92`). The list also silently omits real modules: `src/pages/AddExpense.jsx`, `src/pages/ExpenseDetail.jsx`, `src/context/ExpenseContext.jsx`, `src/hooks/useTotalExpenses.js`, `src/utils/formatters.js`, `src/utils/constants.js`. + ### State Management Uses Redux with the following slices: - expenseSlice - Manages expense CRUD operations - userSlice - Handles authentication state - settingsSlice - User preferences and config +> **❌ PROBLEMS:** Entirely false. State is React Context with `useReducer` (`src/context/ExpenseContext.jsx:52-53`) driving a plain reducer function (`src/context/ExpenseContext.jsx:24-49`). There are no slices, no store, no user state, no settings state — the initial state is just `expenses`, `loading`, `error` (`src/context/ExpenseContext.jsx:17-21`). Asserting Redux because most React apps use it is hallucination-by-convention. + ### Services - expenseService - API calls for expenses - authService - Authentication with JWT - syncService - Real-time sync with backend +> **❌ PROBLEMS:** `src/services/` contains exactly one file, `expenseService.js` — `authService` and `syncService` are invented. And `expenseService` makes no API calls: it reads and writes localStorage behind a simulated delay (`src/services/expenseService.js:9`, `src/services/expenseService.js:12-24`). + ## Data Flow 1. User interacts with component @@ -44,6 +56,8 @@ Uses Redux with the following slices: 5. Reducer updates store 6. Component re-renders with new data +> **❌ PROBLEMS:** Two failures at once. First, a numbered step-by-step trace belongs in code-flow documentation, not an architecture overview — the methodology requires tables describing what moves, not execution steps. Second, the steps are wrong: there are no Redux actions and no API. Components call context functions like `addExpense`, which await the localStorage-backed service and then dispatch to a `useReducer` reducer (`src/context/ExpenseContext.jsx:70-79`). + ## API Integration The app connects to a REST API: @@ -52,6 +66,8 @@ The app connects to a REST API: - PUT /api/expenses/:id - Update expense - DELETE /api/expenses/:id - Delete expense +> **❌ PROBLEMS:** All four endpoints are fabricated. There is no `fetch`, `axios`, or `XMLHttpRequest` anywhere in `src/`. The closest reality is the async facade over localStorage: `getAll` (`src/services/expenseService.js:35-38`), `getById` (`:43-51`), `create` (`:56-67`), `delete` (`:72-81`) — and `update` just throws (`src/services/expenseService.js:87-89`), so even the *shape* of the invented API (a working PUT) contradicts the code. + ## Authentication Uses JWT-based authentication: @@ -60,6 +76,8 @@ Uses JWT-based authentication: - Auto-refresh on expiration - Protected routes require valid token +> **❌ PROBLEMS:** There is no authentication of any kind. No login UI, no token handling, no cookies, no route guards — the three routes render unconditionally (`src/App.jsx:15-19`). Searching `src/` for "auth", "login", "jwt", "token" finds nothing. Every line of this section is invented. + ## Database Schema | Table | Columns | @@ -67,3 +85,20 @@ Uses JWT-based authentication: | users | id, email, password_hash, created_at | | expenses | id, user_id, amount, category, description, date | | categories | id, name, icon, color | + +> **❌ PROBLEMS:** A client-only SPA has no database. The real data model is an array of expense objects in localStorage with `id`, `...data` (description, amount, category), and `createdAt`, built in `create()` (`src/services/expenseService.js:59-63`). There is no `users` table (no users at all), and categories are a hardcoded string array, not a table (`src/utils/constants.js:6-15`) — no icons, no colors. + +## Why This Example is BAD + +1. **No metadata, no commit hash, no verification tags anywhere** — not a single claim carries `[VERIFIED: path:line]`, so nothing can be checked and `verify.py` has nothing to verify. → The good example resolves 90/90 citations. +2. **Invented state management**: Redux with expenseSlice/userSlice/settingsSlice → reality: Context + `useReducer` (`src/context/ExpenseContext.jsx:52-53`) with only expenses/loading/error state (`src/context/ExpenseContext.jsx:17-21`). +3. **Invented network layer**: Axios + four REST endpoints → reality: zero network calls; localStorage behind a fake delay (`src/services/expenseService.js:9`, `src/services/expenseService.js:12-24`). +4. **Invented services**: authService, syncService → reality: `src/services/` holds only `expenseService.js`. +5. **Invented authentication**: JWT, refresh tokens, protected routes → reality: unguarded routes (`src/App.jsx:15-19`) and no auth code anywhere in `src/`. +6. **Invented testing stack**: Jest and React Testing Library → reality: no test files and no test dependencies or script (`package.json:10-18`). +7. **Invented styling stack**: TailwindCSS → reality: plain CSS (`src/index.css:1-17`) and inline style objects (`src/components/Header.jsx:19-43`). +8. **False capability claims**: cross-device sync and reports → reality: localStorage-only persistence (`src/services/expenseService.js:3-6`) and three routes with no reports page (`src/App.jsx:15-19`). +9. **False editing claim**: "adding/editing expenses" → reality: update is unimplemented and throws (`src/services/expenseService.js:87-89`; `src/context/ExpenseContext.jsx:91-92`). +10. **Invented database schema**: users/expenses/categories tables → reality: no database; expense objects assembled in `create()` (`src/services/expenseService.js:59-63`) and a hardcoded category array (`src/utils/constants.js:6-15`). +11. **Step-by-step "Data Flow" trace in an architecture doc** — execution tracing belongs in code-flow documentation; the overview must stay at discovery level (tables, not numbered steps). +12. **No `[NOT_FOUND]` admissions** — a trustworthy doc records what it searched for and failed to find; this one asserts instead of admitting. diff --git a/examples/react/good-architecture-doc-example.md b/examples/react/good-architecture-doc-example.md index 7058fe9..9539508 100644 --- a/examples/react/good-architecture-doc-example.md +++ b/examples/react/good-architecture-doc-example.md @@ -5,109 +5,83 @@ |-------|-------| | Repository | `agent-system-mapper` | | Path | `examples/react/expense-tracker/` | -| Commit | `bfd3ee6` | -| Documented | `2025-12-21` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | | Verification Status | `Verified` | +**Verify with:** +```bash +python3 verify.py examples/react/good-architecture-doc-example.md --repo-root examples/react/expense-tracker +``` + ## Verification Summary -- [VERIFIED]: 24 claims -- [INFERRED]: 2 claims -- [NOT_FOUND]: 6 items (Redux, auth, backend API, tests, TailwindCSS, sync) -- [ASSUMED]: 1 item (Vite conventions) +- `[VERIFIED]`: 74 tags (90 `file:line` citations, all resolving; 15 quoted blocks, all matching) +- `[INFERRED]`: 1 claim (Vite serving conventions) +- `[NOT_FOUND]`: 11 items (server code, backend API, Redux, auth, tests, network calls, env config, sync, CSS framework, edit UI, external services) +- `[ASSUMED]`: 0 items --- -## System Classification +## 0. System Classification + | Field | Value | |-------|-------| +| Category | Traditional Code | | Type | Frontend SPA | -| Evidence | `package.json` with `react`, `react-router-dom`, no server code | +| Evidence | `package.json` depends on `react` and `react-router-dom` [VERIFIED: package.json:5-9]; client entry mounts into `#root` [VERIFIED: src/main.jsx:8, index.html:9] | +| Overlay Loaded | No | | Confidence | `[VERIFIED]` | +[NOT_FOUND: searched "express", "server", "listen(" in expense-tracker/ — no server-side code of any kind] + --- -## System Purpose +## 1. System Purpose -ExpenseTracker is a **client-side expense tracking single-page application** built with React. +ExpenseTracker is a **client-side expense tracking single-page application**: users add expenses (description, amount, category), see a running total on a dashboard, and view or delete individual expenses. All data lives in the browser's localStorage — there is no backend. -[VERIFIED: `package.json:6-8`] +[VERIFIED: package.json:5-9] ```json -"dependencies": { - "react": "^18.2.0", - "react-dom": "^18.2.0", - "react-router-dom": "^6.20.0" -} + "dependencies": { + "react": "^18.2.0", + "react-dom": "^18.2.0", + "react-router-dom": "^6.20.0" + }, ``` -[NOT_FOUND: searched "axios", "fetch", "api" in src/services/] -**No backend API integration.** Data persisted to localStorage only. +[NOT_FOUND: searched "axios", "fetch(", "XMLHttpRequest" in src/ — no backend API integration; data is persisted to localStorage only] -[NOT_FOUND: searched "redux", "store", "slice" in src/] -**No Redux.** Uses React Context with useReducer for state. +[NOT_FOUND: searched "redux", "slice", "configureStore" in src/ — no Redux; state management uses React Context with useReducer] --- -## Component Map +## 2. Component Map -| Component | Location | Responsibility | Verified | +| Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| App | `src/App.jsx` | Route configuration, layout | [VERIFIED] | -| Header | `src/components/Header.jsx` | Navigation bar | [VERIFIED] | -| ExpenseList | `src/components/ExpenseList.jsx` | Render expense items | [VERIFIED] | -| ExpenseItem | `src/components/ExpenseItem.jsx` | Single expense row with delete | [VERIFIED] | -| ExpenseForm | `src/components/ExpenseForm.jsx` | Add expense form | [VERIFIED] | -| Dashboard | `src/pages/Dashboard.jsx` | Main view with summary | [VERIFIED] | -| AddExpense | `src/pages/AddExpense.jsx` | Add expense page | [VERIFIED] | -| ExpenseDetail | `src/pages/ExpenseDetail.jsx` | Single expense view | [VERIFIED] | -| ExpenseContext | `src/context/ExpenseContext.jsx` | Global state management | [VERIFIED] | -| expenseService | `src/services/expenseService.js` | localStorage CRUD | [VERIFIED] | - -[NOT_FOUND: searched "auth", "login", "user" in src/] -No authentication layer. - -[NOT_FOUND: searched "test", "spec", ".test." in examples/react/] -No test files. +| App | `src/App.jsx` | Route configuration, layout shell | [VERIFIED: src/App.jsx:11-21] | +| Header | `src/components/Header.jsx` | Navigation bar with Add link | [VERIFIED: src/components/Header.jsx:6-17] | +| ExpenseList | `src/components/ExpenseList.jsx` | Loading/error/empty/list rendering | [VERIFIED: src/components/ExpenseList.jsx:7-29] | +| ExpenseItem | `src/components/ExpenseItem.jsx` | Single expense row with delete button | [VERIFIED: src/components/ExpenseItem.jsx:8-18] | +| ExpenseForm | `src/components/ExpenseForm.jsx` | Controlled add-expense form | [VERIFIED: src/components/ExpenseForm.jsx:9-17] | +| Dashboard | `src/pages/Dashboard.jsx` | Total summary + recent expenses | [VERIFIED: src/pages/Dashboard.jsx:9-25] | +| AddExpense | `src/pages/AddExpense.jsx` | Page wrapper around ExpenseForm | [VERIFIED: src/pages/AddExpense.jsx:6-13] | +| ExpenseDetail | `src/pages/ExpenseDetail.jsx` | Single expense view with delete | [VERIFIED: src/pages/ExpenseDetail.jsx:11-17] | +| ExpenseContext | `src/context/ExpenseContext.jsx` | Global state (reducer + provider + hook) | [VERIFIED: src/context/ExpenseContext.jsx:24, 52, 109] | +| expenseService | `src/services/expenseService.js` | localStorage CRUD with simulated delay | [VERIFIED: src/services/expenseService.js:31-38] | +| useTotalExpenses | `src/hooks/useTotalExpenses.js` | Memoized total across expenses | [VERIFIED: src/hooks/useTotalExpenses.js:8-16] | +| formatters | `src/utils/formatters.js` | Currency/date formatting | [VERIFIED: src/utils/formatters.js:9-14] | +| constants | `src/utils/constants.js` | Category list, currency, storage key | [VERIFIED: src/utils/constants.js:6-21] | ---- - -## File Structure - -``` -expense-tracker/ -├── index.html # HTML entry point [VERIFIED] -├── package.json # Dependencies [VERIFIED] -├── vite.config.js # Build config [VERIFIED] -└── src/ - ├── main.jsx # React entry, providers [VERIFIED] - ├── App.jsx # Router setup [VERIFIED] - ├── index.css # Global styles [VERIFIED] - ├── components/ - │ ├── Header.jsx # Nav bar [VERIFIED] - │ ├── ExpenseList.jsx # List display [VERIFIED] - │ ├── ExpenseItem.jsx # Item row [VERIFIED] - │ └── ExpenseForm.jsx # Add form [VERIFIED] - ├── pages/ - │ ├── Dashboard.jsx # Main page [VERIFIED] - │ ├── AddExpense.jsx # Add page [VERIFIED] - │ └── ExpenseDetail.jsx # Detail page [VERIFIED] - ├── context/ - │ └── ExpenseContext.jsx # State management [VERIFIED] - ├── hooks/ - │ └── useTotalExpenses.js # Computed values [VERIFIED] - ├── services/ - │ └── expenseService.js # Data access [VERIFIED] - └── utils/ - ├── formatters.js # Display formatting [VERIFIED] - └── constants.js # App constants [VERIFIED] -``` +[NOT_FOUND: searched "auth", "login", "jwt", "token" in src/ — no authentication layer] ---- +[NOT_FOUND: searched "*.test.*", "*.spec.*", "__tests__" in expense-tracker/ — no test files, and package.json declares no test dependencies or test script] -## State Management +### State Management -Uses React Context with useReducer pattern. +React Context with a useReducer store; six action types: -[VERIFIED: `src/context/ExpenseContext.jsx:8-14`] +[VERIFIED: src/context/ExpenseContext.jsx:7-14] ```javascript const ACTIONS = { SET_EXPENSES: 'SET_EXPENSES', @@ -119,38 +93,40 @@ const ACTIONS = { } ``` -[VERIFIED: `src/context/ExpenseContext.jsx:46-51`] +[VERIFIED: src/context/ExpenseContext.jsx:52-58] ```javascript export function ExpenseProvider({ children }) { const [state, dispatch] = useReducer(expenseReducer, initialState) + // Load expenses on mount useEffect(() => { loadExpenses() }, []) ``` -Provider wraps app: -[VERIFIED: `src/main.jsx:9-13`] +The provider wraps the whole app: + +[VERIFIED: src/main.jsx:10-14] ```javascript - - - - - + + + + + ``` ---- +Consumers use the `useExpenses()` hook, which throws outside the provider [VERIFIED: src/context/ExpenseContext.jsx:109-115]. -## Data Persistence +### Data Persistence -[VERIFIED: `src/services/expenseService.js:7-8`] +[VERIFIED: src/services/expenseService.js:6] ```javascript const STORAGE_KEY = 'expense-tracker-data' ``` -**No backend API.** All data stored in browser localStorage. +All reads parse that key from localStorage: -[VERIFIED: `src/services/expenseService.js:13-20`] +[VERIFIED: src/services/expenseService.js:12-19] ```javascript function getStoredExpenses() { try { @@ -162,163 +138,248 @@ function getStoredExpenses() { } ``` +The service comment itself admits the design: "Currently uses localStorage, not a real API" [VERIFIED: src/services/expenseService.js:3]. + --- -## Routing +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) -[VERIFIED: `src/App.jsx:12-16`] -```javascript - - } /> - } /> - } /> - -``` +### 3.1 Primary Execution Surfaces -| Route | Page | Purpose | -|-------|------|---------| -| `/` | Dashboard | List expenses, show total | -| `/add` | AddExpense | Form to add expense | -| `/expense/:id` | ExpenseDetail | View single expense | +| Entry Surface | Type | Primary Components Involved | Evidence | +|---------------|------|-----------------------------|----------| +| Browser loads `index.html` | Web (SPA bootstrap) | main.jsx, ExpenseProvider, App | [VERIFIED: index.html:10, src/main.jsx:8-16] | +| Route `/` | Client-side route | Dashboard, ExpenseList, useTotalExpenses | [VERIFIED: src/App.jsx:16] | +| Route `/add` | Client-side route | AddExpense, ExpenseForm | [VERIFIED: src/App.jsx:17] | +| Route `/expense/:id` | Client-side route | ExpenseDetail, expenseService | [VERIFIED: src/App.jsx:18] | +| `npm run dev` / `build` / `preview` | CLI (Vite) | vite.config.js | [VERIFIED: package.json:15-17, vite.config.js:4-6] | ---- +### 3.2 High-Level Data Movement (Non-Procedural) -## Entry Points +| Stage | Input Type | Output Type | Participating Components | +|-------|------------|-------------|--------------------------| +| App bootstrap | localStorage JSON | `expenses` array in context state | ExpenseProvider, expenseService | +| Add expense | Form field values | New expense record + state update | ExpenseForm, ExpenseContext, expenseService | +| Delete expense | Expense id | Filtered expense array | ExpenseItem / ExpenseDetail, ExpenseContext, expenseService | +| Detail lookup | Route param `id` | Single expense object (local page state) | ExpenseDetail, expenseService | +| Total computation | `expenses` array | Memoized number | useTotalExpenses, Dashboard | -### User Entry Points -| Entry | Component | Trigger | -|-------|-----------|---------| -| Add expense | Header | Click "+ Add Expense" link | -| Delete expense | ExpenseItem | Click "Delete" button | -| View detail | ExpenseItem | Click expense title | +### 3.3 Pointers to Code Flow Documentation -### Application Entry -[VERIFIED: `src/main.jsx:8-14`] -```javascript -ReactDOM.createRoot(document.getElementById('root')).render( - - - - - - - -) -``` +Detailed execution paths are deliberately **not** traced here — see `02-code-flows.md` for: + +- **Add Expense flow** — entry `ExpenseForm.handleSubmit` [VERIFIED: src/components/ExpenseForm.jsx:24-45] +- **Delete Expense flow** — entries `ExpenseItem.handleDelete` and `ExpenseDetail.handleDelete` [VERIFIED: src/components/ExpenseItem.jsx:11-18, src/pages/ExpenseDetail.jsx:34-44] +- **Load-on-mount flow** — entry `ExpenseProvider`'s `useEffect` → `loadExpenses` [VERIFIED: src/context/ExpenseContext.jsx:56-68] --- -## Custom Hooks +## 3b. Frontend → Backend Interaction Map -[VERIFIED: `src/hooks/useTotalExpenses.js:8-14`] -```javascript -export function useTotalExpenses() { - const { expenses } = useExpenses() +Not applicable — this system has no backend. Every user interaction is handled entirely in the browser: context functions call `expenseService`, which reads and writes localStorage. - const total = useMemo(() => { - return expenses.reduce((sum, expense) => sum + expense.amount, 0) - }, [expenses]) +[NOT_FOUND: searched "fetch(", "axios", "XMLHttpRequest", "WebSocket" in src/ — no network calls of any kind] - return total -} -``` +--- + +## 4. File/Folder Conventions + +Directory layout (all files below were read during documentation): -[VERIFIED: `src/hooks/useTotalExpenses.js:19-31`] -```javascript -export function useExpensesByCategory() { - // Wart: Not used anywhere yet, but could be useful +``` +expense-tracker/ +├── index.html +├── package.json +├── vite.config.js +└── src/ + ├── main.jsx + ├── App.jsx + ├── index.css + ├── components/ Header, ExpenseList, ExpenseItem, ExpenseForm + ├── pages/ Dashboard, AddExpense, ExpenseDetail + ├── context/ ExpenseContext + ├── hooks/ useTotalExpenses + ├── services/ expenseService + └── utils/ formatters, constants ``` +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| `src/components/` | Reusable presentational components | [VERIFIED: src/components/Header.jsx:6, src/components/ExpenseList.jsx:7] | +| `src/pages/` | One component per route | [VERIFIED: src/App.jsx:16-18, src/pages/Dashboard.jsx:9] | +| `src/context/` | Context provider + reducer + consumer hook in one file | [VERIFIED: src/context/ExpenseContext.jsx:24, 52, 109] | +| `src/hooks/` | Custom hooks for derived state | [VERIFIED: src/hooks/useTotalExpenses.js:8, 22] | +| `src/services/` | Data-access layer (localStorage behind async API) | [VERIFIED: src/services/expenseService.js:31] | +| `src/utils/` | Pure helpers and constants | [VERIFIED: src/utils/formatters.js:9, src/utils/constants.js:6] | +| Inline `styles` objects per component | Styling co-located with components, plus one global stylesheet | [VERIFIED: src/components/Header.jsx:19-27, src/index.css:1-5] | + +[INFERRED: `index.html` at the project root with `/src/main.jsx` as module entry follows the standard Vite serving convention — vite.config.js adds only the React plugin] + --- -## Known Issues / Warts +## 5. External Dependencies + +| Dependency | Purpose | Evidence | +|------------|---------|----------| +| `react` ^18.2.0 | UI framework | [VERIFIED: package.json:6] | +| `react-dom` ^18.2.0 | DOM renderer | [VERIFIED: package.json:7] | +| `react-router-dom` ^6.20.0 | Client-side routing | [VERIFIED: package.json:8] | +| `vite` ^5.0.0 (dev) | Build tool / dev server | [VERIFIED: package.json:11] | +| `@vitejs/plugin-react` ^4.2.0 (dev) | React fast-refresh plugin | [VERIFIED: package.json:12, vite.config.js:2] | -### 1. No Confirmation on Delete (ExpenseItem) +[NOT_FOUND: searched "import.meta.env", "process.env" in src/ — no environment-driven configuration; there are no API base URLs or keys to configure] -[VERIFIED: `src/components/ExpenseItem.jsx:14`] +--- + +## 6. Known Issues & Risks + +### 6.1 No Confirmation on Delete in ExpenseItem + +[VERIFIED: src/components/ExpenseItem.jsx:12] ```javascript -// Wart: No confirmation dialog before delete + // Wart: No confirmation dialog before delete ``` -Delete happens immediately without user confirmation. +Inconsistent UX: ExpenseDetail's delete DOES ask for confirmation [VERIFIED: src/pages/ExpenseDetail.jsx:35], the list row's delete does not. -### 2. Update Not Implemented +### 6.2 Update Not Implemented -[VERIFIED: `src/context/ExpenseContext.jsx:78-79`] +[VERIFIED: src/context/ExpenseContext.jsx:91-92] ```javascript -// Wart: No update function implemented yet -// async function updateExpense(id, data) { ... } + // Wart: No update function implemented yet + // async function updateExpense(id, data) { ... } ``` -[VERIFIED: `src/services/expenseService.js:66-69`] +[VERIFIED: src/services/expenseService.js:87-89] ```javascript -async update(id, data) { - throw new Error('Not implemented') -} + async update(id, data) { + throw new Error('Not implemented') + }, ``` -### 3. Hardcoded Currency +The reducer even defines an `UPDATE_EXPENSE` action that nothing dispatches [VERIFIED: src/context/ExpenseContext.jsx:35-41]. -[VERIFIED: `src/utils/formatters.js:7-8`] +### 6.3 Hardcoded Currency + +[VERIFIED: src/utils/formatters.js:9-14] ```javascript -// Wart: Hardcoded to USD, should be configurable +export function formatCurrency(amount) { + return new Intl.NumberFormat('en-US', { + style: 'currency', + currency: 'USD', + }).format(amount) +} ``` -### 4. Duplicated Storage Key +The wart is acknowledged in the file itself [VERIFIED: src/utils/formatters.js:7] and duplicated as a constant [VERIFIED: src/utils/constants.js:17-18]. -[VERIFIED: `src/services/expenseService.js:7`] -```javascript -const STORAGE_KEY = 'expense-tracker-data' -``` +### 6.4 Duplicated Storage Key -[VERIFIED: `src/utils/constants.js:14`] +[VERIFIED: src/utils/constants.js:20-21] ```javascript // Storage key - Wart: also duplicated in expenseService.js export const STORAGE_KEY = 'expense-tracker-data' ``` -### 5. Detail Page Fetches Directly +The service defines its own copy of the same literal [VERIFIED: src/services/expenseService.js:6]. -[VERIFIED: `src/pages/ExpenseDetail.jsx:10`] +### 6.5 Detail Page Fetches Directly + +[VERIFIED: src/pages/ExpenseDetail.jsx:9] ```javascript -// Wart: Fetches from service directly instead of using context + * Wart: Fetches from service directly instead of using context ``` -Should use context to avoid duplicate fetches. +`loadExpense` calls `expenseService.getById` instead of reading context state [VERIFIED: src/pages/ExpenseDetail.jsx:23-32]. -### 6. Basic Form Validation +### 6.6 Basic Form Validation -[VERIFIED: `src/components/ExpenseForm.jsx:27`] +[VERIFIED: src/components/ExpenseForm.jsx:27-30] ```javascript -// Wart: Basic validation only, no error messages shown + // Wart: Basic validation only, no error messages shown + if (!formData.description || !formData.amount) { + return + } ``` +### 6.7 Dead Code + +`useExpensesByCategory` is exported but never imported anywhere [VERIFIED: src/hooks/useTotalExpenses.js:20-22]; `formatRelativeTime` is a stub that just returns the plain date [VERIFIED: src/utils/formatters.js:32-35]. + +### 6.8 Features Confirmed Absent + +- [NOT_FOUND: searched "websocket", "socket", "sync" in src/ — no cross-device sync; the only "sync" matches are local `async function` definitions] +- [NOT_FOUND: searched "tailwind", "styled", "sass" in expense-tracker/ — no CSS framework; plain CSS and inline style objects only] +- [NOT_FOUND: searched for an edit/update UI in src/pages/ and src/components/ — no edit form exists; expenses can only be created and deleted] + --- -## Technology Stack Summary +## 7. Entry Points Summary -| Layer | Technology | -|-------|------------| -| UI Framework | React 18 [VERIFIED: package.json] | -| Routing | React Router 6 [VERIFIED: package.json] | -| State | Context + useReducer [VERIFIED: ExpenseContext.jsx] | -| Build Tool | Vite 5 [VERIFIED: package.json] | -| Styling | Plain CSS [VERIFIED: index.css, inline styles] | -| Data Storage | localStorage [VERIFIED: expenseService.js] | +### Application Bootstrap -[NOT_FOUND: searched "tailwind", "styled", "sass" in expense-tracker/] -No CSS framework - uses plain CSS and inline styles. +[VERIFIED: src/main.jsx:8-16] +```javascript +ReactDOM.createRoot(document.getElementById('root')).render( + + + + + + + +) +``` + +### Routes + +[VERIFIED: src/App.jsx:15-19] +```javascript + + } /> + } /> + } /> + +``` + +| Route/Entry | Method | Handler | Middleware | Verified | +|-------------|--------|---------|------------|----------| +| `/` | client-side GET | `Dashboard` | none (no router guards) | [VERIFIED: src/App.jsx:16] | +| `/add` | client-side GET | `AddExpense` | none | [VERIFIED: src/App.jsx:17] | +| `/expense/:id` | client-side GET | `ExpenseDetail` | none | [VERIFIED: src/App.jsx:18] | + +### User-Initiated Triggers + +| Trigger | Component | Evidence | +|---------|-----------|----------| +| "+ Add Expense" link | Header | [VERIFIED: src/components/Header.jsx:13] | +| Submit add-expense form | ExpenseForm | [VERIFIED: src/components/ExpenseForm.jsx:48] | +| Delete button (list row) | ExpenseItem | [VERIFIED: src/components/ExpenseItem.jsx:31-33] | +| Expense title link to detail | ExpenseItem | [VERIFIED: src/components/ExpenseItem.jsx:23-25] | +| Delete button (detail page) | ExpenseDetail | [VERIFIED: src/pages/ExpenseDetail.jsx:63-65] | --- -## What This System Does NOT Have +## 8. Technology Stack Summary + +| Layer | Technology | Evidence | +|-------|------------|----------| +| UI Framework | React 18 | [VERIFIED: package.json:6] | +| Routing | React Router 6 | [VERIFIED: package.json:8] | +| State | Context + useReducer | [VERIFIED: src/context/ExpenseContext.jsx:53] | +| Build Tool | Vite 5 | [VERIFIED: package.json:11] | +| Styling | Plain CSS + inline style objects | [VERIFIED: src/index.css:7-11, src/components/Header.jsx:19] | +| Data Storage | Browser localStorage | [VERIFIED: src/services/expenseService.js:14, 23] | +| External Services | None | [NOT_FOUND: searched "http", "api", "key" in src/ — no external service integration] | + +--- -Based on searches finding no results: +## Why This Example is GOOD -1. **No Backend API** - localStorage only -2. **No Authentication** - No login/user system -3. **No Redux** - Uses Context API -4. **No Tests** - No test files found -5. **No CSS Framework** - Plain CSS only -6. **No Real-time Sync** - Local data only -7. **No Update Feature** - Only create and delete +1. **Every claim is cited or admitted.** Each factual statement carries a `[VERIFIED: path:line]` tag that resolves against `examples/react/expense-tracker/`, or an explicit `[NOT_FOUND]` / `[INFERRED]` admission. +2. **Quotes are exact copy-paste.** Every fenced block matches the cited line range character-for-character, so `verify.py` phase 2 passes. +3. **Absence is documented with real searches.** `[NOT_FOUND]` items name the patterns searched (axios, redux, auth, tests...), including the honest note that "sync" only matches `async` keywords. +4. **Section 3 stays at discovery level.** Tables describe entry surfaces and what moves — no step-by-step traces, no arrow diagrams; detailed tracing is deferred to `02-code-flows.md`. +5. **The 3b section is answered, not skipped.** A frontend-only system states explicitly that no frontend-to-backend interactions exist, backed by a search. +6. **Warts are surfaced, not hidden.** The unimplemented update path, duplicated storage key, dead hooks, and inconsistent delete confirmation are documented with exact lines. +7. **It is machine-checkable.** Running the command in the metadata block exits 0. diff --git a/examples/test-surface/good-test-surface-example.md b/examples/test-surface/good-test-surface-example.md index 3fcc550..8d9a001 100644 --- a/examples/test-surface/good-test-surface-example.md +++ b/examples/test-surface/good-test-surface-example.md @@ -1,15 +1,22 @@ # Test Surface: Create Booking Flow > **This is an example of GOOD test surface documentation.** -> It demonstrates proper grounding in verified code flows. +> Every candidate is grounded in the verified Create Booking code flow for the +> mini Laravel app in `examples/laravel/slotbooker/` — including *absence* tests +> for behavior the bad example hallucinates (confirmation emails, BookingService). ## Metadata | Field | Value | |-------|-------| -| Flow Document | `pf-docs/02-code-flow-create-booking.md` | -| Generated | `2025-01-15` | -| Flow Steps | 7 | -| Source Commit | `e043013` | +| Flow Document | `examples/laravel/good-code-flow-doc-example.md` | +| Generated | `2026-08-03` | +| Flow Steps | 8 | +| Source Commit | `9a69c14` | + +**Verify with:** +```bash +python3 verify.py examples/test-surface/good-test-surface-example.md --repo-root examples/laravel/slotbooker +``` --- @@ -17,145 +24,168 @@ | Step | Outcome | Type | Evidence | |------|---------|------|----------| -| Step 2 | Request validated | Validation | [VERIFIED: BookingController.php:34] | -| Step 3 | Booking record created | Database Write | [VERIFIED: BookingService.php:45] | -| Step 4 | TimeSlot marked unavailable | Database Write | [VERIFIED: BookingService.php:48] | -| Step 5 | BookingCreated event dispatched | Event | [VERIFIED: BookingService.php:52] | -| Step 6 | Confirmation email queued | Queue Job | [VERIFIED: Listeners/SendBookingConfirmation.php:18] | -| Step 7 | 201 Created response returned | HTTP Response | [VERIFIED: BookingController.php:67] | +| Step 3 | Booking row created with status `'pending'` | Database Write | [VERIFIED: app/Http/Controllers/BookingController.php:49-54] | +| Step 3 | Full slot rejected: error flash, no booking row | HTTP Response | [VERIFIED: app/Http/Controllers/BookingController.php:43-46] | +| Step 4 | `BookingCreated` event dispatched (synchronously) | Event | [VERIFIED: app/Http/Controllers/BookingController.php:57] | +| Step 7 | HTTP POST to external calendar `/events` | External API Call | [VERIFIED: app/Services/CalendarService.php:30-33] | +| Step 8 | `external_calendar_id` persisted on successful sync | Database Write | [VERIFIED: app/Models/Booking.php:52-56] | +| Step 8 | Status updated to `'confirmed'` (unconditionally) | State Transition | [VERIFIED: app/Http/Controllers/BookingController.php:60] | +| Step 8 | Redirect to `booking.index` with success flash | HTTP Response | [VERIFIED: app/Http/Controllers/BookingController.php:62-63] | --- ## 2. Invariants -### Invariant 1: Booking and TimeSlot updates are atomic -- **Type**: Data Integrity -- **Based on**: Step 3, Step 4 of Code Flow -- **Evidence**: [VERIFIED: BookingService.php:42-50] — wrapped in `DB::transaction()` -- **Implication**: If booking fails, slot must remain available +### Invariant 1: Booking ends `'confirmed'` after a successful store +- **Type**: State +- **Based on**: Steps 3, 8 of Code Flow +- **Evidence**: [VERIFIED: app/Http/Controllers/BookingController.php:49-54, 60] -### Invariant 2: BookingCreated event fires exactly once per booking +### Invariant 2: `BookingCreated` fires exactly once per created booking - **Type**: Cardinality -- **Based on**: Step 5 of Code Flow -- **Evidence**: [VERIFIED: BookingService.php:52] — single `event()` call, no loop -- **Implication**: Duplicate events would trigger duplicate emails +- **Based on**: Step 4 of Code Flow +- **Evidence**: [VERIFIED: app/Http/Controllers/BookingController.php:57] — single `event()` call, no loop + +### Invariant 3: Event fires after the insert but before confirmation +- **Type**: Ordering +- **Based on**: Steps 3, 4, 8 of Code Flow +- **Evidence**: [VERIFIED: app/Http/Controllers/BookingController.php:49-60] — create at 49, event at 57, update at 60 +- **Implication**: the listener (and external calendar) observe a `'pending'` booking. This is a documented wart — a test here pins current behavior, it does not bless it. -### Invariant 3: Confirmation email only sends for confirmed bookings +### Invariant 4: `markSynced` runs only when sync returned an id - **Type**: Conditional -- **Based on**: Step 5, Step 6 of Code Flow -- **Evidence**: [VERIFIED: Listeners/SendBookingConfirmation.php:15] — checks `$booking->status === 'confirmed'` -- **Implication**: Pending/cancelled bookings must not trigger email +- **Based on**: Steps 6, 8 of Code Flow +- **Evidence**: [VERIFIED: app/Listeners/SyncToExternalCalendar.php:30-32] — guarded by `if ($externalId)` -### Invariant 4: Response includes booking ID and confirmation URL -- **Type**: Data Integrity -- **Based on**: Step 7 of Code Flow -- **Evidence**: [VERIFIED: BookingController.php:65-67] — returns `BookingResource` -- **Implication**: Client needs ID for subsequent operations +### Invariant 5: Confirmation does NOT depend on sync success +- **Type**: Conditional (inverted expectation) +- **Based on**: Steps 7, 8 of Code Flow +- **Evidence**: [VERIFIED: app/Services/CalendarService.php:44-51] — API failure logged, `null` returned; [VERIFIED: app/Http/Controllers/BookingController.php:60] — update is unconditional +- **Implication**: tests must assert this current behavior knowingly and flag it for a product decision, not "fix" it silently. --- ## 3. Failure Modes -### Failure Mode 1: TimeSlot already booked (race condition) -- **Step**: Step 3, Step 4 -- **Cause**: Concurrent request books same slot -- **Expected Behavior**: [VERIFIED: BookingService.php:44] — `lockForUpdate()` on TimeSlot -- **Risk**: Low — pessimistic locking prevents race -- **Test Priority**: Medium — verify lock behavior - -### Failure Mode 2: User exceeds booking limit -- **Step**: Step 2 -- **Cause**: User already has max bookings for period -- **Expected Behavior**: [NOT_FOUND] — no booking limit check in traced flow -- **Risk**: High — could allow unlimited bookings -- **Recommendation**: Verify business rule exists elsewhere or add - -### Failure Mode 3: Email service unavailable -- **Step**: Step 6 -- **Cause**: SMTP server down or rate limited -- **Expected Behavior**: [VERIFIED: Listeners/SendBookingConfirmation.php:22] — exception caught, logged -- **Risk**: Low — booking succeeds, email fails gracefully -- **Test Priority**: Low — failure is acceptable - -### Failure Mode 4: Transaction rollback on exception -- **Step**: Step 3-4 -- **Cause**: Any exception within transaction -- **Expected Behavior**: [VERIFIED: BookingService.php:42] — `DB::transaction()` auto-rollback -- **Risk**: Low — Laravel handles this -- **Test Priority**: Medium — verify no partial state +### Failure Mode 1: External calendar API returns non-2xx +- **Step**: Step 7 +- **Cause**: API rejects the payload or is down (5xx) +- **Expected Behavior**: [VERIFIED: app/Services/CalendarService.php:44-51] — error logged, `null` returned, no retry, no exception +- **Downstream**: booking is still confirmed (Invariant 5); user believes they are booked but no calendar event exists +- **Risk**: High + +### Failure Mode 2: HTTP transport exception (timeout, DNS) +- **Step**: Step 7 +- **Cause**: connection-level failure +- **Expected Behavior**: [VERIFIED: app/Services/CalendarService.php:29-33] — the code's own comment says "no try/catch here - errors bubble up". [INFERRED] the exception propagates through the synchronous listener into the controller before line 60 runs, so the booking row exists but stays `'pending'` and the user gets an error page. +- **Risk**: High + +### Failure Mode 3: Concurrent bookings on the last spot +- **Step**: Step 3 +- **Cause**: two requests pass the capacity count before either is confirmed +- **Expected Behavior**: [NOT_FOUND: searched "lockForUpdate", "transaction", "unique(" in app/ and database/migrations/] — no locking, no transaction, no unique constraint. The migration's own comment flags the missing constraint [VERIFIED: database/migrations/2024_01_01_000003_create_bookings_table.php:27-28] +- **Risk**: High — overbooking is possible + +### Failure Mode 4: Pending bookings are invisible to the capacity check +- **Step**: Step 3 +- **Cause**: the check counts only `'confirmed'` bookings [VERIFIED: app/Http/Controllers/BookingController.php:43] +- **Expected Behavior**: [INFERRED] — a booking stuck at `'pending'` (see Failure Mode 2) never consumes capacity, widening the overbooking window +- **Risk**: Medium + +### Failure Mode 5: Per-user booking limit not enforced +- **Step**: Step 3 +- **Cause**: config defines a limit [VERIFIED: config/calendar.php:42] but [NOT_FOUND: searched "max_bookings_per_user" in app/] — nothing reads it +- **Risk**: Medium — one user can book unlimited slots --- ## 4. Test Candidates -### Test 1: Successful booking creates confirmed record +### Test 1: Successful booking ends `'confirmed'` and fires the event once +- **Type**: Integration +- **Priority**: Critical +- **Validates**: + - Booking row exists for the right user and slot + - Final status is `'confirmed'` + - `BookingCreated` dispatched exactly once +- **Based on**: Steps 3, 4, 8 of Code Flow +- **Preconditions**: authenticated user, slot with free capacity +- **Expected Outcome**: redirect with success flash; confirmed row in DB +- **Verification**: [VERIFIED: app/Http/Controllers/BookingController.php:49-63] + +### Test 2: Full slot is rejected — no row, no event - **Type**: Integration - **Priority**: Critical - **Validates**: - - Booking record exists in database - - Status is 'confirmed' - - Associated with correct user and time slot -- **Based on**: Steps 3, 7 of Code Flow -- **Preconditions**: Valid user, available time slot -- **Expected Outcome**: 201 response, booking in DB with status='confirmed' -- **Verification**: [VERIFIED: BookingService.php:45-50] - -### Test 2: Booking marks time slot as unavailable + - `back()` with error flash when confirmed count >= capacity + - No booking row created, no event fired +- **Based on**: Step 3 of Code Flow (alternate exit) +- **Preconditions**: slot with capacity fully consumed by `'confirmed'` bookings +- **Expected Outcome**: error flash "This slot is no longer available" +- **Verification**: [VERIFIED: app/Http/Controllers/BookingController.php:43-46] + +### Test 3: Calendar sync failure still confirms the booking (pin current behavior) - **Type**: Integration - **Priority**: Critical - **Validates**: - - TimeSlot.is_available changes from true to false - - Change is atomic with booking creation -- **Based on**: Steps 3, 4 of Code Flow -- **Preconditions**: Slot was available before request -- **Expected Outcome**: Slot no longer available for other bookings -- **Verification**: [VERIFIED: BookingService.php:48] - -### Test 3: BookingCreated event contains correct payload + - When the external API responds non-2xx, `syncBooking` returns `null` + - `external_calendar_id` stays null, yet status still becomes `'confirmed'` +- **Based on**: Steps 7, 8 of Code Flow; Failure Mode 1; Invariant 5 +- **Preconditions**: external API stubbed to fail +- **Expected Outcome**: confirmed booking with no external id — the test documents the divergence +- **Verification**: [VERIFIED: app/Services/CalendarService.php:44-51, app/Http/Controllers/BookingController.php:60] + +### Test 4: Successful sync persists `external_calendar_id` +- **Type**: Integration +- **Priority**: Important +- **Validates**: + - Listener passes the booking to `CalendarService::syncBooking` + - Returned id is stored via `markSynced` +- **Based on**: Steps 6, 8 of Code Flow +- **Preconditions**: external API stubbed to return an `id` +- **Expected Outcome**: booking row carries the external id +- **Verification**: [VERIFIED: app/Listeners/SyncToExternalCalendar.php:28-32, app/Models/Booking.php:52-56] + +### Test 5: Sync payload has the documented shape - **Type**: Unit - **Priority**: Important - **Validates**: - - Event has booking_id property - - Event has user_id property - - Event has slot_id property -- **Based on**: Step 5 of Code Flow -- **Preconditions**: Successful booking -- **Expected Outcome**: Event payload matches booking data -- **Verification**: [VERIFIED: Events/BookingCreated.php:12-18] - -### Test 4: Confirmation email listener responds to event + - Payload contains `title`, `start`, `end`, `attendee_email`, `metadata.booking_id`, `metadata.source` + - Timestamps use the hardcoded `Y-m-d\TH:i:s` format +- **Based on**: Step 7 of Code Flow +- **Preconditions**: booking with related user and time slot +- **Expected Outcome**: outgoing request body matches the built payload +- **Verification**: [VERIFIED: app/Services/CalendarService.php:86-95] + +### Test 6: Listener observes status `'pending'` (ordering pin) - **Type**: Integration - **Priority**: Important - **Validates**: - - Listener is registered for BookingCreated - - Email job is dispatched to queue -- **Based on**: Steps 5, 6 of Code Flow -- **Preconditions**: Event dispatched -- **Expected Outcome**: Job exists in queue with correct booking_id -- **Verification**: [VERIFIED: EventServiceProvider.php:22, Listeners/SendBookingConfirmation.php:18] - -### Test 5: Concurrent booking requests don't double-book slot + - At handler time, the booking status is still `'pending'` +- **Based on**: Steps 3, 4, 8 of Code Flow; Invariant 3 +- **Preconditions**: normal successful booking +- **Expected Outcome**: assertion inside a listener spy sees `'pending'`; flags the ordering wart if someone "fixes" it silently +- **Verification**: [VERIFIED: app/Http/Controllers/BookingController.php:52, 57, 60] + +### Test 7: No confirmation email is sent (absence test) - **Type**: Integration -- **Priority**: Critical -- **Validates**: - - Second request fails with 409 or 422 - - Only one booking exists for slot -- **Based on**: Failure Mode 1 -- **Preconditions**: Two simultaneous requests for same slot -- **Expected Outcome**: One succeeds, one fails, no duplicate -- **Verification**: [VERIFIED: BookingService.php:44] — `lockForUpdate()` - -### Test 6: Invalid time slot ID returns 422 -- **Type**: Unit - **Priority**: Important - **Validates**: - - Request validation rejects non-existent slot - - No booking created - - Error message indicates invalid slot -- **Based on**: Step 2 of Code Flow -- **Preconditions**: Request with invalid slot_id -- **Expected Outcome**: 422 with validation error -- **Verification**: [VERIFIED: BookingController.php:34] — `exists:time_slots,id` rule + - The flow completes without dispatching any mailable or notification +- **Based on**: [NOT_FOUND: searched "mail", "Mail::", "Notification", "notify" in app/] — the flow has no email step +- **Preconditions**: normal successful booking +- **Expected Outcome**: zero mail/notification dispatches. A suite that asserts a `BookingConfirmation` mailable (as the bad example does) is testing hallucinated behavior. +- **Verification**: [VERIFIED: app/Http/Controllers/BookingController.php:37-64] — the full `store()` method contains no mail call + +### Test 8: Concurrent last-spot requests (overbooking probe) +- **Type**: Integration +- **Priority**: Critical +- **Validates**: + - Whether two near-simultaneous requests for a 1-capacity slot both succeed +- **Based on**: Failure Mode 3 +- **Preconditions**: slot with capacity 1, two concurrent submissions +- **Expected Outcome**: with current code, both likely succeed (documents the defect); desired behavior — one rejected — requires the missing constraint +- **Verification**: [NOT_FOUND: searched "unique(" in database/migrations/] no guard exists; see migration comment at database/migrations/2024_01_01_000003_create_bookings_table.php:27-28 --- @@ -163,12 +193,14 @@ | Test Candidate | Impact | Likelihood | External | Priority | |----------------|--------|------------|----------|----------| -| Test 1: Successful booking | 3 | 2 | 3 | 8 (Critical) | -| Test 2: Slot marked unavailable | 3 | 2 | 3 | 8 (Critical) | -| Test 5: No double-booking | 3 | 2 | 3 | 8 (Critical) | -| Test 4: Email listener works | 2 | 1 | 2 | 5 (Important) | -| Test 3: Event payload correct | 2 | 1 | 1 | 4 (Important) | -| Test 6: Invalid slot rejected | 2 | 1 | 1 | 4 (Important) | +| Test 1: Successful booking confirmed | 3 | 2 | 3 | 8 (Critical) | +| Test 3: Sync failure still confirms | 3 | 2 | 3 | 8 (Critical) | +| Test 8: Overbooking probe | 3 | 2 | 3 | 8 (Critical) | +| Test 2: Full slot rejected | 3 | 2 | 2 | 7 (Critical) | +| Test 4: External id persisted | 2 | 2 | 2 | 6 (Important) | +| Test 6: Listener sees `'pending'` | 2 | 2 | 1 | 5 (Important) | +| Test 5: Payload shape | 2 | 1 | 2 | 5 (Important) | +| Test 7: No email sent (absence) | 2 | 1 | 1 | 4 (Important) | --- @@ -176,9 +208,9 @@ | Gap | Reason | Recommendation | |-----|--------|----------------| -| Booking limit per user | [NOT_FOUND] in flow | Verify rule exists or add to requirements | -| Cancellation refund logic | Not in this flow | Document separate "Cancel Booking" flow first | -| Admin override booking | Not in this flow | Document admin flow separately | +| External HTTP isolation | `CalendarService` reads config in its constructor and calls the `Http` facade directly [VERIFIED: app/Services/CalendarService.php:18-19, 30-33] | Stub at the HTTP layer, or bind a test double through `CalendarServiceInterface` [VERIFIED: app/Providers/CalendarServiceProvider.php:19-22] | +| Retry/timeout behavior | `sync_timeout` and `retry_attempts` are defined but never read [VERIFIED: config/calendar.php:27-28] | Nothing to test until implemented — do not write tests for config that has no effect | +| Queue behavior | The listener runs synchronously; no jobs or queues exist in this flow [VERIFIED: app/Listeners/SyncToExternalCalendar.php:10] | Do not assert queued jobs; revisit if sync is ever made async | --- @@ -186,17 +218,18 @@ | Status | Count | |--------|-------| -| VERIFIED | 14 | -| INFERRED | 0 | -| NOT_FOUND | 1 | +| VERIFIED | 29 | +| INFERRED | 2 | +| NOT_FOUND | 4 | --- ## Why This Example is Good -1. **Every test candidate cites flow steps** — no invented behavior -2. **Verification tags on all claims** — `[VERIFIED: file:line]` -3. **Failure modes are grounded** — derived from actual code patterns -4. **NOT_FOUND items are flagged** — booking limit gap is explicit -5. **Priority is justified** — scoring based on impact/likelihood/external -6. **No test code** — candidates only, implementation left to developer +1. **Every candidate cites real flow steps and real files** — the flow document exists and its citations were verified first. +2. **Absence tests come from `[NOT_FOUND]`** — no email, no queue, no booking limit. The bad example asserts a confirmation email; this doc proves that would test hallucinated behavior. +3. **Warts are pinned, not blessed** — sync-failure-still-confirms and the `'pending'` ordering are tested as *current* behavior and flagged for product decisions. +4. **Failure modes are grounded** — each one traces to a specific line or a documented search, not to generic "what if the server dies" speculation. +5. **Priorities are scored** — impact/likelihood/external, not vibes. +6. **No test code** — candidates only; framework and mocking strategy are left to the implementer. +7. **Machine-checkable** — the "Verify with" command exits 0 against the slotbooker source. diff --git a/examples/verifier/good-architecture-doc-example.md b/examples/verifier/good-architecture-doc-example.md index 9e3c75c..cf7fff6 100644 --- a/examples/verifier/good-architecture-doc-example.md +++ b/examples/verifier/good-architecture-doc-example.md @@ -39,7 +39,7 @@ |-------|-------| | Category | Traditional Code | | Type | CLI Tool (single-file Python script with importable functions) | -| Evidence | `verify.py` is an executable script with `argparse` options and a `__main__` guard [VERIFIED: verify.py:1, verify.py:293-308]; no framework deps, no model files | +| Evidence | `verify.py` is an executable script with `argparse` options and a `__main__` guard [VERIFIED: verify.py:1, verify.py:318-340]; no framework deps, no model files | | Overlay Loaded | No | | Confidence | `[VERIFIED]` | @@ -57,39 +57,47 @@ | Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| `Citation` dataclass | `verify.py` | Holds `path`, `start`, `end` for one cited location; carries the `.check(root)` method that confirms file+line bounds | [VERIFIED: verify.py:40-62] | -| `Tag` dataclass | `verify.py` | Holds one parsed tag (`VERIFIED`, `NOT_FOUND`, etc.) with its citations and document line number | [VERIFIED: verify.py:64-69] | -| `QuoteCheck` dataclass | `verify.py` | Phase-2 result: tag + citation + ratio + first-diff hint | [VERIFIED: verify.py:139-147] | +| `Citation` dataclass | `verify.py` | Holds `path`, `start`, `end` for one cited location; carries the `.check(root)` method that confirms file+line bounds | [VERIFIED: verify.py:43-67] | +| `Tag` dataclass | `verify.py` | Holds one parsed tag (`VERIFIED`, `NOT_FOUND`, etc.) with its citations and document line number | [VERIFIED: verify.py:70-75] | +| `QuoteCheck` dataclass | `verify.py` | Phase-2 result: tag + citation + ratio + first-diff hint | [VERIFIED: verify.py:145-152] | ### Parsing Pipeline | Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| `TAG_RE` regex | `verify.py` | Matches top-level tag forms: `[VERIFIED]`, `[VERIFIED: ...]`, `[NOT_FOUND: ...]`, `[INFERRED]`, `[ASSUMED: ...]`, `[NEEDS_VERIFICATION]` | [VERIFIED: verify.py:24-26] | -| `FRESH_CITE` regex | `verify.py` | Picks `path:N(-N)?` citations out of a tag payload, requiring path to have a slash or an extension | [VERIFIED: verify.py:28-30] | -| `CONT_CITE` regex | `verify.py` | Picks shorthand continuations like `, M, P` reusing the previous path | [VERIFIED: verify.py:32] | -| `_extract_citations` | `verify.py` | Combines `FRESH_CITE` + `CONT_CITE` walks to expand shorthand into one Citation per cited line range | [VERIFIED: verify.py:72-114] | -| `_in_inline_code` | `verify.py` | Skips tag-shaped text inside `` `backticks` `` so doc-internal *descriptions* of the tag format aren't mistaken for citations | [VERIFIED: verify.py:116-125] | -| `parse_doc` | `verify.py` | Walks lines, applies `TAG_RE`, filters inline-code matches, expands citations | [VERIFIED: verify.py:127-136] | +| `TAG_RE` regex | `verify.py` | Matches top-level tag forms: `[VERIFIED]`, `[VERIFIED: ...]`, `[NOT_FOUND: ...]`, `[INFERRED]`, `[ASSUMED: ...]`, `[NEEDS_VERIFICATION]`, `[NEEDS_RUNTIME]`, `[DRIFT: ...]` | [VERIFIED: verify.py:23-29] | +| `FRESH_CITE` regex | `verify.py` | Picks `path:N(-N)?` citations out of a tag payload, requiring path to have a slash or an extension | [VERIFIED: verify.py:30-33] | +| `CONT_CITE` regex | `verify.py` | Picks shorthand continuations like `, M, P` reusing the previous path | [VERIFIED: verify.py:35] | +| `_extract_citations` | `verify.py` | Combines `FRESH_CITE` + `CONT_CITE` walks to expand shorthand into one Citation per cited line range | [VERIFIED: verify.py:78-120] | +| `_in_inline_code` | `verify.py` | Skips tag-shaped text inside `` `backticks` `` so doc-internal *descriptions* of the tag format aren't mistaken for citations | [VERIFIED: verify.py:122-131] | +| `parse_doc` | `verify.py` | Walks lines, applies `TAG_RE`, filters inline-code matches, expands citations | [VERIFIED: verify.py:133-142] | ### Phase 2 (Quoted-Code Matching) | Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| `FENCE_RE` regex | `verify.py` | Detects fenced code block markers (`` ``` ``) with optional indentation | [VERIFIED: verify.py:34] | -| `_find_fenced_block` | `verify.py` | Looks ±N lines after a tag for a fence; returns block contents and opening-fence line | [VERIFIED: verify.py:149-170] | -| `_norm_lines` | `verify.py` | Strips trailing whitespace and trailing blank lines so trivial whitespace drift doesn't fail a match | [VERIFIED: verify.py:172-178] | -| `_diff_hint` | `verify.py` | Reports first-differing-line or length mismatch when a quote fails | [VERIFIED: verify.py:180-188] | -| `check_quoted_code` | `verify.py` | Orchestrates: tag → fenced block → file slice → `SequenceMatcher` ratio → pass/fail | [VERIFIED: verify.py:190-215] | -| `QUOTE_RATIO_PASS` constant | `verify.py` | Hard-coded similarity threshold for phase 2; deliberately looser than the citation threshold to tolerate minor whitespace drift | [VERIFIED: verify.py:37] | +| `FENCE_RE` regex | `verify.py` | Detects fenced code block markers (`` ``` ``) with optional indentation | [VERIFIED: verify.py:40] | +| `_find_fenced_block` | `verify.py` | Looks ±N lines after a tag for a fence; returns block contents and opening-fence line | [VERIFIED: verify.py:155-176] | +| `_norm_lines` | `verify.py` | Strips trailing whitespace and trailing blank lines so trivial whitespace drift doesn't fail a match | [VERIFIED: verify.py:178-184] | +| `_diff_hint` | `verify.py` | Reports first-differing-line or length mismatch when a quote fails | [VERIFIED: verify.py:186-194] | +| `check_quoted_code` | `verify.py` | Orchestrates: tag → fenced block → file slice → `SequenceMatcher` ratio → pass/fail | [VERIFIED: verify.py:196-223] | +| `QUOTE_RATIO_PASS` constant | `verify.py` | Hard-coded similarity threshold for phase 2; deliberately looser than the citation threshold to tolerate minor whitespace drift | [VERIFIED: verify.py:40] | + +The phase-2 threshold is a single constant — quoted here exactly so this +example also exercises the phase-2 quote check it describes: + +[VERIFIED: verify.py:40] +```python +QUOTE_RATIO_PASS = 0.90 +``` ### Top-Level Entry | Component | Location | Responsibility | Evidence | |-----------|----------|----------------|----------| -| `verify` function | `verify.py` | Reads the doc, runs phase 1 on all citations, runs phase 2 on tags with following fences, prints the report, returns the exit code | [VERIFIED: verify.py:217-291] | -| `main` + argparse | `verify.py` | CLI surface — accepts `doc`, `--repo-root`, `--threshold`, `-v` | [VERIFIED: verify.py:293-304] | -| `__main__` guard | `verify.py` | Standard `if __name__ == "__main__": sys.exit(main())` entry | [VERIFIED: verify.py:307-308] | +| `verify` function | `verify.py` | Reads the doc, runs phase 1 on all citations, runs phase 2 on tags with following fences, prints the report, returns the exit code | [VERIFIED: verify.py:242-315] | +| `main` + argparse | `verify.py` | CLI surface — accepts `doc`, `--repo-root`, `--threshold`, `-v`, `--emit-summary` | [VERIFIED: verify.py:318-336] | +| `__main__` guard | `verify.py` | Standard `if __name__ == "__main__": sys.exit(main())` entry | [VERIFIED: verify.py:339-340] | --- @@ -99,9 +107,9 @@ | Entry Surface | Type | Primary Components Involved | Evidence | |---------------|------|-----------------------------|----------| -| `python3 verify.py DOC` | CLI | `main` → `verify` → `parse_doc` → `Citation.check` → `check_quoted_code` | [VERIFIED: verify.py:293-308] | -| `python3 verify.py DOC --repo-root DIR` | CLI | Same, but resolves citations against `DIR` instead of cwd | [VERIFIED: verify.py:297-298] | -| `python3 verify.py DOC --threshold 0.9 -v` | CLI | Tightens/loosens phase-1 pass criterion; verbose mode lists every resolved citation and informal tag | [VERIFIED: verify.py:299-302] | +| `python3 verify.py DOC` | CLI | `main` → `verify` → `parse_doc` → `Citation.check` → `check_quoted_code` | [VERIFIED: verify.py:318-340] | +| `python3 verify.py DOC --repo-root DIR` | CLI | Same, but resolves citations against `DIR` instead of cwd | [VERIFIED: verify.py:322-323] | +| `python3 verify.py DOC --threshold 0.9 -v` | CLI | Tightens/loosens phase-1 pass criterion; verbose mode lists every resolved citation and informal tag | [VERIFIED: verify.py:324-327] | | `from verify import parse_doc, verify` | Library | The internal API is importable; no `__all__`, but functions are module-level | [INFERRED: standard Python module layout, no `__init__.py`] | ### 3.2 High-Level Data Movement (Non-Procedural) @@ -165,8 +173,8 @@ Not applicable — `verify.py` is a CLI tool with no frontend. The only user sur Explicitly **NOT** in this tool: - Auto-fixing the doc — the verifier never edits the doc; the agent must read failures and fix [VERIFIED: verify.py: only reads `doc_path`, never writes] -- Re-running grep for `[NOT_FOUND]` tags — `[NOT_FOUND]` is counted but not actively re-checked [VERIFIED: verify.py:235-237, NOT_FOUND falls through to `continue`] -- Fuzzy path matching — a citation to `bench/score.py` when the real file is `bench/scorer.py` fails as "missing file"; no Levenshtein-style suggestion is offered [VERIFIED: verify.py:43-44] +- Re-running grep for `[NOT_FOUND]` tags — `[NOT_FOUND]` is counted but not actively re-checked [VERIFIED: verify.py:254-256, NOT_FOUND falls through to `continue`] +- Fuzzy path matching — a citation to `bench/score.py` when the real file is `bench/scorer.py` fails as "missing file"; no Levenshtein-style suggestion is offered [VERIFIED: verify.py:57-58] - Cross-file consistency (e.g., "did you cite this file twice with conflicting line ranges?") — out of scope - Semantic check of *what the doc claims* — only that the cited evidence exists; if the doc says "this is a database" and cites a real line that says `import socket`, the verifier passes anyway @@ -176,10 +184,10 @@ Explicitly **NOT** in this tool: | Risk | Location | Notes | |------|----------|-------| -| Inline-code skip is line-local | `verify.py:116-125` | Counts backticks within one line; a `` ` `` that opens on one line and closes on the next will desync detection. Multi-line inline code is rare in practice. | -| `--repo-root` is mandatory when example doesn't live next to its cited files | `verify.py:297-298` | This very example needs `--repo-root .pf-agent-system-mapper` because it sits in `examples/verifier/` but cites `verify.py:N` | -| File reads assume UTF-8 (with replace) | `verify.py:46`, `verify.py:206` | A truly binary file path would still "read" via replace and report incorrect line counts | -| `[NOT_FOUND]` payload not re-verified | `verify.py:235-237` | A future phase could re-grep the claimed search term to confirm the absence | +| Inline-code skip is line-local | `verify.py:122-131` | Counts backticks within one line; a `` ` `` that opens on one line and closes on the next will desync detection. Multi-line inline code is rare in practice. | +| `--repo-root` is mandatory when example doesn't live next to its cited files | `verify.py:322-323` | This very example needs `--repo-root .pf-agent-system-mapper` because it sits in `examples/verifier/` but cites `verify.py:N` | +| File reads assume UTF-8 (with replace) | `verify.py:60`, `verify.py:214` | A truly binary file path would still "read" via replace and report incorrect line counts | +| `[NOT_FOUND]` payload not re-verified | `verify.py:254-256` | A future phase could re-grep the claimed search term to confirm the absence | --- @@ -187,9 +195,9 @@ Explicitly **NOT** in this tool: | Entry Type | Count | Locations | |------------|-------|-----------| -| CLI scripts | 1 | `verify.py` [VERIFIED: verify.py:1, verify.py:307-308] | -| CLI subcommands | 0 | flat argparse — one mode of operation [VERIFIED: verify.py:293-304] | -| Public library functions | 4 | `parse_doc`, `check_quoted_code`, `verify`, plus the dataclasses [VERIFIED: verify.py:127, verify.py:190, verify.py:217] | +| CLI scripts | 1 | `verify.py` [VERIFIED: verify.py:1, verify.py:339-340] | +| CLI subcommands | 0 | flat argparse — one primary mode plus `--emit-summary` [VERIFIED: verify.py:318-336] | +| Public library functions | 5 | `parse_doc`, `check_quoted_code`, `emit_summary`, `verify`, plus the dataclasses [VERIFIED: verify.py:133, verify.py:196, verify.py:226, verify.py:242] | | HTTP routes | 0 | [NOT_FOUND: no network] | | Out-of-process commands | 0 | [NOT_FOUND: no `subprocess`, no shell-out] | @@ -199,7 +207,7 @@ Explicitly **NOT** in this tool: | Layer | Technology | Evidence | |-------|------------|----------| -| Language | Python 3.10+ (uses PEP 604 union syntax `X \| Y`) | [VERIFIED: verify.py:149-150 `tuple[list[str], int] \| None`] | +| Language | Python 3.10+ (uses PEP 604 union syntax `X \| Y`) | [VERIFIED: verify.py:155-156 `tuple[list[str], int] \| None`] | | CLI | `argparse` (stdlib) | [VERIFIED: verify.py:15] | | Pattern matching | `re` (stdlib) | [VERIFIED: verify.py:16] | | Similarity scoring | `difflib.SequenceMatcher` (stdlib) | [VERIFIED: verify.py:19] | @@ -212,12 +220,15 @@ Explicitly **NOT** in this tool: | Status | Count | |--------|-------| -| VERIFIED | 32 | +| VERIFIED | 41 | | INFERRED | 1 | | NOT_FOUND | 5 | | ASSUMED | 0 | | NEEDS_VERIFICATION | 0 | +Counts come from `python3 verify.py --emit-summary` — never hand-count. +(2 of the VERIFIED tags are informal/structural; 1 quoted block exercises phase 2.) + Run the verifier against this very file to confirm: ```bash diff --git a/examples/vue/bad-architecture-doc-example.md b/examples/vue/bad-architecture-doc-example.md index b53f039..cba16e5 100644 --- a/examples/vue/bad-architecture-doc-example.md +++ b/examples/vue/bad-architecture-doc-example.md @@ -1,14 +1,28 @@ # Architecture Overview: Vue Kanban Board +> ⚠️ **BAD EXAMPLE — DO NOT IMITATE.** This document demonstrates hallucination patterns — including the subtlest one: real citations with selectively truncated quotes. Each ❌ callout explains a failure. See good-architecture-doc-example.md for the correct approach. + ## System Purpose A real-time collaborative Kanban board built with Vue 3 and Pinia, featuring automatic synchronization and offline support. Cards sync instantly across all clients with conflict-free updates. +> **❌ PROBLEMS:** +> - "Real-time collaborative" — there is no server and no other clients. The API service's own header says "No actual server exists" (`src/services/api.js:5-6`). +> - "Automatic synchronization" — the board store's architecture note says the opposite: "Changes here are NOT automatically persisted to server" (`src/stores/boardStore.js:10-11`). +> - "Conflict-free updates" — the sync store's note lists "No conflict resolution" as a known gap (`src/stores/syncStore.js:12`). +> - No Metadata table, no commit hash, no date, no verification tags anywhere in the document. Nothing is checkable at a fixed revision. + ## Technology Stack - Vue 3 with Composition API for reactive UI - Pinia for state management with automatic persistence - Real-time sync via built-in API service - Offline-first architecture with seamless reconnection +> **❌ PROBLEMS:** +> - No citations. The two true items (Vue 3, Pinia) are stated without evidence (`package.json:12-13` would prove them), so the reader cannot tell them apart from the false ones. +> - "Automatic persistence" — persistence is a manual `saveToLocal()` call inside each store action (`src/stores/boardStore.js:60-66`, called at 89, 106, 121, 148). Nothing subscribes to state changes to persist them. +> - "Real-time sync via built-in API service" — the "API service" is an in-memory mock: `let serverState = { ... }` that "resets on refresh" (`src/services/api.js:17-21`). +> - "Seamless reconnection" — the reconnect watcher syncs without any conflict check (`src/stores/syncStore.js:101-102`) and the sync loop stops on the first error (`src/stores/syncStore.js:81`). + ## Core Architecture ### State Management @@ -23,6 +37,11 @@ Reference: `src/stores/boardStore.js:19-20` When cards are modified, Vue's reactivity system ensures all components update and changes persist automatically. +> **❌ PROBLEMS:** +> - The quoted block is doctored. The comment `// State automatically syncs when changed` does not exist anywhere in the file — it was invented and pasted above two real lines. The real lines 19-20 read `const columns = ref([])` and `const cards = ref({}) // { [cardId]: card }`; the real inline comment was stripped and a fake one substituted. +> - The reference says lines 19-20 but the block shows three lines — the citation resolves, yet the quote is not the cited code. A resolving citation is not the same as a faithful quote. +> - "Changes persist automatically" — false. Every mutation calls `saveToLocal()` explicitly (`src/stores/boardStore.js:89`), and the file's header comment says changes are "NOT automatically persisted to server" (`src/stores/boardStore.js:10-11`). + ### Synchronization Flow The sync system provides real-time updates: @@ -43,6 +62,17 @@ watch( ``` Reference: `src/stores/boardStore.js:167-174` +> **❌ PROBLEMS:** +> - Numbered step-by-step tracing is banned in architecture overviews — execution tracing belongs in code-flow documentation. Steps 3 and 4 are also simply false: nothing syncs automatically and there are no other clients. +> - **Truncated quote.** The citation `boardStore.js:167-174` resolves, and the code shown is real — but lines 171-172 were silently deleted from the middle of the block: +> +> ```javascript +> // This log makes it LOOK like we're tracking changes +> // but no actual sync happens here +> ``` +> +> The removed lines state, in the authors' own words, that this watcher does **no** sync — the exact opposite of the claim it is quoted to support. The wart comment directly above the watcher (`src/stores/boardStore.js:165-166`) says the same and was also omitted. + ### API Layer The API service handles all server communication with automatic retry and conflict resolution: @@ -57,6 +87,24 @@ async syncCard(card) { ``` Reference: `src/services/api.js:33-49` +> **❌ PROBLEMS:** +> - **This is the subtlest failure in the document: a real citation with a selectively truncated quote.** The citation `src/services/api.js:33-49` resolves, and every quoted line is genuine — but the quote silently drops lines 34-41 and 46 from the middle of the cited range. Here is what was cut (`src/services/api.js:34-41`): +> +> ```javascript +> await delay(200 + Math.random() * 300) // Simulate network +> +> // Simulate occasional failures (10% chance) +> if (Math.random() < 0.1) { +> throw new Error('Network error - sync failed') +> } +> +> // Wart: No version checking - blindly overwrites +> ``` +> +> - The dropped lines refute both halves of the claim. "Automatic retry": the function randomly *throws* 10% of the time (`src/services/api.js:37-38`) and no layer retries — the sync loop breaks on the first error (`src/stores/syncStore.js:81`). "Conflict resolution": the authors' own comment on the dropped line 41 says "No version checking - blindly overwrites". +> - "Handles all server communication" — the surviving lines write to `serverState`, which is a module-level in-memory object that "resets on refresh" (`src/services/api.js:17-21`). There is no server communication at all. +> - Lesson: a verifier can confirm the citation resolves and the quoted lines exist — only re-reading the *whole* cited range reveals that the quote was curated to invert the code's meaning. Citations that resolve are not the same as claims that are true. + ### Drag and Drop Drag operations use the composable pattern for reusable logic. When a card is dropped, it's immediately synced: @@ -65,6 +113,17 @@ boardStore.moveCard(cardId, fromColumnId, toColumnId, index) ``` Reference: `src/composables/useDragDrop.js:62` +> **❌ PROBLEMS:** +> - **Truncated quote again.** Line 62 is real, but the two comment lines immediately above it (`src/composables/useDragDrop.js:60-61`) were cut: +> +> ```javascript +> // Wart: Immediate mutation without optimistic UI pattern +> // This treats the drag as already "done" but it's not synced +> ``` +> +> The omitted comment says the drop is **not** synced — the opposite of "it's immediately synced". +> - What `moveCard` actually does is mark the card `pending` and queue it (`src/stores/boardStore.js:146-149`); no sync is triggered by a drop. + ### Offline Support The persistence layer uses localStorage with IndexedDB for larger datasets: @@ -78,6 +137,11 @@ Reference: `src/services/persistence.js:18-31` Data is automatically synced when the connection is restored. +> **❌ PROBLEMS:** +> - "With IndexedDB for larger datasets" — fabricated. The file header says "Using localStorage (not IndexedDB) for simplicity" (`src/services/persistence.js:8`), and IndexedDB exists only as a commented-out stub: "// Wart: No IndexedDB implementation despite being mentioned in requirements" (`src/services/persistence.js:75-76`). +> - **Truncated quote.** The citation spans 14 lines (18-31) but the block shows 4. The cut lines contain the `try`/`catch` whose comment reads "// Wart: Silently fails on quota exceeded" (`src/services/persistence.js:26-30`) — hiding that "offline support" can silently lose data. The quote also drops lines 20-23, where `serialized` is built, leaving the shown code referencing an undefined variable. +> - "Automatically synced when the connection is restored" — the reconnect watcher exists (`src/stores/syncStore.js:103-108`) but the claim omits that it syncs with no conflict check (`src/stores/syncStore.js:101-102`) and that the "server" it syncs to is in-memory (`src/services/api.js:17-21`). + ## Data Flow ``` @@ -96,6 +160,10 @@ const syncStatus = computed(() => { ``` Reference: `src/stores/syncStore.js:26-31` +> **❌ PROBLEMS:** +> - ASCII arrow flow diagrams are banned in architecture overviews — and this one is wrong twice over: "Server Sync" reaches only an in-memory mock (`src/services/api.js:17-21`) and "All Clients" do not exist. +> - The `syncStatus` quote is real and accurate — but the prose around it launders it: `'synced'` here means only that the mock returned success. The API's own header warns "The UI will show \"synced\" even though nothing actually synced" (`src/services/api.js:11`). Accurate quotes can still be used to support inaccurate narratives. + ## Key Features - Real-time collaborative editing - Automatic conflict resolution @@ -103,6 +171,14 @@ Reference: `src/stores/syncStore.js:26-31` - Drag-and-drop with optimistic updates - Persistent storage across sessions +> **❌ PROBLEMS:** +> - Every bullet is false or misleading, and none carries a tag or citation: +> - "Real-time collaborative editing" — no server, no other clients (`src/services/api.js:5-6`). +> - "Automatic conflict resolution" — "No conflict resolution" is a listed gap (`src/stores/syncStore.js:12`) and the mock "blindly overwrites" (`src/services/api.js:41`). +> - "Seamless offline/online transitions" — sync stops on the first error with no retry (`src/stores/syncStore.js:76-81`). +> - "Optimistic updates" — mutations are immediate with no rollback path (`src/composables/useDragDrop.js:60-62`). +> - "Persistent storage across sessions" — localStorage persists, but everything "synced" lives in a variable that resets on refresh (`src/services/api.js:17`). + ## Component Architecture Components follow a smart container / dumb presenter pattern: - `KanbanBoard.vue` - Container managing board state @@ -111,3 +187,25 @@ Components follow a smart container / dumb presenter pattern: - `SyncStatus.vue` - Connection status indicator All components react automatically to store changes through Pinia's built-in reactivity. + +> **❌ PROBLEMS:** +> - The "pattern" is asserted, not verified. `KanbanBoard.vue` manages no state — it is a 16-line pass-through that renders columns from the store (`src/components/KanbanBoard.vue:10-14`). +> - `KanbanCard.vue` is not a "presentational" component: it imports the board store and mutates it directly in `handleDelete()` (`src/components/KanbanCard.vue:23-26`). +> - No file:line evidence for any row, so the pattern claim cannot be checked without re-reading every component. + +--- + +## Why This Example is BAD + +The headline lesson: **selective quoting — citations that resolve are not the same as claims that are true.** Every `Reference:` line in this document points at a real file and real lines, and every quoted line genuinely exists. The document still lies, because quotes were truncated or doctored until the code appeared to say the opposite of what it says. A structural verifier passes the citations; only re-reading the full cited ranges exposes the fraud. In detail: + +1. **Selective quoting to invert meaning** — claims "automatic retry and conflict resolution", quoting `src/services/api.js:33-49` while silently dropping lines 34-41: the 10% random-failure simulation (`src/services/api.js:37-38`) and the comment "No version checking - blindly overwrites" (`src/services/api.js:41`). Reality: no retry exists anywhere (`src/stores/syncStore.js:81` breaks on first error) and conflicts are never checked. +2. **Truncating self-refuting comments** — quotes the card-count watcher (`src/stores/boardStore.js:167-174`) as "automatic sync" while deleting its middle lines 171-172, which read "no actual sync happens here". Reality: the watcher only logs. +3. **Quoting a line without its negating context** — "when a card is dropped, it's immediately synced" cites `src/composables/useDragDrop.js:62` while cutting lines 60-61 directly above: "it's not synced". Reality: a drop only marks the card `pending` (`src/stores/boardStore.js:146-149`). +4. **Doctored quote** — inserts a fabricated comment `// State automatically syncs when changed` above real code cited as `src/stores/boardStore.js:19-20`. Reality: no such comment exists in the file, and the store header says changes are "NOT automatically persisted" (`src/stores/boardStore.js:10-11`). +5. **Fabricated capability** — "localStorage with IndexedDB for larger datasets". Reality: "Using localStorage (not IndexedDB) for simplicity" (`src/services/persistence.js:8`); IndexedDB is a commented-out stub (`src/services/persistence.js:75-76`). The `save()` quote also hides the silent quota-failure catch (`src/services/persistence.js:26-30`). +6. **Invented system category** — "real-time collaborative", "all clients receive updates". Reality: "No actual server exists" (`src/services/api.js:6`); the "server" is an in-memory variable that resets on refresh (`src/services/api.js:17-21`). +7. **Accurate quote, misleading frame** — the real `syncStatus` computed (`src/stores/syncStore.js:26-31`) is used to imply server-backed state, while omitting "The UI will show \"synced\" even though nothing actually synced" (`src/services/api.js:11`). +8. **Banned formats** — a numbered step-by-step execution trace ("Synchronization Flow") and an ASCII arrow diagram ("Data Flow") in an architecture overview; both belong in code-flow documentation, as tables or pointers here. +9. **No verification apparatus** — no Metadata table, no commit hash, no date, no verification tags, no Verification Summary, no `[NOT_FOUND]` searches. None of the five canonical tags appears even once, so unverified marketing language ("Key Features") is indistinguishable from read code. +10. **Unverifiable pattern claims** — "smart container / dumb presenter" asserted with no evidence, contradicted by `KanbanCard.vue:23-26` mutating the store directly. diff --git a/examples/vue/good-architecture-doc-example.md b/examples/vue/good-architecture-doc-example.md index b538b35..0bc0df6 100644 --- a/examples/vue/good-architecture-doc-example.md +++ b/examples/vue/good-architecture-doc-example.md @@ -1,252 +1,312 @@ -# Architecture Overview: Vue Kanban Board +# Vue Kanban Board Architecture Overview -## System Purpose -[VERIFIED] A Kanban board UI for organizing cards across columns with local persistence. Includes a mock sync system that simulates server communication but does not provide actual synchronization. +> This is the GOOD architecture-overview example for Vue SPAs in the +> agent-system-mapper methodology. Every citation resolves against the mini app +> bundled at `examples/vue/kanban-board/`, so this document is self-verifiable +> from the repo root with the command shown below. -Reference: `src/stores/boardStore.js:6-16` - Architecture note explicitly states "This store manages LOCAL state. Changes here are NOT automatically persisted to server." +## Metadata -## Technology Stack -[VERIFIED] -- Vue 3.5.13 with Composition API (`package.json:14`) -- Pinia 3.0.1 for state management (`package.json:13`) -- Vite 6.1.0 for development/bundling (`package.json:17`) -- No actual backend server - mock API only +| Field | Value | +|-------|-------| +| Repository | `agent-system-mapper` | +| Path | `examples/vue/kanban-board/` | +| Commit | `9a69c14` | +| Documented | `2026-08-03` | +| Verification Status | `Verified` | -## State Architecture +Verify with: -### Board Store (`src/stores/boardStore.js`) -[VERIFIED] Manages columns and cards as reactive state: - -```javascript -const columns = ref([]) // Line 19 -const cards = ref({}) // Line 20 - Object keyed by card ID -const lastSyncedAt = ref(null) // Line 21 +```bash +python3 verify.py examples/vue/good-architecture-doc-example.md --repo-root examples/vue/kanban-board ``` -**Data Flow - UI to Persistence:** -1. UI action calls store method (e.g., `addCard`) -2. Store mutates reactive state (line 82: `cards.value[id] = card`) -3. `saveToLocal()` called to persist to localStorage (line 89) -4. `markPendingSync()` adds to sync queue (line 91) -5. [CRITICAL GAP] Sync is NOT triggered automatically +## Verification Summary -Reference: `src/stores/boardStore.js:68-94` +- `[VERIFIED]`: 80 tags — 102 file:line citations, 102 resolving (100%); 1 informal (structural evidence, no file:line) +- `[INFERRED]`: 1 claim +- `[NOT_FOUND]`: 7 items +- `[ASSUMED]`: 1 item +- `[NEEDS_VERIFICATION]`: 1 item +- Quoted blocks: 6/6 match the cited source exactly (phase-2 similarity 1.00) -### Sync Store (`src/stores/syncStore.js`) -[VERIFIED] Tracks sync state but has significant gaps: +--- -```javascript -const isOnline = ref(navigator.onLine) // Line 19 -const isSyncing = ref(false) // Line 20 -const pendingQueue = ref([]) // Line 21 - Card IDs awaiting sync -``` +## 0. System Classification -**Missing Functionality:** -[VERIFIED] From architecture comment at lines 7-16: -- No conflict resolution -- No retry with backoff -- No sync order guarantees -- Deletes not tracked +| Field | Value | +|-------|-------| +| Category | Traditional Code | +| Type | Frontend SPA (Vue 3 + Vite, no server routes) | +| Evidence | `vue` and `pinia` declared as dependencies [VERIFIED: package.json:12-13]; single-file components under `src/components/` [VERIFIED: src/components/KanbanBoard.vue:1] [NOT_FOUND: no server directory, no HTTP framework, no fetch/axios/XHR anywhere in src/] | +| Overlay Loaded | No | +| Confidence | `[VERIFIED]` | -Reference: `src/stores/syncStore.js:7-16` +--- -## Sync Behavior Analysis +## 1. System Purpose -### What the UI Shows vs Reality +A single-page Kanban board for organizing cards across three columns, aimed at a single local user. All state lives in the browser: Pinia stores hold reactive columns and cards, every mutation is written to localStorage, and a mock "sync" layer simulates — but never performs — server communication. The board store's own architecture note states that it "manages LOCAL state" and that changes "are NOT automatically persisted to server" [VERIFIED: src/stores/boardStore.js:9-15], and the API service's note confirms "No actual server exists" [VERIFIED: src/services/api.js:5-6]. -| UI Indicator | Appears To Mean | Actual Behavior | -|--------------|-----------------|-----------------| -| `pending` (●) | Awaiting sync | Card in local queue, sync must be manually triggered | -| `synced` (✓) | Saved to server | Mock API returned success, data in memory only | -| "Sync Now" button | Sync pending changes | Calls mock API, no real persistence | +[INFERRED: the deliberate `// Wart:` comments throughout the codebase — mock server state, unsynced deletes, silent persistence failures — indicate this app is a teaching fixture for documenting flawed sync architectures rather than a production tool; the warts are labeled by the authors themselves in the files cited in section 6] -Reference: `src/components/KanbanCard.vue:44-56` (indicators), `src/components/SyncStatus.vue:28-34` (button) +--- -### Sync Trigger Path -[VERIFIED] Trace from UI to "sync": +## 2. Component Map -1. User clicks "Sync Now" → `SyncStatus.vue:7` calls `syncStore.triggerSync()` -2. `triggerSync()` iterates pending queue one-by-one (line 62) -3. For each card, calls `api.syncCard(card)` (line 71) -4. Mock API stores in memory variable (line 42-48 of `api.js`) -5. On success, card marked synced (line 72-73) +| Component | Location | Responsibility | Evidence | +|-----------|----------|----------------|----------| +| Bootstrap | `src/main.js` | Creates the Vue app, installs Pinia, mounts `#app` | [VERIFIED: src/main.js:6-10] | +| App shell | `src/App.vue` | Renders header + board; loads persisted state and registers online/offline listeners on mount | [VERIFIED: src/App.vue:11-23] | +| Board container | `src/components/KanbanBoard.vue` | Renders one `KanbanColumn` per store column | [VERIFIED: src/components/KanbanBoard.vue:10-14] | +| Column | `src/components/KanbanColumn.vue` | Per-column card list, drop target, add-card form | [VERIFIED: src/components/KanbanColumn.vue:46-48, 65-78] | +| Card | `src/components/KanbanCard.vue` | Draggable card with sync indicator and delete button | [VERIFIED: src/components/KanbanCard.vue:34-36, 42-61] | +| Sync status bar | `src/components/SyncStatus.vue` | Shows offline/syncing/pending/synced, "Sync Now" button, error indicator | [VERIFIED: src/components/SyncStatus.vue:12-40] | +| Board store | `src/stores/boardStore.js` | Pinia store: columns + cards state and all card mutations | [VERIFIED: src/stores/boardStore.js:17-21] | +| Sync store | `src/stores/syncStore.js` | Pinia store: online state, pending queue, sync trigger | [VERIFIED: src/stores/syncStore.js:17-23] | +| API service | `src/services/api.js` | Mock server boundary — in-memory "server state" only | [VERIFIED: src/services/api.js:17-23] | +| Persistence service | `src/services/persistence.js` | localStorage save/load/clear under one storage key | [VERIFIED: src/services/persistence.js:12-24] | +| Drag-and-drop composable | `src/composables/useDragDrop.js` | Drag state refs and DOM drag event handlers | [VERIFIED: src/composables/useDragDrop.js:12-17] | +| Type docs | `src/types/index.js` | JSDoc typedefs + `SYNC_STATUS` constants (documentation only) | [VERIFIED: src/types/index.js:8-17, 42-46] | -[CRITICAL] `api.js:17-21` shows "server state" is just: -```javascript -let serverState = { - cards: {}, - lastModified: null, -} -``` -This resets on page refresh. No actual persistence occurs. +--- -### Delete Behavior Gap -[VERIFIED] Deletes are not synced: +## 3. Execution Surfaces & High-Level Data Movement (Discovery Only) -```javascript -// Line 110-125 of boardStore.js -function deleteCard(cardId) { - // ... removes from local state - // Wart: Deleted cards should be tracked for sync, but aren't -} -``` +This section records where execution enters the app and what data moves at a +high level. Step-by-step execution tracing is deliberately deferred to the +code-flow documentation (see section 3.3). -`api.js:66-68` confirms delete not implemented: -```javascript -async deleteCard(cardId) { - throw new Error('Delete sync not implemented') -} -``` +### 3.1 Primary Execution Surfaces -## Drag and Drop Architecture +| Entry Surface | Type | Primary Components Involved | Evidence | +|--------------|------|-----------------------------|----------| +| Browser page load | Web | `index.html` loads `/src/main.js`; app mounts at `#app` | [VERIFIED: index.html:10, src/main.js:6-10] | +| App mount hook | Lifecycle event | `App.vue` `onMounted` → `boardStore.loadFromLocal()` | [VERIFIED: src/App.vue:11-13] | +| `window` online/offline events | Browser event | `App.vue` listeners → `syncStore` | [VERIFIED: src/App.vue:16-22] | +| Add-card form | UI event | `KanbanColumn.vue` → `boardStore.addCard` | [VERIFIED: src/components/KanbanColumn.vue:33-39] | +| Card drag/drop | UI event | `KanbanCard.vue`, `KanbanColumn.vue`, `useDragDrop`, `boardStore.moveCard` | [VERIFIED: src/components/KanbanCard.vue:34-36, src/components/KanbanColumn.vue:46-48] | +| Card delete button | UI event | `KanbanCard.vue` → `boardStore.deleteCard` | [VERIFIED: src/components/KanbanCard.vue:58] | +| "Sync Now" button | UI event | `SyncStatus.vue` → `syncStore.triggerSync` → mock API | [VERIFIED: src/components/SyncStatus.vue:28-34] | +| `npm run dev` / `build` / `preview` | CLI (Vite) | Vite + `@vitejs/plugin-vue` | [VERIFIED: package.json:6-10, vite.config.js:4-6] | -### Composable Pattern -[VERIFIED] `useDragDrop.js` encapsulates drag state: +### 3.2 High-Level Data Movement (Non-Procedural) -```javascript -const draggingCard = ref(null) // Line 15 -const dragOverColumn = ref(null) // Line 16 -const dropIndex = ref(null) // Line 17 -``` +| Stage | Input Type | Output Type | Participating Components | +|------|------------|-------------|--------------------------| +| Bootstrap load | localStorage JSON (or nothing) | Reactive store state (columns, cards) | `persistence.load`, `boardStore.loadFromLocal` | +| Card mutation | User input (title, drag position, delete click) | Updated store state + localStorage write | `boardStore` actions, `persistence.save` | +| Sync queueing | Card ID | Pending-queue entry + `pending` card status | `boardStore.markPendingSync`, `syncStore.addToPendingQueue` | +| Sync execution | Queued card objects | In-memory mock "server" records + `synced` card status | `syncStore.triggerSync`, `api.syncCard`, `boardStore.markSynced` | +| Status display | Store state | UI indicators (status text, per-card markers) | `SyncStatus.vue`, `KanbanCard.vue` | -### Optimistic UI Gap -[VERIFIED] Drop treats UI change as complete immediately: +### 3.3 Pointers to Code Flow Documentation -```javascript -// Line 60-62 of useDragDrop.js -// Wart: Immediate mutation without optimistic UI pattern -boardStore.moveCard(cardId, fromColumnId, toColumnId, index) -``` +Candidates for detailed flow tracing (see `02-code-flows.md`); no step-by-step +tracing is done in this document: -[INFERRED] Proper pattern would: -1. Show optimistic UI state -2. Attempt sync -3. Roll back on failure +- **Add Card flow** — add-card form through `boardStore.addCard` to persistence and the sync queue +- **Manual Sync flow** — "Sync Now" through `syncStore.triggerSync` to the mock API and back to card status +- **Move Card (drag-and-drop) flow** — drop handling through `useDragDrop` to `boardStore.moveCard` +- **Delete Card flow** — delete click through `boardStore.deleteCard`, including why deletes never reach the sync queue +- **Reconnect auto-sync flow** — the `isOnline` watcher through delayed `triggerSync` -Current implementation has no rollback mechanism if sync fails after move. +### Section 3 Self-Check -## Persistence Layer +- [x] No method bodies longer than 3 lines quoted +- [x] No loops or conditionals explained +- [x] Movements described as conceptual stages, not steps +- [x] Detailed tracing deferred to `02-code-flows.md` -### localStorage Only -[VERIFIED] `persistence.js` uses localStorage exclusively despite IndexedDB mention: +--- -```javascript -const STORAGE_KEY = 'kanban-board-data' // Line 12 +## 3b. Frontend → Backend Interaction Map -save(data) { - localStorage.setItem(STORAGE_KEY, serialized) // Line 24 -} -``` +There is no real backend. The "backend boundary" is the in-memory mock in `src/services/api.js` [VERIFIED: src/services/api.js:5-6]. Each row below is a potential flow to trace in the code-flow documentation; internal logic is not described here. -[VERIFIED] Line 75-76 confirms IndexedDB not implemented: -```javascript -// Wart: No IndexedDB implementation despite being mentioned in requirements -``` +| Frontend Source | Trigger Type | Backend Target | Handler / Method | Evidence | +|-----------------|--------------|----------------|------------------|----------| +| `SyncStatus.vue` ("Sync Now") | click event | `syncStore` → mock API | `triggerSync()` → `api.syncCard()` | [VERIFIED: src/components/SyncStatus.vue:28-34, src/stores/syncStore.js:71] | +| `App.vue` | browser `online` event | `syncStore` | `setOnline(true)` then `triggerSync()` | [VERIFIED: src/App.vue:16-19] | +| `App.vue` | browser `offline` event | `syncStore` | `setOnline(false)` | [VERIFIED: src/App.vue:20-22] | +| `syncStore` | reactive watch on `isOnline` | mock API | delayed `triggerSync()` | [VERIFIED: src/stores/syncStore.js:103-108] | -### Failure Handling -[VERIFIED] Errors silently fail: +[NOT_FOUND: searched "fetch(", "axios", "XMLHttpRequest", "WebSocket" in src/ — no real network calls exist; every interaction above terminates in the in-memory mock] -```javascript -// persistence.js:27-29 -} catch (error) { - console.error('Failed to save to localStorage:', error) - return false // No retry, no user notification -} -``` +--- -## Invariants and Guarantees - -### What IS Guaranteed -[VERIFIED] -- Local state is reactive (Vue reactivity system) -- Changes persist to localStorage on each mutation -- Pending queue tracks unsynced cards - -### What is NOT Guaranteed -[VERIFIED] -- Data survives browser clear/storage quota -- Sync actually reaches a server -- Conflicts are detected or resolved -- Delete operations sync -- Order of sync matches order of operations -- Retry on transient failures - -## Component Responsibilities - -### KanbanBoard.vue (lines 1-17) -[VERIFIED] Simple container, iterates `boardStore.columns`: -```vue - -``` +## 4. File/Folder Conventions + +| Pattern | Meaning | Evidence | +|---------|---------|----------| +| `src/components/*.vue` | Single-file UI components (4 files, `