From c63f3f0aff0114c1f6395adc6e2e97a64092b1f7 Mon Sep 17 00:00:00 2001 From: Abdulrahman Date: Mon, 3 Aug 2026 18:40:48 +0300 Subject: [PATCH 1/4] docs: define Phase 2A opportunity architecture Record the architecture decisions Phase 2A implementation depends on, before any schema, adapter, or UI code is written. - ADR-016: TypeScript for opportunity ingestion and deterministic scoring, superseding ADR-003 for that scope only. Phase 1B resume parsing and Phase 6 market aggregation remain open, separately-decided questions. - ADR-017: local-first ingestion execution (an explicit, owner-run CLI, not a GitHub Actions schedule), because no hosted Supabase project exists yet and a cloud runner cannot reach the loopback-only local database. Records the later migration path (Supabase Edge Function, then Cloudflare Worker) and its trigger conditions, none of which are provisioned now. - ADR-018: manually entered opportunities are private, user-owned rows in a new `private_opportunities` table, structurally separate from the shared, service-role-written opportunity-identity domain -- not a nullable-owner column on `opportunities`. Retires the informal "manual source" sketch from earlier planning in favor of the smallest-blast-radius RLS shape. - Cross-reference notes added to ADR-002, ADR-006, and ADR-010 recording how each relates to the new decisions, without altering their own accepted or proposed decisions. - ARCHITECTURE.md, DATA_MODEL.md, RLS_POLICY_MATRIX.md, SECURITY_AND_PRIVACY.md, TESTING_STRATEGY.md, USER_WORKFLOWS.md, DEVELOPMENT_ROADMAP.md, DEPLOYMENT_STRATEGY.md, COST_MODEL.md, OBSERVABILITY.md, MVP_SCOPE.md, and DATA_FLOW.md updated so no diagram or prose still presents scheduled GitHub Actions Python ingestion as active or guaranteed; corrected fragilities (e.g. the Supabase pause-mitigation assumption) are called out honestly rather than left stale. - New docs/INGESTION_ARCHITECTURE.md: the adapter/orchestrator interface, identity/dedup and freshness/closure rules, sanitization, the first-adapter choice (Greenhouse), the SSRF-adjacent manual-import boundary, and the binding source-onboarding checklist. - Phase 2A PR sequence recorded in DEVELOPMENT_ROADMAP.md: 2A.0 (this PR, docs only) -> 2A.1 (foundation) -> 2A.2 (Greenhouse adapter + local ingestion, the minimum-useful milestone) -> 2A.3 (explainable ECE market relevance). No schema, dependency, adapter, HTTP request, or runtime code is introduced by this commit. --- docs/ARCHITECTURE.md | 189 ++++++++----- docs/COST_MODEL.md | 25 +- docs/DATA_FLOW.md | 70 +++-- docs/DATA_MODEL.md | 16 ++ docs/DEPLOYMENT_STRATEGY.md | 74 ++++-- docs/DEVELOPMENT_ROADMAP.md | 73 +++-- docs/INGESTION_ARCHITECTURE.md | 251 ++++++++++++++++++ docs/MVP_SCOPE.md | 4 +- docs/OBSERVABILITY.md | 57 ++-- docs/RLS_POLICY_MATRIX.md | 1 + docs/SECURITY_AND_PRIVACY.md | 48 +++- docs/TESTING_STRATEGY.md | 9 +- docs/USER_WORKFLOWS.md | 32 ++- ...ADR-002-frontend-and-backend-boundaries.md | 6 + docs/adr/ADR-003-python-ingestion-pipeline.md | 6 +- docs/adr/ADR-006-posting-version-history.md | 7 + ...identity-and-cross-source-deduplication.md | 10 + .../ADR-016-typescript-ingestion-runtime.md | 146 ++++++++++ ...ADR-017-local-first-ingestion-execution.md | 114 ++++++++ .../ADR-018-private-manual-opportunities.md | 168 ++++++++++++ docs/adr/README.md | 5 +- 21 files changed, 1133 insertions(+), 178 deletions(-) create mode 100644 docs/INGESTION_ARCHITECTURE.md create mode 100644 docs/adr/ADR-016-typescript-ingestion-runtime.md create mode 100644 docs/adr/ADR-017-local-first-ingestion-execution.md create mode 100644 docs/adr/ADR-018-private-manual-opportunities.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2f19f03..55c6b51 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -20,34 +20,48 @@ explains the *why*, not just the *what* — read it alongside [docs/adr/](adr/RE ## 2. Components +**Status note (Phase 2A.0)**: the diagram below reflects the current design after +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript ingestion) and +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution), which supersede +this document's earlier depiction of scheduled GitHub Actions running a Python ingestion job. No +ingestion code exists yet and no hosted Supabase project exists yet — the diagram distinguishes +**implemented**, **approved near-term**, and **possible later** components explicitly, per those +ADRs. + ```mermaid flowchart TB - subgraph Client["Browser (user device)"] + subgraph Client["Browser (user device) — implemented"] SPA["React + TypeScript SPA\n(Vite build)"] end - subgraph CF["Cloudflare Pages"] + subgraph CF["Cloudflare Pages — implemented"] Static["Static asset hosting\n(the built SPA)"] end - subgraph Supabase["Supabase (managed)"] - Auth["Auth\n(email/magic-link)"] + subgraph Supabase["Supabase — implemented (local dev stack only; no hosted project yet)"] + Auth["Auth\n(email/password)"] PG["Postgres\n(app data + RLS)"] Storage["Storage\n(resume files deferred to Phase 1B;\nno raw external content by default)"] - Edge["Edge Functions (Deno)\n[reserved, not used at MVP]"] + Edge["Edge Functions (Deno)\n[reserved; candidate future ingestion host,\nsee ADR-017 stage 2 — not built]"] + end + + subgraph Local["Owner's local machine — approved near-term design (ADR-016/ADR-017)\nno code implemented yet"] + Ingest["TypeScript ingestion CLI\n(source adapters, run on demand)"] + Score["TypeScript scoring CLI\n(deterministic ECE relevance, run on demand)"] end - subgraph GHA["GitHub Actions (scheduled)"] - Ingest["Python ingestion job\n(source adapters)"] - Score["Python scoring job\n(deterministic pipeline)"] - Parse["Python resume-parsing job\n(deferred to Phase 1B)"] + subgraph FutureHosted["Possible later hosted execution — not provisioned\n(ADR-017 migration path, evaluated only when triggered)"] + FutureEdge["Supabase Edge Function on a schedule"] + FutureWorker["Cloudflare Worker + Cron Trigger"] end - subgraph Ext["External sources"] - ATS["Greenhouse/Lever job-board APIs"] - Gov["Public government/EU job feeds"] - Events["Public event/university calendars"] - UserLinks["User-submitted links"] + subgraph Deferred["Deferred — Phase 1B, execution model not yet decided"] + Parse["Resume-parsing job\n(language/host TBD; ADR-016 does not decide this)"] + end + + subgraph Ext["External sources — Phase 2A"] + ATS["Greenhouse Job Board API\n(first adapter, ADR-016/ingestion architecture)"] + Manual["User-confirmed manual entries\n(no automated fetch, see ADR-018)"] end SPA <--> Static @@ -55,15 +69,16 @@ flowchart TB SPA <--> PG SPA <--> Storage - Ingest --> Ext - Ingest -->|service-role key, GitHub secret| PG - Score -->|reads opportunities + profile, writes scores| PG + Ingest --> ATS + Ingest -->|service-role key, local .env, never committed| PG + Score -->|reads opportunities, writes deterministic scores| PG + Manual -.->|browser-confirmed, private row, see ADR-018| PG + + FutureEdge -.->|later migration stage, not built| PG + FutureWorker -.->|later migration stage, not built| PG + Parse -.->|future: reads uploaded resume| Storage Parse -.->|future: writes reviewable suggestions| PG - - GHA -.->|cron trigger| Ingest - GHA -.->|cron trigger| Score - GHA -.->|cron trigger| Parse ``` ### Frontend — React + TypeScript + Vite, hosted on Cloudflare Pages @@ -147,35 +162,56 @@ raw external HTML/JSON as it was originally fetched. Concretely: ingestion run doubles as a keep-alive, but this is documented explicitly as a real operational risk in [COST_MODEL.md](COST_MODEL.md) and [OBSERVABILITY.md](OBSERVABILITY.md), not hidden. -### Ingestion, parsing, and scoring — Python, run by GitHub Actions on a schedule +### Ingestion and scoring — TypeScript, run as a local CLI (Phase 2A; approved near-term design, not yet implemented) + +**Status**: this section reflects [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (runtime +language) and [ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (execution model), which +supersede this document's earlier Python/GitHub-Actions description for ingestion and opportunity +scoring specifically. Resume parsing's language and execution model remain a separate, undecided +Phase 1B question (see below) — neither ADR resolves it. -Three related but distinct jobs, all Python, all triggered by GitHub Actions cron: +Two related but distinct TypeScript jobs, run today by the repository owner as an explicit local +CLI, not on any schedule: 1. **Ingestion**: runs each configured source adapter, normalizes results into the common opportunity schema, resolves cross-source identity (auto-linking on strong evidence, flagging uncertain matches for review, otherwise creating a new canonical opportunity — see [ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)), diffs against the last known version, and writes new/updated source listings, opportunities, and version - history to Postgres using a service-role key held in GitHub Actions secrets (never exposed to - the browser). -2. **Resume parsing**: a future Phase 1B capability. It will extract separately reviewable - suggestions, never overwrite confirmed profile records, and must revisit field/suggestion-level - provenance ([ADR-014](adr/ADR-014-confirmed-profile-versus-imported-suggestions.md)). -3. **Scoring**: recomputes market-relevance and personal-competitiveness scores whenever - opportunities or the profile change meaningfully, writing explainable, versioned score records. - -Chosen as Python specifically for its data/text-processing and future ML ecosystem (resume -parsing, NLP-assisted extraction, pandas-based market analysis), independent of what language the -(currently nonexistent) synchronous backend would use if one is added later. - -**Where it runs**: GitHub Actions, on standard GitHub-hosted runners. The repository is public, so -these minutes are free and unlimited (see [COST_MODEL.md](COST_MODEL.md)) — not always-on; each -job is a short-lived process that starts, does work, writes to Postgres, and exits. + history to Postgres using a service-role key read from a gitignored local `.env.local` (never + committed, never held by any hosted compute at this stage). +2. **Scoring**: recomputes deterministic, profile-independent market-relevance scores + ([SCORING_AND_MATCHING_MODEL.md](SCORING_AND_MATCHING_MODEL.md) Ranking 1) whenever + opportunities change meaningfully, writing explainable, versioned score records. Personal + competitiveness (Ranking 2) is later work, deferred until the profile domains it needs exist. + +Chosen as TypeScript specifically because the actual Phase 2A workload (fetch JSON, validate, +normalize, sanitize, hash, upsert) does not need Python's data/ML ecosystem, and because the +realistic future hosted execution targets (Supabase Edge Functions, Cloudflare Workers) both run +TypeScript natively — see [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) for the full +reasoning, including what this does **not** decide about Phase 1B resume parsing. + +**Where it runs today**: the repository owner's own machine, invoked on demand +(`node --env-file=.env.local dist/cli.js ingest ...`), against the local (or, once one exists, a +real hosted) Supabase stack — not GitHub Actions, and not any other scheduled or hosted compute. +See [ADR-017](adr/ADR-017-local-first-ingestion-execution.md) for why, and for the explicit, +trigger-gated later migration path (Supabase Edge Function → Cloudflare Worker → last-resort +scheduled container), none of which is provisioned yet. + +**Resume parsing** (a future Phase 1B capability, unaffected by ADR-016/ADR-017): it will extract +separately reviewable suggestions, never overwrite confirmed profile records, and must revisit +field/suggestion-level provenance +([ADR-014](adr/ADR-014-confirmed-profile-versus-imported-suggestions.md)). Its language and +execution model are open questions for that phase's own design pass — Python remains a plausible, +but not yet decided, candidate (see [OPEN_QUESTIONS.md](OPEN_QUESTIONS.md)). **Failure cases**: a failed adapter run must not corrupt existing data — writes are additive (new versions), and a run that errors partway logs the failure (see -[OBSERVABILITY.md](OBSERVABILITY.md)) and is safely retried on the next schedule without -duplicating data, using source-provided stable IDs or URL+content hashing for idempotency. +[OBSERVABILITY.md](OBSERVABILITY.md)) and is safely re-run by the owner without duplicating data, +using source-provided stable IDs or URL+content hashing for idempotency. Because runs are manual, +"best-effort cron timing" does not apply here the way it does to the future hosted stages; freshness +is exactly as current as the owner's last run, and the UI must present `stale` accordingly (see +[DATA_FLOW.md](DATA_FLOW.md)). ### Reserved, not built at MVP: a synchronous API/worker layer @@ -183,21 +219,30 @@ Some future features (instant resume processing on upload, synchronous LLM expla receivers for email-based alerts) need a request/response server, not a batch job. The architecture reserves Supabase Edge Functions (Deno, colocated with the database, generous free tier) as the first place such logic would go, with a move to a small containerized FastAPI service -(Python, consistent with the ingestion codebase) as the fallback if Edge Functions' constraints -(execution time, language) don't fit. This is discussed, not built — see -[ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md). +(Python) as the fallback if Edge Functions' constraints (execution time, language) don't fit. This +is discussed, not built — see [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md). (Note: +ADR-002's original fallback rationale described this Python service as "consistent with the +ingestion codebase" — since [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) moved ingestion +to TypeScript, that specific justification no longer holds; ADR-002's actual decision, no +synchronous backend at MVP, is otherwise unaffected and remains Accepted.) -## 3. Repository layout (Phase 0: implemented as shown) +## 3. Repository layout (Phase 0: implemented as shown; `ingestion/` updated by Phase 2A) ``` careeros/ app/ React + TypeScript + Vite frontend (implemented, Phase 0) - ingestion/ Python: source adapters, normalization, scoring, resume parsing - (placeholder only -- no code yet, see ingestion/README.md; Phase 2+) + ingestion/ TypeScript: source adapters, normalization, deterministic scoring + (placeholder only as of Phase 2A.0 -- no code yet, see ingestion/README.md + and ADR-016; implemented starting Phase 2A.1/2A.2. Resume parsing is a + separate, undecided Phase 1B question -- not assumed to live here.) supabase/ config.toml, migrations/, tests/database/ (pgTAP), scripts/ (local-only dev tooling) -- implemented, Phase 0 docs/ this documentation set - .github/workflows/ ci.yml (Phase 0); ingestion/parsing/scoring schedules are Phase 2+ + .github/workflows/ ci.yml (Phase 0; gains an `ingestion` lint/typecheck/test job in + Phase 2A.2). No ingestion/scoring *schedule* is planned in GitHub + Actions -- ingestion runs as a local CLI (see ADR-017); scheduled + hosted execution, if ever adopted, targets Supabase Edge Functions or + Cloudflare Workers, not GitHub Actions cron. ``` A monorepo, not separate repositories, because the frontend, database schema, and ingestion @@ -239,12 +284,18 @@ abstract. | Option | When it would be the better choice | Why not chosen for the MVP | |---|---|---| -| **No backend; direct Supabase access + batch Python (chosen)** | Fits a single-user tool where almost every operation is CRUD-on-your-own-data (RLS-safe) or a scheduled batch job | — | +| **No backend; direct Supabase access + batch ingestion/scoring (chosen; TypeScript per [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md), local CLI per [ADR-017](adr/ADR-017-local-first-ingestion-execution.md))** | Fits a single-user tool where almost every operation is CRUD-on-your-own-data (RLS-safe) or an owner-invoked batch job | — | | **FastAPI (Python) as a real API server** | Once synchronous, user-triggered server-side logic is unavoidable (instant resume parsing, LLM explanation endpoints with rate limiting, webhook receivers) | Needs an always-on host; free tiers for that (Render, Fly.io, Railway) come with sleep/cold-start behavior or limited free hours, adding operational complexity not justified yet. This is the **documented target** for when the batch-only model proves insufficient — see [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md) | -| **Node.js backend** | If the team wanted one language across the whole stack | Python was already the right choice for ingestion/parsing/scoring (see below); running two backend languages is worse than running Python everywhere non-frontend | +| **Node.js backend** | If the team wanted one language across the whole stack | This synchronous-backend question is independent of the ingestion-language decision below; no synchronous backend is built at MVP either way (this section's own Decision) | ### Ingestion/analysis language +**Status (Phase 2A.0)**: superseded for opportunity ingestion and deterministic scoring by +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) — see that ADR for the current decision and +reasoning. The paragraph below is preserved as the original planning-phase rationale, which +correctly anticipated Phase 1B resume parsing and Phase 6 market aggregation; those remain open, +separately-decided questions, not resolved by ADR-016. + Python was specified as a requirement in the brief and is confirmed here: mature HTML parsing/sanitization libraries, pandas for market-skill aggregation, and the most direct path to future ML-assisted resume parsing or embeddings work. @@ -276,10 +327,10 @@ would be a second system to keep in sync with Postgres for no clear win. | Option | When it would be the better choice | Why not chosen for the MVP | |---|---|---| -| **Cloudflare Pages (chosen) for the frontend; Supabase managed for data/auth/storage; GitHub Actions for scheduled compute** | Fits a $0, single-user, low-traffic tool with no always-on compute need | — | +| **Cloudflare Pages (chosen) for the frontend; Supabase managed for data/auth/storage; local CLI for ingestion/scoring today, with a hosted scheduled stage (Supabase Edge Function, then Cloudflare Worker) evaluated later — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)** | Fits a $0, single-user, low-traffic tool with no always-on compute need | — | | **AWS / Azure / GCP** | Needed once the project needs fine-grained IAM, VPC-level network control, managed queues at scale, or the student specifically wants cloud-provider experience with real infrastructure spend controls | Overkill for current scale; meaningfully higher operational and cost-management burden for a single maintainer. Worth revisiting deliberately at the public-launch phase, and separately, as a *learning* exercise if desired — but not required by the product | -| **Self-hosted** (home server, student cloud credits via GitHub Student Developer Pack) | If GitHub Actions free minutes or Supabase free-tier limits are actually hit | A real $0-adjacent option given the user's student status; documented as a fallback in [COST_MODEL.md](COST_MODEL.md) rather than the default, to avoid taking on server maintenance before it's needed | -| **Long-running background worker** (always-on process, not scheduled batch) | If ingestion needs near-real-time polling (many sources checked every few minutes) | Not needed at MVP ingestion frequency (daily/few-times-daily is enough for job postings and events); scheduled GitHub Actions jobs are simpler to reason about and free | +| **Self-hosted** (home server, student cloud credits via GitHub Student Developer Pack) | If Supabase free-tier limits are actually hit | A real $0-adjacent option given the user's student status; documented as a fallback in [COST_MODEL.md](COST_MODEL.md) rather than the default, to avoid taking on server maintenance before it's needed | +| **Long-running background worker** (always-on process, not scheduled batch) | If ingestion needs near-real-time polling (many sources checked every few minutes) | Not needed at MVP ingestion frequency (daily/few-times-daily is enough for job postings and events, and Phase 2A doesn't even run on a schedule yet); a future *scheduled* hosted stage (Supabase Edge Function or Cloudflare Worker, not GitHub Actions — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) is simpler to reason about and free | | **Dedicated serverless functions (Vercel/Cloudflare Workers) beyond Supabase Edge Functions** | If synchronous compute needs to live outside Supabase's own edge runtime for latency, language, or ecosystem reasons | Supabase Edge Functions are colocated with the database (fewer network hops, one fewer provider to manage) and are the first thing evaluated when synchronous compute becomes necessary | ## 5. Where each component runs (summary) @@ -289,22 +340,28 @@ would be a second system to keep in sync with Postgres for no clear win. | React SPA | User's browser | Untrusted execution environment; never holds secrets beyond the user's own short-lived session token | | Cloudflare Pages | Cloudflare's CDN | Serves static assets only; no application logic or secrets | | Supabase Auth/Postgres/Storage | Supabase-managed cloud (AWS-backed) | Authoritative data store; RLS is the authorization boundary | -| Python ingestion/parsing/scoring | GitHub Actions runners (ephemeral, per-run VMs) | Holds the Postgres service-role key as a GitHub secret; never exposed to the browser or logs | +| TypeScript ingestion/scoring (Phase 2A; [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)) | The repository owner's own machine, run on demand as a local CLI ([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) — not GitHub Actions | Holds the Postgres service-role key in a gitignored local `.env.local`; never committed, never exposed to the browser or logs. A later hosted stage (Supabase Edge Function, Cloudflare Worker) would hold it in that platform's own secret storage instead | +| Resume parsing (Phase 1B, deferred) | Not yet decided — language and execution model are open questions ADR-016/ADR-017 do not resolve | Not yet decided | | External sources | Third-party infrastructure | Fully untrusted; every response is treated as untrusted data, sanitized before storage | ## 6. Scalability limits of this design This design comfortably handles one user and a realistic ingestion volume (low thousands of -opportunity records, daily-to-hourly ingestion cadence). Known ceilings, documented honestly rather +opportunity records). Under Phase 2A's manual local-CLI execution +([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)), cadence is simply "however often the +owner runs it" rather than a target frequency; the daily-to-hourly figures below describe a +possible future scheduled stage, not the current design. Known ceilings, documented honestly rather than solved preemptively: -- **GitHub Actions minutes** are free and unlimited on standard runners for this public - repository; the practical ceiling is runner concurrency and job duration, not a minute quota — - not a problem at daily/few-times-daily cadence, would become a design question at near-real-time - polling of many sources. -- **Supabase free-tier project pausing on inactivity** requires a keep-alive strategy (the - scheduled jobs themselves largely cover this, but this is a real fragility, not a solved - problem — see [COST_MODEL.md](COST_MODEL.md)). +- **CI (GitHub Actions) minutes** are free and unlimited on standard runners for this public + repository; the practical ceiling is runner concurrency and job duration, not a minute quota. This + applies to CI today; if a future scheduled hosted stage for ingestion is ever adopted, it targets + Supabase Edge Functions or Cloudflare Workers, not GitHub Actions cron — see + [ADR-017](adr/ADR-017-local-first-ingestion-execution.md). +- **Supabase free-tier project pausing on inactivity** requires a keep-alive strategy. The original + plan assumed scheduled ingestion jobs would largely cover this; under Phase 2A's manual execution + model that mitigation does not apply, so this is currently a **more exposed** fragility than + originally planned, not a solved one — see [COST_MODEL.md](COST_MODEL.md) §3. - **No synchronous backend** means any feature requiring instant server-side response (real-time chat-style LLM interaction, instant resume feedback) is out of scope until the Edge Function/FastAPI layer is added. @@ -314,9 +371,15 @@ than solved preemptively: ## 7. GitHub Actions reliability characteristics (read before relying on cadence) +**Scope note (Phase 2A.0)**: opportunity ingestion and scoring do **not** currently run on GitHub +Actions at all — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md); they run as a local +CLI on demand. This section's cautions apply to CI itself (which does run on every push/PR, see +`.github/workflows/ci.yml`) and would apply again to any *future* scheduled workflow this project +adopts — they are not a description of how ingestion currently behaves. + The cadences in [DATA_FLOW.md](DATA_FLOW.md#5-data-freshness-expectations-proposal) are targets, -not guarantees. This is not a caveat to skim past — it shapes how every scheduled job and every -UI status indicator must be built: +not guarantees for whichever execution stage is current. This is not a caveat to skim past — it +shapes how every scheduled job and every UI status indicator must be built: - **Scheduled (`cron`) triggers are best-effort and can start late.** GitHub's own documentation states that scheduled workflows may be delayed during periods of high load, most commonly at the diff --git a/docs/COST_MODEL.md b/docs/COST_MODEL.md index b6ce16b..0d61e30 100644 --- a/docs/COST_MODEL.md +++ b/docs/COST_MODEL.md @@ -9,8 +9,9 @@ relying on it operationally, and updated when it drifts. | Component | Provider | Free tier basis | Realistic MVP usage | Risk of exceeding | |---|---|---|---|---| | Frontend hosting | Cloudflare Pages | Generous free tier for static sites, unlimited requests/bandwidth on the free plan as published | One small SPA, single user | Very low | -| Database + Auth + Storage | Supabase free tier | Free project with published limits on database size, monthly active users, storage, and auto-pauses after a period of inactivity | Low data volume (low thousands of rows), one user, small files (resumes are KB-scale) | **Real risk**: free-tier project pausing after inactivity — mitigated by scheduled jobs acting as a keep-alive, but this is a genuine fragility, not fully solved (see §3). Storage growth is also bounded on purpose: only sanitized opportunity snapshots are retained indefinitely (see [ARCHITECTURE.md](ARCHITECTURE.md#raw-external-content-retention-policy)) — raw external responses are not stored by default, and the optional debug table has a short (14-day, proposal) retention with scheduled cleanup, so it cannot grow unbounded even if left enabled | -| Scheduled compute | GitHub Actions | The repository is public, so standard GitHub-hosted runner minutes are free and unlimited (no monthly quota, unlike a private repository on the free plan) | Short jobs (minutes each), daily-to-hourly cadence | Low at proposed cadence on standard runners; larger/specialized runners and some storage or advanced Actions features can still be billable even on a public repository — not used here | +| Database + Auth + Storage | Supabase free tier | Free project with published limits on database size, monthly active users, storage, and auto-pauses after a period of inactivity | Low data volume (low thousands of rows), one user, small files (resumes are KB-scale) | **Real risk, currently higher than originally assumed**: free-tier project pausing after inactivity was previously expected to be mitigated by *scheduled* ingestion jobs acting as a keep-alive; per [ADR-017](adr/ADR-017-local-first-ingestion-execution.md), Phase 2A ingestion is a **manually run local CLI, not a schedule**, so that mitigation does not apply until a later hosted/scheduled execution stage is adopted — see §3 for the restated fragility. Storage growth is also bounded on purpose: only sanitized opportunity snapshots are retained indefinitely (see [ARCHITECTURE.md](ARCHITECTURE.md#raw-external-content-retention-policy)) — raw external responses are not stored by default, and the optional debug table has a short (14-day, proposal) retention with scheduled cleanup, so it cannot grow unbounded even if left enabled | +| CI compute | GitHub Actions | The repository is public, so standard GitHub-hosted runner minutes are free and unlimited (no monthly quota, unlike a private repository on the free plan) | `frontend`/`database` jobs today; a third `ingestion` job from Phase 2A.2 (install, lint, typecheck, test — no live network) | Low. **Not** used for scheduled ingestion/scoring compute — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md); larger/specialized runners and some storage or advanced Actions features can still be billable even on a public repository, but nothing here uses them | +| Ingestion/scoring compute | The repository owner's own machine (Phase 2A; local CLI) | $0 — no provider, no hosted compute | Manual, on-demand runs | None at this stage. A later hosted/scheduled stage (Supabase Edge Function, then Cloudflare Worker — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) would move this row onto whichever free tier that provider publishes, evaluated at that time | | Domain | Optional custom domain | N/A — `*.pages.dev` is free | Not required for functionality | N/A unless a custom domain is deliberately purchased | | LLM API calls (deferred feature) | Not yet selected | Most providers require payment beyond a small trial credit | Not used until Phase 7 | This is the most likely first genuinely-paid line item once LLM explanation prose is added — budget and provider choice deferred to that phase, and must be re-evaluated against [SCORING_AND_MATCHING_MODEL.md](SCORING_AND_MATCHING_MODEL.md)'s constraint that LLM use only follows grounded scoring | @@ -20,18 +21,26 @@ Every "free tier" above is a business decision by a for-profit company and can c architecture's resilience to this isn't "these will always be free" but "each component is swappable without a rewrite": Cloudflare Pages could be replaced by any static host; Supabase's Postgres+Auth+Storage bundle could, at real cost in migration effort, be replaced by a self-hosted -Postgres plus a different auth provider if its terms ever became unworkable; GitHub Actions could -be replaced by any scheduler that can run a Python script. This swappability is a consequence of +Postgres plus a different auth provider if its terms ever became unworkable; whichever execution +stage runs ingestion (local CLI today; possibly a Supabase Edge Function or Cloudflare Worker +later — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) could be replaced by any +runtime that can execute the same TypeScript adapter code +([ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)). This swappability is a consequence of keeping business logic in the application/adapter code rather than in provider-specific glue, consistent with [ARCHITECTURE.md](ARCHITECTURE.md). ## 3. Known fragility: Supabase free-tier project pausing Free Supabase projects pause after a period of inactivity and require a manual or API-triggered -unpause. Scheduled ingestion/scoring jobs hitting the database regularly should prevent this in -practice, but this must be explicitly monitored (see [OBSERVABILITY.md](OBSERVABILITY.md)) rather -than assumed solved — a paused database silently breaks every scheduled job until someone notices -and unpauses it manually. +unpause. The original mitigation assumed *scheduled* ingestion/scoring jobs would hit the database +regularly enough to prevent this; per +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md), Phase 2A ingestion is a **manually +run local CLI**, so this fragility is currently **more exposed, not less**, than the original +planning assumed — a hosted Supabase project left unused between the owner's manual runs can pause +on its own schedule regardless of how often ingestion happens to run. This must be explicitly +monitored (see [OBSERVABILITY.md](OBSERVABILITY.md)) rather than assumed solved, and is a concrete +reason to revisit ADR-017's later execution stages once a hosted project actually exists — a +scheduled stage would restore the original keep-alive mitigation. ## 4. Real, legitimately-free options given the user's student status diff --git a/docs/DATA_FLOW.md b/docs/DATA_FLOW.md index ca6c035..0343565 100644 --- a/docs/DATA_FLOW.md +++ b/docs/DATA_FLOW.md @@ -3,30 +3,46 @@ Status: **proposal**, consistent with [ARCHITECTURE.md](ARCHITECTURE.md) and [DATA_MODEL.md](DATA_MODEL.md). -## 0. A standing caveat: GitHub Actions is a scheduler, not a reliable interactive system - -Every flow below runs on GitHub Actions cron. Cron triggers are **best-effort and can start -late** — GitHub documents that scheduled workflows may be delayed, especially at the top of the -hour when load is highest, and can occasionally be dropped entirely during platform incidents. -Every flow in this document is therefore designed to be **idempotent, safely retryable, and -tolerant of a job simply not running exactly on schedule** — nothing here assumes cron fires at a -precise time, and no user-facing feature should imply otherwise. See +## 0. Execution-model status (Phase 2A.0) + +**Opportunity ingestion and scoring currently run as an explicit local CLI, invoked on demand by +the repository owner — not on any GitHub Actions schedule.** See +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript) and +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution), which supersede +this document's earlier "every flow below runs on GitHub Actions cron" framing for the ingestion +flow in §1. Under this model, freshness is exactly as current as the owner's last run — there is no +cron schedule to be "best-effort" about, and the `stale` status in §1's flow reflects "not run +recently by the owner," not a missed cron trigger. + +The cron-reliability cautions below remain relevant for two things: this repository's own CI +(`.github/workflows/ci.yml`, which does run on every push/PR) and **any future scheduled workflow** +this project might adopt at a later execution-model stage +([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)'s migration path) — they are not a +description of how ingestion behaves today. + +**Historical framing, preserved for that future-scheduled case**: cron triggers are **best-effort +and can start late** — GitHub documents that scheduled workflows may be delayed, especially at the +top of the hour when load is highest, and can occasionally be dropped entirely during platform +incidents. Any future scheduled flow in this document must remain **idempotent, safely retryable, +and tolerant of a job simply not running exactly on schedule** — nothing here should assume cron +fires at a precise time, and no user-facing feature should imply otherwise. See [ARCHITECTURE.md](ARCHITECTURE.md#7-github-actions-reliability-characteristics-read-before-relying-on-cadence) -for the full set of reliability characteristics this constrains. +for the full set of reliability characteristics this constrains, if and when a scheduled stage is +adopted. ## 1. Opportunity ingestion, deduplication, and scoring flow ```mermaid sequenceDiagram - participant Cron as GitHub Actions (cron, best-effort timing) - participant Adapter as Source adapter (Python) + participant Owner as Repository owner (local CLI, on demand) + participant Adapter as Source adapter (TypeScript) participant Norm as Normalizer participant Dedup as Identity/dedup step participant DB as Postgres (Supabase) - participant Score as Scoring engine (Python) + participant Score as Scoring engine (TypeScript) participant SPA as React SPA - Cron->>Adapter: run scheduled ingestion job (may start later than scheduled) + Owner->>Adapter: run ingestion CLI (owner-invoked, no schedule) Adapter->>Adapter: fetch from source (compliant API/feed) Adapter->>Norm: raw source record Norm->>Norm: map to common opportunity schema,\nsanitize content, classify category/subtype @@ -50,7 +66,7 @@ sequenceDiagram end end Note over DB: a source_listing no longer found in a fresh fetch\nis marked status = removed on the listing;\nthe canonical opportunity is marked removed only when\nevery linked listing is removed - Cron->>Score: run scoring job (after ingestion; also best-effort timing) + Owner->>Score: run scoring CLI (after ingestion, owner-invoked) Score->>DB: read opportunities whose current_version_id changed\nsince last scoring run, + current user profile Score->>Score: compute market-relevance score (Ranking 1) Score->>Score: compute personal-competitiveness score (Ranking 2) @@ -133,7 +149,13 @@ its current-version pointer moves on. ## 4. Market intelligence aggregation -A scheduled (weekly, proposal) Python job reads recent opportunity versions and their structured +**Status**: Phase 6, not yet designed or built. Its language and execution model are **not** +decided by [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (scoped explicitly to opportunity +ingestion and deterministic opportunity scoring only) — Python remains a plausible candidate here, +alongside TypeScript for consistency with ingestion, to be decided at that phase's own planning +pass. The description below is preserved as the original planning-phase proposal. + +A scheduled (weekly, proposal) job reads recent opportunity versions and their structured required/preferred qualifications, tallies skill/tool/tooling mentions per engineering-area category, and writes a compact aggregate snapshot to Postgres (not recomputed on every page load). **Aggregation counts distinct canonical `opportunity_id`s, excluding any with `status = @@ -149,11 +171,13 @@ expensive aggregation off the request path entirely. | Data | Update cadence | Rationale | |---|---|---| -| Opportunity ingestion | Daily, or a few times daily per source | Balances freshness against GitHub Actions free-minute budget and source politeness | -| Resume parsing | Every 10-15 minutes, eventually consistent | Fast enough to feel responsive without needing a live server — but never guaranteed to hit that interval exactly, per §0 | -| Scoring recompute | After every ingestion run that changes a `current_version_id`, and after any profile edit | Scores must never silently go stale relative to the profile or version they claim to explain | -| Market intelligence snapshot | Weekly | Aggregate trends don't meaningfully shift faster than this at MVP data volume | - -These cadences are targets the schedule aims for, not commitments the system can guarantee — see -§0 and [OBSERVABILITY.md](OBSERVABILITY.md) for how staleness itself is monitored and surfaced to -the user rather than silently assumed. +| Opportunity ingestion | Whenever the owner runs the local CLI (Phase 2A; see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) — a daily or few-times-daily target *once* a later hosted/scheduled stage is adopted | No schedule exists yet; there is no cron budget to balance against today. Source politeness still governs the self-imposed per-run request budget regardless of cadence | +| Resume parsing | Every 10-15 minutes, eventually consistent (Phase 1B, not yet designed) | Fast enough to feel responsive without needing a live server — but never guaranteed to hit that interval exactly, and its execution model is a separate, undecided question (see [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)'s scope note) | +| Scoring recompute | Whenever the owner runs the local scoring CLI after an ingestion run that changed a `current_version_id` (Phase 2A); after every ingestion run automatically, once a scheduled stage exists | Scores must never silently go stale relative to the opportunity version they claim to explain | +| Market intelligence snapshot | Weekly (Phase 6, not yet built) | Aggregate trends don't meaningfully shift faster than this at MVP data volume | + +Today's cadences are simply "whenever the owner runs the CLI," not a schedule at all. The +daily/weekly figures above are targets for the later, explicitly-triggered scheduled stage in +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) — see §0 and +[OBSERVABILITY.md](OBSERVABILITY.md) for how staleness itself is monitored and surfaced to the user +rather than silently assumed, under either execution model. diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index 5b536d3..de30e0e 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -35,6 +35,18 @@ This revision corrects two gaps identified in review: suggestion structures. `profiles.profile_complete` has no authoritative role; completeness is derived from [PROFILE_COMPLETENESS_SPEC.md](PROFILE_COMPLETENESS_SPEC.md). +## Revision note (Phase 2A.0 architecture pass) + +Manually entered opportunities are **not** part of the shared opportunity-identity domain below. +[ADR-018](adr/ADR-018-private-manual-opportunities.md) adds `private_opportunities` (see the User +activity domain table) as a structurally separate, user-owned table — private by default, ordinary +mutable CRUD rather than versioned, and outside [ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)'s +deduplication scope. This is a documentation-and-decision update only; no migration exists yet +(that is Phase 2A.1). [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) and +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) additionally settle the ingestion runtime +language and execution model referenced throughout [ARCHITECTURE.md](ARCHITECTURE.md) and +[DATA_FLOW.md](DATA_FLOW.md); neither changes this document's schema. + ## Entity-relationship overview ```mermaid @@ -73,6 +85,9 @@ erDiagram OPPORTUNITY_VERSIONS ||--o{ SAVED_OPPORTUNITIES : snapshot_of USERS ||--o{ APPLICATIONS : tracks SAVED_OPPORTUNITIES ||--o| APPLICATIONS : becomes + + USERS ||--o{ PRIVATE_OPPORTUNITIES : enters + PRIVATE_OPPORTUNITIES ||--o| OPPORTUNITIES : promoted_to APPLICATIONS ||--o{ TASKS : generates APPLICATIONS ||--o{ INTERVIEW_PREP_NOTES : has @@ -243,6 +258,7 @@ distinct, timestamped event, not a mutation of the number the user already saw. | `applications` | `id`, `user_id`, `saved_opportunity_id`, `status` (`identified`\|`preparing`\|`applied`\|`interview_scheduled`\|`interview_complete`\|`offer`\|`rejected`\|`withdrawn`), `applied_at`, `resume_id_used` | | | `tasks` | `id`, `user_id`, `application_id` (nullable), `opportunity_id` (nullable), `title`, `due_date`, `status` (`open`\|`done`), `origin` (`manual`\|`suggested`) | Deadline-derived tasks reference the opportunity directly even before an application exists | | `interview_prep_notes` | `id`, `application_id`, `content`, `created_at` | | +| `private_opportunities` | `id`, `user_id`, `source_url`, `title`, `organization_name`, `location_text`, `opportunity_kind`, `employment_type`, `remote_mode`, `description_text`, `posted_at`, `application_deadline`, `application_url`, `dismissed_at`, `promoted_to_opportunity_id` (nullable, trusted-write only), `created_at`, `updated_at`, `last_confirmed_at` | **New (Phase 2A, see [ADR-018](adr/ADR-018-private-manual-opportunities.md))**: manually entered opportunities, structurally and permission-wise separate from the shared identity domain above — not written through the ingestion/dedup pipeline, not versioned like `opportunity_versions` (ordinary mutable CRUD, like `work_experience`), private by default. `promoted_to_opportunity_id` is a future, trusted-only, additive link to a canonical shared opportunity if one is later discovered by an adapter; no promotion logic exists yet. Exact column types/constraints are finalized at Phase 2A.1 implementation time | ## Raw external-content retention (debug-only, not a core table) diff --git a/docs/DEPLOYMENT_STRATEGY.md b/docs/DEPLOYMENT_STRATEGY.md index 832d3e3..8980699 100644 --- a/docs/DEPLOYMENT_STRATEGY.md +++ b/docs/DEPLOYMENT_STRATEGY.md @@ -76,38 +76,60 @@ onboarded before a broader public launch. - RLS policies are part of the migration files, not a separately-managed dashboard setting, for the same reproducibility reason. -## 4. Scheduled compute deployment - -- GitHub Actions workflows in `.github/workflows/` define the ingestion, resume-parsing, and - scoring jobs on cron schedules (see cadences in - [DATA_FLOW.md](DATA_FLOW.md#5-data-freshness-expectations-proposal)). **GitHub Actions cron is a - best-effort scheduler, not a reliable interactive execution system** — see - [ARCHITECTURE.md](ARCHITECTURE.md#7-github-actions-reliability-characteristics-read-before-relying-on-cadence) +## 4. Ingestion/scoring compute deployment + +**Status (Phase 2A.0)**: this section previously assumed GitHub Actions cron would run scheduled +Python ingestion/parsing/scoring jobs. [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) +(TypeScript) and [ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution) +supersede that assumption for ingestion and scoring. **No GitHub Actions workflow runs ingestion or +scoring**, today or in the currently planned near-term design. + +**Current design (Phase 2A)**: ingestion and scoring run as an explicit local CLI, invoked by the +repository owner on demand against the local (or, once one exists, a real hosted) Supabase stack: + +``` +cd ingestion && node --env-file=.env.local dist/cli.js ingest --source greenhouse: [--dry-run] +cd ingestion && node --env-file=.env.local dist/cli.js score --all +``` + +- The service-role key lives in a gitignored `ingestion/.env.local`, never a GitHub Actions + repository secret, never committed, and never held by any hosted compute at this stage. +- Every run must still be idempotent and safely re-runnable, since the owner may re-run after a + failure or simply run more or less often than any target cadence — nothing here assumes any + particular run frequency. +- `.github/workflows/ci.yml` remains the only GitHub Actions workflow in this repository, and it + never performs ingestion or holds the service-role key — it runs `frontend`/`database` jobs (and, + from Phase 2A.2, a third `ingestion` job: install, lint, typecheck, test, no live network call). + +**Later, explicitly-triggered migration path** (not provisioned now — see +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) for the full trigger conditions): a hosted +Supabase Edge Function on a schedule, then a Cloudflare Worker + Cron Trigger if Edge Function +limits are hit, with GitHub Actions considered only as a last resort and only with a dedicated +least-privilege database role — never the full service-role key. If and when a scheduled stage is +adopted, the historical cron-hygiene guidance below becomes relevant again: + +- **GitHub Actions cron is a best-effort scheduler, not a reliable interactive execution system** — + see [ARCHITECTURE.md](ARCHITECTURE.md#7-github-actions-reliability-characteristics-read-before-relying-on-cadence) for the full set of consequences this has for job design. - **Cron expressions must avoid the top of the hour** (e.g., `7,22,37,52 * * * *`, not `0,15,30,45 * * * *`), since GitHub documents that `:00`-aligned schedules see the most congestion-related delay. -- Each workflow installs Python dependencies (pinned via a lockfile), runs the relevant module, - and exits — no persistent process. Every workflow must be idempotent and safely re-runnable, - since a late or dropped scheduled run may need to be caught up by the next run or a manual one - without duplicating data. -- Secrets (Supabase service-role key, any source API keys) are GitHub Actions repository secrets, - injected as environment variables only inside the job run. -- A manual `workflow_dispatch` trigger should exist alongside the cron schedule for every job, so - ingestion or scoring can be re-run on demand while debugging. **This is a developer control, - invoked with repository write access via the GitHub UI or `gh` CLI — it is not, and must never - become, a button exposed in the frontend that the browser triggers using a stored GitHub token.** - Embedding any token with workflow-trigger permission in browser-reachable code is a secret- - exposure risk this project does not accept. A genuine user-facing "process this now" action - belongs to the synchronous Edge Function/worker layer reserved in - [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md), introduced only once actually needed - — not to GitHub Actions. +- A manual `workflow_dispatch` trigger should exist alongside any future cron schedule, so a run can + be re-triggered on demand while debugging. **This is a developer control, invoked with repository + write access via the GitHub UI or `gh` CLI — it is not, and must never become, a button exposed in + the frontend that the browser triggers using a stored GitHub token.** Embedding any token with + workflow-trigger permission in browser-reachable code is a secret-exposure risk this project does + not accept. A genuine user-facing "process this now" action belongs to the synchronous Edge + Function/worker layer reserved in [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md), + introduced only once actually needed — not to GitHub Actions. - **The repository is public**, so standard GitHub-hosted runner minutes for public repositories are free (no monthly minute quota to track, unlike a private repository on the free plan) — see - [COST_MODEL.md](COST_MODEL.md). Larger/specialized runners and some storage or advanced features - can still be billable even on a public repository; the ingestion cadence should still be - reviewed against actual usage once Phase 2's scheduled jobs exist, but not because of a private- - repo minute quota. + [COST_MODEL.md](COST_MODEL.md). This currently applies only to CI (`ci.yml`), since ingestion runs + locally, not on any GitHub-hosted runner. + +**Resume parsing** (Phase 1B, deferred) is unaffected by ADR-016/ADR-017: its language and execution +model remain a separate, undecided question for that phase's own design pass (see +[OPEN_QUESTIONS.md](OPEN_QUESTIONS.md)). ## 5. Release process (proposal) diff --git a/docs/DEVELOPMENT_ROADMAP.md b/docs/DEVELOPMENT_ROADMAP.md index 11c49d2..7acb37c 100644 --- a/docs/DEVELOPMENT_ROADMAP.md +++ b/docs/DEVELOPMENT_ROADMAP.md @@ -65,30 +65,62 @@ unimplemented checks are shown as product-unavailable rather than user omissions **Exit condition**: a resume produces a truthful, reviewable draft without altering confirmed records unless the user explicitly accepts a change. -## Phase 2 — First ingestion pipeline, unscored - -- Source-adapter interface implemented ([ADR-004](adr/ADR-004-source-adapter-architecture.md)), - producing `source_listings`, not opportunities directly. -- The identity/deduplication step ([ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)) - is built alongside the first adapter, even though a single adapter alone can't yet produce a - cross-source duplicate — it's the shared pipeline every subsequent adapter plugs into, and - retrofitting it after a second adapter exists would mean re-processing already-ingested data. -- One or two ATS (Greenhouse/Lever) adapters built first, since they offer the best - coverage-to-effort ratio (see [DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md)). -- Normalization into the complete common schema (see +## Phase 2A — Opportunity discovery MVP + +**Status**: planning and source research complete (Phase 2A.0, this document's own architecture +decisions); no schema, adapter, or UI code implemented yet. Runtime and execution-model decisions +are recorded in [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript, superseding +[ADR-003](adr/ADR-003-python-ingestion-pipeline.md) for this scope), +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-CLI execution first), and +[ADR-018](adr/ADR-018-private-manual-opportunities.md) (manual entries are private, owned rows). +Source compliance findings are recorded in +[DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md); the adapter/orchestrator design is +in [INGESTION_ARCHITECTURE.md](INGESTION_ARCHITECTURE.md). + +Sequenced as four PRs, each independently reviewable: + +- **Phase 2A.0 — Architecture and compliance** (this PR): ADRs, source-compliance registry, + ingestion architecture, shared-vs-private opportunity data decision, implementation boundaries. + No runtime implementation. +- **Phase 2A.1 — Opportunity foundation**: shared opportunity identity/version schema + ([DATA_MODEL.md](DATA_MODEL.md)), the `private_opportunities` table + ([ADR-018](adr/ADR-018-private-manual-opportunities.md)), the ingestion-source registry, exact + RLS/grants ([RLS_POLICY_MATRIX.md](RLS_POLICY_MATRIX.md)), manual metadata entry, list/detail UI, + search and filters (Postgres full-text search), save/hide. Synthetic fixtures only; no + source-network code. +- **Phase 2A.2 — Greenhouse adapter and local ingestion** (the first minimum-useful milestone): the + TypeScript local ingestion CLI ([ADR-016](adr/ADR-016-typescript-ingestion-runtime.md), + [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)); the source-adapter interface + ([ADR-004](adr/ADR-004-source-adapter-architecture.md)) implemented for Greenhouse first, since it + offers unauthenticated, robots-permitted, complete-response public access with the strongest + observed timestamp coverage (see [DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md)); + the identity/deduplication step ([ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)) + built alongside this first adapter, even though one adapter alone can't yet produce a cross-source + duplicate — it's the shared pipeline every subsequent adapter plugs into; normalization into the + complete common schema (see [DATA_MODEL.md](DATA_MODEL.md#opportunity_versions--complete-field-list)), full version history, - `removed` detection at the listing level. -- A plain, unranked opportunity browse/search view (Postgres full-text search) in the frontend. + conservative closure detection at the listing level; no live third-party call in CI. +- **Phase 2A.3 — Explainable ECE market relevance**: deterministic taxonomy and rule-based scoring + (Ranking 1 only — profile-independent; personal competitiveness/Ranking 2 remains Phase 3), a + ranking UI, synthetic evaluation vectors. No acceptance probability, per + [SCORING_AND_MATCHING_MODEL.md](SCORING_AND_MATCHING_MODEL.md). -**Exit condition**: real German ECE-relevant postings are visible and searchable, with working -version history, before any scoring exists. +**Exit condition (met at Phase 2A.2)**: real German ECE-relevant postings from at least one +compliant, official source are visible and searchable, with working version history and +conservative freshness/closure tracking, alongside any manually entered private opportunities — +before any relevance ranking exists. Phase 2A.3 makes the list explainably ranked rather than +merely searchable. -## Phase 3 — Explainable scoring +## Phase 3 — Personal-competitiveness scoring + +**Note**: Ranking 1 (profile-independent market/degree relevance) moved earlier, into +**Phase 2A.3**, since it needs no profile data and makes Phase 2A's opportunity list explainably +ranked rather than merely searchable. Phase 3 covers what still requires profile domains that don't +exist until Phase 1A is fully built out. - Hard eligibility rules. -- Ranking 1 (market/degree relevance) and Ranking 2 (personal competitiveness), with the - `evidence_jsonb` explainability payload and the "My Fit" view from - [USER_WORKFLOWS.md](USER_WORKFLOWS.md). +- Ranking 2 (personal competitiveness), with the `evidence_jsonb` explainability payload and the + "My Fit" view from [USER_WORKFLOWS.md](USER_WORKFLOWS.md). - Golden-fixture regression tests for scoring (see [TESTING_STRATEGY.md](TESTING_STRATEGY.md)). **Exit condition**: the user can look at a real posting and get a trustworthy, evidence-linked @@ -140,7 +172,8 @@ Candidates, not commitments, roughly in likely priority order: ## Explicit sequencing rules -- Scoring (Phase 3) never starts before ingestion (Phase 2) produces real data — scoring logic +- Ranking 1 (Phase 2A.3) never starts before ingestion (Phase 2A.2) produces real data, and + personal-competitiveness scoring (Phase 3) never starts before Ranking 1 exists — scoring logic built against imagined data tends to fit reality poorly. - No LLM-based feature is built before the deterministic scoring pipeline exists and passes its regression tests, per [ADR-005](adr/ADR-005-explainable-ranking-before-llm-ranking.md). diff --git a/docs/INGESTION_ARCHITECTURE.md b/docs/INGESTION_ARCHITECTURE.md new file mode 100644 index 0000000..88e8f51 --- /dev/null +++ b/docs/INGESTION_ARCHITECTURE.md @@ -0,0 +1,251 @@ +# Ingestion Architecture + +Status: **proposal** — the architecture new source adapters and the local ingestion CLI must +follow, per [ADR-004](adr/ADR-004-source-adapter-architecture.md) (adapter interface), +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript runtime), +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution), and +[ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md) (identity/dedup). No +code exists yet — this document governs Phase 2A.2's implementation, not a description of +something already built. + +## 1. What exists today + +Nothing. `ingestion/` contains only a placeholder `README.md`. This document, together with the +ADRs above and [DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md), is the design that +Phase 2A.1 (schema) and Phase 2A.2 (the first adapter and CLI) implement against. + +## 2. Repository shape + +`ingestion/` becomes a second, independent Node/TypeScript project — its own `package.json`, +`package-lock.json`, `tsconfig.json` — pinned to the same Node version as `app/` (`.nvmrc`). No npm +workspace and no package shared by import between `app/` and `ingestion/` at this stage; the +Postgres schema is the contract between them (see [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)). + +``` +ingestion/ + src/ + adapters/ one module per adapter (e.g. greenhouse.ts) + orchestrator.ts shared fetch -> normalize -> identify -> upsert -> closure pipeline + sanitize.ts HTML sanitization (sanitize-html) + plain-text derivation + identity.ts URL normalization, content hashing + cli.ts entry point: `ingest`, `score` subcommands + test/ + fixtures/ synthetic, public-safe recorded-shape fixtures (never real raw responses) + package.json + tsconfig.json +``` + +## 3. Source-adapter interface (per ADR-004) + +```ts +interface SourceAdapter { + readonly adapterKey: string // e.g. 'greenhouse' + readonly adapterVersion: string // e.g. 'greenhouse/1.0.0' + readonly politeness: { + minIntervalMs: number // e.g. >= 1000ms between requests to one source + maxRequestsPerRun: number + userAgent: string // honest, non-spoofed, includes a contact URL + timeoutMs: number // e.g. 15_000 + maxResponseBytes: number // e.g. 8 MiB + } + + validateConfig(config: unknown): AdapterConfig // throws on unknown/extra keys + fetchPage(cfg: AdapterConfig, cursor?: string): Promise<{ + records: unknown[] + nextCursor?: string + pageComplete: boolean // did this page represent the source's full result set? + }> + parse(raw: unknown): Raw // hand-written type guard; throws on shape mismatch + normalize(raw: Raw, cfg: AdapterConfig): NormalizedOpportunityVersion // sanitizes, classifies + identity(raw: Raw, cfg: AdapterConfig): { + sourceExternalId: string + canonicalUrl: string + applicationUrl: string | null + } + provenance(cfg: AdapterConfig): { sourceKey: string; baseUrl: string } +} +``` + +An adapter's job is narrow, per [ADR-004](adr/ADR-004-source-adapter-architecture.md): turn one +source's data into the common schema. It never decides whether a listing is a new canonical +opportunity, a duplicate, or a match for an existing one — that is the identity/dedup step below, +which runs once, generically, after every adapter. + +**Schema validation without a new dependency**: `parse()` is a hand-written type-guard function per +adapter, not a `zod` schema. One adapter with a modest field count does not justify a runtime +validation dependency, per [AGENTS.md](../AGENTS.md)'s "don't add dependencies casually." Adopt +`zod` only once a third adapter, or the first adapter with genuinely nested/branching response +shapes, makes hand-written guards unwieldy. + +## 4. Orchestrator responsibilities (shared, written once) + +The orchestrator — not any individual adapter — owns: + +- Retry with exponential backoff **only on 429/5xx/network errors**, never on 4xx, honoring a + `Retry-After` header when present. +- Enforcing each source's configured `politeness` interval and per-run request cap. +- Conditional requests (`If-None-Match`) where a source returns an `ETag`. +- The sanitization gate (see §7) before anything reaches normalization output. +- Content hashing and idempotent upsert, keyed on `(source_id, source_external_id)`. +- The identity/deduplication decision (§5). +- Version diffing against the current version (§6). +- Closure/freshness evaluation (§8). +- Writing one `ingestion_runs` row per invocation, with `completeness` set from whether every + configured page was fetched successfully. +- Per-record failure isolation: one malformed record increments `error_count` and is skipped; it + never fails the whole run. +- `--dry-run`: runs every step, including normalization, sanitization, and hashing, but wraps all + writes in a transaction that is rolled back at the end — nothing persists. + +**Logging discipline** (per [SECURITY_AND_PRIVACY.md](SECURITY_AND_PRIVACY.md) §9): IDs, counts, +timings, and error *types*/messages only. Never description text, never a full raw payload. +`ingestion_runs.error_summary` is capped (proposal: ≤ 32 entries, ≤ 8 KB total) and truncated +messages are marked as such, never silently cut. + +## 5. Identity and deduplication (per ADR-010) + +Applied once, generically, to every new or changed listing from every adapter: + +1. **Already linked to a canonical opportunity?** Check whether captured content changed + (`content_hash`); if so, insert a new `opportunity_version`; if not, only touch + `source_listings.last_seen_at`. +2. **New, unlinked listing — strong deterministic match** (exact shared identifier scoped to the + same employer, or byte-identical normalized application URL): auto-link to the existing + canonical opportunity; insert a new version under it. +3. **Weak/uncertain match** (same organization, similar title, no exact identifier): **never + auto-merge** — create a new, independent opportunity and a `potential_duplicate_links` row + (`status = pending_review`). +4. **No match signal**: create a new opportunity and its first version. + +No fuzzy/similarity-based automatic merging is implemented at any point in Phase 2A — a false merge +silently combines two postings' histories, which is worse than a visible, manually-resolvable +duplicate (see [ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md) for the +full reasoning). The review-queue UI for confirming/rejecting a `pending_review` pair is deferred +past Phase 2A.2 — per [DEVELOPMENT_ROADMAP.md](DEVELOPMENT_ROADMAP.md), a second adapter that could +plausibly cover an already-covered employer is not enabled until that queue exists and works. + +**URL normalization**: lowercase scheme+host, strip default port/fragment/userinfo, collapse +duplicate slashes, strip one trailing slash. Tracking-parameter handling is **per-adapter, via a +significance allowlist**, not a global blocklist — a query parameter that looks like tracking on one +source can be load-bearing identity on another (e.g., a job-ID query parameter). Each adapter +declares which parameters are significant; everything else is dropped and the remainder sorted for +stable comparison. + +## 6. Versioning (per ADR-006) + +Every mutable, historically-meaningful field lives on `opportunity_versions`, never on the stable +`opportunities` row (see [DATA_MODEL.md](DATA_MODEL.md)). A content change produces a new version +row; nothing is ever updated or deleted in place, at any role's privilege level — this is enforced +at the grant level (see [RLS_POLICY_MATRIX.md](RLS_POLICY_MATRIX.md)), not merely by adapter +discipline. + +## 7. Sanitization (per SECURITY_AND_PRIVACY.md §6) + +- HTML entities are decoded once (some sources double-encode), then sanitized with an **allowlist** + (`sanitize-html`, npm — added under `ingestion/` only, justified per + [AGENTS.md](../AGENTS.md) because hand-rolled HTML sanitization is a well-known way to ship an + XSS bug): `p, br, ul, ol, li, strong, em, b, i, h3, h4, h5, h6, a[href]`. `a[href]` accepts only + `https:`/`mailto:`; every other scheme (including `javascript:`, `data:`) drops the link, keeping + its text. +- Unconditionally removed: `