diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2f19f03..b0ad091 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -6,6 +6,10 @@ explains the *why*, not just the *what* — read it alongside [docs/adr/](adr/RE ## 1. Guiding constraints +- Personal-first engineering priority: until the owner validates real-world usefulness, prioritize + working product features over preparation for hypothetical future users — see + [MVP_SCOPE.md](MVP_SCOPE.md#current-product-mode-and-engineering-priority) for the exact policy + and the safeguards that remain required regardless of product stage. - Single user today, must not require a rewrite to support more (see [ADR-008](adr/ADR-008-private-first-multi-user-later.md)). - $0 recurring cost by default (see [ADR-009](adr/ADR-009-zero-cost-mvp-deployment.md) and @@ -20,34 +24,48 @@ explains the *why*, not just the *what* — read it alongside [docs/adr/](adr/RE ## 2. Components +**Status note (Phase 2A.0)**: the diagram below reflects the current design after +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript ingestion) and +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution), which supersede +this document's earlier depiction of scheduled GitHub Actions running a Python ingestion job. No +ingestion code exists yet and no hosted Supabase project exists yet — the diagram distinguishes +**implemented**, **approved near-term**, and **possible later** components explicitly, per those +ADRs. + ```mermaid flowchart TB - subgraph Client["Browser (user device)"] + subgraph Client["Browser (user device) — implemented"] SPA["React + TypeScript SPA\n(Vite build)"] end - subgraph CF["Cloudflare Pages"] + subgraph CF["Cloudflare Pages — implemented"] Static["Static asset hosting\n(the built SPA)"] end - subgraph Supabase["Supabase (managed)"] - Auth["Auth\n(email/magic-link)"] + subgraph Supabase["Supabase — implemented (local dev stack only; no hosted project yet)"] + Auth["Auth\n(email/password)"] PG["Postgres\n(app data + RLS)"] Storage["Storage\n(resume files deferred to Phase 1B;\nno raw external content by default)"] - Edge["Edge Functions (Deno)\n[reserved, not used at MVP]"] + Edge["Edge Functions (Deno)\n[reserved; candidate future ingestion host,\nsee ADR-017 stage 2 — not built]"] + end + + subgraph Local["Owner's local machine — approved near-term design (ADR-016/ADR-017)\nno code implemented yet"] + Ingest["TypeScript ingestion CLI\n(source adapters, run on demand)"] + Score["TypeScript scoring CLI\n(deterministic ECE relevance, run on demand)"] end - subgraph GHA["GitHub Actions (scheduled)"] - Ingest["Python ingestion job\n(source adapters)"] - Score["Python scoring job\n(deterministic pipeline)"] - Parse["Python resume-parsing job\n(deferred to Phase 1B)"] + subgraph FutureHosted["Possible later hosted execution — not provisioned\n(ADR-017 migration path, evaluated only when triggered)"] + FutureEdge["Supabase Edge Function on a schedule"] + FutureWorker["Cloudflare Worker + Cron Trigger"] end - subgraph Ext["External sources"] - ATS["Greenhouse/Lever job-board APIs"] - Gov["Public government/EU job feeds"] - Events["Public event/university calendars"] - UserLinks["User-submitted links"] + subgraph Deferred["Deferred — Phase 1B, execution model not yet decided"] + Parse["Resume-parsing job\n(language/host TBD; ADR-016 does not decide this)"] + end + + subgraph Ext["External sources — Phase 2A"] + ATS["Greenhouse Job Board API\n(first adapter, ADR-016/ingestion architecture)"] + Manual["User-confirmed manual entries\n(no automated fetch, see ADR-018)"] end SPA <--> Static @@ -55,15 +73,16 @@ flowchart TB SPA <--> PG SPA <--> Storage - Ingest --> Ext - Ingest -->|service-role key, GitHub secret| PG - Score -->|reads opportunities + profile, writes scores| PG + Ingest --> ATS + Ingest -->|service-role key, local .env, never committed| PG + Score -->|reads opportunities, writes deterministic scores| PG + Manual -.->|browser-confirmed, private row, see ADR-018| PG + + FutureEdge -.->|later migration stage, not built| PG + FutureWorker -.->|later migration stage, not built| PG + Parse -.->|future: reads uploaded resume| Storage Parse -.->|future: writes reviewable suggestions| PG - - GHA -.->|cron trigger| Ingest - GHA -.->|cron trigger| Score - GHA -.->|cron trigger| Parse ``` ### Frontend — React + TypeScript + Vite, hosted on Cloudflare Pages @@ -147,35 +166,56 @@ raw external HTML/JSON as it was originally fetched. Concretely: ingestion run doubles as a keep-alive, but this is documented explicitly as a real operational risk in [COST_MODEL.md](COST_MODEL.md) and [OBSERVABILITY.md](OBSERVABILITY.md), not hidden. -### Ingestion, parsing, and scoring — Python, run by GitHub Actions on a schedule +### Ingestion and scoring — TypeScript, run as a local CLI (Phase 2A; approved near-term design, not yet implemented) + +**Status**: this section reflects [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (runtime +language) and [ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (execution model), which +supersede this document's earlier Python/GitHub-Actions description for ingestion and opportunity +scoring specifically. Resume parsing's language and execution model remain a separate, undecided +Phase 1B question (see below) — neither ADR resolves it. -Three related but distinct jobs, all Python, all triggered by GitHub Actions cron: +Two related but distinct TypeScript jobs, run today by the repository owner as an explicit local +CLI, not on any schedule: 1. **Ingestion**: runs each configured source adapter, normalizes results into the common opportunity schema, resolves cross-source identity (auto-linking on strong evidence, flagging uncertain matches for review, otherwise creating a new canonical opportunity — see [ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)), diffs against the last known version, and writes new/updated source listings, opportunities, and version - history to Postgres using a service-role key held in GitHub Actions secrets (never exposed to - the browser). -2. **Resume parsing**: a future Phase 1B capability. It will extract separately reviewable - suggestions, never overwrite confirmed profile records, and must revisit field/suggestion-level - provenance ([ADR-014](adr/ADR-014-confirmed-profile-versus-imported-suggestions.md)). -3. **Scoring**: recomputes market-relevance and personal-competitiveness scores whenever - opportunities or the profile change meaningfully, writing explainable, versioned score records. - -Chosen as Python specifically for its data/text-processing and future ML ecosystem (resume -parsing, NLP-assisted extraction, pandas-based market analysis), independent of what language the -(currently nonexistent) synchronous backend would use if one is added later. - -**Where it runs**: GitHub Actions, on standard GitHub-hosted runners. The repository is public, so -these minutes are free and unlimited (see [COST_MODEL.md](COST_MODEL.md)) — not always-on; each -job is a short-lived process that starts, does work, writes to Postgres, and exits. + history to Postgres using a service-role key read from a gitignored local `.env.local` (never + committed, never held by any hosted compute at this stage). +2. **Scoring**: recomputes deterministic, profile-independent market-relevance scores + ([SCORING_AND_MATCHING_MODEL.md](SCORING_AND_MATCHING_MODEL.md) Ranking 1) whenever + opportunities change meaningfully, writing explainable, versioned score records. Personal + competitiveness (Ranking 2) is later work, deferred until the profile domains it needs exist. + +Chosen as TypeScript specifically because the actual Phase 2A workload (fetch JSON, validate, +normalize, sanitize, hash, upsert) does not need Python's data/ML ecosystem, and because the +realistic future hosted execution targets (Supabase Edge Functions, Cloudflare Workers) both run +TypeScript natively — see [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) for the full +reasoning, including what this does **not** decide about Phase 1B resume parsing. + +**Where it runs today**: the repository owner's own machine, invoked on demand +(`node --env-file=.env.local dist/cli.js ingest ...`), against the local (or, once one exists, a +real hosted) Supabase stack — not GitHub Actions, and not any other scheduled or hosted compute. +See [ADR-017](adr/ADR-017-local-first-ingestion-execution.md) for why, and for the explicit, +trigger-gated later migration path (Supabase Edge Function → Cloudflare Worker → last-resort +scheduled container), none of which is provisioned yet. + +**Resume parsing** (a future Phase 1B capability, unaffected by ADR-016/ADR-017): it will extract +separately reviewable suggestions, never overwrite confirmed profile records, and must revisit +field/suggestion-level provenance +([ADR-014](adr/ADR-014-confirmed-profile-versus-imported-suggestions.md)). Its language and +execution model are open questions for that phase's own design pass — Python remains a plausible, +but not yet decided, candidate (see [OPEN_QUESTIONS.md](OPEN_QUESTIONS.md)). **Failure cases**: a failed adapter run must not corrupt existing data — writes are additive (new versions), and a run that errors partway logs the failure (see -[OBSERVABILITY.md](OBSERVABILITY.md)) and is safely retried on the next schedule without -duplicating data, using source-provided stable IDs or URL+content hashing for idempotency. +[OBSERVABILITY.md](OBSERVABILITY.md)) and is safely re-run by the owner without duplicating data, +using source-provided stable IDs or URL+content hashing for idempotency. Because runs are manual, +"best-effort cron timing" does not apply here the way it does to the future hosted stages; freshness +is exactly as current as the owner's last run, and the UI must present `stale` accordingly (see +[DATA_FLOW.md](DATA_FLOW.md)). ### Reserved, not built at MVP: a synchronous API/worker layer @@ -183,21 +223,30 @@ Some future features (instant resume processing on upload, synchronous LLM expla receivers for email-based alerts) need a request/response server, not a batch job. The architecture reserves Supabase Edge Functions (Deno, colocated with the database, generous free tier) as the first place such logic would go, with a move to a small containerized FastAPI service -(Python, consistent with the ingestion codebase) as the fallback if Edge Functions' constraints -(execution time, language) don't fit. This is discussed, not built — see -[ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md). +(Python) as the fallback if Edge Functions' constraints (execution time, language) don't fit. This +is discussed, not built — see [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md). (Note: +ADR-002's original fallback rationale described this Python service as "consistent with the +ingestion codebase" — since [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) moved ingestion +to TypeScript, that specific justification no longer holds; ADR-002's actual decision, no +synchronous backend at MVP, is otherwise unaffected and remains Accepted.) -## 3. Repository layout (Phase 0: implemented as shown) +## 3. Repository layout (Phase 0: implemented as shown; `ingestion/` updated by Phase 2A) ``` careeros/ app/ React + TypeScript + Vite frontend (implemented, Phase 0) - ingestion/ Python: source adapters, normalization, scoring, resume parsing - (placeholder only -- no code yet, see ingestion/README.md; Phase 2+) + ingestion/ TypeScript: source adapters, normalization, deterministic scoring + (placeholder only as of Phase 2A.0 -- no code yet, see ingestion/README.md + and ADR-016; implemented starting Phase 2A.1. Resume parsing is a + separate, undecided Phase 1B question -- not assumed to live here.) supabase/ config.toml, migrations/, tests/database/ (pgTAP), scripts/ (local-only dev tooling) -- implemented, Phase 0 docs/ this documentation set - .github/workflows/ ci.yml (Phase 0); ingestion/parsing/scoring schedules are Phase 2+ + .github/workflows/ ci.yml (Phase 0; gains an `ingestion` lint/typecheck/test job in + Phase 2A.1). No ingestion/scoring *schedule* is planned in GitHub + Actions -- ingestion runs as a local CLI (see ADR-017); scheduled + hosted execution, if ever adopted, targets Supabase Edge Functions or + Cloudflare Workers, not GitHub Actions cron. ``` A monorepo, not separate repositories, because the frontend, database schema, and ingestion @@ -239,12 +288,18 @@ abstract. | Option | When it would be the better choice | Why not chosen for the MVP | |---|---|---| -| **No backend; direct Supabase access + batch Python (chosen)** | Fits a single-user tool where almost every operation is CRUD-on-your-own-data (RLS-safe) or a scheduled batch job | — | +| **No backend; direct Supabase access + batch ingestion/scoring (chosen; TypeScript per [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md), local CLI per [ADR-017](adr/ADR-017-local-first-ingestion-execution.md))** | Fits a single-user tool where almost every operation is CRUD-on-your-own-data (RLS-safe) or an owner-invoked batch job | — | | **FastAPI (Python) as a real API server** | Once synchronous, user-triggered server-side logic is unavoidable (instant resume parsing, LLM explanation endpoints with rate limiting, webhook receivers) | Needs an always-on host; free tiers for that (Render, Fly.io, Railway) come with sleep/cold-start behavior or limited free hours, adding operational complexity not justified yet. This is the **documented target** for when the batch-only model proves insufficient — see [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md) | -| **Node.js backend** | If the team wanted one language across the whole stack | Python was already the right choice for ingestion/parsing/scoring (see below); running two backend languages is worse than running Python everywhere non-frontend | +| **Node.js backend** | If the team wanted one language across the whole stack | This synchronous-backend question is independent of the ingestion-language decision below; no synchronous backend is built at MVP either way (this section's own Decision) | ### Ingestion/analysis language +**Status (Phase 2A.0)**: superseded for opportunity ingestion and deterministic scoring by +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) — see that ADR for the current decision and +reasoning. The paragraph below is preserved as the original planning-phase rationale, which +correctly anticipated Phase 1B resume parsing and Phase 6 market aggregation; those remain open, +separately-decided questions, not resolved by ADR-016. + Python was specified as a requirement in the brief and is confirmed here: mature HTML parsing/sanitization libraries, pandas for market-skill aggregation, and the most direct path to future ML-assisted resume parsing or embeddings work. @@ -276,10 +331,10 @@ would be a second system to keep in sync with Postgres for no clear win. | Option | When it would be the better choice | Why not chosen for the MVP | |---|---|---| -| **Cloudflare Pages (chosen) for the frontend; Supabase managed for data/auth/storage; GitHub Actions for scheduled compute** | Fits a $0, single-user, low-traffic tool with no always-on compute need | — | +| **Cloudflare Pages (chosen) for the frontend; Supabase managed for data/auth/storage; local CLI for ingestion/scoring today, with a hosted scheduled stage (Supabase Edge Function, then Cloudflare Worker) evaluated later — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)** | Fits a $0, single-user, low-traffic tool with no always-on compute need | — | | **AWS / Azure / GCP** | Needed once the project needs fine-grained IAM, VPC-level network control, managed queues at scale, or the student specifically wants cloud-provider experience with real infrastructure spend controls | Overkill for current scale; meaningfully higher operational and cost-management burden for a single maintainer. Worth revisiting deliberately at the public-launch phase, and separately, as a *learning* exercise if desired — but not required by the product | -| **Self-hosted** (home server, student cloud credits via GitHub Student Developer Pack) | If GitHub Actions free minutes or Supabase free-tier limits are actually hit | A real $0-adjacent option given the user's student status; documented as a fallback in [COST_MODEL.md](COST_MODEL.md) rather than the default, to avoid taking on server maintenance before it's needed | -| **Long-running background worker** (always-on process, not scheduled batch) | If ingestion needs near-real-time polling (many sources checked every few minutes) | Not needed at MVP ingestion frequency (daily/few-times-daily is enough for job postings and events); scheduled GitHub Actions jobs are simpler to reason about and free | +| **Self-hosted** (home server, student cloud credits via GitHub Student Developer Pack) | If Supabase free-tier limits are actually hit | A real $0-adjacent option given the user's student status; documented as a fallback in [COST_MODEL.md](COST_MODEL.md) rather than the default, to avoid taking on server maintenance before it's needed | +| **Long-running background worker** (always-on process, not scheduled batch) | If ingestion needs near-real-time polling (many sources checked every few minutes) | Not needed at MVP ingestion frequency (daily/few-times-daily is enough for job postings and events, and Phase 2A doesn't even run on a schedule yet); a future *scheduled* hosted stage (Supabase Edge Function or Cloudflare Worker, not GitHub Actions — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) is simpler to reason about and free | | **Dedicated serverless functions (Vercel/Cloudflare Workers) beyond Supabase Edge Functions** | If synchronous compute needs to live outside Supabase's own edge runtime for latency, language, or ecosystem reasons | Supabase Edge Functions are colocated with the database (fewer network hops, one fewer provider to manage) and are the first thing evaluated when synchronous compute becomes necessary | ## 5. Where each component runs (summary) @@ -289,22 +344,28 @@ would be a second system to keep in sync with Postgres for no clear win. | React SPA | User's browser | Untrusted execution environment; never holds secrets beyond the user's own short-lived session token | | Cloudflare Pages | Cloudflare's CDN | Serves static assets only; no application logic or secrets | | Supabase Auth/Postgres/Storage | Supabase-managed cloud (AWS-backed) | Authoritative data store; RLS is the authorization boundary | -| Python ingestion/parsing/scoring | GitHub Actions runners (ephemeral, per-run VMs) | Holds the Postgres service-role key as a GitHub secret; never exposed to the browser or logs | +| TypeScript ingestion/scoring (Phase 2A; [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)) | The repository owner's own machine, run on demand as a local CLI ([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) — not GitHub Actions | Holds the Postgres service-role key in a gitignored local `.env.local`; never committed, never exposed to the browser or logs. A later hosted stage (Supabase Edge Function, Cloudflare Worker) would hold it in that platform's own secret storage instead | +| Resume parsing (Phase 1B, deferred) | Not yet decided — language and execution model are open questions ADR-016/ADR-017 do not resolve | Not yet decided | | External sources | Third-party infrastructure | Fully untrusted; every response is treated as untrusted data, sanitized before storage | ## 6. Scalability limits of this design This design comfortably handles one user and a realistic ingestion volume (low thousands of -opportunity records, daily-to-hourly ingestion cadence). Known ceilings, documented honestly rather +opportunity records). Under Phase 2A's manual local-CLI execution +([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)), cadence is simply "however often the +owner runs it" rather than a target frequency; the daily-to-hourly figures below describe a +possible future scheduled stage, not the current design. Known ceilings, documented honestly rather than solved preemptively: -- **GitHub Actions minutes** are free and unlimited on standard runners for this public - repository; the practical ceiling is runner concurrency and job duration, not a minute quota — - not a problem at daily/few-times-daily cadence, would become a design question at near-real-time - polling of many sources. -- **Supabase free-tier project pausing on inactivity** requires a keep-alive strategy (the - scheduled jobs themselves largely cover this, but this is a real fragility, not a solved - problem — see [COST_MODEL.md](COST_MODEL.md)). +- **CI (GitHub Actions) minutes** are free and unlimited on standard runners for this public + repository; the practical ceiling is runner concurrency and job duration, not a minute quota. This + applies to CI today; if a future scheduled hosted stage for ingestion is ever adopted, it targets + Supabase Edge Functions or Cloudflare Workers, not GitHub Actions cron — see + [ADR-017](adr/ADR-017-local-first-ingestion-execution.md). +- **Supabase free-tier project pausing on inactivity** requires a keep-alive strategy. The original + plan assumed scheduled ingestion jobs would largely cover this; under Phase 2A's manual execution + model that mitigation does not apply, so this is currently a **more exposed** fragility than + originally planned, not a solved one — see [COST_MODEL.md](COST_MODEL.md) §3. - **No synchronous backend** means any feature requiring instant server-side response (real-time chat-style LLM interaction, instant resume feedback) is out of scope until the Edge Function/FastAPI layer is added. @@ -314,9 +375,15 @@ than solved preemptively: ## 7. GitHub Actions reliability characteristics (read before relying on cadence) +**Scope note (Phase 2A.0)**: opportunity ingestion and scoring do **not** currently run on GitHub +Actions at all — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md); they run as a local +CLI on demand. This section's cautions apply to CI itself (which does run on every push/PR, see +`.github/workflows/ci.yml`) and would apply again to any *future* scheduled workflow this project +adopts — they are not a description of how ingestion currently behaves. + The cadences in [DATA_FLOW.md](DATA_FLOW.md#5-data-freshness-expectations-proposal) are targets, -not guarantees. This is not a caveat to skim past — it shapes how every scheduled job and every -UI status indicator must be built: +not guarantees for whichever execution stage is current. This is not a caveat to skim past — it +shapes how every scheduled job and every UI status indicator must be built: - **Scheduled (`cron`) triggers are best-effort and can start late.** GitHub's own documentation states that scheduled workflows may be delayed during periods of high load, most commonly at the diff --git a/docs/COST_MODEL.md b/docs/COST_MODEL.md index b6ce16b..376e09d 100644 --- a/docs/COST_MODEL.md +++ b/docs/COST_MODEL.md @@ -9,8 +9,9 @@ relying on it operationally, and updated when it drifts. | Component | Provider | Free tier basis | Realistic MVP usage | Risk of exceeding | |---|---|---|---|---| | Frontend hosting | Cloudflare Pages | Generous free tier for static sites, unlimited requests/bandwidth on the free plan as published | One small SPA, single user | Very low | -| Database + Auth + Storage | Supabase free tier | Free project with published limits on database size, monthly active users, storage, and auto-pauses after a period of inactivity | Low data volume (low thousands of rows), one user, small files (resumes are KB-scale) | **Real risk**: free-tier project pausing after inactivity — mitigated by scheduled jobs acting as a keep-alive, but this is a genuine fragility, not fully solved (see §3). Storage growth is also bounded on purpose: only sanitized opportunity snapshots are retained indefinitely (see [ARCHITECTURE.md](ARCHITECTURE.md#raw-external-content-retention-policy)) — raw external responses are not stored by default, and the optional debug table has a short (14-day, proposal) retention with scheduled cleanup, so it cannot grow unbounded even if left enabled | -| Scheduled compute | GitHub Actions | The repository is public, so standard GitHub-hosted runner minutes are free and unlimited (no monthly quota, unlike a private repository on the free plan) | Short jobs (minutes each), daily-to-hourly cadence | Low at proposed cadence on standard runners; larger/specialized runners and some storage or advanced Actions features can still be billable even on a public repository — not used here | +| Database + Auth + Storage | Supabase free tier | Free project with published limits on database size, monthly active users, storage, and auto-pauses after a period of inactivity | Low data volume (low thousands of rows), one user, small files (resumes are KB-scale) | **Real risk, currently higher than originally assumed**: free-tier project pausing after inactivity was previously expected to be mitigated by *scheduled* ingestion jobs acting as a keep-alive; per [ADR-017](adr/ADR-017-local-first-ingestion-execution.md), Phase 2A ingestion is a **manually run local CLI, not a schedule**, so that mitigation does not apply until a later hosted/scheduled execution stage is adopted — see §3 for the restated fragility. Storage growth is also bounded on purpose: only sanitized opportunity snapshots are retained indefinitely (see [ARCHITECTURE.md](ARCHITECTURE.md#raw-external-content-retention-policy)) — raw external responses are not stored by default, and the optional debug table has a short (14-day, proposal) retention with scheduled cleanup, so it cannot grow unbounded even if left enabled | +| CI compute | GitHub Actions | The repository is public, so standard GitHub-hosted runner minutes are free and unlimited (no monthly quota, unlike a private repository on the free plan) | `frontend`/`database` jobs today; a third `ingestion` job from Phase 2A.1 (install, lint, typecheck, test — no live network) | Low. **Not** used for scheduled ingestion/scoring compute — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md); larger/specialized runners and some storage or advanced Actions features can still be billable even on a public repository, but nothing here uses them | +| Ingestion/scoring compute | The repository owner's own machine (Phase 2A; local CLI) | $0 — no provider, no hosted compute | Manual, on-demand runs | None at this stage. A later hosted/scheduled stage (Supabase Edge Function, then Cloudflare Worker — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) would move this row onto whichever free tier that provider publishes, evaluated at that time | | Domain | Optional custom domain | N/A — `*.pages.dev` is free | Not required for functionality | N/A unless a custom domain is deliberately purchased | | LLM API calls (deferred feature) | Not yet selected | Most providers require payment beyond a small trial credit | Not used until Phase 7 | This is the most likely first genuinely-paid line item once LLM explanation prose is added — budget and provider choice deferred to that phase, and must be re-evaluated against [SCORING_AND_MATCHING_MODEL.md](SCORING_AND_MATCHING_MODEL.md)'s constraint that LLM use only follows grounded scoring | @@ -20,18 +21,26 @@ Every "free tier" above is a business decision by a for-profit company and can c architecture's resilience to this isn't "these will always be free" but "each component is swappable without a rewrite": Cloudflare Pages could be replaced by any static host; Supabase's Postgres+Auth+Storage bundle could, at real cost in migration effort, be replaced by a self-hosted -Postgres plus a different auth provider if its terms ever became unworkable; GitHub Actions could -be replaced by any scheduler that can run a Python script. This swappability is a consequence of +Postgres plus a different auth provider if its terms ever became unworkable; whichever execution +stage runs ingestion (local CLI today; possibly a Supabase Edge Function or Cloudflare Worker +later — see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) could be replaced by any +runtime that can execute the same TypeScript adapter code +([ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)). This swappability is a consequence of keeping business logic in the application/adapter code rather than in provider-specific glue, consistent with [ARCHITECTURE.md](ARCHITECTURE.md). ## 3. Known fragility: Supabase free-tier project pausing Free Supabase projects pause after a period of inactivity and require a manual or API-triggered -unpause. Scheduled ingestion/scoring jobs hitting the database regularly should prevent this in -practice, but this must be explicitly monitored (see [OBSERVABILITY.md](OBSERVABILITY.md)) rather -than assumed solved — a paused database silently breaks every scheduled job until someone notices -and unpauses it manually. +unpause. The original mitigation assumed *scheduled* ingestion/scoring jobs would hit the database +regularly enough to prevent this; per +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md), Phase 2A ingestion is a **manually +run local CLI**, so this fragility is currently **more exposed, not less**, than the original +planning assumed — a hosted Supabase project left unused between the owner's manual runs can pause +on its own schedule regardless of how often ingestion happens to run. This must be explicitly +monitored (see [OBSERVABILITY.md](OBSERVABILITY.md)) rather than assumed solved, and is a concrete +reason to revisit ADR-017's later execution stages once a hosted project actually exists — a +scheduled stage would restore the original keep-alive mitigation. ## 4. Real, legitimately-free options given the user's student status diff --git a/docs/DATA_FLOW.md b/docs/DATA_FLOW.md index ca6c035..7d4b38e 100644 --- a/docs/DATA_FLOW.md +++ b/docs/DATA_FLOW.md @@ -3,30 +3,46 @@ Status: **proposal**, consistent with [ARCHITECTURE.md](ARCHITECTURE.md) and [DATA_MODEL.md](DATA_MODEL.md). -## 0. A standing caveat: GitHub Actions is a scheduler, not a reliable interactive system - -Every flow below runs on GitHub Actions cron. Cron triggers are **best-effort and can start -late** — GitHub documents that scheduled workflows may be delayed, especially at the top of the -hour when load is highest, and can occasionally be dropped entirely during platform incidents. -Every flow in this document is therefore designed to be **idempotent, safely retryable, and -tolerant of a job simply not running exactly on schedule** — nothing here assumes cron fires at a -precise time, and no user-facing feature should imply otherwise. See +## 0. Execution-model status (Phase 2A.0) + +**Opportunity ingestion and scoring currently run as an explicit local CLI, invoked on demand by +the repository owner — not on any GitHub Actions schedule.** See +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript) and +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution), which supersede +this document's earlier "every flow below runs on GitHub Actions cron" framing for the ingestion +flow in §1. Under this model, freshness is exactly as current as the owner's last run — there is no +cron schedule to be "best-effort" about, and the `stale` status in §1's flow reflects "not run +recently by the owner," not a missed cron trigger. + +The cron-reliability cautions below remain relevant for two things: this repository's own CI +(`.github/workflows/ci.yml`, which does run on every push/PR) and **any future scheduled workflow** +this project might adopt at a later execution-model stage +([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)'s migration path) — they are not a +description of how ingestion behaves today. + +**Historical framing, preserved for that future-scheduled case**: cron triggers are **best-effort +and can start late** — GitHub documents that scheduled workflows may be delayed, especially at the +top of the hour when load is highest, and can occasionally be dropped entirely during platform +incidents. Any future scheduled flow in this document must remain **idempotent, safely retryable, +and tolerant of a job simply not running exactly on schedule** — nothing here should assume cron +fires at a precise time, and no user-facing feature should imply otherwise. See [ARCHITECTURE.md](ARCHITECTURE.md#7-github-actions-reliability-characteristics-read-before-relying-on-cadence) -for the full set of reliability characteristics this constrains. +for the full set of reliability characteristics this constrains, if and when a scheduled stage is +adopted. ## 1. Opportunity ingestion, deduplication, and scoring flow ```mermaid sequenceDiagram - participant Cron as GitHub Actions (cron, best-effort timing) - participant Adapter as Source adapter (Python) + participant Owner as Repository owner (local CLI, on demand) + participant Adapter as Source adapter (TypeScript) participant Norm as Normalizer participant Dedup as Identity/dedup step participant DB as Postgres (Supabase) - participant Score as Scoring engine (Python) + participant Score as Scoring engine (TypeScript) participant SPA as React SPA - Cron->>Adapter: run scheduled ingestion job (may start later than scheduled) + Owner->>Adapter: run ingestion CLI (owner-invoked, no schedule) Adapter->>Adapter: fetch from source (compliant API/feed) Adapter->>Norm: raw source record Norm->>Norm: map to common opportunity schema,\nsanitize content, classify category/subtype @@ -50,7 +66,7 @@ sequenceDiagram end end Note over DB: a source_listing no longer found in a fresh fetch\nis marked status = removed on the listing;\nthe canonical opportunity is marked removed only when\nevery linked listing is removed - Cron->>Score: run scoring job (after ingestion; also best-effort timing) + Owner->>Score: run scoring CLI (after ingestion, owner-invoked) Score->>DB: read opportunities whose current_version_id changed\nsince last scoring run, + current user profile Score->>Score: compute market-relevance score (Ranking 1) Score->>Score: compute personal-competitiveness score (Ranking 2) @@ -119,7 +135,7 @@ flowchart LR A["User views opportunity + fit explanation\n(scoped to current_version_id)"] --> B["User clicks Save"] B --> C["saved_opportunities row created,\npinning opportunity_id AND\nsaved_opportunity_version_id"] C --> D{"User moves to\napplication pipeline?"} - D -- yes --> E["applications row created\n(status = identified)"] + D -- yes --> E["applications row created\n(status = preparing,\nsaved_opportunity_id set)"] E --> F["User adds/accepts suggested tasks"] F --> G["tasks rows created,\nlinked to application"] D -- not yet --> H["Stays saved only"] @@ -131,9 +147,23 @@ as they saw it — including which `source_listing_id` (and therefore which sour even if the canonical opportunity later gains new versions, gets merged with another duplicate, or its current-version pointer moves on. +A manually entered `private_opportunities` row (see +[USER_WORKFLOWS.md](USER_WORKFLOWS.md#workflow-9--manually-adding-an-opportunity-phase-2a-see-adr-018)) +follows the same "moves to application pipeline" step, but there is no separate save step first — +the private row is already the user's own saved record — and the `applications` row it creates sets +`private_opportunity_id` plus `private_opportunity_snapshot_jsonb` instead of `saved_opportunity_id` +(see [DATA_MODEL.md](DATA_MODEL.md#user-activity-domain-user-owned-rls-protected)). Exactly one of +`saved_opportunity_id` / `private_opportunity_id` is set on any given `applications` row. + ## 4. Market intelligence aggregation -A scheduled (weekly, proposal) Python job reads recent opportunity versions and their structured +**Status**: Phase 6, not yet designed or built. Its language and execution model are **not** +decided by [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (scoped explicitly to opportunity +ingestion and deterministic opportunity scoring only) — Python remains a plausible candidate here, +alongside TypeScript for consistency with ingestion, to be decided at that phase's own planning +pass. The description below is preserved as the original planning-phase proposal. + +A scheduled (weekly, proposal) job reads recent opportunity versions and their structured required/preferred qualifications, tallies skill/tool/tooling mentions per engineering-area category, and writes a compact aggregate snapshot to Postgres (not recomputed on every page load). **Aggregation counts distinct canonical `opportunity_id`s, excluding any with `status = @@ -149,11 +179,13 @@ expensive aggregation off the request path entirely. | Data | Update cadence | Rationale | |---|---|---| -| Opportunity ingestion | Daily, or a few times daily per source | Balances freshness against GitHub Actions free-minute budget and source politeness | -| Resume parsing | Every 10-15 minutes, eventually consistent | Fast enough to feel responsive without needing a live server — but never guaranteed to hit that interval exactly, per §0 | -| Scoring recompute | After every ingestion run that changes a `current_version_id`, and after any profile edit | Scores must never silently go stale relative to the profile or version they claim to explain | -| Market intelligence snapshot | Weekly | Aggregate trends don't meaningfully shift faster than this at MVP data volume | - -These cadences are targets the schedule aims for, not commitments the system can guarantee — see -§0 and [OBSERVABILITY.md](OBSERVABILITY.md) for how staleness itself is monitored and surfaced to -the user rather than silently assumed. +| Opportunity ingestion | Whenever the owner runs the local CLI (Phase 2A; see [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) — a daily or few-times-daily target *once* a later hosted/scheduled stage is adopted | No schedule exists yet; there is no cron budget to balance against today. Source politeness still governs the self-imposed per-run request budget regardless of cadence | +| Resume parsing | Every 10-15 minutes, eventually consistent (Phase 1B, not yet designed) | Fast enough to feel responsive without needing a live server — but never guaranteed to hit that interval exactly, and its execution model is a separate, undecided question (see [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)'s scope note) | +| Scoring recompute | Whenever the owner runs the local scoring CLI after an ingestion run that changed a `current_version_id` (Phase 2A); after every ingestion run automatically, once a scheduled stage exists | Scores must never silently go stale relative to the opportunity version they claim to explain | +| Market intelligence snapshot | Weekly (Phase 6, not yet built) | Aggregate trends don't meaningfully shift faster than this at MVP data volume | + +Today's cadences are simply "whenever the owner runs the CLI," not a schedule at all. The +daily/weekly figures above are targets for the later, explicitly-triggered scheduled stage in +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) — see §0 and +[OBSERVABILITY.md](OBSERVABILITY.md) for how staleness itself is monitored and surfaced to the user +rather than silently assumed, under either execution model. diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index 5b536d3..ed5be95 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -35,6 +35,18 @@ This revision corrects two gaps identified in review: suggestion structures. `profiles.profile_complete` has no authoritative role; completeness is derived from [PROFILE_COMPLETENESS_SPEC.md](PROFILE_COMPLETENESS_SPEC.md). +## Revision note (Phase 2A.0 architecture pass) + +Manually entered opportunities are **not** part of the shared opportunity-identity domain below. +[ADR-018](adr/ADR-018-private-manual-opportunities.md) adds `private_opportunities` (see the User +activity domain table) as a structurally separate, user-owned table — private by default, ordinary +mutable CRUD rather than versioned, and outside [ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)'s +deduplication scope. This is a documentation-and-decision update only; no migration exists yet +(that is Phase 2A.1). [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) and +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) additionally settle the ingestion runtime +language and execution model referenced throughout [ARCHITECTURE.md](ARCHITECTURE.md) and +[DATA_FLOW.md](DATA_FLOW.md); neither changes this document's schema. + ## Entity-relationship overview ```mermaid @@ -73,6 +85,9 @@ erDiagram OPPORTUNITY_VERSIONS ||--o{ SAVED_OPPORTUNITIES : snapshot_of USERS ||--o{ APPLICATIONS : tracks SAVED_OPPORTUNITIES ||--o| APPLICATIONS : becomes + + USERS ||--o{ PRIVATE_OPPORTUNITIES : enters + PRIVATE_OPPORTUNITIES ||--o| OPPORTUNITIES : promoted_to APPLICATIONS ||--o{ TASKS : generates APPLICATIONS ||--o{ INTERVIEW_PREP_NOTES : has @@ -240,9 +255,10 @@ distinct, timestamped event, not a mutation of the number the user already saw. | Table | Key fields | Notes | |---|---|---| | `saved_opportunities` | `id`, `user_id`, `opportunity_id`, `saved_opportunity_version_id`, `notes`, `saved_at` | Snapshot pinning: even if the opportunity gets new versions later, this remembers which version — and, transitively via that version's `source_listing_id`, which source content — the user actually saved and reacted to | -| `applications` | `id`, `user_id`, `saved_opportunity_id`, `status` (`identified`\|`preparing`\|`applied`\|`interview_scheduled`\|`interview_complete`\|`offer`\|`rejected`\|`withdrawn`), `applied_at`, `resume_id_used` | | +| `applications` | `id`, `user_id`, `saved_opportunity_id` (nullable FK to `saved_opportunities`, set for a shared/ingested opportunity), `private_opportunity_id` (nullable FK to `private_opportunities`, set for a manually entered one — exactly one of the two is set), `private_opportunity_snapshot_jsonb` (populated only when `private_opportunity_id` is set — see note below), `status` (`preparing`\|`applied`\|`awaiting_response`\|`interview_scheduled`\|`interview_complete`\|`offer`\|`accepted`\|`rejected`\|`withdrawn`\|`closed`), `applied_at`, `status_updated_at`, `next_action` (nullable), `next_action_due_at` (nullable), `notes`, `contact_note` (nullable), `resume_id_used` | Current status plus timestamps only — no event-sourced status-history table at MVP (see [MVP_SCOPE.md](MVP_SCOPE.md#current-product-mode-and-engineering-priority)); "saved" is a state prior to and outside this enum, held on `saved_opportunities`/`private_opportunities` | | `tasks` | `id`, `user_id`, `application_id` (nullable), `opportunity_id` (nullable), `title`, `due_date`, `status` (`open`\|`done`), `origin` (`manual`\|`suggested`) | Deadline-derived tasks reference the opportunity directly even before an application exists | -| `interview_prep_notes` | `id`, `application_id`, `content`, `created_at` | | +| `interview_prep_notes` | `id`, `application_id`, `content`, `created_at` | Free-text notes tied to the application — likely questions, topics to revise, recruiter notes, post-interview reflections all fit in `content`; no structured sub-fields at MVP | +| `private_opportunities` | `id`, `user_id`, `source_url`, `title`, `organization_name`, `location_text`, `opportunity_kind`, `employment_type`, `remote_mode`, `description_text`, `posted_at`, `application_deadline`, `application_url`, `dismissed_at`, `promoted_to_opportunity_id` (nullable, trusted-write only), `created_at`, `updated_at`, `last_confirmed_at` | **New (Phase 2A, see [ADR-018](adr/ADR-018-private-manual-opportunities.md))**: manually entered opportunities, structurally and permission-wise separate from the shared identity domain above — not written through the ingestion/dedup pipeline, not versioned like `opportunity_versions` (ordinary mutable CRUD, like `work_experience`), private by default. `promoted_to_opportunity_id` is a future, trusted-only, additive link to a canonical shared opportunity if one is later discovered by an adapter; no promotion logic exists yet. Exact column types/constraints are finalized at Phase 2A.1 implementation time | ## Raw external-content retention (debug-only, not a core table) @@ -264,6 +280,17 @@ entirely to `opportunity_versions`). latest": a user's saved reasoning and fit explanation refer to the posting as it existed when they looked at it. If requirements change later, the user should see that explicitly as a diff, not have their prior judgment silently reinterpreted against new text. +- **Why `applications` carries its own `private_opportunity_snapshot_jsonb`** rather than reading + `private_opportunities` live: a shared opportunity already gets this guarantee for free through + `saved_opportunities.saved_opportunity_version_id` pointing at an immutable + `opportunity_versions` row. `private_opportunities` has no such immutable version history — it is + ordinary mutable CRUD, like `work_experience` — so without its own snapshot, editing a private + entry after applying (fixing a typo, updating a copied deadline) would silently rewrite the exact + listing content the application and interview-prep notes refer to. The snapshot is captured once, + at `applications` creation time, from `private_opportunities`' fields as they stood at that + moment, and is never updated afterward — this is the smallest relationship that gives both the + shared and private paths the same "what I applied to is preserved" guarantee without introducing + version history for `private_opportunities` itself. - **Why `evidence_jsonb` exists on every score table**: this is the mechanism that satisfies the hard explainability requirement in [PRODUCT_REQUIREMENTS.md](PRODUCT_REQUIREMENTS.md) — every sub-score must be traceable to the specific profile fields and posting requirement lines that diff --git a/docs/DATA_SOURCES_AND_COMPLIANCE.md b/docs/DATA_SOURCES_AND_COMPLIANCE.md index 3d2685b..e9201a0 100644 --- a/docs/DATA_SOURCES_AND_COMPLIANCE.md +++ b/docs/DATA_SOURCES_AND_COMPLIANCE.md @@ -1,8 +1,16 @@ # Data Sources and Compliance -Status: **proposal** for the source categories and specific example sources; the exact source list -for the MVP should be finalized (and its terms of service actually read) before any adapter is -built — see [OPEN_QUESTIONS.md](OPEN_QUESTIONS.md). +Status: **dated source-compliance registry**, current as of **2026-08-03** (Phase 2A.0 research +pass). Terms of service, `robots.txt` directives, and API behavior can change after this date — +this document should be re-verified against each source's current terms before its adapter is +enabled or re-enabled, not assumed permanently accurate. Findings below are labeled by confidence: +**confirmed** (verified against official documentation or a live, manual, low-volume request during +this pass), **inferred** (reasonable technical inference, not itself verified), or **unresolved** +(a real legal/compliance question this pass did not settle). No permission is invented where the +evidence does not support it — the precise phrases used below (*technically public*, *officially +documented*, *intended programmatic use*, *terms not fully resolved*, *blocked pending +clarification*, *manual import only*, *prohibited by identified terms*) are chosen deliberately and +are not interchangeable with "legally cleared." ## 1. Hard rule @@ -14,93 +22,208 @@ overridden by user request — see [AGENTS.md](../AGENTS.md). ## 2. Source-adapter architecture Every source is isolated behind an adapter implementing a common interface (fetch → normalize → -classify → return common-schema records). See -[ADR-004](adr/ADR-004-source-adapter-architecture.md) for the interface shape. This means adding -or removing a source never touches scoring, storage, or frontend code, and each source's specific -legal/operational profile is documented independently, below. - -## 3. Source categories, prioritized as instructed - -### Direct company career pages - -- **What**: individually scraping/parsing a specific company's own public careers page, or using - the company's own published feed/API if one exists. -- **Legal basis**: a company's own public career page generally permits normal, low-volume, - robots.txt-respecting access for the purpose of finding their own job postings; this must be - confirmed per-site (check `robots.txt` and terms of use) before an adapter is added, not assumed. -- **Operational concerns**: no shared schema across companies — every company page needs its own - parsing logic, which is the highest-maintenance source type. Pages change layout without notice. -- **Reliability**: lowest — most likely to break silently. Ingestion must log parse failures per - adapter clearly (see [OBSERVABILITY.md](OBSERVABILITY.md)) rather than fail the whole run. -- **Recommended MVP use**: pick a small number (2-5) of specific target companies/employers - relevant to the user's engineering areas and Bremen/Hamburg/international focus, rather than - attempting broad coverage. - -### Public applicant-tracking-system (ATS) job-board endpoints - -- **What**: Greenhouse and Lever both publish structured, public job-board JSON endpoints intended - for exactly this kind of consumption (many companies expose `boards-api.greenhouse.io/v1/boards//jobs` - style endpoints, and similarly for Lever); some other ATS platforms (Workday, SmartRecruiters, - Personio) offer comparable public or semi-public listing endpoints depending on the employer's - configuration. -- **Legal basis**: these are the ATS vendor's own public API surface, designed for external - consumption. Still confirm each vendor's API terms of use and any documented rate limits. -- **Operational concerns**: one adapter per ATS vendor (not per company) covers every company using - that vendor — this is the best effort-to-coverage ratio of any source category and should be the - first adapters built. -- **Reliability**: high — stable, documented, structured JSON. -- **Recommended MVP use**: primary source category for the MVP. - -### Public government or European job sources - -- **What**: e.g., Germany's Bundesagentur für Arbeit job search (which has historically offered - API access for developers), EURES (the EU's public employment portal), or similar official - sources. -- **Legal basis**: public-sector job portals generally intend third-party/API consumption but - usually require registration for an API key and have documented usage terms — must be confirmed - and, where required, an API key obtained before building the adapter. -- **Operational concerns**: broader coverage but noisier relevance (not engineering- or - student-specific by default) — normalization/classification logic carries more weight here. -- **Reliability**: generally high (public institutional infrastructure), but subject to - government-system quirks (maintenance windows, slower API evolution). - -### Public event APIs or feeds, university and organization calendars - -- **What**: hackathon/competition listing platforms with public APIs, Constructor University's own - public event pages/calendars, professional-society (e.g., VDE, IEEE student branch) event - listings where publicly accessible. -- **Legal basis**: public calendar/event pages are typically intended for public consumption; - confirm terms where an API key or registration is involved. -- **Operational concerns**: primary source for development opportunities (hackathons, - scholarships, workshops, career fairs). -- **Reliability**: variable by source; university calendar formats and small-organization sites - change without notice more than established ATS platforms. - -### User-submitted links - -- **What**: the user pastes a URL to a specific posting the pipeline hasn't ingested from a - configured source. -- **Legal basis**: the user is the one initiating the individual fetch of a page they chose to - view, for their own personal archival use — closer to a personal bookmarking/archiving action - than automated bulk extraction, but the fetch must still respect the target site's `robots.txt` - and must never be used to bulk-crawl beyond the single submitted page. -- **Operational concerns**: needs its own lightweight parser (best-effort structured extraction, - falling back to storing the sanitized raw text if structured fields can't be reliably - identified) since it isn't tied to any known adapter's schema. -- **Security concerns**: this is the primary SSRF risk surface — the fetch must validate the URL - (reject internal/private IP ranges, non-http(s) schemes) before making any server-side request. - See [SECURITY_AND_PRIVACY.md](SECURITY_AND_PRIVACY.md). - -### User-authorized email alerts - -- **What**: the user forwards or connects an email-based job-alert subscription they already - receive (e.g., a company's own "new jobs" newsletter they signed up for). -- **Legal basis**: the user already consented to receive this content directly; parsing an email - the user explicitly forwards or grants access to is materially different from scraping a - platform that prohibits it. -- **Operational concerns**: deferred past MVP — requires either an email-forwarding intake address - or an OAuth mail-read integration, both of which are real scope (see - [MVP_SCOPE.md](MVP_SCOPE.md)). Documented here as a future adapter category, not built yet. +classify → return common-schema records). See [ADR-004](adr/ADR-004-source-adapter-architecture.md) +for the interface shape and [INGESTION_ARCHITECTURE.md](INGESTION_ARCHITECTURE.md) for the full +adapter/orchestrator design, lifecycle rules, and the source-onboarding checklist (§4 of that +document) — not duplicated here. + +## 3. Source-compliance matrix + +### Applicant-tracking-system (ATS) sources + +#### Greenhouse Job Board API — **Implement first** + +| Field | Finding | +|---|---| +| Category | Public ATS job-board API | +| Official documentation | `https://developers.greenhouse.io/job-board.html` | +| Access mechanism | `GET https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs?content=true` | +| Authentication | None for GET endpoints — **confirmed**: the documentation states "Job Board data is publicly available, so authentication is not required for any GET endpoints." Only the application-submission POST endpoint requires a key | +| Intended use per documentation | "Build a custom job board or career site to post your jobs publicly for candidate applications"; export job-board data for custom career/application sites — **confirmed** | +| Geography/coverage | Global, per-employer board token; no cross-company discovery — one token per employer | +| Structured fields | `id`, `internal_job_id`, `requisition_id`, `title`, `absolute_url`, `location.name`, `offices[]`, `departments[]`, `updated_at`, `first_published`, `application_deadline`, `company_name`, `language`, `content` (HTML, entity-encoded), `metadata` — **confirmed**, observed live against a public board | +| Pagination/feed behavior | None needed — a single request returns the board's complete job list (observed: 129/129 jobs in one response for one board) — **confirmed** | +| Update/closure signals | `updated_at`, `first_published`, `application_deadline` present; closed posts simply disappear from the complete list, which is a reliable closure signal given the complete-response behavior — **confirmed / inferred for reliability** | +| Technical reliability | High — stable documented JSON, CDN-fronted — **inferred** | +| Terms/reuse status | **Technically public, officially documented, intended for career-site/job-board reuse.** No separate Job Board API terms-of-use document beyond the developer docs was found distinct from the intended-use description above — **unresolved**: whether a private, single-user archival use is squarely within "intended use" is not explicitly stated either way | +| Rate-limit risk | No documented rate limit for the Job Board API specifically — **confirmed that none is documented**; a conservative self-imposed limit is required regardless | +| Attribution requirement | None stated — **confirmed that none is stated** | +| `robots.txt` | `boards-api.greenhouse.io/robots.txt`: `User-agent: *` / `Disallow: /embed/` only — the Job Board API path is **not** disallowed — **confirmed** | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Implement first.** The adapter is restricted to explicitly configured, reviewed board tokens only (see [INGESTION_ARCHITECTURE.md](INGESTION_ARCHITECTURE.md) §10) and must never enumerate or probe arbitrary tokens | +| Reason | Unauthenticated, `robots.txt`-permitted, officially documented as intended for exactly this kind of reuse, complete-response semantics make closure detection reliable, and the strongest observed timestamp/identifier coverage of any source examined | + +#### Lever Postings API — **Candidate for later** + +| Field | Finding | +|---|---| +| Category | Public ATS job-board API | +| Official documentation | `https://github.com/lever/postings-api` | +| Access mechanism | `GET https://api.lever.co/v0/postings/{site}?mode=json` (separate EU instance: `api.eu.lever.co`) | +| Authentication | None for GET — **confirmed**. Application-submission POST requires an API key | +| Intended use per documentation | The official documentation explicitly documents this as a public postings API intended for third-party consumption and discusses building custom career pages against it — **confirmed** | +| Geography/coverage | Per-employer site name; separate global and EU instances must be configured explicitly | +| Structured fields | `id`, `text` (title), `categories{location,team,commitment,department,level}`, `description`, `hostedUrl`, `applyUrl`, `workplaceType`, optional `salaryRange` — **confirmed** from documentation. No documented update timestamp field | +| Pagination | `skip`/`limit` offset paging — **confirmed** | +| Update/closure signals | No documented update timestamp; closure inferred only from disappearance across paginated fetches — weaker than Greenhouse | +| Rate-limit risk | Documented only for the application-submission POST (429 above 2 req/s); no documented GET limit — **confirmed** | +| Terms/reuse status | **Officially documented, intended programmatic use** — the clearest "yes, third parties may consume this" documentation found among ATS sources | +| Attribution requirement | None found in the documentation reviewed | +| `robots.txt` | `api.lever.co/robots.txt`: `User-agent: *` / `Allow: /` / `Crawl-delay: 1` — explicitly permissive, with a stated delay to honor — **confirmed** | +| Coverage check | 15 plausible EU/German engineering-employer site names probed (single low-volume request each) — all returned HTTP 404; Lever's own demo board returned an empty result set — **confirmed** | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Candidate for later**, not the first adapter | +| Reason | The access basis is actually cleaner than Greenhouse's (explicit third-party-consumption documentation, a stated `Crawl-delay`), but Greenhouse has stronger observed timestamp coverage (`updated_at`, `application_deadline`) for the first implementation, and no relevant German/EU employer board was found live during this pass. Revisit once a Lever-hosted employer relevant to this user's geography/engineering areas is identified | + +#### Workable public XML feed — **Later or rejected for the MVP** + +| Field | Finding | +|---|---| +| Category | Public aggregate feed (all Workable customers) | +| Official documentation | Workable Help Center, "Utilizing the XML Job Feed" | +| Access mechanism | `GET https://www.workable.com/boards/workable.xml` | +| Authentication | None | +| Intended use per documentation | **Officially documented as intended for programmatic consumption** — explicitly described as a mechanism "to share this with a job board/partner Workable doesn't integrate with to programmatically get your job postings" | +| Geography/coverage | All published jobs across **every** Workable customer worldwide — the only genuinely cross-company discovery source examined | +| Structured fields | `title`, `date`, `referencenumber`, `url`, `company`, `city/state/country`, `remote`, `postalcode`, `description`, `education`, `jobtype`, `category`, `experience`, `website` — **confirmed** from documentation. No update timestamp beyond `date`, no deadline field | +| Pagination/feed behavior | **None** — the documentation states explicitly: "we don't support any filtering parameters on the feed; you should read/download the full file and filter on your end" — **confirmed** | +| Measured feed size (dated observation, not a contractual value) | **644,138,283 bytes uncompressed / 107,803,152 bytes gzipped, 132,490 `` records**, measured 2026-08-03 in a single request; updated hourly per the documentation | +| Attribution requirement | Yes — **confirmed**: "job URLs must not be altered, as this affects source attribution" | +| `robots.txt` | `workable.com/robots.txt` does not disallow the feed path; also declares `Content-Signal: search=yes, ai-input=yes, ai-train=no` — **confirmed** | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Rejected for the first adapter; later reconsideration possible** | +| Reason | Bandwidth, parsing cost, and maintenance burden — extracting a handful of Bremen/Hamburg-relevant postings currently requires downloading and filtering ~108 MB gzipped locally on every refresh, which is a real, measured operational cost, not a theoretical one. This is a **dated operational observation about feed size**, not a permanent contractual limit — reconsider if streaming XML parsing is implemented and a clear coverage value is identified, or if Workable ever publishes a filtered/per-company endpoint | + +#### SmartRecruiters Posting API — **Blocked pending clarification** + +| Field | Finding | +|---|---| +| Category | Public ATS posting API, described for customers' own career sites | +| Official documentation | `https://developers.smartrecruiters.com/docs/posting-api`, `/docs/endpoints`, `/docs/authentication` | +| Access mechanism | `GET https://api.smartrecruiters.com/v1/companies/{companyIdentifier}/postings` (list, with `country`/`region`/`city`/`department`/`language` filters) and `/postings/{postingId}` (detail) | +| Authentication per documentation | The Authentication page states API-key authentication is the supported method for the platform's APIs; the Posting API's own overview describes it as intended for **SmartRecruiters customers** building their own career sites and for **partners** building widgets | +| Observed behavior (single, low-volume, manual verification request per employer) | The list endpoint answered **without any credential** and returned real data — e.g. `BoschGroup` → `totalFound: 4714` (867 with `country=de`), `Continental` → `890` — **confirmed by direct observation** | +| Structured fields | `id`, `uuid`, `name`, `releasedDate`, `language`, `location{city,region,country,postalCode,latitude,longitude,remote,hybrid}`, `company{identifier,name}`, `department`, `industry`, `function`, `typeOfEmployment`, `experienceLevel`; detail adds `applyUrl`, `active`, `jobAd.sections{...}` — **confirmed**, observed live | +| Pagination | `limit`/`offset`; a requested `limit=200` was silently clamped to 100 — **confirmed** | +| Update/closure signals | No update timestamp beyond `releasedDate`; `active` boolean on the detail endpoint is the explicit closure signal | +| Known false-negative trap | An unknown/misspelled company identifier returns **HTTP 200 with `totalFound: 0`**, not 404 — verified with a deliberately fake identifier — **confirmed**. A zero result is not distinguishable from a wrong identifier by status code alone | +| `robots.txt` | `api.smartrecruiters.com/robots.txt`: `User-agent: LinkedInBot` → `Allow: /v1/companies/`; **`User-agent: * → Disallow: /`** — **confirmed** | +| Terms/reuse status | **Terms not fully resolved** — the documented API exists and answers unauthenticated requests in practice, but the documentation frames intended use as customer/partner career-site building, and the API host's own `robots.txt` disallows every user agent except one named crawler | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Blocked pending clarification. Do not implement.** Do not attempt to bypass the `robots.txt` restriction or argue around it by technical interpretation | +| Reason | A documented public API answering unauthenticated requests is not, by itself, sufficient authorization when the API domain's own `robots.txt` excludes general automated access. This is the single highest-coverage source found for this user's geography (Bosch, Continental) and is explicitly not implemented until one of: (a) written clarification from SmartRecruiters, (b) an appropriate credentialed/partner relationship, or (c) an explicit `robots.txt` change | + +#### Personio XML feeds — **Candidate for later** + +| Field | Finding | +|---|---| +| Category | Per-tenant public XML feed | +| Access mechanism | `GET https://{tenant}.jobs.personio.de/xml` (some tenants redirect to a custom domain or `.personio.com`) | +| Authentication | None observed | +| Structured fields (observed) | `id`, `name`, `jobDescriptions`, `department`, `office`, `additionalOffices`, `employmentType`, `schedule`, `seniority`, `yearsOfExperience`, `occupation`, `occupationCategory`, `recruitingCategory`, `subcompany`, `createdAt` | +| Update/closure signals | `createdAt` only — no update timestamp, no deadline field; weaker than Greenhouse | +| Terms/reuse status | **Terms not fully resolved** — no dedicated public API terms document was located during this pass; feed accessibility was only spot-checked, not reviewed against Personio's own terms | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Candidate for later**, pending a proper terms review | +| Reason | Personio is widely used by German small/mid-size engineering employers — exactly the Bremen/Hamburg gap Greenhouse does not fill — but a terms review has not yet been done, and its weaker timestamp fields make closure detection less reliable than Greenhouse's | + +#### Workday (undocumented internal `/wday/cxs/` endpoint) — **Do not use** + +| Field | Finding | +|---|---| +| Observed behavior | `POST https://ag.wd3.myworkdayjobs.com/wday/cxs/ag/Airbus/jobs` answered without authentication, `userAuthenticated: false`, `total: 2000`, minimal fields (`title`, `externalPath`, `locationsText`, `postedOn` as a relative string, `bulletFields`) | +| What this is | The internal endpoint that powers Workday's own career-site single-page application — **not** a published, documented third-party API | +| Terms/reuse status | **Public browser accessibility does not constitute a supported API.** No Workday developer documentation describing third-party use of this endpoint was found | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Do not use.** Do not implement this or any other undocumented SPA endpoint merely because it responds without authentication | +| Reason | This is precisely the case this registry exists to guard against: an endpoint being reachable is not the same as it being an intended, documented integration point | + +#### SAP SuccessFactors career sites (e.g. Fraunhofer) — **Manual import only** + +| Field | Finding | +|---|---| +| Observed | `jobs.fraunhofer.de` follows the standard SuccessFactors Recruiting Marketing career-site pattern (`/search/?createNewAlert=false`) | +| RSS/feed check | Three plausible RSS paths (`/rss`, `/rssfeed`, `/search/?...&rss=1`) all returned `text/html`, i.e. **no RSS** | +| Terms/reuse status | No vendor-level public posting API exists for SuccessFactors; each tenant's data is HTML only | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Manual import only** | + +### Official public-sector and European sources + +#### Bundesagentur für Arbeit Jobsuche — **Do not use** + +| Field | Finding | +|---|---| +| What was tested | `GET https://rest.arbeitsagentur.de/jobboerse/jobsuche-service/pc/v6/jobs` with header `X-API-Key: jobboerse-jobsuche` — answered successfully (`maxErgebnisse: 777` for a sample query), with rich German-specific fields | +| Official API status | **No official public API exists.** The interface and its constant API key are documented only by a community project (`bundesAPI/jobsuche-api`); that project's own README states the BA "bietet ... bis heute keine offizielle API an" (offers no official API to date) | +| Terms of use (official, `arbeitsagentur.de/en/terms-of-use` §2a(3)) | **Explicitly prohibited by identified terms.** Quoted directly: *"use robots, web spiders or similar technologies, or to use existing communication or programming interfaces contrary to the BA's intended purpose, and thus to read out content from the portal or apps for the purpose of data collection and evaluation."* This describes exactly the intended use case here | +| Additional context | Public reporting (netzpolitik.org, 2021) documents the BA objecting to third-party mass use of this interface | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Do not use.** Do not use this or any unofficial wrapper, and do not use the extracted application API key | +| Reason | This is the single largest German vacancy database, and the coverage loss is real — recorded explicitly so a future contributor finding the community wrapper does not mistake technical availability for permission | + +#### Make it in Germany — **Do not use** + +| Field | Finding | +|---|---| +| `robots.txt` | Permissive (`Allow: /`) | +| Data source | Its job listings are sourced from the Bundesagentur für Arbeit Jobsuche database — **confirmed** from the portal's own description of itself | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Do not use** — inherits the BA data-access concern one hop removed; using an intermediary does not change the underlying compliance question, per [ADR-007](adr/ADR-007-compliant-data-sources.md) | + +#### EURES — **Licensed/partner-only** + +| Field | Finding | +|---|---| +| What was tested | `GET https://europa.eu/eures/api` → HTTP 401 | +| Official access | The genuinely documented EURES APIs are **input** APIs used by national public employment services to push vacancies *into* EURES, not a public retrieval API for third parties. Community "EURES API documentation" projects found during research explicitly disclaim European Commission affiliation | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Licensed/partner-only; do not use** for the MVP | +| Reason | No documented, acceptable public retrieval mechanism was found. Revisit only if the European Commission publishes one | + +#### EURAXESS — **Manual import only** + +| Field | Finding | +|---|---| +| Content relevance | High — research assistant, PhD, fellowship, and thesis-adjacent postings across Europe, filterable by country, research field, and career stage (R1–R4) | +| RSS/API check | `GET https://euraxess.ec.europa.eu/jobs/search/rss` → HTTP 404; no documented API or export was found | +| `robots.txt` | A stock Drupal file disallowing only `/core/` and `/profiles/` — permission-neutral, not itself a license to bulk-extract | +| Date reviewed | 2026-08-03 | +| **MVP decision** | **Manual import only** | +| Reason | No documented programmatic access exists; a future adapter needs an explicit reuse enquiry to the European Commission first, which this pass did not initiate | + +#### European Job Days — **Manual import only** + +`europeanjobdays.eu/robots.txt` is accessible (HTTP 200, permissive); no documented API or feed was +found. Event-shaped content belongs to the deferred development-opportunity category (roadmap +Phase 5), not Phase 2A. Date reviewed: 2026-08-03. + +### Company career sites (target-employer research) + +Single, low-volume, manual verification request per employer (HEAD/redirect check, plus a +SmartRecruiters/Greenhouse/Lever probe where plausible). Date reviewed: 2026-08-03. + +| Employer | Finding | MVP decision | +|---|---|---| +| Bosch | SmartRecruiters `BoschGroup` (4,714 postings, 867 DE); own site `jobs.bosch.com/robots.txt` disallows `/en/` for all agents | Blocked (SmartRecruiters, see above) | +| Continental | SmartRecruiters `Continental` (890 postings) | Blocked (SmartRecruiters, see above) | +| Airbus | Undocumented Workday CXS endpoint (`ag.wd3.myworkdayjobs.com`) | Do not use (undocumented endpoint); manual import only | +| Siemens | `jobs.siemens.com` custom/Avature-style career site; no ATS API match | Manual import only | +| Infineon | `jobs.infineon.com/careers`; no ATS API match | Manual import only | +| Texas Instruments | `careers.ti.com` (Oracle Recruiting Cloud-shaped path); no ATS API match | Manual import only | +| NXP | No public path found matching any probed ATS; guessed careers path returned 404 | Manual import only | +| Deutsche Bahn | `karriere.deutschebahn.com` → `db.jobs`, custom site | Manual import only | +| Deutsche Telekom, Mercedes-Benz, BMW, Beckhoff, Phoenix Contact, Rohde & Schwarz, OHB, Rheinmetall, Hensoldt | Custom career sites; no ATS API match found | Manual import only | +| Fraunhofer institutes | SAP SuccessFactors, no RSS (see above) | Manual import only | +| DLR, University of Bremen, TU Hamburg (TUHH), Constructor University | Bespoke CMS career pages; guessed paths returned 404 | Manual import only | +| Greenhouse-hosted German deep-tech employers found during research | `helsing` (129 jobs, 107 German-location, 3 student-titled), `marvelfusion` (12, 10 German), `konux` (4, 4 German) | **Implement first** (Greenhouse) | + +**Honest coverage caveat**: the confirmed Greenhouse German boards found during this pass skew +Munich/Berlin deep-tech, with very few student-level titles. A Greenhouse-only adapter delivers real +automated discovery but **not** meaningful Bremen/Hamburg student-role coverage on its own — that +gap is filled by manual import (Workflow 9, +[USER_WORKFLOWS.md](USER_WORKFLOWS.md#workflow-9--manually-adding-an-opportunity-phase-2a-see-adr-018)), +which is why both ship together at Phase 2A.1 rather than the adapter alone. ## 4. Rate-limiting and politeness (applies to every adapter) @@ -108,51 +231,31 @@ legal/operational profile is documented independently, below. - Identify the ingestion job with a clear, honest User-Agent string, not a spoofed browser identity. - Cache/reuse `last_checked_at` to avoid re-fetching unchanged pages more often than the ingestion - cadence requires (see [DATA_FLOW.md](DATA_FLOW.md)). + cadence requires — under Phase 2A's manual local-CLI execution + ([ADR-017](adr/ADR-017-local-first-ingestion-execution.md)), this simply means not re-running + more often than useful, not a schedule to tune. - Back off and log, rather than retry aggressively, on 429/5xx responses. -## 5. Adding a new source (process, for future agents/contributors) - -1. Read the source's terms of use / API documentation; confirm automated access is permitted. -2. Record the source, its legal basis, and any rate-limit constraints in the table below. -3. Implement one adapter module conforming to the common interface - ([ADR-004](adr/ADR-004-source-adapter-architecture.md)). -4. Add it to the `sources` table (see [DATA_MODEL.md](DATA_MODEL.md)) with `enabled = false` until - manually verified against real data. - -## 6. Target-employer watchlist (proposed, not approved) - -The user has named the following employers as ones of real interest, reflecting the ECE -engineering-area focus and the Bremen/Hamburg/international priorities in -[PRODUCT_REQUIREMENTS.md](PRODUCT_REQUIREMENTS.md): **Airbus, Siemens, NXP, Texas Instruments, -OHB, Rohde & Schwarz, Bosch, Infineon, Mercedes-Benz, DLR, Fraunhofer institutes**, and other -relevant employers around Bremen and Hamburg. - -**This is a watchlist of candidates to investigate, not a list of cleared or implemented data -sources.** None of these companies' recruitment systems have been verified as compliant sources, -and no adapter for any of them exists. Before any one of them is turned into an adapter: - -1. Identify what recruitment/ATS platform they actually use today (a company on this list may use - its own custom career site, or a third-party ATS such as Greenhouse, Lever, Workday, - SmartRecruiters, SAP SuccessFactors, or another — this varies per company and changes over - time, so it must be checked, not assumed). -2. If they use a known ATS with a public job-board API (see §3's "Public - applicant-tracking-system (ATS) job-board endpoints" category), the existing generic adapter - for that ATS vendor likely covers them with just a configuration entry, not new code. -3. If they run a custom career page, read that page's `robots.txt` and terms of use, and treat it - under §3's "Direct company career pages" category — including its higher maintenance and - reliability cost. -4. Record the outcome in the source registry below, with `enabled = false` until the adapter is - actually built and manually verified. - -Large multinational engineering employers (Airbus, Siemens, Bosch, Infineon, NXP, Texas -Instruments, Mercedes-Benz) are more likely to run on a standard ATS with a public API; German -public research/space organizations (DLR, Fraunhofer) and mid-size specialists (OHB, Rohde & -Schwarz) may be more likely to run custom career pages requiring per-site verification — but this -is a planning-time expectation to verify, not a confirmed fact about any of them. - -## 7. Source registry (to be filled in during implementation) +## 5. Adding a new source + +See [INGESTION_ARCHITECTURE.md](INGESTION_ARCHITECTURE.md) §10 for the full, binding +source-onboarding checklist. In summary: read terms and `robots.txt`, determine intended audience, +record authentication/rate-limit/attribution requirements, define allowed endpoints/identifiers, +add synthetic fixtures and tests with no live-CI dependency, record a dated compliance entry in this +document, add the source with `enabled = false`, and receive explicit repository-owner approval +before enabling any real configuration. + +## 6. Target-employer watchlist — resolved by this pass + +The prior watchlist (Airbus, Siemens, NXP, Texas Instruments, OHB, Rohde & Schwarz, Bosch, Infineon, +Mercedes-Benz, DLR, Fraunhofer institutes, and other Bremen/Hamburg-area employers) has been +investigated; see the company career-sites table in §3 above for the per-employer finding and +decision. None of them are implemented as adapters yet — Bosch and Continental are the +highest-value target once the SmartRecruiters clarification (§3) is resolved; every other watchlist +employer is manual-import-only pending its own custom-site terms review if one is ever undertaken. + +## 7. Source registry (implementation-time; none configured yet) | Source | Category | Legal basis confirmed | Rate limit | Status | |---|---|---|---|---| -| *(none configured yet, including none of the watchlist employers above — this is a planning-phase placeholder table, not a claim that any adapter exists or that any source's compliance has been verified)* | | | | | +| *(none configured yet — this table is populated at Phase 2A.1 implementation time, one row per registered `sources` entry, `enabled = false` until manually verified)* | | | | | diff --git a/docs/DEPLOYMENT_STRATEGY.md b/docs/DEPLOYMENT_STRATEGY.md index 832d3e3..839601f 100644 --- a/docs/DEPLOYMENT_STRATEGY.md +++ b/docs/DEPLOYMENT_STRATEGY.md @@ -76,38 +76,60 @@ onboarded before a broader public launch. - RLS policies are part of the migration files, not a separately-managed dashboard setting, for the same reproducibility reason. -## 4. Scheduled compute deployment - -- GitHub Actions workflows in `.github/workflows/` define the ingestion, resume-parsing, and - scoring jobs on cron schedules (see cadences in - [DATA_FLOW.md](DATA_FLOW.md#5-data-freshness-expectations-proposal)). **GitHub Actions cron is a - best-effort scheduler, not a reliable interactive execution system** — see - [ARCHITECTURE.md](ARCHITECTURE.md#7-github-actions-reliability-characteristics-read-before-relying-on-cadence) +## 4. Ingestion/scoring compute deployment + +**Status (Phase 2A.0)**: this section previously assumed GitHub Actions cron would run scheduled +Python ingestion/parsing/scoring jobs. [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) +(TypeScript) and [ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution) +supersede that assumption for ingestion and scoring. **No GitHub Actions workflow runs ingestion or +scoring**, today or in the currently planned near-term design. + +**Current design (Phase 2A)**: ingestion and scoring run as an explicit local CLI, invoked by the +repository owner on demand against the local (or, once one exists, a real hosted) Supabase stack: + +``` +cd ingestion && node --env-file=.env.local dist/cli.js ingest --source greenhouse: [--dry-run] +cd ingestion && node --env-file=.env.local dist/cli.js score --all +``` + +- The service-role key lives in a gitignored `ingestion/.env.local`, never a GitHub Actions + repository secret, never committed, and never held by any hosted compute at this stage. +- Every run must still be idempotent and safely re-runnable, since the owner may re-run after a + failure or simply run more or less often than any target cadence — nothing here assumes any + particular run frequency. +- `.github/workflows/ci.yml` remains the only GitHub Actions workflow in this repository, and it + never performs ingestion or holds the service-role key — it runs `frontend`/`database` jobs (and, + from Phase 2A.1, a third `ingestion` job: install, lint, typecheck, test, no live network call). + +**Later, explicitly-triggered migration path** (not provisioned now — see +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) for the full trigger conditions): a hosted +Supabase Edge Function on a schedule, then a Cloudflare Worker + Cron Trigger if Edge Function +limits are hit, with GitHub Actions considered only as a last resort and only with a dedicated +least-privilege database role — never the full service-role key. If and when a scheduled stage is +adopted, the historical cron-hygiene guidance below becomes relevant again: + +- **GitHub Actions cron is a best-effort scheduler, not a reliable interactive execution system** — + see [ARCHITECTURE.md](ARCHITECTURE.md#7-github-actions-reliability-characteristics-read-before-relying-on-cadence) for the full set of consequences this has for job design. - **Cron expressions must avoid the top of the hour** (e.g., `7,22,37,52 * * * *`, not `0,15,30,45 * * * *`), since GitHub documents that `:00`-aligned schedules see the most congestion-related delay. -- Each workflow installs Python dependencies (pinned via a lockfile), runs the relevant module, - and exits — no persistent process. Every workflow must be idempotent and safely re-runnable, - since a late or dropped scheduled run may need to be caught up by the next run or a manual one - without duplicating data. -- Secrets (Supabase service-role key, any source API keys) are GitHub Actions repository secrets, - injected as environment variables only inside the job run. -- A manual `workflow_dispatch` trigger should exist alongside the cron schedule for every job, so - ingestion or scoring can be re-run on demand while debugging. **This is a developer control, - invoked with repository write access via the GitHub UI or `gh` CLI — it is not, and must never - become, a button exposed in the frontend that the browser triggers using a stored GitHub token.** - Embedding any token with workflow-trigger permission in browser-reachable code is a secret- - exposure risk this project does not accept. A genuine user-facing "process this now" action - belongs to the synchronous Edge Function/worker layer reserved in - [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md), introduced only once actually needed - — not to GitHub Actions. +- A manual `workflow_dispatch` trigger should exist alongside any future cron schedule, so a run can + be re-triggered on demand while debugging. **This is a developer control, invoked with repository + write access via the GitHub UI or `gh` CLI — it is not, and must never become, a button exposed in + the frontend that the browser triggers using a stored GitHub token.** Embedding any token with + workflow-trigger permission in browser-reachable code is a secret-exposure risk this project does + not accept. A genuine user-facing "process this now" action belongs to the synchronous Edge + Function/worker layer reserved in [ADR-002](adr/ADR-002-frontend-and-backend-boundaries.md), + introduced only once actually needed — not to GitHub Actions. - **The repository is public**, so standard GitHub-hosted runner minutes for public repositories are free (no monthly minute quota to track, unlike a private repository on the free plan) — see - [COST_MODEL.md](COST_MODEL.md). Larger/specialized runners and some storage or advanced features - can still be billable even on a public repository; the ingestion cadence should still be - reviewed against actual usage once Phase 2's scheduled jobs exist, but not because of a private- - repo minute quota. + [COST_MODEL.md](COST_MODEL.md). This currently applies only to CI (`ci.yml`), since ingestion runs + locally, not on any GitHub-hosted runner. + +**Resume parsing** (Phase 1B, deferred) is unaffected by ADR-016/ADR-017: its language and execution +model remain a separate, undecided question for that phase's own design pass (see +[OPEN_QUESTIONS.md](OPEN_QUESTIONS.md)). ## 5. Release process (proposal) diff --git a/docs/DEVELOPMENT_ROADMAP.md b/docs/DEVELOPMENT_ROADMAP.md index 11c49d2..0f91b6b 100644 --- a/docs/DEVELOPMENT_ROADMAP.md +++ b/docs/DEVELOPMENT_ROADMAP.md @@ -65,44 +65,82 @@ unimplemented checks are shown as product-unavailable rather than user omissions **Exit condition**: a resume produces a truthful, reviewable draft without altering confirmed records unless the user explicitly accepts a change. -## Phase 2 — First ingestion pipeline, unscored - -- Source-adapter interface implemented ([ADR-004](adr/ADR-004-source-adapter-architecture.md)), - producing `source_listings`, not opportunities directly. -- The identity/deduplication step ([ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)) - is built alongside the first adapter, even though a single adapter alone can't yet produce a - cross-source duplicate — it's the shared pipeline every subsequent adapter plugs into, and - retrofitting it after a second adapter exists would mean re-processing already-ingested data. -- One or two ATS (Greenhouse/Lever) adapters built first, since they offer the best - coverage-to-effort ratio (see [DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md)). -- Normalization into the complete common schema (see - [DATA_MODEL.md](DATA_MODEL.md#opportunity_versions--complete-field-list)), full version history, - `removed` detection at the listing level. -- A plain, unranked opportunity browse/search view (Postgres full-text search) in the frontend. - -**Exit condition**: real German ECE-relevant postings are visible and searchable, with working -version history, before any scoring exists. - -## Phase 3 — Explainable scoring +## Phase 2A — Opportunity discovery MVP + +**Status**: architecture and source research complete (Phase 2A.0); no schema, adapter, or UI code +implemented yet. Runtime and execution-model decisions are recorded in +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript, superseding +[ADR-003](adr/ADR-003-python-ingestion-pipeline.md) for this scope), +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-CLI execution first), and +[ADR-018](adr/ADR-018-private-manual-opportunities.md) (manual entries are private, owned rows). +Source compliance findings are recorded in +[DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md); the adapter/orchestrator design is +in [INGESTION_ARCHITECTURE.md](INGESTION_ARCHITECTURE.md). + +**Sequencing note**: the original Phase 2A.0 plan split "opportunity foundation" (schema, RLS, +empty UI) and "Greenhouse adapter" into two separate PRs. Per the personal-first priority in +[MVP_SCOPE.md](MVP_SCOPE.md#current-product-mode-and-engineering-priority), those are combined +below into one vertical milestone — a feature is not useful merely because tables and empty screens +exist; the owner needs to actually browse and act on real postings before the next slice starts. + +- **Phase 2A.0 — Architecture and compliance** (this PR): ADRs, source-compliance registry, + ingestion architecture, shared-vs-private opportunity data decision, implementation boundaries. + No runtime implementation. +- **Phase 2A.1 — Personal Opportunity MVP** (the first minimum-useful milestone; one vertical PR or + tightly coordinated implementation branch): shared opportunity identity/version schema + ([DATA_MODEL.md](DATA_MODEL.md)), the `private_opportunities` table + ([ADR-018](adr/ADR-018-private-manual-opportunities.md)), the ingestion-source registry, exact + RLS/grants ([RLS_POLICY_MATRIX.md](RLS_POLICY_MATRIX.md)); the TypeScript local ingestion CLI + ([ADR-016](adr/ADR-016-typescript-ingestion-runtime.md), + [ADR-017](adr/ADR-017-local-first-ingestion-execution.md)) implementing the source-adapter + interface ([ADR-004](adr/ADR-004-source-adapter-architecture.md)) for Greenhouse first, since it + offers unauthenticated, robots-permitted, complete-response public access with the strongest + observed timestamp coverage (see [DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md)) + against a small, reviewed set of configured boards; the identity/deduplication step + ([ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md)) built alongside + this first adapter, even though one adapter alone can't yet produce a cross-source duplicate — + it's the shared pipeline every subsequent adapter plugs into; normalization into the complete + common schema (see [DATA_MODEL.md](DATA_MODEL.md#opportunity_versions--complete-field-list)), full + version history, conservative closure detection; manual metadata entry for unsupported sources; + opportunity list/detail UI, search and essential filters (Postgres full-text search), save/hide; + and application tracking end to end — status pipeline, pinned listing snapshots (shared and + private, see [DATA_MODEL.md](DATA_MODEL.md#user-activity-domain-user-owned-rls-protected)), and + interview-preparation notes. No live third-party call in CI. +- **Phase 2A.2 — Explainable ECE market relevance**: deterministic taxonomy and rule-based scoring + (Ranking 1 only — profile-independent; personal competitiveness/Ranking 2 remains Phase 3), a + ranking UI, synthetic evaluation vectors. No acceptance probability, per + [SCORING_AND_MATCHING_MODEL.md](SCORING_AND_MATCHING_MODEL.md). + +**Exit condition (met at Phase 2A.1)**: the owner can run one local command, import real Greenhouse +opportunities, browse and search them in CareerOS alongside any manually entered private +opportunities, save a listing, mark it applied, update its status, and later inspect the exact +pinned listing content used for interview preparation — before any relevance ranking exists. +Phase 2A.2 makes the list explainably ranked rather than merely searchable. + +## Phase 3 — Personal-competitiveness scoring + +**Note**: Ranking 1 (profile-independent market/degree relevance) moved earlier, into +**Phase 2A.2**, since it needs no profile data and makes Phase 2A's opportunity list explainably +ranked rather than merely searchable. Phase 3 covers what still requires profile domains that don't +exist until Phase 1A is fully built out. - Hard eligibility rules. -- Ranking 1 (market/degree relevance) and Ranking 2 (personal competitiveness), with the - `evidence_jsonb` explainability payload and the "My Fit" view from - [USER_WORKFLOWS.md](USER_WORKFLOWS.md). +- Ranking 2 (personal competitiveness), with the `evidence_jsonb` explainability payload and the + "My Fit" view from [USER_WORKFLOWS.md](USER_WORKFLOWS.md). - Golden-fixture regression tests for scoring (see [TESTING_STRATEGY.md](TESTING_STRATEGY.md)). **Exit condition**: the user can look at a real posting and get a trustworthy, evidence-linked explanation of their fit. -## Phase 4 — Save, track, act +## Phase 4 — Save, track, act — **absorbed into Phase 2A.1** -- Saved opportunities with pinned snapshots. -- Application pipeline (status tracking). -- Manual/suggested tasks and interview-prep notes. - -**Exit condition**: Workflows 4 and 6 in [USER_WORKFLOWS.md](USER_WORKFLOWS.md) work end to end — -this is the point at which the MVP definition of done in [MVP_SCOPE.md](MVP_SCOPE.md) is mostly -satisfied for the career-opportunity half of the product. +Saved opportunities with pinned snapshots, the application pipeline, and interview-prep notes are +no longer a separate later phase — per the personal-first priority in +[MVP_SCOPE.md](MVP_SCOPE.md#current-product-mode-and-engineering-priority), they are part of Phase +2A.1's exit condition directly, since a searchable-but-untrackable opportunity list is not yet a +genuinely useful product. This phase number is kept (not renumbered away) so existing references +and history stay traceable; there is no separate Phase 4 scope remaining. Workflows 4 and 6 in +[USER_WORKFLOWS.md](USER_WORKFLOWS.md) now work end to end at Phase 2A.1. ## Phase 5 — Development opportunities @@ -140,7 +178,8 @@ Candidates, not commitments, roughly in likely priority order: ## Explicit sequencing rules -- Scoring (Phase 3) never starts before ingestion (Phase 2) produces real data — scoring logic +- Ranking 1 (Phase 2A.2) never starts before ingestion (Phase 2A.1) produces real data, and + personal-competitiveness scoring (Phase 3) never starts before Ranking 1 exists — scoring logic built against imagined data tends to fit reality poorly. - No LLM-based feature is built before the deterministic scoring pipeline exists and passes its regression tests, per [ADR-005](adr/ADR-005-explainable-ranking-before-llm-ranking.md). diff --git a/docs/INGESTION_ARCHITECTURE.md b/docs/INGESTION_ARCHITECTURE.md new file mode 100644 index 0000000..36023e5 --- /dev/null +++ b/docs/INGESTION_ARCHITECTURE.md @@ -0,0 +1,252 @@ +# Ingestion Architecture + +Status: **proposal** — the architecture new source adapters and the local ingestion CLI must +follow, per [ADR-004](adr/ADR-004-source-adapter-architecture.md) (adapter interface), +[ADR-016](adr/ADR-016-typescript-ingestion-runtime.md) (TypeScript runtime), +[ADR-017](adr/ADR-017-local-first-ingestion-execution.md) (local-first execution), and +[ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md) (identity/dedup). No +code exists yet — this document governs Phase 2A.1's implementation, not a description of +something already built. + +## 1. What exists today + +Nothing. `ingestion/` contains only a placeholder `README.md`. This document, together with the +ADRs above and [DATA_SOURCES_AND_COMPLIANCE.md](DATA_SOURCES_AND_COMPLIANCE.md), is the design that +Phase 2A.1 (schema, first adapter, and local CLI, shipped together — see +[DEVELOPMENT_ROADMAP.md](DEVELOPMENT_ROADMAP.md)) implements against. + +## 2. Repository shape + +`ingestion/` becomes a second, independent Node/TypeScript project — its own `package.json`, +`package-lock.json`, `tsconfig.json` — pinned to the same Node version as `app/` (`.nvmrc`). No npm +workspace and no package shared by import between `app/` and `ingestion/` at this stage; the +Postgres schema is the contract between them (see [ADR-016](adr/ADR-016-typescript-ingestion-runtime.md)). + +``` +ingestion/ + src/ + adapters/ one module per adapter (e.g. greenhouse.ts) + orchestrator.ts shared fetch -> normalize -> identify -> upsert -> closure pipeline + sanitize.ts HTML sanitization (sanitize-html) + plain-text derivation + identity.ts URL normalization, content hashing + cli.ts entry point: `ingest`, `score` subcommands + test/ + fixtures/ synthetic, public-safe recorded-shape fixtures (never real raw responses) + package.json + tsconfig.json +``` + +## 3. Source-adapter interface (per ADR-004) + +```ts +interface SourceAdapter { + readonly adapterKey: string // e.g. 'greenhouse' + readonly adapterVersion: string // e.g. 'greenhouse/1.0.0' + readonly politeness: { + minIntervalMs: number // e.g. >= 1000ms between requests to one source + maxRequestsPerRun: number + userAgent: string // honest, non-spoofed, includes a contact URL + timeoutMs: number // e.g. 15_000 + maxResponseBytes: number // e.g. 8 MiB + } + + validateConfig(config: unknown): AdapterConfig // throws on unknown/extra keys + fetchPage(cfg: AdapterConfig, cursor?: string): Promise<{ + records: unknown[] + nextCursor?: string + pageComplete: boolean // did this page represent the source's full result set? + }> + parse(raw: unknown): Raw // hand-written type guard; throws on shape mismatch + normalize(raw: Raw, cfg: AdapterConfig): NormalizedOpportunityVersion // sanitizes, classifies + identity(raw: Raw, cfg: AdapterConfig): { + sourceExternalId: string + canonicalUrl: string + applicationUrl: string | null + } + provenance(cfg: AdapterConfig): { sourceKey: string; baseUrl: string } +} +``` + +An adapter's job is narrow, per [ADR-004](adr/ADR-004-source-adapter-architecture.md): turn one +source's data into the common schema. It never decides whether a listing is a new canonical +opportunity, a duplicate, or a match for an existing one — that is the identity/dedup step below, +which runs once, generically, after every adapter. + +**Schema validation without a new dependency**: `parse()` is a hand-written type-guard function per +adapter, not a `zod` schema. One adapter with a modest field count does not justify a runtime +validation dependency, per [AGENTS.md](../AGENTS.md)'s "don't add dependencies casually." Adopt +`zod` only once a third adapter, or the first adapter with genuinely nested/branching response +shapes, makes hand-written guards unwieldy. + +## 4. Orchestrator responsibilities (shared, written once) + +The orchestrator — not any individual adapter — owns: + +- Retry with exponential backoff **only on 429/5xx/network errors**, never on 4xx, honoring a + `Retry-After` header when present. +- Enforcing each source's configured `politeness` interval and per-run request cap. +- Conditional requests (`If-None-Match`) where a source returns an `ETag`. +- The sanitization gate (see §7) before anything reaches normalization output. +- Content hashing and idempotent upsert, keyed on `(source_id, source_external_id)`. +- The identity/deduplication decision (§5). +- Version diffing against the current version (§6). +- Closure/freshness evaluation (§8). +- Writing one `ingestion_runs` row per invocation, with `completeness` set from whether every + configured page was fetched successfully. +- Per-record failure isolation: one malformed record increments `error_count` and is skipped; it + never fails the whole run. +- `--dry-run`: runs every step, including normalization, sanitization, and hashing, but wraps all + writes in a transaction that is rolled back at the end — nothing persists. + +**Logging discipline** (per [SECURITY_AND_PRIVACY.md](SECURITY_AND_PRIVACY.md) §9): IDs, counts, +timings, and error *types*/messages only. Never description text, never a full raw payload. +`ingestion_runs.error_summary` is capped (proposal: ≤ 32 entries, ≤ 8 KB total) and truncated +messages are marked as such, never silently cut. + +## 5. Identity and deduplication (per ADR-010) + +Applied once, generically, to every new or changed listing from every adapter: + +1. **Already linked to a canonical opportunity?** Check whether captured content changed + (`content_hash`); if so, insert a new `opportunity_version`; if not, only touch + `source_listings.last_seen_at`. +2. **New, unlinked listing — strong deterministic match** (exact shared identifier scoped to the + same employer, or byte-identical normalized application URL): auto-link to the existing + canonical opportunity; insert a new version under it. +3. **Weak/uncertain match** (same organization, similar title, no exact identifier): **never + auto-merge** — create a new, independent opportunity and a `potential_duplicate_links` row + (`status = pending_review`). +4. **No match signal**: create a new opportunity and its first version. + +No fuzzy/similarity-based automatic merging is implemented at any point in Phase 2A — a false merge +silently combines two postings' histories, which is worse than a visible, manually-resolvable +duplicate (see [ADR-010](adr/ADR-010-opportunity-identity-and-cross-source-deduplication.md) for the +full reasoning). The review-queue UI for confirming/rejecting a `pending_review` pair is deferred +past Phase 2A.1 — per [DEVELOPMENT_ROADMAP.md](DEVELOPMENT_ROADMAP.md), a second adapter that could +plausibly cover an already-covered employer is not enabled until that queue exists and works. + +**URL normalization**: lowercase scheme+host, strip default port/fragment/userinfo, collapse +duplicate slashes, strip one trailing slash. Tracking-parameter handling is **per-adapter, via a +significance allowlist**, not a global blocklist — a query parameter that looks like tracking on one +source can be load-bearing identity on another (e.g., a job-ID query parameter). Each adapter +declares which parameters are significant; everything else is dropped and the remainder sorted for +stable comparison. + +## 6. Versioning (per ADR-006) + +Every mutable, historically-meaningful field lives on `opportunity_versions`, never on the stable +`opportunities` row (see [DATA_MODEL.md](DATA_MODEL.md)). A content change produces a new version +row; nothing is ever updated or deleted in place, at any role's privilege level — this is enforced +at the grant level (see [RLS_POLICY_MATRIX.md](RLS_POLICY_MATRIX.md)), not merely by adapter +discipline. + +## 7. Sanitization (per SECURITY_AND_PRIVACY.md §6) + +- HTML entities are decoded once (some sources double-encode), then sanitized with an **allowlist** + (`sanitize-html`, npm — added under `ingestion/` only, justified per + [AGENTS.md](../AGENTS.md) because hand-rolled HTML sanitization is a well-known way to ship an + XSS bug): `p, br, ul, ol, li, strong, em, b, i, h3, h4, h5, h6, a[href]`. `a[href]` accepts only + `https:`/`mailto:`; every other scheme (including `javascript:`, `data:`) drops the link, keeping + its text. +- Unconditionally removed: `