From 4b3a00e97d3614d2886b4881788c6fe640bb76f0 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 18:49:18 +0900 Subject: [PATCH 01/13] docs: cloudfront insight (visit stats) design spec (#43) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design for the first vertical slice of `hostdoc insight `: cloudfront mode end-to-end (v2 delivery-pipeline logging → dedicated log bucket with 30-day expiry → CLI-side gzip/W3C parse → per-code hits + approx unique visitors). Records the v1-vs-v2 decision (v2 chosen: ACL-free bucket-policy delivery fits hostdoc's hardened private OAC buckets) and the metric scope. Refs #43 --- .../2026-07-11-cloudfront-insight-design.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-11-cloudfront-insight-design.md diff --git a/docs/superpowers/specs/2026-07-11-cloudfront-insight-design.md b/docs/superpowers/specs/2026-07-11-cloudfront-insight-design.md new file mode 100644 index 0000000..1f7755b --- /dev/null +++ b/docs/superpowers/specs/2026-07-11-cloudfront-insight-design.md @@ -0,0 +1,117 @@ +# 발행 페이지 인사이트 — cloudfront 슬라이스 (`hostdoc insight `) + +- **이슈**: [#43](https://github.com/jkas2016/hostdoc/issues/43) — 발행 페이지 방문 통계, 접근 로그 기반, 세 모드 대칭 +- **날짜**: 2026-07-11 +- **모드 영향**: 이번 스펙은 **cloudfront 모드만** (수직 슬라이스 = MVP). s3-website·self-hosted는 각자 후속 스펙 +- **선행 결정**: 서버사이드 접근 로그 방식(A) 확정 — JS 비콘(B)은 HTML 원본 훼손 + 서버 필요라 hostdoc 철학 위반(이슈에서 배제) + +## 범위 (이 스펙) + +첫 슬라이스는 **cloudfront 한 모드를 로깅 활성화부터 조회까지 끝단**으로 완성한다. Acceptance criteria의 "최소 cloudfront에서 문서별 히트 수 표 출력"과 일치. + +- **지표**: 문서별 총 **히트 수** + **근사 순방문자**(고유 client IP) + 봇/프리페치 필터. (referer·지역·시간대 대시보드는 Non-goal) +- **조회 계층**: **CLI 직접 파싱**만 (추가 인프라 0). Athena는 후순위 별도 이슈. + +## 결정 사항 (브레인스토밍 확정) + +### D1. CloudFront 로깅은 표준 로그 **v2 (delivery 파이프라인)** 로 확정 + +v1 legacy(`logging_config`)와 v2(CloudWatch vended-logs delivery)를 비교한 결과, **hostdoc 아키텍처에는 v2가 정답**이다. + +| | v1 legacy `logging_config` | **v2 delivery (채택)** | +|---|---|---| +| 로그 버킷 권한 | **S3 ACL 필수**(`awslogsdelivery`에 FULL_CONTROL) | **버킷 정책**(`delivery.logs.amazonaws.com`), ACL 불필요 | +| 필드 | 전체 고정 | **필드 선택 가능** | +| 파티셔닝 | 파일명 날짜만 | 파일명 날짜 + hive/경로 파티션 | +| Terraform | 배포 리소스 내 블록 1개 | 리소스 3종 + 버킷 정책 | +| 과금 | S3 스토리지만 | S3 배달 무료, 스토리지만 (Parquet만 예외) | + +**근거**: hostdoc cloudfront는 **private S3 + OAC + 하드닝 버킷**([#20](https://github.com/jkas2016/hostdoc/issues/20))이다. 모던 S3 기본값은 ACL 비활성(bucket-owner-enforced)인데 v1 legacy는 로그 버킷에 ACL을 요구해 이 보안 태세와 충돌한다. v2는 버킷 정책 기반이라 ACL 없이 동작하고, 필드 선택으로 필요한 필드만 남겨 로그 크기·PII를 줄인다. +공식 문서: [Configure standard logging (v2)](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logging.html), [legacy(v1)](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logging-legacy-s3.html). + +### D2. 전용 로그 버킷 + 30일 자동 만료 + +- 콘텐츠 버킷과 **분리된 전용 로그 버킷**을 Terraform으로 신규 생성(served content에 로그가 섞이지 않게). private, 버킷 정책으로 delivery 서비스 쓰기만 허용. +- S3 lifecycle로 **30일 후 자동 만료** — 비용 상한 + PII 최소화. insight 기본 스캔 범위와 자연히 일치(30일 지난 로그는 애초에 없음). + +### D3. 로깅 필드 최소셋 + +delivery `recordFields`로 다음만 선택: `date`, `time`, `c-ip`, `cs-uri-stem`, `sc-status`, `cs(User-Agent)`, `x-edge-result-type`. (referer·cookie 등 미로깅 → PII·크기 최소) + +## 로그 포맷 (검증됨) + +v2 w3c/S3 출력은 legacy와 동일한 **gzip + 탭 구분 W3C**: +- 파일: gzip(`.gz`), 파티션 경로 `...////
//...` +- 각 파일: 헤더 2줄(`#Version`, `#Fields`) + 탭 구분 레코드, 시간순 보장 안 됨 +- 특수문자 URL 인코딩(공백=`%20` 등) → 파싱 시 디코드 +- 필드는 `#Fields` 헤더로 위치를 **동적 매핑**(고정 인덱스 가정 금지 — 필드셋이 바뀌어도 안전) + +## 설계 + +### 변경 대상 + +**프로비저닝 (Terraform, 로컬 전용)** +- `infra/main.tf` — ① 전용 로그 버킷(private, 버킷 정책 = delivery 서비스 쓰기, 30일 lifecycle) ② `aws_cloudwatch_log_delivery_source`(distribution ARN, `log_type=ACCESS_LOGS`) ③ `aws_cloudwatch_log_delivery_destination`(로그 버킷, `output_format="w3c"`) ④ `aws_cloudwatch_log_delivery`(source↔destination, `record_fields`=최소셋, `s3_delivery_configuration` 파티션). delivery API는 us-east-1. +- `infra/outputs.tf` — `log_bucket_name` output 추가 +- `infra/variables.tf` — 필요 시 `log_retention_days` 변수(기본 30) + +**config / init** +- `src/lib/config.ts` — `Config`에 `logBucket?: string` 추가(cloudfront 전용). mode 파생 로직 무변경. +- `src/commands/init.ts` — `readTerraformOutputs`에서 `log_bucket_name`을 읽어 `cfg.logBucket`에 저장. **하위호환**: 기존(로그 없는) infra에는 이 output이 없으므로 **optional**로 처리(없으면 insight가 안내 메시지). + +**조회 계층 (신규)** +- `src/lib/cflog.ts` — 순수 파서. 입력: gzip 버퍼(또는 해제된 텍스트). `#Fields` 헤더로 컬럼 매핑 → 레코드 배열. URL 디코드. 네트워크·AWS 의존 없음(단위 테스트 용이). +- `src/lib/insight.ts` — 집계 로직(순수): 레코드 + `code` → `{ hits, uniqueVisitors, since, until }`. `cs-uri-stem`이 `//`로 시작 필터, `c-ip` set으로 순방문자 근사, User-Agent 봇 패턴 + `x-edge-result-type` 프리페치성 제외. +- `src/commands/insight.ts` — `runInsight({ code, days?, since?, profile? } & Overrides)`. `resolveConfig`로 cloudfront config 확인(아니면 명확한 에러). `logBucket` 없으면 "재프로비저닝 필요" 안내. 로그 버킷에서 날짜 파티션 prefix로 객체 나열(기본=전체≈30일, `--days`/`--since`로 축소) → 각 gz get → `cflog.parse` → `insight.aggregate` → 표 출력. +- `src/index.ts` — `insight ` 커맨드 등록(`withCommon` + `--days`/`--since`). +- 로그 버킷 읽기: 콘텐츠 버킷과 다른 버킷이므로 `S3Backend`(bucket 바인딩)를 **로그 버킷으로 별도 인스턴스화**하거나 소형 전용 리더 사용. 기존 발행/조회 backend와 분리 → `publish`/`list`/`rm` 무회귀. + +**문서** +- `README.md` + 문서 — `insight` 사용법, **프라이버시 고지**(방문자 IP 저장·순방문자 근사에만 사용·최소 필드), 소급 불가·지연(수 분~수 시간) 고지, 기존 사용자 `provision` 재실행 안내. + +### 출력 (예시) + +``` +$ hostdoc insight team/q1/report +team/q1/report — logging since 2026-07-11 (cloudfront) + hits 128 + unique visitors ~34 (approx, by client IP) + range 2026-07-11 → 2026-07-25 (14 log files scanned) + note CloudFront logs lag minutes–hours; bots/prefetch filtered. +``` + +### 기존 사용자 (소급) + +로깅 리소스는 신규다. 기존 cloudfront 사용자는 `hostdoc provision` 재실행으로 로그 버킷+delivery를 추가해야 하고, **활성화 이후 트래픽만** 잡힌다. insight 출력에 "logging since"를 명시하고, `logBucket` 미설정 시 재프로비저닝을 안내한다. + +## 테스트 (test-plan first) + +`aws-sdk-client-mock` + gzip 픽스처, **네트워크·Terraform·AWS creds 없이**: + +1. **`cflog.parse`** — 2줄 헤더 스킵, `#Fields` 동적 매핑, 탭 분리, URL 디코드, 빈/깨진 파일 graceful. +2. **`insight.aggregate`** — `//` prefix 필터(다른 code 배제), 히트 카운트, 고유 IP 순방문자, 봇/프리페치 제외, 중첩 slug(`team/q1/report`) 정확. +3. **`runInsight`** — S3 `ListObjectsV2`(날짜 파티션 prefix) + `GetObject`(gz 바디) 목킹 → 표 출력. cloudfront 아닌 config → 에러. `logBucket` 없음 → 재프로비저닝 안내. `--days`/`--since` 범위 축소가 나열 prefix에 반영. +4. **회귀** — `publish`/`list`/`rm` 무영향. CI(무 AWS·무 Terraform) 통과 유지. + +## 비목표 (이 스펙) + +- s3-website·self-hosted 로그(각자 후속 스펙 — 로그 소스·포맷이 전혀 다름). +- JS 비콘/HTML 주입, 실시간 대시보드, 서드파티 분석, Athena 리치 분석(referer/지역/시간대). +- 로그 파일 자체의 서드파티 반출. + +## 리스크 + +| Risk | Mitigation | +|---|---| +| 소급 불가 | `provision` 재실행 경로 + "logging since" 명시 | +| 로그 지연(수 분~수 시간) | 실시간 배제 Non-goal, 출력에 지연 고지 | +| 방문자 IP 저장(프라이버시) | 문서 고지, IP는 순방문자 근사에만, 최소 필드, 30일 만료 | +| 고트래픽 시 CLI 파싱 느림 | 날짜 파티션으로 스캔 축소, `--days`/`--since`, Athena 후순위 | +| 순방문자 정밀도 한계 | IP 근사임을 출력 명시, 봇/프리페치 필터 | +| 하위호환(기존 infra에 log output 없음) | `logBucket` optional, 미설정 시 안내 | + +## 참고 + +- 코드: `infra/main.tf`·`infra/outputs.tf`, `src/commands/init.ts`, `src/lib/config.ts`(`Config`), `src/lib/backend.ts`(`S3Backend`), `src/lib/url.ts`·`src/lib/meta.ts`(`/` prefix 모델) +- AWS 공식: [standard logging v2](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logging.html) · [legacy v1](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logging-legacy-s3.html) · [CloudFront pricing](https://aws.amazon.com/cloudfront/pricing/)(표준 로그 S3 배달 무료) +- 관련 이슈: [#41](https://github.com/jkas2016/hostdoc/issues/41)(self-hosted 모드), [#20](https://github.com/jkas2016/hostdoc/issues/20)(infra 하드닝) From 0b7bc30db1b0dd7b623bbb07f718ded55608bd64 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 18:56:06 +0900 Subject: [PATCH 02/13] docs: cloudfront insight implementation plan (#43) TDD, bite-sized 8-task plan for the cloudfront insight slice: pure W3C parser (T1) + aggregation/log-key selection (T2), getObjectBytes (T3), config.logBucket + init import (T4), runInsight command (T5), CLI registration (T6), v2 logging Terraform (T7), README privacy/usage (T8). Refs #43 --- .../plans/2026-07-11-cloudfront-insight.md | 951 ++++++++++++++++++ 1 file changed, 951 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-11-cloudfront-insight.md diff --git a/docs/superpowers/plans/2026-07-11-cloudfront-insight.md b/docs/superpowers/plans/2026-07-11-cloudfront-insight.md new file mode 100644 index 0000000..e9506fd --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-cloudfront-insight.md @@ -0,0 +1,951 @@ +# CloudFront Insight (Visit Stats) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add `hostdoc insight ` that reports per-document hits + approximate unique visitors for cloudfront-mode sites, from CloudFront standard-logging (v2) access logs the user provisions in their own AWS. + +**Architecture:** Terraform provisions a dedicated private log bucket (30-day expiry) and a CloudWatch vended-logs delivery pipeline (source→destination→delivery) that writes gzipped W3C access logs to it. The CLI lists those log objects, gunzips them, parses the tab-separated W3C records with a pure parser, filters to the document's `//` prefix, and aggregates hits + unique client IPs (bots/errors filtered). Parsing and aggregation are pure functions; only the command touches AWS. + +**Tech Stack:** TypeScript (ESM, `.js` import suffix in source), Node `node:zlib` gunzip, `@aws-sdk/client-s3`, Terraform (`hashicorp/aws >= 5.0`), vitest + `aws-sdk-client-mock`. + +## Global Constraints + +- ESM: relative imports in `.ts` source MUST use a `.js` extension (e.g. `./insight.js`). +- CI runs build→typecheck→test with **no AWS creds, no network, no Terraform** — every test must pass offline. Terraform is validated locally only. +- Build stays on `tsc` (`npm run build`); typecheck is `npm run typecheck` (tsgo, no emit). +- Tests use `aws-sdk-client-mock`; `test/setup-env.ts` repoints `XDG_CONFIG_HOME`; state is set via `HOSTDOC_*` env vars. +- Mode is **derived** in `resolveConfig` and never stored; do not add mode-derivation logic. `insight` is cloudfront-only for this slice. +- CloudFront standard logging **v2** (delivery pipeline), not v1 legacy. Log bucket is ACL-free (bucket-policy delivery). Delivery resources must be created in `us-east-1` (use the existing `aws.us_east_1` provider alias). +- Privacy: log only the minimal field set; client IP is used for unique-visitor approximation only; document the IP-retention notice. + +--- + +## File Structure + +- **Create** `src/lib/cflog.ts` — pure CloudFront W3C log parser (text → records). No I/O. +- **Create** `src/lib/insight.ts` — pure aggregation + log-key date selection. No I/O. +- **Create** `src/commands/insight.ts` — `runInsight` orchestration: resolve config, list/get/gunzip log objects, parse, aggregate, format. +- **Modify** `src/lib/aws.ts` — add `getObjectBytes` (raw bytes for gunzip). +- **Modify** `src/lib/config.ts` — add `logBucket?: string` to `Config`. +- **Modify** `src/commands/init.ts` — read optional `log_bucket_name` Terraform output into `cfg.logBucket`. +- **Modify** `src/index.ts` — register the `insight ` command. +- **Modify** `infra/main.tf`, `infra/outputs.tf`, `infra/variables.tf` — log bucket + delivery pipeline + output + retention variable. +- **Modify** `README.md` — usage + privacy notice. +- **Create** tests: `test/cflog.test.ts`, `test/insight.test.ts`, `test/insight-command.test.ts`; **modify** `test/aws` coverage via command test; **modify** `test/init.test.ts`. + +--- + +## Task 1: CloudFront W3C log parser (`src/lib/cflog.ts`) + +**Files:** +- Create: `src/lib/cflog.ts` +- Test: `test/cflog.test.ts` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `interface LogRecord { date: string; time: string; cIp: string; uriStem: string; status: string; userAgent: string; edgeResult: string; }` + - `function parseCloudFrontLog(text: string): LogRecord[]` + +Field mapping is **dynamic**: read the `#Fields:` header line and map by CloudFront field name (`date`, `time`, `c-ip`, `cs-uri-stem`, `sc-status`, `cs(User-Agent)`, `x-edge-result-type`). Never assume fixed column positions. Values are URL-decoded (CloudFront percent-encodes spaces/specials); a malformed escape falls back to the raw value. Lines starting with `#` and blank lines are skipped. No `#Fields` header → `[]`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/cflog.test.ts +import { describe, it, expect } from "vitest"; +import { parseCloudFrontLog } from "../src/lib/cflog.js"; + +const HEADER = + "#Version: 1.0\n" + + "#Fields: date time c-ip cs-uri-stem sc-status cs(User-Agent) x-edge-result-type\n"; + +describe("parseCloudFrontLog", () => { + it("maps fields by the #Fields header, not fixed positions", () => { + const log = + HEADER + + "2026-07-11\t21:34:06\t203.0.113.5\t/abc/index.html\t200\tMozilla%2F5.0\tHit\n"; + const recs = parseCloudFrontLog(log); + expect(recs).toHaveLength(1); + expect(recs[0]).toEqual({ + date: "2026-07-11", + time: "21:34:06", + cIp: "203.0.113.5", + uriStem: "/abc/index.html", + status: "200", + userAgent: "Mozilla/5.0", // %2F decoded + edgeResult: "Hit", + }); + }); + + it("skips comment and blank lines and returns [] without a #Fields header", () => { + expect(parseCloudFrontLog("#Version: 1.0\n\n")).toEqual([]); + expect(parseCloudFrontLog("")).toEqual([]); + }); + + it("tolerates a malformed percent-escape by keeping the raw value", () => { + const log = HEADER + "2026-07-11\t00:00:00\t1.1.1.1\t/x/%\t200\tUA\tMiss\n"; + expect(parseCloudFrontLog(log)[0].uriStem).toBe("/x/%"); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/cflog.test.ts` +Expected: FAIL — `parseCloudFrontLog` not found / module missing. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/lib/cflog.ts +export interface LogRecord { + date: string; + time: string; + cIp: string; + uriStem: string; + status: string; + userAgent: string; + edgeResult: string; +} + +// CloudFront field name -> LogRecord key. +const FIELD_MAP: Record = { + date: "date", + time: "time", + "c-ip": "cIp", + "cs-uri-stem": "uriStem", + "sc-status": "status", + "cs(User-Agent)": "userAgent", + "x-edge-result-type": "edgeResult", +}; + +function safeDecode(v: string): string { + if (v === "-" || v === "") return v === "-" ? "" : v; + try { + return decodeURIComponent(v); + } catch { + return v; + } +} + +/** Parse a decompressed CloudFront standard-log (W3C, tab-separated) file. */ +export function parseCloudFrontLog(text: string): LogRecord[] { + const lines = text.split("\n"); + let cols: (keyof LogRecord | null)[] | null = null; + const out: LogRecord[] = []; + + for (const line of lines) { + if (line.startsWith("#Fields:")) { + const names = line.slice("#Fields:".length).trim().split(/\s+/); + cols = names.map((n) => FIELD_MAP[n] ?? null); + continue; + } + if (line.startsWith("#") || line.trim() === "" || !cols) continue; + + const parts = line.split("\t"); + const rec: LogRecord = { + date: "", + time: "", + cIp: "", + uriStem: "", + status: "", + userAgent: "", + edgeResult: "", + }; + for (let i = 0; i < cols.length; i++) { + const key = cols[i]; + if (key) rec[key] = safeDecode(parts[i] ?? ""); + } + out.push(rec); + } + return out; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/cflog.test.ts` +Expected: PASS (3 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/cflog.ts test/cflog.test.ts +git commit -m "feat: pure CloudFront W3C access-log parser (#43)" +``` + +--- + +## Task 2: Insight aggregation + log-key selection (`src/lib/insight.ts`) + +**Files:** +- Create: `src/lib/insight.ts` +- Test: `test/insight.test.ts` + +**Interfaces:** +- Consumes: `LogRecord` from `./cflog.js`. +- Produces: + - `interface InsightResult { code: string; hits: number; uniqueVisitors: number; since: string | null; until: string | null; }` + - `function aggregateInsight(records: LogRecord[], code: string): InsightResult` + - `function cloudFrontLogKeyDate(key: string): string | null` — extracts `YYYY-MM-DD` from a partitioned log key (`...//YYYY/MM/DD/HH/...`), else `null`. + - `function filterLogKeys(keys: string[], opts: { since?: string }): string[]` — keeps `.gz` keys whose partition date is `>= since` (or all if no `since` / undatable). + - `function dateDaysAgo(days: number, now: Date): string` — `YYYY-MM-DD` for `now - days`. + +**Metric definition (refines the spec's "prefix filter" to be a page-view count, not asset noise):** a *hit* is a successful (`sc-status` 2xx/3xx) request from a non-bot User-Agent whose `cs-uri-stem` is under `//` **and** is a page view (ends in `.html` or `/`). Asset requests (`.css`, `.js`, images) are excluded so the count reflects page opens. *Unique visitors* = distinct `c-ip` among those hits. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/insight.test.ts +import { describe, it, expect } from "vitest"; +import { + aggregateInsight, + cloudFrontLogKeyDate, + filterLogKeys, + dateDaysAgo, +} from "../src/lib/insight.js"; +import type { LogRecord } from "../src/lib/cflog.js"; + +function rec(p: Partial): LogRecord { + return { + date: "2026-07-11", + time: "00:00:00", + cIp: "1.1.1.1", + uriStem: "/abc/index.html", + status: "200", + userAgent: "Mozilla/5.0", + edgeResult: "Hit", + ...p, + }; +} + +describe("aggregateInsight", () => { + it("counts page-view hits under the code prefix and approx unique IPs", () => { + const recs = [ + rec({ cIp: "1.1.1.1", uriStem: "/abc/index.html" }), + rec({ cIp: "2.2.2.2", uriStem: "/abc/index.html" }), + rec({ cIp: "1.1.1.1", uriStem: "/abc/index.html", date: "2026-07-12" }), + rec({ uriStem: "/abc/style.css" }), // asset -> excluded + rec({ uriStem: "/other/index.html" }), // different code -> excluded + ]; + const r = aggregateInsight(recs, "abc"); + expect(r.hits).toBe(3); + expect(r.uniqueVisitors).toBe(2); + expect(r.since).toBe("2026-07-11"); + expect(r.until).toBe("2026-07-12"); + }); + + it("excludes bots and non-2xx/3xx statuses", () => { + const recs = [ + rec({ userAgent: "Googlebot/2.1" }), + rec({ status: "404" }), + rec({ cIp: "9.9.9.9" }), + ]; + const r = aggregateInsight(recs, "abc"); + expect(r.hits).toBe(1); + expect(r.uniqueVisitors).toBe(1); + }); + + it("returns zeros and null range when nothing matches", () => { + expect(aggregateInsight([], "abc")).toEqual({ + code: "abc", + hits: 0, + uniqueVisitors: 0, + since: null, + until: null, + }); + }); + + it("counts nested-slug page views", () => { + const r = aggregateInsight( + [rec({ uriStem: "/team/q1/report/index.html" })], + "team/q1/report", + ); + expect(r.hits).toBe(1); + }); +}); + +describe("log key selection", () => { + it("extracts the partition date from a log key", () => { + expect( + cloudFrontLogKeyDate("E123ABC/2026/07/11/14/E123ABC.2026-07-11-14.abc.gz"), + ).toBe("2026-07-11"); + expect(cloudFrontLogKeyDate("no-date-here.gz")).toBeNull(); + }); + + it("filters to .gz keys on/after `since`", () => { + const keys = [ + "d/2026/07/10/00/a.gz", + "d/2026/07/11/00/b.gz", + "d/2026/07/12/00/c.gz", + "d/2026/07/12/00/note.txt", + ]; + expect(filterLogKeys(keys, { since: "2026-07-11" })).toEqual([ + "d/2026/07/11/00/b.gz", + "d/2026/07/12/00/c.gz", + ]); + expect(filterLogKeys(keys, {})).toEqual([ + "d/2026/07/10/00/a.gz", + "d/2026/07/11/00/b.gz", + "d/2026/07/12/00/c.gz", + ]); + }); + + it("computes a date N days before a fixed now", () => { + expect(dateDaysAgo(7, new Date("2026-07-11T00:00:00Z"))).toBe("2026-07-04"); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/insight.test.ts` +Expected: FAIL — module/exports missing. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/lib/insight.ts +import type { LogRecord } from "./cflog.js"; + +export interface InsightResult { + code: string; + hits: number; + uniqueVisitors: number; + since: string | null; + until: string | null; +} + +const BOT_RE = + /bot|crawl|spider|slurp|bingpreview|facebookexternalhit|headless|monitor|curl|wget|python-requests|pingdom|uptime/i; + +function isPageView(uriStem: string, prefix: string): boolean { + return ( + uriStem.startsWith(prefix) && + (uriStem.endsWith(".html") || uriStem.endsWith("/")) + ); +} + +/** Per-document page-view hits + approximate unique visitors (by client IP). */ +export function aggregateInsight( + records: LogRecord[], + code: string, +): InsightResult { + const prefix = `/${code}/`; + const ips = new Set(); + let hits = 0; + let since: string | null = null; + let until: string | null = null; + + for (const r of records) { + if (!/^[23]/.test(r.status)) continue; + if (BOT_RE.test(r.userAgent)) continue; + if (!isPageView(r.uriStem, prefix)) continue; + hits++; + if (r.cIp) ips.add(r.cIp); + if (since === null || r.date < since) since = r.date; + if (until === null || r.date > until) until = r.date; + } + + return { code, hits, uniqueVisitors: ips.size, since, until }; +} + +/** Extract YYYY-MM-DD from a partitioned CloudFront log key, else null. */ +export function cloudFrontLogKeyDate(key: string): string | null { + const m = key.match(/\/(\d{4})\/(\d{2})\/(\d{2})\//); + return m ? `${m[1]}-${m[2]}-${m[3]}` : null; +} + +/** Keep .gz log keys whose partition date is >= since (all if no since). */ +export function filterLogKeys( + keys: string[], + opts: { since?: string }, +): string[] { + return keys.filter((k) => { + if (!k.endsWith(".gz")) return false; + if (!opts.since) return true; + const d = cloudFrontLogKeyDate(k); + return d === null ? true : d >= opts.since; + }); +} + +/** YYYY-MM-DD for (now - days), UTC. */ +export function dateDaysAgo(days: number, now: Date): string { + const d = new Date(now.getTime() - days * 24 * 60 * 60 * 1000); + return d.toISOString().slice(0, 10); +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/insight.test.ts` +Expected: PASS (7 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/insight.ts test/insight.test.ts +git commit -m "feat: insight aggregation + log-key date selection (#43)" +``` + +--- + +## Task 3: Raw-bytes S3 getter (`src/lib/aws.ts`) + +**Files:** +- Modify: `src/lib/aws.ts` (add after `getJson`) +- Test: covered by Task 5's command test (mock returns gzipped bytes). + +**Interfaces:** +- Produces: `function getObjectBytes(s3: S3Client, bucket: string, key: string): Promise` + +- [ ] **Step 1: Add the implementation** + +```ts +// src/lib/aws.ts — append after getJson +export async function getObjectBytes( + s3: S3Client, + bucket: string, + key: string, +): Promise { + const res = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key })); + if (!res.Body) { + throw new Error(`Empty response body for ${key}`); + } + const bytes = await ( + res.Body as { transformToByteArray(): Promise } + ).transformToByteArray(); + return Buffer.from(bytes); +} +``` + +- [ ] **Step 2: Typecheck** + +Run: `npm run typecheck` +Expected: no errors. + +- [ ] **Step 3: Commit** + +```bash +git add src/lib/aws.ts +git commit -m "feat: getObjectBytes S3 helper for gzip log reads (#43)" +``` + +--- + +## Task 4: Config `logBucket` + init import (`src/lib/config.ts`, `src/commands/init.ts`) + +**Files:** +- Modify: `src/lib/config.ts` (add field to `Config`) +- Modify: `src/commands/init.ts` (read optional output) +- Test: `test/init.test.ts` + +**Interfaces:** +- Produces: `Config.logBucket?: string` (cloudfront-only, optional for backward compat). +- Consumes: `readTerraformOutputs` output map in `init.ts`. + +- [ ] **Step 1: Write the failing test** + +Inspect `test/init.test.ts` for how outputs are mocked (it stubs `readTerraformOutputs` or `execFileSync`). Add a test asserting `log_bucket_name` flows into config, and that its absence leaves `logBucket` undefined. Example (adapt to the file's existing mock style): + +```ts +// test/init.test.ts — new cases +it("imports log_bucket_name into the config when present", () => { + // Arrange the same output-mock mechanism the other init tests use, adding: + // log_bucket_name: { value: "my-logs" } alongside bucket_name/region/distribution_id/site_domain + const cfg = runInit({ dir: "/fake" }); + expect(cfg.logBucket).toBe("my-logs"); +}); + +it("leaves logBucket undefined when the output is absent (legacy infra)", () => { + // Mock outputs WITHOUT log_bucket_name. + const cfg = runInit({ dir: "/fake" }); + expect(cfg.logBucket).toBeUndefined(); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/init.test.ts` +Expected: FAIL — `logBucket` is undefined in the present case / property not set. + +- [ ] **Step 3: Implement** + +```ts +// src/lib/config.ts — add to interface Config, in the cloudfront group + distributionId?: string; // cloudfront + domain?: string; // cloudfront + logBucket?: string; // cloudfront access-log bucket (optional; enables `insight`) +``` + +```ts +// src/commands/init.ts — after building cfg, before saveConfig + const logBucket = outputs["log_bucket_name"]?.value; + if (typeof logBucket === "string" && logBucket) { + cfg.logBucket = logBucket; + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/init.test.ts` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/lib/config.ts src/commands/init.ts test/init.test.ts +git commit -m "feat: import log_bucket_name output into cloudfront config (#43)" +``` + +--- + +## Task 5: `runInsight` command (`src/commands/insight.ts`) + +**Files:** +- Create: `src/commands/insight.ts` +- Test: `test/insight-command.test.ts` + +**Interfaces:** +- Consumes: `resolveConfig`/`Overrides` (`../lib/config.js`), `isValidPath`/`isValidCode` (`../lib/code.js`), `makeS3`/`listKeys`/`getObjectBytes` (`../lib/aws.js`), `parseCloudFrontLog` (`../lib/cflog.js`), `aggregateInsight`/`filterLogKeys`/`dateDaysAgo` (`../lib/insight.js`), `gunzipSync` (`node:zlib`). +- Produces: + - `interface InsightArgs { code: string; days?: number; since?: string; profile?: string; }` + - `function runInsight(args: InsightArgs & Overrides): Promise` — returns the formatted table. + +Behavior: resolve config; require `mode === "cloudfront"` else throw a clear error; require `logBucket` else throw a re-provision hint; validate `code`; compute `since` = `args.since ?? (args.days ? dateDaysAgo(args.days, new Date()) : undefined)`; list log keys, `filterLogKeys`; get + gunzip + parse each; aggregate; format. A single unreadable/corrupt log object is skipped (a warning to stderr), never fatal. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/insight-command.test.ts +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mockClient } from "aws-sdk-client-mock"; +import { gzipSync } from "node:zlib"; +import { + S3Client, + ListObjectsV2Command, + GetObjectCommand, +} from "@aws-sdk/client-s3"; +import { runInsight } from "../src/commands/insight.js"; + +const s3mock = mockClient(S3Client); + +const LOG = + "#Version: 1.0\n" + + "#Fields: date time c-ip cs-uri-stem sc-status cs(User-Agent) x-edge-result-type\n" + + "2026-07-11\t00:00:00\t1.1.1.1\t/abc/index.html\t200\tMozilla%2F5.0\tHit\n" + + "2026-07-11\t00:00:01\t2.2.2.2\t/abc/index.html\t200\tMozilla%2F5.0\tHit\n" + + "2026-07-11\t00:00:02\t9.9.9.9\t/abc/style.css\t200\tMozilla%2F5.0\tHit\n"; + +function gzBody() { + const buf = gzipSync(Buffer.from(LOG)); + return { transformToByteArray: async () => new Uint8Array(buf) } as any; +} + +beforeEach(() => { + s3mock.reset(); + process.env.HOSTDOC_DOMAIN = "shared.example.com"; + process.env.HOSTDOC_DISTRIBUTION = "DIST1"; + process.env.HOSTDOC_BUCKET = "content-bucket"; + process.env.HOSTDOC_REGION = "us-east-1"; + process.env.HOSTDOC_LOG_BUCKET = "log-bucket"; +}); +afterEach(() => { + for (const k of [ + "HOSTDOC_DOMAIN", + "HOSTDOC_DISTRIBUTION", + "HOSTDOC_BUCKET", + "HOSTDOC_REGION", + "HOSTDOC_LOG_BUCKET", + ]) + delete process.env[k]; +}); + +describe("runInsight", () => { + it("aggregates hits + unique visitors from gzipped logs", async () => { + s3mock + .on(ListObjectsV2Command) + .resolves({ Contents: [{ Key: "DIST1/2026/07/11/00/a.gz" }], IsTruncated: false }); + s3mock.on(GetObjectCommand).resolves({ Body: gzBody() }); + + const out = await runInsight({ code: "abc" }); + expect(out).toMatch(/hits\s+2/); + expect(out).toMatch(/unique visitors\s+~2/); + expect(out).toMatch(/abc/); + }); + + it("errors clearly when the resolved mode is not cloudfront", async () => { + delete process.env.HOSTDOC_DOMAIN; + delete process.env.HOSTDOC_DISTRIBUTION; + await expect(runInsight({ code: "abc" })).rejects.toThrow(/cloudfront/i); + }); + + it("hints re-provision when no log bucket is configured", async () => { + delete process.env.HOSTDOC_LOG_BUCKET; + await expect(runInsight({ code: "abc" })).rejects.toThrow(/provision/i); + }); +}); +``` + +Note: `HOSTDOC_LOG_BUCKET` is used only to drive this test through env. Wire it in Task 6 via `resolveConfig`? No — `logBucket` is not an `Overrides` field. Instead, read it in `runInsight` as `args`-independent config: resolve `logBucket` from `cfg.logBucket` OR `process.env.HOSTDOC_LOG_BUCKET` (env escape hatch for testing/BYO). Implement exactly as below. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/insight-command.test.ts` +Expected: FAIL — module missing. + +- [ ] **Step 3: Implement** + +```ts +// src/commands/insight.ts +import { gunzipSync } from "node:zlib"; +import { resolveConfig, type Overrides } from "../lib/config.js"; +import { isValidPath, isValidCode } from "../lib/code.js"; +import { makeS3, listKeys, getObjectBytes } from "../lib/aws.js"; +import { parseCloudFrontLog, type LogRecord } from "../lib/cflog.js"; +import { + aggregateInsight, + filterLogKeys, + dateDaysAgo, + type InsightResult, +} from "../lib/insight.js"; + +export interface InsightArgs { + code: string; + days?: number; + since?: string; + profile?: string; +} + +function formatInsight(r: InsightResult, filesScanned: number): string { + const range = + r.since && r.until ? `${r.since} → ${r.until}` : "no matching requests yet"; + return [ + `${r.code} — cloudfront`, + ` hits ${r.hits}`, + ` unique visitors ~${r.uniqueVisitors} (approx, by client IP)`, + ` range ${range} (${filesScanned} log file(s) scanned)`, + ` note CloudFront logs lag minutes–hours; bots filtered; unique is an IP approximation.`, + ].join("\n"); +} + +export async function runInsight(args: InsightArgs & Overrides): Promise { + const cfg = resolveConfig(args); + if (cfg.mode !== "cloudfront") { + throw new Error( + `insight currently supports cloudfront mode only (resolved mode: ${cfg.mode}).`, + ); + } + const logBucket = cfg.logBucket ?? process.env.HOSTDOC_LOG_BUCKET; + if (!logBucket) { + throw new Error( + "No access-log bucket configured. Re-run `hostdoc provision` to enable " + + "CloudFront access logging, then `hostdoc init --from-terraform `.", + ); + } + if (!isValidPath(args.code) && !isValidCode(args.code)) { + throw new Error(`Invalid code: ${args.code}`); + } + + const since = + args.since ?? (args.days ? dateDaysAgo(args.days, new Date()) : undefined); + + const s3 = makeS3({ region: cfg.region, profile: args.profile }); + const allKeys = await listKeys(s3, logBucket, ""); + const keys = filterLogKeys(allKeys, { since }); + + const records: LogRecord[] = []; + for (const key of keys) { + try { + const gz = await getObjectBytes(s3, logBucket, key); + records.push(...parseCloudFrontLog(gunzipSync(gz).toString("utf8"))); + } catch (err) { + process.stderr.write( + `hostdoc: skipping unreadable log ${key}: ${(err as Error).message}\n`, + ); + } + } + + return formatInsight(aggregateInsight(records, args.code), keys.length); +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/insight-command.test.ts` +Expected: PASS (3 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/commands/insight.ts test/insight-command.test.ts +git commit -m "feat: runInsight command reads+aggregates CloudFront logs (#43)" +``` + +--- + +## Task 6: Register `insight` in the CLI (`src/index.ts`) + +**Files:** +- Modify: `src/index.ts` (import + command registration) +- Test: `test/smoke.test.ts` (extend the help/commands assertion if present) or add a minimal registration check. + +**Interfaces:** +- Consumes: `runInsight` from `./commands/insight.js`. + +- [ ] **Step 1: Add the import and command** + +```ts +// src/index.ts — add with the other command imports +import { runInsight } from "./commands/insight.js"; +``` + +```ts +// src/index.ts — add after the `open` command block, before `config` +withCommon(program.command("insight ")) + .description("Show visit stats (hits + approx unique visitors) for a published document (cloudfront)") + .option("--days ", "only scan the last N days of logs") + .option("--since ", "only scan logs on or after YYYY-MM-DD") + .action(async (code, opts) => { + try { + console.log( + await runInsight({ + code, + days: opts.days ? Number(opts.days) : undefined, + since: opts.since, + ...overrides(opts), + }), + ); + } catch (err) { + fail(err); + } + }); +``` + +- [ ] **Step 2: Verify it fails first (if adding a smoke assertion)** + +If `test/smoke.test.ts` asserts the list of commands in `--help`, add `"insight"` to the expected set and run: +Run: `npx vitest run test/smoke.test.ts` +Expected: FAIL before the `src/index.ts` edit, PASS after. If no such assertion exists, skip to Step 3. + +- [ ] **Step 3: Verify the command is wired end to end (dev CLI)** + +Run: +```bash +HOSTDOC_DOMAIN=d.example.com HOSTDOC_DISTRIBUTION=E1 HOSTDOC_BUCKET=b HOSTDOC_REGION=us-east-1 \ + npm run dev --silent -- insight abc 2>&1 | head -3 +``` +Expected: the re-provision hint (`No access-log bucket configured...`) — proves the command is registered and reaches `runInsight` (no log bucket set). Then with `HOSTDOC_LOG_BUCKET=x` it proceeds to an AWS call (expected creds error offline) — proves wiring. + +- [ ] **Step 4: Full suite + typecheck** + +Run: `npm run typecheck && npx vitest run` +Expected: all pass except the pre-existing environment-dependent `test/skill.test.ts` preflight case (fails identically on `main` when local AWS creds exist; passes in CI). + +- [ ] **Step 5: Commit** + +```bash +git add src/index.ts test/smoke.test.ts +git commit -m "feat: register `hostdoc insight ` command (#43)" +``` + +--- + +## Task 7: CloudFront v2 logging infrastructure (Terraform) + +**Files:** +- Modify: `infra/main.tf` (log bucket + delivery pipeline) +- Modify: `infra/outputs.tf` (`log_bucket_name` output) +- Modify: `infra/variables.tf` (`log_retention_days`) + +**Note on testing:** CI has no Terraform. Verification for this task is `terraform fmt -check` + `terraform validate` locally (they need only provider init, no AWS calls). A real end-to-end check requires `terraform apply` against a live account — call that out in the PR as manual, not CI. + +- [ ] **Step 1: Add the retention variable** + +```hcl +# infra/variables.tf — append +variable "log_retention_days" { + description = "Days to retain CloudFront access logs before S3 lifecycle expiry." + type = number + default = 30 +} +``` + +- [ ] **Step 2: Add the log bucket + delivery pipeline** + +```hcl +# infra/main.tf — append + +# --- Access-log bucket (private; CloudFront standard logging v2 destination) --- +resource "aws_s3_bucket" "logs" { + bucket = "${local.bucket_name}-logs" +} + +resource "aws_s3_bucket_public_access_block" "logs" { + bucket = aws_s3_bucket.logs.id + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true +} + +resource "aws_s3_bucket_lifecycle_configuration" "logs" { + bucket = aws_s3_bucket.logs.id + rule { + id = "expire-logs" + status = "Enabled" + filter {} + expiration { + days = var.log_retention_days + } + } +} + +# Allow the CloudWatch vended-logs delivery service to write objects. +resource "aws_s3_bucket_policy" "logs" { + depends_on = [aws_s3_bucket_public_access_block.logs] + bucket = aws_s3_bucket.logs.id + policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Sid = "AWSLogsDeliveryWrite" + Effect = "Allow" + Principal = { Service = "delivery.logs.amazonaws.com" } + Action = "s3:PutObject" + Resource = "${aws_s3_bucket.logs.arn}/*" + Condition = { + StringEquals = { + "aws:SourceAccount" = data.aws_caller_identity.current.account_id + "s3:x-amz-acl" = "bucket-owner-full-control" + } + ArnLike = { + "aws:SourceArn" = "arn:aws:logs:us-east-1:${data.aws_caller_identity.current.account_id}:delivery-source:*" + } + } + }] + }) +} + +# --- Standard logging (v2) delivery pipeline (must be us-east-1) --- +resource "aws_cloudwatch_log_delivery_source" "cf" { + provider = aws.us_east_1 + name = "${replace(local.site_domain, ".", "-")}-access-logs" + log_type = "ACCESS_LOGS" + resource_arn = aws_cloudfront_distribution.site.arn +} + +resource "aws_cloudwatch_log_delivery_destination" "cf" { + provider = aws.us_east_1 + name = "${replace(local.site_domain, ".", "-")}-s3-dest" + output_format = "w3c" + + delivery_destination_configuration { + destination_resource_arn = aws_s3_bucket.logs.arn + } +} + +resource "aws_cloudwatch_log_delivery" "cf" { + provider = aws.us_east_1 + delivery_source_name = aws_cloudwatch_log_delivery_source.cf.name + delivery_destination_arn = aws_cloudwatch_log_delivery_destination.cf.arn + field_delimiter = "\t" + + record_fields = [ + "date", + "time", + "c-ip", + "cs-uri-stem", + "sc-status", + "cs(User-Agent)", + "x-edge-result-type", + ] + + s3_delivery_configuration { + enable_hive_compatible_path = false + suffix_path = "{DistributionId}/{yyyy}/{MM}/{dd}/{HH}" + } + + depends_on = [aws_s3_bucket_policy.logs] +} +``` + +- [ ] **Step 3: Add the output** + +```hcl +# infra/outputs.tf — append +output "log_bucket_name" { + value = aws_s3_bucket.logs.id +} +``` + +- [ ] **Step 4: Validate locally** + +Run (from `infra/`, requires terraform + provider init, no AWS calls): +```bash +terraform fmt -check +terraform init -backend=false +terraform validate +``` +Expected: `fmt` clean, `validate` reports "Success! The configuration is valid." If `validate` flags an unknown argument on the delivery resources, reconcile against the provider docs for your pinned `hashicorp/aws` version (schema verified against current provider main). + +- [ ] **Step 5: Commit** + +```bash +git add infra/main.tf infra/outputs.tf infra/variables.tf +git commit -m "feat: provision CloudFront v2 access logging to a private log bucket (#43)" +``` + +--- + +## Task 8: Documentation (README privacy + usage) + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Add an `insight` section** + +Insert a section documenting: +- Usage: `hostdoc insight ` (+ `--days N` / `--since YYYY-MM-DD`), cloudfront-only for now. +- Enabling: logging is provisioned by `hostdoc provision`; existing cloudfront users must **re-run** `hostdoc provision` then `hostdoc init --from-terraform ` to gain a log bucket. Only traffic **after** enabling is captured. +- Latency: CloudFront standard logs lag minutes–hours; not real-time. +- **Privacy notice:** access logs store visitor IP addresses in the user's own S3 log bucket; hostdoc uses IPs only to approximate unique-visitor counts and does not transmit them anywhere; only a minimal field set is logged; logs auto-expire after `log_retention_days` (default 30). + +Example block to include: + +```markdown +### Visit stats — `hostdoc insight ` (cloudfront) + + hostdoc insight team/q1/report # all available logs (~30 days) + hostdoc insight team/q1/report --days 7 # last 7 days only + +Shows page-view hits and an approximate unique-visitor count (by client IP) +for a published document, read from the CloudFront access logs in **your own** +S3 bucket. + +**Enabling:** access logging is set up by `hostdoc provision`. If you provisioned +before this feature, re-run `hostdoc provision` and then +`hostdoc init --from-terraform ` to add a log bucket. Only traffic after +enabling is recorded, and CloudFront logs lag minutes to hours (not real-time). + +**Privacy:** logs are stored only in your S3 log bucket and contain visitor IP +addresses. hostdoc uses IPs solely to approximate unique visitors and never +sends them anywhere. Only a minimal field set is logged, and logs auto-expire +after 30 days (`log_retention_days`). +``` + +- [ ] **Step 2: Commit** + +```bash +git add README.md +git commit -m "docs: document `hostdoc insight` usage + privacy notice (#43)" +``` + +--- + +## Self-Review (completed) + +- **Spec coverage:** v2-delivery provisioning (T7) ✓; dedicated log bucket + 30-day expiry (T7) ✓; minimal fields (T7 `record_fields`) ✓; `logBucket` config + init back-compat (T4) ✓; W3C parse with dynamic `#Fields` mapping (T1) ✓; hits + approx unique + bot filter (T2) ✓; `/` prefix incl. nested slug (T2) ✓; date-window scan `--days`/`--since` (T2 + T5) ✓; cloudfront-only + re-provision hint (T5) ✓; table output + latency/IP notes (T5) ✓; offline tests (T1–T6) ✓; privacy docs (T8) ✓; no-regression + CI (T6 Step 4) ✓. +- **Placeholder scan:** none — every code step shows complete code; the one env-reconciliation note (T7 Step 4) is a validation fallback, not a code placeholder. +- **Type consistency:** `LogRecord`, `InsightResult`, `parseCloudFrontLog`, `aggregateInsight`, `filterLogKeys`, `dateDaysAgo`, `getObjectBytes`, `runInsight`/`InsightArgs`, `Config.logBucket` used identically across tasks. +- **Refinement flagged:** T2 counts page views (`.html`/`/`) rather than every request under the prefix, so "hits" reflect page opens, not asset fetches — a deliberate tightening of the spec's "prefix filter", documented in T8. From e32c532de7869e654e3f772206c038237b71b98d Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:21:55 +0900 Subject: [PATCH 03/13] =?UTF-8?q?docs:=20fix=20plan=20Task=204=20=E2=80=94?= =?UTF-8?q?=20thread=20logBucket=20through=20resolveConfig=20(#43)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/2026-07-11-cloudfront-insight.md | 80 ++++++++++++++----- 1 file changed, 61 insertions(+), 19 deletions(-) diff --git a/docs/superpowers/plans/2026-07-11-cloudfront-insight.md b/docs/superpowers/plans/2026-07-11-cloudfront-insight.md index e9506fd..5281692 100644 --- a/docs/superpowers/plans/2026-07-11-cloudfront-insight.md +++ b/docs/superpowers/plans/2026-07-11-cloudfront-insight.md @@ -434,26 +434,54 @@ git commit -m "feat: getObjectBytes S3 helper for gzip log reads (#43)" --- -## Task 4: Config `logBucket` + init import (`src/lib/config.ts`, `src/commands/init.ts`) +## Task 4: Config `logBucket` — interface + resolveConfig threading + init import **Files:** -- Modify: `src/lib/config.ts` (add field to `Config`) +- Modify: `src/lib/config.ts` (add field to `Config`; thread `logBucket` through the cloudfront branches of `resolveConfig`) - Modify: `src/commands/init.ts` (read optional output) -- Test: `test/init.test.ts` +- Test: `test/config.test.ts`, `test/init.test.ts` **Interfaces:** -- Produces: `Config.logBucket?: string` (cloudfront-only, optional for backward compat). +- Produces: `Config.logBucket?: string` (cloudfront-only, optional for backward compat), populated by `resolveConfig` from `HOSTDOC_LOG_BUCKET` env or the config file, and written by `init` from the `log_bucket_name` Terraform output. - Consumes: `readTerraformOutputs` output map in `init.ts`. -- [ ] **Step 1: Write the failing test** +**Why resolveConfig must thread it:** `resolveConfig` reconstructs `Config` from merged fields and its cloudfront branches (forced-mode and derived) currently return only `{ mode, bucket, region, domain, distributionId }`. Without adding `logBucket` here, a value saved in `config.json` by `init` is dropped at runtime and `cfg.logBucket` is always `undefined`. `logBucket` is NOT an `Overrides`/CLI-flag field for this slice — env + file only (consistent with hostdoc's precedence for non-flag fields like `serveRoot`). + +- [ ] **Step 1: Write the failing tests** -Inspect `test/init.test.ts` for how outputs are mocked (it stubs `readTerraformOutputs` or `execFileSync`). Add a test asserting `log_bucket_name` flows into config, and that its absence leaves `logBucket` undefined. Example (adapt to the file's existing mock style): +```ts +// test/config.test.ts — add near the other cloudfront resolveConfig tests +it("threads logBucket from the config file into a cloudfront config", () => { + saveConfig({ + mode: "cloudfront", + bucket: "b", + region: "us-east-1", + domain: "d.example.com", + distributionId: "E1", + logBucket: "b-logs", + }); + expect(resolveConfig({}).logBucket).toBe("b-logs"); +}); + +it("lets HOSTDOC_LOG_BUCKET override the file logBucket", () => { + saveConfig({ + mode: "cloudfront", + bucket: "b", + region: "us-east-1", + domain: "d.example.com", + distributionId: "E1", + logBucket: "file-logs", + }); + process.env.HOSTDOC_LOG_BUCKET = "env-logs"; + expect(resolveConfig({}).logBucket).toBe("env-logs"); +}); +``` ```ts -// test/init.test.ts — new cases +// test/init.test.ts — new cases (adapt to the file's existing output-mock style) it("imports log_bucket_name into the config when present", () => { - // Arrange the same output-mock mechanism the other init tests use, adding: - // log_bucket_name: { value: "my-logs" } alongside bucket_name/region/distribution_id/site_domain + // Add log_bucket_name: { value: "my-logs" } alongside the existing mocked + // outputs (bucket_name/region/distribution_id/site_domain). const cfg = runInit({ dir: "/fake" }); expect(cfg.logBucket).toBe("my-logs"); }); @@ -465,10 +493,10 @@ it("leaves logBucket undefined when the output is absent (legacy infra)", () => }); ``` -- [ ] **Step 2: Run test to verify it fails** +- [ ] **Step 2: Run tests to verify they fail** -Run: `npx vitest run test/init.test.ts` -Expected: FAIL — `logBucket` is undefined in the present case / property not set. +Run: `npx vitest run test/config.test.ts test/init.test.ts` +Expected: FAIL — `logBucket` undefined (resolveConfig drops it; init doesn't set it). - [ ] **Step 3: Implement** @@ -479,6 +507,20 @@ Expected: FAIL — `logBucket` is undefined in the present case / property not s logBucket?: string; // cloudfront access-log bucket (optional; enables `insight`) ``` +```ts +// src/lib/config.ts — in resolveConfig, alongside the other field merges +// (near serveRoot/host/port/scheme), add: + const logBucket = process.env.HOSTDOC_LOG_BUCKET ?? file?.logBucket; +``` + +```ts +// src/lib/config.ts — add `logBucket` to BOTH cloudfront return objects: +// 1) the forced-mode branch: `if (forced === "cloudfront") { ... return {` +// 2) the derived branch: `if (domain && distributionId) { ... return {` +// Each becomes: + return { mode: "cloudfront", bucket, region, domain, distributionId, logBucket }; +``` + ```ts // src/commands/init.ts — after building cfg, before saveConfig const logBucket = outputs["log_bucket_name"]?.value; @@ -487,16 +529,16 @@ Expected: FAIL — `logBucket` is undefined in the present case / property not s } ``` -- [ ] **Step 4: Run test to verify it passes** +- [ ] **Step 4: Run tests to verify they pass** -Run: `npx vitest run test/init.test.ts` -Expected: PASS. +Run: `npx vitest run test/config.test.ts test/init.test.ts` +Expected: PASS (including the existing config/init cases — no regression). - [ ] **Step 5: Commit** ```bash -git add src/lib/config.ts src/commands/init.ts test/init.test.ts -git commit -m "feat: import log_bucket_name output into cloudfront config (#43)" +git add src/lib/config.ts src/commands/init.ts test/config.test.ts test/init.test.ts +git commit -m "feat: thread cloudfront logBucket through resolveConfig + init (#43)" ``` --- @@ -588,7 +630,7 @@ describe("runInsight", () => { }); ``` -Note: `HOSTDOC_LOG_BUCKET` is used only to drive this test through env. Wire it in Task 6 via `resolveConfig`? No — `logBucket` is not an `Overrides` field. Instead, read it in `runInsight` as `args`-independent config: resolve `logBucket` from `cfg.logBucket` OR `process.env.HOSTDOC_LOG_BUCKET` (env escape hatch for testing/BYO). Implement exactly as below. +Note: `HOSTDOC_LOG_BUCKET` here is honored by `resolveConfig` (Task 4) — it populates `cfg.logBucket`, so `runInsight` reads only `cfg.logBucket` (no separate env read). The env var is the legitimate override path for `logBucket`, mirroring how the test sets the other cloudfront fields via `HOSTDOC_*`. - [ ] **Step 2: Run test to verify it fails** @@ -637,7 +679,7 @@ export async function runInsight(args: InsightArgs & Overrides): Promise `insight currently supports cloudfront mode only (resolved mode: ${cfg.mode}).`, ); } - const logBucket = cfg.logBucket ?? process.env.HOSTDOC_LOG_BUCKET; + const logBucket = cfg.logBucket; if (!logBucket) { throw new Error( "No access-log bucket configured. Re-run `hostdoc provision` to enable " + From ae3bc6b023f9fe6484bcef5969df84a683599bbc Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:23:27 +0900 Subject: [PATCH 04/13] feat: pure CloudFront W3C access-log parser (#43) --- src/lib/cflog.ts | 62 ++++++++++++++++++++++++++++++++++++++++++++++ test/cflog.test.ts | 35 ++++++++++++++++++++++++++ 2 files changed, 97 insertions(+) create mode 100644 src/lib/cflog.ts create mode 100644 test/cflog.test.ts diff --git a/src/lib/cflog.ts b/src/lib/cflog.ts new file mode 100644 index 0000000..24c313b --- /dev/null +++ b/src/lib/cflog.ts @@ -0,0 +1,62 @@ +export interface LogRecord { + date: string; + time: string; + cIp: string; + uriStem: string; + status: string; + userAgent: string; + edgeResult: string; +} + +// CloudFront field name -> LogRecord key. +const FIELD_MAP: Record = { + date: "date", + time: "time", + "c-ip": "cIp", + "cs-uri-stem": "uriStem", + "sc-status": "status", + "cs(User-Agent)": "userAgent", + "x-edge-result-type": "edgeResult", +}; + +function safeDecode(v: string): string { + if (v === "-" || v === "") return v === "-" ? "" : v; + try { + return decodeURIComponent(v); + } catch { + return v; + } +} + +/** Parse a decompressed CloudFront standard-log (W3C, tab-separated) file. */ +export function parseCloudFrontLog(text: string): LogRecord[] { + const lines = text.split("\n"); + let cols: (keyof LogRecord | null)[] | null = null; + const out: LogRecord[] = []; + + for (const line of lines) { + if (line.startsWith("#Fields:")) { + const names = line.slice("#Fields:".length).trim().split(/\s+/); + cols = names.map((n) => FIELD_MAP[n] ?? null); + continue; + } + if (line.startsWith("#") || line.trim() === "" || !cols) continue; + + const parts = line.split("\t"); + const rec: LogRecord = { + date: "", + time: "", + cIp: "", + uriStem: "", + status: "", + userAgent: "", + edgeResult: "", + }; + for (let i = 0; i < cols.length; i++) { + const key = cols[i]; + if (key) rec[key] = safeDecode(parts[i] ?? ""); + } + out.push(rec); + } + return out; +} diff --git a/test/cflog.test.ts b/test/cflog.test.ts new file mode 100644 index 0000000..126789d --- /dev/null +++ b/test/cflog.test.ts @@ -0,0 +1,35 @@ +import { describe, it, expect } from "vitest"; +import { parseCloudFrontLog } from "../src/lib/cflog.js"; + +const HEADER = + "#Version: 1.0\n" + + "#Fields: date time c-ip cs-uri-stem sc-status cs(User-Agent) x-edge-result-type\n"; + +describe("parseCloudFrontLog", () => { + it("maps fields by the #Fields header, not fixed positions", () => { + const log = + HEADER + + "2026-07-11\t21:34:06\t203.0.113.5\t/abc/index.html\t200\tMozilla%2F5.0\tHit\n"; + const recs = parseCloudFrontLog(log); + expect(recs).toHaveLength(1); + expect(recs[0]).toEqual({ + date: "2026-07-11", + time: "21:34:06", + cIp: "203.0.113.5", + uriStem: "/abc/index.html", + status: "200", + userAgent: "Mozilla/5.0", // %2F decoded + edgeResult: "Hit", + }); + }); + + it("skips comment and blank lines and returns [] without a #Fields header", () => { + expect(parseCloudFrontLog("#Version: 1.0\n\n")).toEqual([]); + expect(parseCloudFrontLog("")).toEqual([]); + }); + + it("tolerates a malformed percent-escape by keeping the raw value", () => { + const log = HEADER + "2026-07-11\t00:00:00\t1.1.1.1\t/x/%\t200\tUA\tMiss\n"; + expect(parseCloudFrontLog(log)[0].uriStem).toBe("/x/%"); + }); +}); From 0ddd737c1173f7f9ea04e9668a2c58f78e3a8704 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:27:03 +0900 Subject: [PATCH 05/13] feat: insight aggregation + log-key date selection (#43) --- src/lib/insight.ts | 68 ++++++++++++++++++++++++++++++ test/insight.test.ts | 98 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+) create mode 100644 src/lib/insight.ts create mode 100644 test/insight.test.ts diff --git a/src/lib/insight.ts b/src/lib/insight.ts new file mode 100644 index 0000000..c628c3d --- /dev/null +++ b/src/lib/insight.ts @@ -0,0 +1,68 @@ +import type { LogRecord } from "./cflog.js"; + +export interface InsightResult { + code: string; + hits: number; + uniqueVisitors: number; + since: string | null; + until: string | null; +} + +const BOT_RE = + /bot|crawl|spider|slurp|bingpreview|facebookexternalhit|headless|monitor|curl|wget|python-requests|pingdom|uptime/i; + +function isPageView(uriStem: string, prefix: string): boolean { + return ( + uriStem.startsWith(prefix) && + (uriStem.endsWith(".html") || uriStem.endsWith("/")) + ); +} + +/** Per-document page-view hits + approximate unique visitors (by client IP). */ +export function aggregateInsight( + records: LogRecord[], + code: string, +): InsightResult { + const prefix = `/${code}/`; + const ips = new Set(); + let hits = 0; + let since: string | null = null; + let until: string | null = null; + + for (const r of records) { + if (!/^[23]/.test(r.status)) continue; + if (BOT_RE.test(r.userAgent)) continue; + if (!isPageView(r.uriStem, prefix)) continue; + hits++; + if (r.cIp) ips.add(r.cIp); + if (since === null || r.date < since) since = r.date; + if (until === null || r.date > until) until = r.date; + } + + return { code, hits, uniqueVisitors: ips.size, since, until }; +} + +/** Extract YYYY-MM-DD from a partitioned CloudFront log key, else null. */ +export function cloudFrontLogKeyDate(key: string): string | null { + const m = key.match(/\/(\d{4})\/(\d{2})\/(\d{2})\//); + return m ? `${m[1]}-${m[2]}-${m[3]}` : null; +} + +/** Keep .gz log keys whose partition date is >= since (all if no since). */ +export function filterLogKeys( + keys: string[], + opts: { since?: string }, +): string[] { + return keys.filter((k) => { + if (!k.endsWith(".gz")) return false; + if (!opts.since) return true; + const d = cloudFrontLogKeyDate(k); + return d === null ? true : d >= opts.since; + }); +} + +/** YYYY-MM-DD for (now - days), UTC. */ +export function dateDaysAgo(days: number, now: Date): string { + const d = new Date(now.getTime() - days * 24 * 60 * 60 * 1000); + return d.toISOString().slice(0, 10); +} diff --git a/test/insight.test.ts b/test/insight.test.ts new file mode 100644 index 0000000..6a486d1 --- /dev/null +++ b/test/insight.test.ts @@ -0,0 +1,98 @@ +import { describe, it, expect } from "vitest"; +import { + aggregateInsight, + cloudFrontLogKeyDate, + filterLogKeys, + dateDaysAgo, +} from "../src/lib/insight.js"; +import type { LogRecord } from "../src/lib/cflog.js"; + +function rec(p: Partial): LogRecord { + return { + date: "2026-07-11", + time: "00:00:00", + cIp: "1.1.1.1", + uriStem: "/abc/index.html", + status: "200", + userAgent: "Mozilla/5.0", + edgeResult: "Hit", + ...p, + }; +} + +describe("aggregateInsight", () => { + it("counts page-view hits under the code prefix and approx unique IPs", () => { + const recs = [ + rec({ cIp: "1.1.1.1", uriStem: "/abc/index.html" }), + rec({ cIp: "2.2.2.2", uriStem: "/abc/index.html" }), + rec({ cIp: "1.1.1.1", uriStem: "/abc/index.html", date: "2026-07-12" }), + rec({ uriStem: "/abc/style.css" }), // asset -> excluded + rec({ uriStem: "/other/index.html" }), // different code -> excluded + ]; + const r = aggregateInsight(recs, "abc"); + expect(r.hits).toBe(3); + expect(r.uniqueVisitors).toBe(2); + expect(r.since).toBe("2026-07-11"); + expect(r.until).toBe("2026-07-12"); + }); + + it("excludes bots and non-2xx/3xx statuses", () => { + const recs = [ + rec({ userAgent: "Googlebot/2.1" }), + rec({ status: "404" }), + rec({ cIp: "9.9.9.9" }), + ]; + const r = aggregateInsight(recs, "abc"); + expect(r.hits).toBe(1); + expect(r.uniqueVisitors).toBe(1); + }); + + it("returns zeros and null range when nothing matches", () => { + expect(aggregateInsight([], "abc")).toEqual({ + code: "abc", + hits: 0, + uniqueVisitors: 0, + since: null, + until: null, + }); + }); + + it("counts nested-slug page views", () => { + const r = aggregateInsight( + [rec({ uriStem: "/team/q1/report/index.html" })], + "team/q1/report", + ); + expect(r.hits).toBe(1); + }); +}); + +describe("log key selection", () => { + it("extracts the partition date from a log key", () => { + expect( + cloudFrontLogKeyDate("E123ABC/2026/07/11/14/E123ABC.2026-07-11-14.abc.gz"), + ).toBe("2026-07-11"); + expect(cloudFrontLogKeyDate("no-date-here.gz")).toBeNull(); + }); + + it("filters to .gz keys on/after `since`", () => { + const keys = [ + "d/2026/07/10/00/a.gz", + "d/2026/07/11/00/b.gz", + "d/2026/07/12/00/c.gz", + "d/2026/07/12/00/note.txt", + ]; + expect(filterLogKeys(keys, { since: "2026-07-11" })).toEqual([ + "d/2026/07/11/00/b.gz", + "d/2026/07/12/00/c.gz", + ]); + expect(filterLogKeys(keys, {})).toEqual([ + "d/2026/07/10/00/a.gz", + "d/2026/07/11/00/b.gz", + "d/2026/07/12/00/c.gz", + ]); + }); + + it("computes a date N days before a fixed now", () => { + expect(dateDaysAgo(7, new Date("2026-07-11T00:00:00Z"))).toBe("2026-07-04"); + }); +}); From 2936e08451246f50955722f001351e2ffee0cde2 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:31:23 +0900 Subject: [PATCH 06/13] feat: getObjectBytes S3 helper for gzip log reads (#43) --- src/lib/aws.ts | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/aws.ts b/src/lib/aws.ts index a6c2a1c..f42806e 100644 --- a/src/lib/aws.ts +++ b/src/lib/aws.ts @@ -96,3 +96,18 @@ export async function getJson( const text = await (res.Body as { transformToString(): Promise }).transformToString(); return JSON.parse(text) as T; } + +export async function getObjectBytes( + s3: S3Client, + bucket: string, + key: string, +): Promise { + const res = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key })); + if (!res.Body) { + throw new Error(`Empty response body for ${key}`); + } + const bytes = await ( + res.Body as { transformToByteArray(): Promise } + ).transformToByteArray(); + return Buffer.from(bytes); +} From 226394f7e1eadba59eab7c34eee9dcbcc8d58539 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:35:48 +0900 Subject: [PATCH 07/13] feat: thread cloudfront logBucket through resolveConfig + init (#43) --- src/commands/init.ts | 4 ++++ src/lib/config.ts | 6 ++++-- test/config.test.ts | 24 ++++++++++++++++++++++++ test/init.test.ts | 20 ++++++++++++++++++++ 4 files changed, 52 insertions(+), 2 deletions(-) diff --git a/src/commands/init.ts b/src/commands/init.ts index 3714307..2d29075 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -39,6 +39,10 @@ export function runInit(args: { dir: string }): Config { distributionId: get("distribution_id"), domain: get("site_domain"), }; + const logBucket = outputs["log_bucket_name"]?.value; + if (typeof logBucket === "string" && logBucket) { + cfg.logBucket = logBucket; + } saveConfig(cfg); return cfg; } diff --git a/src/lib/config.ts b/src/lib/config.ts index 90ad2db..afdec09 100644 --- a/src/lib/config.ts +++ b/src/lib/config.ts @@ -14,6 +14,7 @@ export interface Config { websiteEndpoint?: string; // s3-website distributionId?: string; // cloudfront domain?: string; // cloudfront + logBucket?: string; // cloudfront access-log bucket (optional; enables `insight`) // self-hosted serveRoot?: string; host?: string; // DDNS/domain/static IP; if unset, public IP is auto-detected @@ -120,6 +121,7 @@ export function resolveConfig( flags.scheme ?? (process.env.HOSTDOC_SCHEME as "http" | "https" | undefined) ?? file?.scheme; + const logBucket = process.env.HOSTDOC_LOG_BUCKET ?? file?.logBucket; // --- Explicit mode pin (B): --mode > HOSTDOC_MODE. Never read from file. --- // Accept any case (e.g. S3-Website, CLOUDFRONT); normalize before matching, @@ -179,7 +181,7 @@ export function resolveConfig( if (serveRoot) { warn("--mode cloudfront: ignoring serveRoot from config/env."); } - return { mode: "cloudfront", bucket, region, domain, distributionId }; + return { mode: "cloudfront", bucket, region, domain, distributionId, logBucket }; } // serveRoot is the self-hosted discriminator. Mixing it with AWS fields is a @@ -217,7 +219,7 @@ export function resolveConfig( "'bucket' set at higher precedence than 'domain', but 'domain' resolves the mode to cloudfront (bucket ignored for mode). Use --mode s3-website to force s3-website.", ); } - return { mode: "cloudfront", bucket, region, domain, distributionId }; + return { mode: "cloudfront", bucket, region, domain, distributionId, logBucket }; } if (bucket && region) { diff --git a/test/config.test.ts b/test/config.test.ts index 5d98a3d..b2588d2 100644 --- a/test/config.test.ts +++ b/test/config.test.ts @@ -31,6 +31,7 @@ const ENV_KEYS = [ "HOSTDOC_PORT", "HOSTDOC_SCHEME", "HOSTDOC_MODE", + "HOSTDOC_LOG_BUCKET", ]; const saved: Record = {}; @@ -118,6 +119,29 @@ describe("resolveConfig", () => { expect(cfg.mode).toBe("cloudfront"); expect(cfg.domain).toBe("shared.example.com"); }); + it("threads logBucket from the config file into a cloudfront config", () => { + saveConfig({ + mode: "cloudfront", + bucket: "b", + region: "us-east-1", + domain: "d.example.com", + distributionId: "E1", + logBucket: "b-logs", + }); + expect(resolveConfig({}).logBucket).toBe("b-logs"); + }); + it("lets HOSTDOC_LOG_BUCKET override the file logBucket", () => { + saveConfig({ + mode: "cloudfront", + bucket: "b", + region: "us-east-1", + domain: "d.example.com", + distributionId: "E1", + logBucket: "file-logs", + }); + process.env.HOSTDOC_LOG_BUCKET = "env-logs"; + expect(resolveConfig({}).logBucket).toBe("env-logs"); + }); it("flags override the file", () => { saveConfig({ mode: "s3-website", diff --git a/test/init.test.ts b/test/init.test.ts index 2f86be4..e5b6b11 100644 --- a/test/init.test.ts +++ b/test/init.test.ts @@ -47,4 +47,24 @@ describe("runInit", () => { mockExec.mockReturnValue(JSON.stringify({ region: { value: "us-east-1" } })); expect(() => runInit({ dir: "./infra" })).toThrow(/bucket_name/); }); + + it("imports log_bucket_name into the config when present", () => { + mockExec.mockReturnValue( + JSON.stringify({ + bucket_name: { value: "shared.example.com", type: "string" }, + region: { value: "us-east-1", type: "string" }, + distribution_id: { value: "E123ABC", type: "string" }, + site_domain: { value: "shared.example.com", type: "string" }, + log_bucket_name: { value: "my-logs", type: "string" }, + }), + ); + const cfg = runInit({ dir: "./infra" }); + expect(cfg.logBucket).toBe("my-logs"); + }); + + it("leaves logBucket undefined when the output is absent (legacy infra)", () => { + mockExec.mockReturnValue(FIXTURE); + const cfg = runInit({ dir: "./infra" }); + expect(cfg.logBucket).toBeUndefined(); + }); }); From 1b6ec3cb237cbecd57bd205cae8f0d4e3fe849fe Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:40:28 +0900 Subject: [PATCH 08/13] feat: runInsight command reads+aggregates CloudFront logs (#43) --- src/commands/insight.ts | 70 ++++++++++++++++++++++++++++++++++++ test/insight-command.test.ts | 67 ++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) create mode 100644 src/commands/insight.ts create mode 100644 test/insight-command.test.ts diff --git a/src/commands/insight.ts b/src/commands/insight.ts new file mode 100644 index 0000000..0e13239 --- /dev/null +++ b/src/commands/insight.ts @@ -0,0 +1,70 @@ +import { gunzipSync } from "node:zlib"; +import { resolveConfig, type Overrides } from "../lib/config.js"; +import { isValidPath, isValidCode } from "../lib/code.js"; +import { makeS3, listKeys, getObjectBytes } from "../lib/aws.js"; +import { parseCloudFrontLog, type LogRecord } from "../lib/cflog.js"; +import { + aggregateInsight, + filterLogKeys, + dateDaysAgo, + type InsightResult, +} from "../lib/insight.js"; + +export interface InsightArgs { + code: string; + days?: number; + since?: string; + profile?: string; +} + +function formatInsight(r: InsightResult, filesScanned: number): string { + const range = + r.since && r.until ? `${r.since} → ${r.until}` : "no matching requests yet"; + return [ + `${r.code} — cloudfront`, + ` hits ${r.hits}`, + ` unique visitors ~${r.uniqueVisitors} (approx, by client IP)`, + ` range ${range} (${filesScanned} log file(s) scanned)`, + ` note CloudFront logs lag minutes–hours; bots filtered; unique is an IP approximation.`, + ].join("\n"); +} + +export async function runInsight(args: InsightArgs & Overrides): Promise { + const cfg = resolveConfig(args); + if (cfg.mode !== "cloudfront") { + throw new Error( + `insight currently supports cloudfront mode only (resolved mode: ${cfg.mode}).`, + ); + } + const logBucket = cfg.logBucket; + if (!logBucket) { + throw new Error( + "No access-log bucket configured. Re-run `hostdoc provision` to enable " + + "CloudFront access logging, then `hostdoc init --from-terraform `.", + ); + } + if (!isValidPath(args.code) && !isValidCode(args.code)) { + throw new Error(`Invalid code: ${args.code}`); + } + + const since = + args.since ?? (args.days ? dateDaysAgo(args.days, new Date()) : undefined); + + const s3 = makeS3({ region: cfg.region, profile: args.profile }); + const allKeys = await listKeys(s3, logBucket, ""); + const keys = filterLogKeys(allKeys, { since }); + + const records: LogRecord[] = []; + for (const key of keys) { + try { + const gz = await getObjectBytes(s3, logBucket, key); + records.push(...parseCloudFrontLog(gunzipSync(gz).toString("utf8"))); + } catch (err) { + process.stderr.write( + `hostdoc: skipping unreadable log ${key}: ${(err as Error).message}\n`, + ); + } + } + + return formatInsight(aggregateInsight(records, args.code), keys.length); +} diff --git a/test/insight-command.test.ts b/test/insight-command.test.ts new file mode 100644 index 0000000..109566f --- /dev/null +++ b/test/insight-command.test.ts @@ -0,0 +1,67 @@ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mockClient } from "aws-sdk-client-mock"; +import { gzipSync } from "node:zlib"; +import { + S3Client, + ListObjectsV2Command, + GetObjectCommand, +} from "@aws-sdk/client-s3"; +import { runInsight } from "../src/commands/insight.js"; + +const s3mock = mockClient(S3Client); + +const LOG = + "#Version: 1.0\n" + + "#Fields: date time c-ip cs-uri-stem sc-status cs(User-Agent) x-edge-result-type\n" + + "2026-07-11\t00:00:00\t1.1.1.1\t/abc/index.html\t200\tMozilla%2F5.0\tHit\n" + + "2026-07-11\t00:00:01\t2.2.2.2\t/abc/index.html\t200\tMozilla%2F5.0\tHit\n" + + "2026-07-11\t00:00:02\t9.9.9.9\t/abc/style.css\t200\tMozilla%2F5.0\tHit\n"; + +function gzBody() { + const buf = gzipSync(Buffer.from(LOG)); + return { transformToByteArray: async () => new Uint8Array(buf) } as any; +} + +beforeEach(() => { + s3mock.reset(); + process.env.HOSTDOC_DOMAIN = "shared.example.com"; + process.env.HOSTDOC_DISTRIBUTION = "DIST1"; + process.env.HOSTDOC_BUCKET = "content-bucket"; + process.env.HOSTDOC_REGION = "us-east-1"; + process.env.HOSTDOC_LOG_BUCKET = "log-bucket"; +}); +afterEach(() => { + for (const k of [ + "HOSTDOC_DOMAIN", + "HOSTDOC_DISTRIBUTION", + "HOSTDOC_BUCKET", + "HOSTDOC_REGION", + "HOSTDOC_LOG_BUCKET", + ]) + delete process.env[k]; +}); + +describe("runInsight", () => { + it("aggregates hits + unique visitors from gzipped logs", async () => { + s3mock + .on(ListObjectsV2Command) + .resolves({ Contents: [{ Key: "DIST1/2026/07/11/00/a.gz" }], IsTruncated: false }); + s3mock.on(GetObjectCommand).resolves({ Body: gzBody() }); + + const out = await runInsight({ code: "abc" }); + expect(out).toMatch(/hits\s+2/); + expect(out).toMatch(/unique visitors\s+~2/); + expect(out).toMatch(/abc/); + }); + + it("errors clearly when the resolved mode is not cloudfront", async () => { + delete process.env.HOSTDOC_DOMAIN; + delete process.env.HOSTDOC_DISTRIBUTION; + await expect(runInsight({ code: "abc" })).rejects.toThrow(/cloudfront/i); + }); + + it("hints re-provision when no log bucket is configured", async () => { + delete process.env.HOSTDOC_LOG_BUCKET; + await expect(runInsight({ code: "abc" })).rejects.toThrow(/provision/i); + }); +}); From 4388bdac2d5feb12aa4ff16b72a1f1cd68747c54 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:45:25 +0900 Subject: [PATCH 09/13] feat: register `hostdoc insight ` command (#43) --- src/index.ts | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/src/index.ts b/src/index.ts index 75707a1..4862d72 100644 --- a/src/index.ts +++ b/src/index.ts @@ -11,6 +11,7 @@ import { runPublish } from "./commands/publish.js"; import { listDocs, formatRows } from "./commands/list.js"; import { runRm } from "./commands/rm.js"; import { runOpen, openPublishedUrl } from "./commands/open.js"; +import { runInsight } from "./commands/insight.js"; import { describeConfig } from "./commands/config.js"; import { runSetupSelfHosted, runInstallSelfHosted } from "./commands/setup-selfhosted.js"; import { onPath } from "./lib/which.js"; @@ -306,6 +307,25 @@ withCommon(program.command("open ")) } }); +withCommon(program.command("insight ")) + .description("Show visit stats (hits + approx unique visitors) for a published document (cloudfront)") + .option("--days ", "only scan the last N days of logs") + .option("--since ", "only scan logs on or after YYYY-MM-DD") + .action(async (code, opts) => { + try { + console.log( + await runInsight({ + code, + days: opts.days ? Number(opts.days) : undefined, + since: opts.since, + ...overrides(opts), + }), + ); + } catch (err) { + fail(err); + } + }); + withCommon(program.command("config")) .description("Show the active configuration") .action((opts) => { From 883feeabcc852dada3b171fa6c36bab3a199604e Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:49:50 +0900 Subject: [PATCH 10/13] feat: provision CloudFront v2 access logging to a private log bucket (#43) --- infra/main.tf | 92 ++++++++++++++++++++++++++++++++++++++++++++++ infra/outputs.tf | 4 ++ infra/variables.tf | 6 +++ 3 files changed, 102 insertions(+) diff --git a/infra/main.tf b/infra/main.tf index 712aaeb..fe206b3 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -200,3 +200,95 @@ resource "aws_iam_user_policy" "publisher" { user = aws_iam_user.publisher[0].name policy = local.publisher_policy_json } + +# --- Access-log bucket (private; CloudFront standard logging v2 destination) --- +resource "aws_s3_bucket" "logs" { + bucket = "${local.bucket_name}-logs" +} + +resource "aws_s3_bucket_public_access_block" "logs" { + bucket = aws_s3_bucket.logs.id + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true +} + +resource "aws_s3_bucket_lifecycle_configuration" "logs" { + bucket = aws_s3_bucket.logs.id + rule { + id = "expire-logs" + status = "Enabled" + filter {} + expiration { + days = var.log_retention_days + } + } +} + +# Allow the CloudWatch vended-logs delivery service to write objects. +resource "aws_s3_bucket_policy" "logs" { + depends_on = [aws_s3_bucket_public_access_block.logs] + bucket = aws_s3_bucket.logs.id + policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Sid = "AWSLogsDeliveryWrite" + Effect = "Allow" + Principal = { Service = "delivery.logs.amazonaws.com" } + Action = "s3:PutObject" + Resource = "${aws_s3_bucket.logs.arn}/*" + Condition = { + StringEquals = { + "aws:SourceAccount" = data.aws_caller_identity.current.account_id + "s3:x-amz-acl" = "bucket-owner-full-control" + } + ArnLike = { + "aws:SourceArn" = "arn:aws:logs:us-east-1:${data.aws_caller_identity.current.account_id}:delivery-source:*" + } + } + }] + }) +} + +# --- Standard logging (v2) delivery pipeline (must be us-east-1) --- +resource "aws_cloudwatch_log_delivery_source" "cf" { + provider = aws.us_east_1 + name = "${replace(local.site_domain, ".", "-")}-access-logs" + log_type = "ACCESS_LOGS" + resource_arn = aws_cloudfront_distribution.site.arn +} + +resource "aws_cloudwatch_log_delivery_destination" "cf" { + provider = aws.us_east_1 + name = "${replace(local.site_domain, ".", "-")}-s3-dest" + output_format = "w3c" + + delivery_destination_configuration { + destination_resource_arn = aws_s3_bucket.logs.arn + } +} + +resource "aws_cloudwatch_log_delivery" "cf" { + provider = aws.us_east_1 + delivery_source_name = aws_cloudwatch_log_delivery_source.cf.name + delivery_destination_arn = aws_cloudwatch_log_delivery_destination.cf.arn + field_delimiter = "\t" + + record_fields = [ + "date", + "time", + "c-ip", + "cs-uri-stem", + "sc-status", + "cs(User-Agent)", + "x-edge-result-type", + ] + + s3_delivery_configuration { + enable_hive_compatible_path = false + suffix_path = "{DistributionId}/{yyyy}/{MM}/{dd}/{HH}" + } + + depends_on = [aws_s3_bucket_policy.logs] +} diff --git a/infra/outputs.tf b/infra/outputs.tf index 5474550..c72aedb 100644 --- a/infra/outputs.tf +++ b/infra/outputs.tf @@ -56,3 +56,7 @@ output "publisher_secret_access_key" { value = var.create_publisher_user ? aws_iam_access_key.publisher[0].secret : null sensitive = true } + +output "log_bucket_name" { + value = aws_s3_bucket.logs.id +} diff --git a/infra/variables.tf b/infra/variables.tf index e2383b3..a196d44 100644 --- a/infra/variables.tf +++ b/infra/variables.tf @@ -28,3 +28,9 @@ variable "create_publisher_user" { default = false description = "Create a dedicated least-privilege IAM publisher user + access key. The secret is stored in plaintext in terraform.tfstate — keep state private and rotate/destroy the key when done (see outputs.tf)." } + +variable "log_retention_days" { + description = "Days to retain CloudFront access logs before S3 lifecycle expiry." + type = number + default = 30 +} From 4492a66f5db9f478df263dfbd4be8d8d8e452eba Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:53:22 +0900 Subject: [PATCH 11/13] docs: document `hostdoc insight` usage + privacy notice (#43) --- README.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/README.md b/README.md index 5dbd24e..39b3951 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,27 @@ non-interactively. Overwriting (`--force`) and `hostdoc rm` automatically invalidate `//*` on the distribution. +### Visit stats — `hostdoc insight ` (cloudfront) + +```bash +hostdoc insight team/q1/report # all available logs (~30 days) +hostdoc insight team/q1/report --days 7 # last 7 days only +``` + +Shows page-view hits and an approximate unique-visitor count (by client IP) +for a published document, read from the CloudFront access logs in **your own** +S3 bucket. + +**Enabling:** access logging is set up by `hostdoc provision`. If you provisioned +before this feature, re-run `hostdoc provision` and then +`hostdoc init --from-terraform ` to add a log bucket. Only traffic after +enabling is recorded, and CloudFront logs lag minutes to hours (not real-time). + +**Privacy:** logs are stored only in your S3 log bucket and contain visitor IP +addresses. hostdoc uses IPs solely to approximate unique visitors and never +sends them anywhere. Only a minimal field set is logged, and logs auto-expire +after 30 days (`log_retention_days`). + ### External (non-Route53) DNS Automated ACM validation and alias records require a Route53 hosted zone. If From e6ccb2690b2ecb39bbe765b9ff86a78499533eb3 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 22:55:26 +0900 Subject: [PATCH 12/13] docs: document insight --since flag (#43) --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 39b3951..506eaab 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,7 @@ Overwriting (`--force`) and `hostdoc rm` automatically invalidate ```bash hostdoc insight team/q1/report # all available logs (~30 days) hostdoc insight team/q1/report --days 7 # last 7 days only +hostdoc insight team/q1/report --since 2026-01-15 # logs on or after a date ``` Shows page-view hits and an approximate unique-visitor count (by client IP) From 8360736da250e3976f8755bdb030899af8336234 Mon Sep 17 00:00:00 2001 From: yeonigi Date: Sat, 11 Jul 2026 23:03:49 +0900 Subject: [PATCH 13/13] fix: validate insight --days is a positive integer (#43) --- src/commands/insight.ts | 6 ++++++ test/insight-command.test.ts | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/src/commands/insight.ts b/src/commands/insight.ts index 0e13239..cd5d6ed 100644 --- a/src/commands/insight.ts +++ b/src/commands/insight.ts @@ -46,6 +46,12 @@ export async function runInsight(args: InsightArgs & Overrides): Promise if (!isValidPath(args.code) && !isValidCode(args.code)) { throw new Error(`Invalid code: ${args.code}`); } + if ( + args.days !== undefined && + (!Number.isInteger(args.days) || args.days <= 0) + ) { + throw new Error("--days must be a positive integer."); + } const since = args.since ?? (args.days ? dateDaysAgo(args.days, new Date()) : undefined); diff --git a/test/insight-command.test.ts b/test/insight-command.test.ts index 109566f..b30da3d 100644 --- a/test/insight-command.test.ts +++ b/test/insight-command.test.ts @@ -64,4 +64,10 @@ describe("runInsight", () => { delete process.env.HOSTDOC_LOG_BUCKET; await expect(runInsight({ code: "abc" })).rejects.toThrow(/provision/i); }); + + it("rejects a non-positive-integer --days value", async () => { + await expect(runInsight({ code: "abc", days: NaN })).rejects.toThrow( + /positive integer/i, + ); + }); });