Add native Top Pages and Traffic Sources with real EmDash screenshots

.github/workflows/ci.yml

@@ -26,3 +26,23 @@ jobs:
           cache: pnpm
       - run: pnpm install --frozen-lockfile
       - run: pnpm run check
+  screenshots:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v7
+      - uses: pnpm/action-setup@v6
+        with:
+          version: 10.32.1
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 24
+          cache: pnpm
+      - run: pnpm install --frozen-lockfile
+      - run: pnpm exec playwright install --with-deps chromium
+      - run: pnpm screenshot
+      - uses: actions/upload-artifact@v7
+        with:
+          name: emdash-openanalytics-screenshots
+          path: docs/screenshots/*.png
+          if-no-files-found: error

.gitignore

@@ -1,6 +1,8 @@
 node_modules/
 dist/
 coverage/
+.astro/
+demo/emdash-env.d.ts
 .turbo/
 .DS_Store
 *.tgz

README.md

@@ -4,7 +4,8 @@ Native EmDash CMS integration for OpenAnalytics, by Black Swamp AI.
 
 Native connection validation, public-site tracker installation, and an analytics
 overview inside EmDash. The admin page uses EmDash Block Kit controls, metric
-cards, notices, and a timeseries chart. Private credentials stay on the server.
+cards, notices, a timeseries chart, Top Pages, and Traffic Sources. Private
+credentials stay on the server.
 
 ## Installation
 
@@ -103,8 +104,19 @@ numbers are easier to interpret. Missing freshness is shown as unavailable.
 Authentication failures, missing analytics scope, suspended service, rate limits,
 and unavailable upstream service have safe messages and validation/retry controls.
 
-Opening the page or changing its range makes two server-side reads: overview and
-timeseries. Revalidation first reads site metadata and refreshes the installation
+Top Pages shows the first ten pages ranked by views, with views and visitors.
+Traffic Sources shows the first ten attribution groups ranked by views, with
+views and visitors. Campaign source, medium, and campaign values appear when
+recorded; rows with the same referrer can represent different campaigns.
+Untagged rows with an empty referrer are shown as Direct / internal. Tagged
+rows keep their recorded UTM source.
+These are per-row visitor counts and should not be summed into the overview.
+Empty reports have their own messages. A failed report leaves other successful
+sections visible, and each section reports its own data caveats.
+
+Opening the page or changing its range makes four server-side reads: overview,
+timeseries, pages, and sources. All reuse one requested interval and timezone.
+Revalidation first reads site metadata and refreshes the installation
 snapshot. There is no polling, automatic retry, or shared analytics cache. The
 private admin route requires `plugins:manage` and EmDash's CSRF protection.
 
@@ -152,14 +164,17 @@ manual tracker URL overrides.
 
 ## Current limitations
 
-- The overview is intentionally small: no top pages, sources, sessions, funnels,
-  revenue, visitor profiles, editor analytics, or realtime polling.
+- The summary is intentionally small: no pagination, custom filters, geography,
+  devices, sessions browser, individual visitors, custom-event reports, funnels,
+  revenue, web vitals, editor analytics, or realtime polling. OAuth and account
+  or site creation are also outside this plugin's current scope.
 - Revalidate after tracker rotation or a collector URL change. Saved installation
   metadata has no automatic expiry; private-key revocation is detected on validation.
 - Static pages receive the snapshot available when they are rendered. Rebuild
   those pages after changing the connection or tracking switch.
 - The tracker independently fetches OpenAnalytics's browser configuration. A
-  suspended site may validate successfully; ingestion follows OpenAnalytics policy.
+  suspended site may validate successfully and still have its tracker installed;
+  collection and analytics access follow OpenAnalytics's suspension policy.
 - Duplicate protection covers EmDash fragments in a placement. Remove any tracker
   tag already installed manually in the theme.
 
@@ -173,6 +188,11 @@ pnpm check
 `check` runs typechecking, linting, formatting checks, tests, build, and package
 verification. CI runs the same checks. No npm publication is performed.
 
+Run `pnpm exec playwright install chromium`, then `pnpm screenshot` to generate
+three screenshots from a local EmDash admin instance using synthetic analytics.
+See the [demo instructions](demo/README.md) for the authentication, fixture,
+artifact safety checks, and screenshot locations. CI also runs this workflow.
+
 Source boundaries are the native plugin entry, settings/configuration, the
 server-side OpenAnalytics client, saved connection state, tracker fragments,
 and native admin page.

demo/README.md

@@ -0,0 +1,9 @@
+# EmDash OpenAnalytics screenshot demo
+
+Run `pnpm screenshot` from the repository root. It builds `dist/`, starts a local OpenAnalytics fixture and an actual EmDash Astro admin using the packaged plugin descriptor, authenticates through EmDash's development-only bypass, configures the fixture connection through the protected settings API, validates it, and captures three screenshots under `docs/screenshots/`.
+
+The fixture and EmDash app bind to `127.0.0.1` on reserved ephemeral ports. Their readiness routes require a per-run random token, and EmDash also refuses to start if its configured port is occupied. The fixture accepts only its synthetic `oa_sk_demo_fixture_only_…` key. The EmDash demo database lives in a newly created temporary directory and is removed on exit. Captures are staged in another temporary directory; before they are copied into the repository, the harness scans rendered DOM text, page HTML, text/JSON browser responses, process output, and PNG bytes for private-key-shaped values and the fixture key. This environment has no OCR tool, so the screenshot check supplements the scan of the exact DOM used to take each image.
+
+The fixture keeps totals, comparison values, report rows, and chart shape fixed. It echoes the selected request range and timezone in response metadata and verifies overview, timeseries, pages, and sources all receive one matching interval. Request bounds and connection-validation time follow the current system clock; no production code is patched to freeze time. The chart generates points within the selected window. This gives stable content and layout while preserving the real admin range behavior.
+
+EmDash's `/_emdash/api/setup/dev-bypass` is used only on the loopback-bound Astro development server. EmDash guards this route behind `import.meta.env.DEV`. No dev bypass is exposed by the screenshot harness on a non-loopback address.

demo/astro.config.mjs

@@ -0,0 +1,34 @@
+import { fileURLToPath } from "node:url";
+
+import node from "@astrojs/node";
+import react from "@astrojs/react";
+import { defineConfig } from "astro/config";
+import emdash from "emdash/astro";
+import { sqlite } from "emdash/db";
+
+import { openAnalytics } from "../dist/index.mjs";
+
+const descriptor = openAnalytics();
+
+export default defineConfig({
+	output: "server",
+	adapter: node({ mode: "standalone" }),
+	devToolbar: { enabled: false },
+	server: {
+		host: "127.0.0.1",
+		port: Number(process.env.EMDASH_DEMO_PORT ?? 4390),
+		strictPort: true,
+	},
+	integrations: [
+		react(),
+		emdash({
+			database: sqlite({ url: process.env.EMDASH_DEMO_DB_URL ?? "file:./data.db" }),
+			plugins: [
+				{
+					...descriptor,
+					entrypoint: fileURLToPath(new URL("../dist/index.mjs", import.meta.url)),
+				},
+			],
+		}),
+	],
+});

demo/src/pages/demo-health.ts

@@ -0,0 +1,12 @@
+import type { APIRoute } from "astro";
+
+export const prerender = false;
+
+export const GET: APIRoute = ({ request }) => {
+	const expected = process.env.EMDASH_DEMO_RUN_ID;
+	const supplied = new URL(request.url).searchParams.get("run");
+	if (!expected || supplied !== expected) return new Response("Not found", { status: 404 });
+	return new Response(JSON.stringify({ run: expected }), {
+		headers: { "Content-Type": "application/json" },
+	});
+};

demo/tsconfig.json

@@ -0,0 +1,3 @@
+{
+	"extends": "astro/tsconfigs/strict"
+}

docs/implementation-footprint.md

@@ -7,18 +7,25 @@ counts include `src/**/*.ts` for EmDash and `nodes/**/*.ts` plus
 fixtures/helpers. Lockfiles, generated code, docs, CI, and package tooling are
 excluded. These are size comparisons, not runtime performance measurements.
 
-| Implementation                | Production LOC | Test LOC | Production modules | OpenAnalytics HTTP endpoints |
-| ----------------------------- | -------------: | -------: | -----------------: | ---------------------------: |
-| EmDash scaffold, PR #1        |            495 |      721 |                  9 |                            1 |
-| EmDash native overview, PR #2 |           1458 |     1923 |                 10 |                            3 |
-| n8n OpenAnalytics 0.1.1       |            607 |      613 |                  5 |                           11 |
+| Implementation                  | Production LOC | Test LOC | Production modules | OpenAnalytics HTTP endpoints |
+| ------------------------------- | -------------: | -------: | -----------------: | ---------------------------: |
+| EmDash scaffold, PR #1          |            495 |      721 |                  9 |                            1 |
+| EmDash native overview, PR #2   |           1458 |     1923 |                 10 |                            3 |
+| EmDash pages and sources, PR #3 |           1770 |     2423 |                 14 |                            5 |
+| n8n OpenAnalytics 0.1.1         |            607 |      613 |                  5 |                           11 |
 
 PR #2 adds **963 production lines** and **1202 test lines**
 over the scaffold. The scaffold tree at `a9e69a2` is identical to the original
 `c7c8efe` tree. The comparison uses n8n commit
 `8caaa20c6385f6fba1fcf2e8c2dbec0a1ac5efb1` from the local
 `@blackswampai/n8n-nodes-openanalytics` checkout.
 
+PR #3 adds **312 production lines** and **500 test lines** over merged PR #2
+(`2865caa`). It supports site metadata, overview, timeseries, pages, and sources.
+The admin coordinator delegates range handling, connection blocks, overview/chart
+rendering, and report tables to four small modules. Successful sections remain
+visible when another read fails.
+
 The EmDash plugin now owns a native admin page, public tracker insertion,
 credential-bound installation snapshots, secure read transport, response
 validation/projection, and useful connection/error/freshness states. The n8n
@@ -28,13 +35,15 @@ The size difference reflects those different responsibilities.
 
 EmDash production modules remain narrowly scoped: plugin/descriptor, connection
 state, configuration/settings, OpenAnalytics transport/errors/types, tracker
-fragment, and one admin page. PR #2 adds no production dependencies, custom
+fragment, and native admin modules. PR #3 adds no production dependencies, custom
 browser bundle, chart library, application framework, polling, or shared cache.
 `@emdash-cms/blocks@1.0.1` is a development dependency for type-only authoring and
 upstream response validation in tests; EmDash provides its renderer at runtime.
 
-Analytics use exactly two HTTP reads per requested overview: overview (including
-its server-provided preceding-period totals) and timeseries. Revalidation adds
+Analytics use exactly four HTTP reads per interaction: overview (including
+its server-provided preceding-period totals), timeseries, pages, and sources.
+All reuse one requested interval and timezone. Pages and sources each request
+the first ten rows ranked by views. Revalidation adds
 one site read. Public page rendering adds no read-key requests. A transient
 revalidation failure returns its error and preserves a matching tracker
 snapshot without starting additional analytics requests.
@@ -43,3 +52,27 @@ To reproduce the line counts, enumerate the directories above, include only
 `.ts` files, and sum their physical lines. For baseline comparison, read those
 same files from `git show a9e69a2:<path>`; for n8n, read files at the pinned commit.
 Apply the same formatting/measurement convention in future PRs.
+
+| Implementation          | Runtime dependencies | Development dependencies | Host peer      |
+| ----------------------- | -------------------: | -----------------------: | -------------- |
+| EmDash PR #2            |                    0 |                        8 | `emdash`       |
+| EmDash PR #3            |                    0 |                       14 | `emdash`       |
+| n8n OpenAnalytics 0.1.1 |                    0 |                        7 | `n8n-workflow` |
+
+PR #3 adds only development tooling: Astro, its Node and React adapters and React peers
+for EmDash's host admin, and Playwright. The plugin still uses native Block Kit
+and has no browser entrypoint. The React dependencies serve EmDash itself in
+the demo; they add no React application to the plugin. The demo runs an actual
+EmDash instance with an isolated temporary SQLite database and a loopback
+OpenAnalytics HTTP fixture. Screenshots come from its authenticated admin page
+and are checked for credential exposure before being retained. Demo code and
+screenshots are excluded from npm.
+
+Package sizes are measured with `npm pack --ignore-scripts --dry-run --json`
+after building; packed size is the compressed tarball, not dependency install
+size. EmDash PR #3 packs to **about 23.7 kB** (decimal), compared with
+**19.3 kB** for PR #2. The existing n8n checkout at the pinned commit packs to
+**8,919 bytes**.
+Its package includes compiled workflow operations, source maps, and icons; the
+EmDash package includes the compiled plugin, declarations, README, and two
+contract/footprint documents. Package size reflects those different contents.

docs/upstream-contracts.md

@@ -51,9 +51,10 @@ The published `@emdash-cms/blocks@1.0.1` types define headers, fields, actions,
 selects/buttons, stats, banners, context, and a native timeseries chart with
 `config.chart_type: "timeseries"` and `[timestamp_ms, value]` points. The host
 owns chart rendering, typography, spacing, navigation, and initial loading.
-Forms, tables (including badge cells), and tabs are available but unnecessary
-for this page. There is no standalone status badge or plugin-owned loading
-block; banners/fields represent connection state.
+Forms, tables (including badge cells), and tabs are available; this page now
+uses native tables for Top Pages and Traffic Sources. There is no standalone
+status badge or plugin-owned loading block; banners/fields represent connection
+state.
 
 Only Block Kit **types** are imported in production; the blocks package is a
 development dependency and its React/chart renderer is not bundled. Tests use
@@ -145,12 +146,83 @@ and [read-key route implementation](https://github.com/OpenLabs-so/openanalytics
   can return HTTP 400.
 
 The CMS guide recommends reads on actual admin use and warns against sharing a
-read-key response cache across administrators. Each overview interaction makes
-only two analytics reads; validation explicitly adds one site read. No polling,
-automatic retries, or extra read endpoints are introduced.
+read-key response cache across administrators. Each analytics page interaction
+reads overview, timeseries, pages, and sources; validation adds one site read.
+No polling or automatic retries are introduced.
 
 The native timeseries chart has no timezone formatting option. OpenAnalytics
 aligns the returned buckets to the configured timezone, and the page formats
 range/freshness text in that timezone, but native chart tick/tooltips use the
 administrator's browser timezone. The UI discloses this and uses a neutral axis
 label; timestamps are never shifted to fake timezone formatting.
+
+## Top pages and traffic sources
+
+Verified again against OpenAnalytics `main` at
+[`f7fc9169f32d48e55eb9106bceae9e87b6aa6bb9`](https://github.com/OpenLabs-so/openanalytics/tree/f7fc9169f32d48e55eb9106bceae9e87b6aa6bb9)
+on 2026-09-29; `git ls-remote origin refs/heads/main` matched this SHA.
+
+The plugin uses these private read-key operations:
+
+```http
+GET /v1/read/analytics/pages?from=...&to=...&timezone=...&limit=10&sort=views
+GET /v1/read/analytics/sources?from=...&to=...&timezone=...&limit=10
+```
+
+Both require `from`, `to`, and `timezone`. The instants are ISO-8601 UTC and
+form a half-open interval: `from` is included and `to` is excluded. `timezone`
+is an IANA timezone used to interpret calendar boundaries. `limit` is optional
+and accepts integers 1 through 500 (default 100); this plugin requests 10 rows.
+Pages also accepts `sort` (`views`, `entrances`, or `exits`, default `views`)
+and both endpoints accept optional session-scoped `filters`. The site selector
+header is not sent with a private site-bound key; it is for OAuth credentials.
+
+Pages returns `{ meta, items }`. A row requires `page_path`, `views`,
+`visitors`, `entrances`, `exits`, `bounces`, and `bounce_rate`. Counts are
+nonnegative integers. Session measures may be null: null means the metric was
+not measured, the session-decoration read did not cover that path, or the row
+is imported; zero means the server measured zero. `entrances` counts sessions
+that began on a path, `exits` sessions ending there, and `bounces` unengaged
+sessions counted on their entry path. `bounce_rate` is `bounces / entrances`
+and is null when no denominator was measured. Pages are ranked and cut by the
+server according to `sort`; re-sorting a views-limited response in the client
+would misrepresent the top-N result.
+
+Sources returns `{ meta, items }`. A row requires the string tuple
+`referrer_domain`, `utm_source`, `utm_medium`, and `utm_campaign`, plus
+nonnegative integer `views` and `visitors`. The canonical external referrer is
+a lowercase host without `www`, port, or path. Empty `referrer_domain` means
+Direct and also includes internal navigation; historical stored spellings were
+not rewritten. UTM fields may be empty. Rows represent attribution tuples, not
+sessions, and the report is ranked/cut by views.
+
+Both responses require metadata describing requested and effective ranges,
+timezone, resolution, data sources, accuracy, freshness, comparison range,
+truncation, cache state, and partial state. For these reports
+`comparison_range` is null. Freshness state distinguishes `no_data`, `ok`,
+`stale`, and `degraded`. Reports can merge live events with imported provider
+data; metadata identifies sources and whether results are exact,
+provider-defined, or estimated. A legitimate empty result is an empty `items`
+array, not an error.
+
+The documented HTTP responses are 400 for invalid/unservable ranges, 401 for
+missing/invalid authentication, 403 for missing analytics scope or suspended
+site, 404 for unknown site, 429 for rate limiting, and 503 when analytics
+storage is unavailable. Error envelopes carry stable codes, including
+`FORBIDDEN`, `SITE_SUSPENDED`, `RANGE_TOO_LARGE`, and
+`SERVICE_UNAVAILABLE`; clients should not display response messages as trusted
+content. A suspended site closes analytics reads with `SITE_SUSPENDED`;
+tracker installation state is a separate concern. Hosted deployments may
+have a limited billing-grace period for ingest, governed by the collector's
+admission policy.
+
+Primary source locations in that pinned revision:
+
+- `packages/contracts/openapi/openapi.yaml:1529-1594` — paths, auth and HTTP responses.
+- `packages/contracts/openapi/openapi.yaml:8468-8505` — analytics metadata and freshness.
+- `packages/contracts/openapi/openapi.yaml:8777-8889` — pages and sources row schemas.
+- `packages/contracts/openapi/openapi.yaml:12030-12079,12152-12161,12218-12237` — site selector, range/timezone, limit and pages sort.
+- `apps/api/src/http/read-key.ts:896-920` — private-key pages read and session decoration.
+- `apps/api/src/analytics/service.ts:1036-1072,1168-1180` — pages session decoration/ranking and sources view ranking.
+- `apps/api/src/http/middleware.ts:183-197` and `apps/api/src/http/read-key.ts:600-608` — suspended-site analytics gate.
+- `apps/tracker/src/core.ts:81-90` and `apps/collector/src/ingest-config-store.ts:215-270` — tracker stand-down and collector admission during suspension.