diff --git a/.env.example b/.env.example index f6bf24d..beea05c 100644 --- a/.env.example +++ b/.env.example @@ -13,6 +13,13 @@ GOOGLE_CLIENT_ID= GOOGLE_CLIENT_SECRET= GITHUB_CLIENT_ID= GITHUB_CLIENT_SECRET= +# Optional GitHub App for connecting repositories to the knowledge base. +GITHUB_APP_ID= +GITHUB_APP_SLUG= +GITHUB_APP_CLIENT_ID= +GITHUB_APP_CLIENT_SECRET= +GITHUB_APP_PRIVATE_KEY= +GITHUB_APP_WEBHOOK_SECRET= # Email RESEND_API_KEY= diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d52b96..6b3f26e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,21 @@ All notable changes to Supo will be documented in this file. ## Unreleased +### GitHub App for the knowledge base + +- **Connect GitHub:** a read-only GitHub App replaces pasting a public repository URL. Private repositories work, and the workspace picks repositories from a list. + - The installing GitHub user is verified through OAuth before an installation is linked, so an installation ID in the callback URL can't be forged. + - The connect `state` is signed, expires after 30 minutes and is bound to the Supo user who started it. +- **What is indexed:** README, changelog and documentation files at any depth (Markdown, MDX, reStructuredText, AsciiDoc), up to 400 files, using one git tree request instead of walking folders. Front matter titles are used, and MDX imports are dropped. + - Release notes (default on) and the 200 most recently updated issues (opt-in). + - Optional branch, include folders and skip folders. + - Source code, tests, dependencies and repository plumbing are skipped. +- **Updates:** a signed webhook (`/api/webhooks/github`) re-syncs on pushes to the tracked branch, releases and issue changes. + - Uninstalling the app, suspending it or removing repository access takes the source out of retrieval. Restoring access re-indexes it. + - Renames and visibility changes are followed. +- **Privacy:** answers from private repositories never link customers to the files. +- **Indexing:** unchanged documents reuse their stored embeddings. A sync only re-embeds the files or pages that changed. This applies to every source type. + ### Review follow-ups (2) - **Job queue:** diff --git a/README.md b/README.md index 98782db..6ebfd7c 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,9 @@ drizzle/ SQL migrations ## Knowledge Base -**Sources:** articles (markdown), single pages, whole websites (crawled via `robots.txt` / `sitemap.xml`, sitemap indexes, or by following links under the start URL; up to 200 pages), GitHub repositories (README and `docs/`), and approved answers learned from conversations. URL, website and GitHub sources re-sync daily and are only re-embedded when their content changes. +**Sources:** articles (markdown), single pages, whole websites (crawled via `robots.txt` / `sitemap.xml`, sitemap indexes, or by following links under the start URL; up to 200 pages), GitHub repositories, and approved answers learned from conversations. URL, website and GitHub sources re-sync daily and are only re-embedded when their content changes; unchanged pages and files reuse their stored embeddings. + +**GitHub App:** repositories connected through the Supo GitHub App (read-only, private repositories included) index README, changelog and documentation files anywhere in the repository (Markdown, MDX, reStructuredText, AsciiDoc), plus release notes and optionally recent issues. Source code, tests and dependencies are skipped. Pushes to the tracked branch, releases and issue changes re-index through the GitHub webhook, and only changed files are re-embedded. Answers from private repositories never link customers to the files. Public repositories can still be added by URL without the app. **Pipeline** (`src/lib/knowledge/*`): @@ -174,7 +176,8 @@ Paid WAF, CAPTCHA, bot-management products, and domain-based email hardening are - Neon Postgres database - Upstash Redis database - Google OAuth app -- GitHub OAuth app +- GitHub OAuth app (sign-in) +- GitHub App (optional, connects repositories to the knowledge base) - Google Gemini API key - Resend API key - Ably API key for realtime features @@ -190,6 +193,7 @@ Notes: - `BETTER_AUTH_ADMIN_USER_IDS` should contain only the trusted owner/admin Better Auth user id. - Upstash Redis is required in production. Local development fails open when Redis env vars are absent. - The email channel is optional. Set `INBOUND_EMAIL_DOMAIN` (a receiving domain in Resend) and `RESEND_WEBHOOK_SECRET` (the signing secret of a Resend webhook for `email.received` pointed at `/api/inbound/email`). +- The GitHub App is optional. Set `GITHUB_APP_ID`, `GITHUB_APP_SLUG`, `GITHUB_APP_CLIENT_ID`, `GITHUB_APP_CLIENT_SECRET`, `GITHUB_APP_PRIVATE_KEY` (the PEM, multi-line or with `\n` escapes) and `GITHUB_APP_WEBHOOK_SECRET`. Configure the app with callback URL `/api/integrations/github/callback` and "Request user authorization (OAuth) during installation" enabled, webhook URL `/api/webhooks/github`, read-only Contents, Metadata and Issues permissions, and the Push, Release, Repository and Issues events. It is separate from the GitHub OAuth app used for sign-in. - `ABLY_API_KEY` enables realtime inbox/widget updates. The widget keeps polling as a fallback if Ably is absent. - `CRON_SECRET` authorizes Vercel Cron and internal job routes. Vercel sends it as `Authorization: Bearer $CRON_SECRET` automatically when the variable is set on the project. - `DATABASE_SSL=disable` (or `sslmode=disable` in the URL) turns off TLS for a local Postgres. diff --git a/drizzle/0028_github_app.sql b/drizzle/0028_github_app.sql new file mode 100644 index 0000000..0d51d6b --- /dev/null +++ b/drizzle/0028_github_app.sql @@ -0,0 +1,24 @@ +-- GitHub App installations linked to workspaces, and GitHub App knowledge sources. +CREATE TABLE IF NOT EXISTS "github_installation" ( + "id" text PRIMARY KEY NOT NULL, + "organization_id" text NOT NULL REFERENCES "organization"("id") ON DELETE cascade, + "installation_id" bigint NOT NULL, + "account_login" text NOT NULL, + "account_type" text NOT NULL, + "account_avatar_url" text, + "repository_selection" text, + "suspended_at" timestamp, + "created_by_id" text REFERENCES "user"("id") ON DELETE set null, + "created_at" timestamp NOT NULL, + "updated_at" timestamp NOT NULL +); +--> statement-breakpoint +CREATE UNIQUE INDEX IF NOT EXISTS "github_installation_org_installation_idx" ON "github_installation" ("organization_id", "installation_id"); +--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "github_installation_installation_idx" ON "github_installation" ("installation_id"); +--> statement-breakpoint +ALTER TABLE "knowledge_source" ADD COLUMN IF NOT EXISTS "github_installation_id" bigint; +--> statement-breakpoint +ALTER TABLE "knowledge_source" ADD COLUMN IF NOT EXISTS "github_repo_id" bigint; +--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "knowledge_source_github_repo_idx" ON "knowledge_source" ("github_repo_id") WHERE "github_repo_id" IS NOT NULL; diff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.json index e311a92..2899ec9 100644 --- a/drizzle/meta/_journal.json +++ b/drizzle/meta/_journal.json @@ -197,6 +197,13 @@ "when": 1778631600000, "tag": "0027_add_widget_config_allowed_origins", "breakpoints": true + }, + { + "idx": 28, + "version": "7", + "when": 1778631660000, + "tag": "0028_github_app", + "breakpoints": true } ] } diff --git a/src/app/(app)/products/[id]/knowledge/AddSourceDialog.tsx b/src/app/(app)/products/[id]/knowledge/AddSourceDialog.tsx index 7a8cf4c..5faf648 100644 --- a/src/app/(app)/products/[id]/knowledge/AddSourceDialog.tsx +++ b/src/app/(app)/products/[id]/knowledge/AddSourceDialog.tsx @@ -1,7 +1,7 @@ "use client"; -import { useState, useTransition } from "react"; -import { useRouter } from "next/navigation"; +import { useEffect, useState, useTransition } from "react"; +import { usePathname, useRouter, useSearchParams } from "next/navigation"; import { toast } from "sonner"; import { Button } from "@/components/ui/button"; @@ -13,6 +13,17 @@ import { Textarea } from "@/components/ui/textarea"; import type { AddSourceInput } from "@/lib/knowledge/sources"; import { addSource, uploadSourceFile } from "./actions"; +import { GitHubSourceForm } from "./GitHubSourceForm"; + +// Results of the GitHub App connect round trip (see /api/integrations/github/callback). +const GITHUB_RESULTS: Record = { + connected: { ok: true, message: "GitHub connected — choose a repository to add" }, + cancelled: { ok: false, message: "GitHub authorization was cancelled" }, + unauthorized: { ok: false, message: "Your GitHub account can't access that installation" }, + "no-installations": { ok: false, message: "Install the GitHub App first, then try again" }, + "not-configured": { ok: false, message: "The GitHub App is not configured on this server" }, + error: { ok: false, message: "Could not connect GitHub. Please try again." }, +}; // "file" is parsed server-side and stored as an article source. type SourceType = AddSourceInput["type"] | "file"; @@ -31,8 +42,21 @@ const segmentInactive = "text-[color:var(--text-secondary)] hover:text-foregroun export function AddSourceDialog({ productId }: { productId: string }) { const router = useRouter(); - const [open, setOpen] = useState(false); - const [type, setType] = useState("article"); + const pathname = usePathname(); + const searchParams = useSearchParams(); + const githubResult = searchParams.get("github"); + // Back from GitHub: reopen the dialog on the GitHub tab to pick a repository. + const [returnedFromGitHub] = useState(() => githubResult === "connected" || githubResult === "no-installations"); + const [open, setOpen] = useState(returnedFromGitHub); + const [type, setType] = useState(returnedFromGitHub ? "github" : "article"); + + useEffect(() => { + if (!githubResult) return; + const result = GITHUB_RESULTS[githubResult]; + if (result?.ok) toast.success(result.message); + else if (result) toast.error(result.message); + router.replace(pathname); + }, [githubResult, pathname, router]); const [name, setName] = useState(""); const [url, setUrl] = useState(""); const [content, setContent] = useState(""); @@ -49,7 +73,7 @@ export function AddSourceDialog({ productId }: { productId: string }) { setFile(null); } - const needsUrl = type !== "article" && type !== "file"; + const needsUrl = type === "website" || type === "url"; function handleUpload() { if (!file) return; @@ -123,106 +147,111 @@ export function AddSourceDialog({ productId }: { productId: string }) { ))} -
- - setName(e.target.value)} + {type === "github" ? ( + { + setOpen(false); + reset(); + router.refresh(); + }} /> -
+ ) : ( + <> +
+ + setName(e.target.value)} + /> +
- {type === "file" && ( -
- - { - const next = e.target.files?.[0] ?? null; - setFile(next); - if (next && !name.trim()) setName(next.name.replace(/\.[a-z0-9]+$/i, "")); - }} - /> -

- PDF, Word (.docx), Markdown, text, CSV or HTML, up to 4 MB. Scanned PDFs without a text layer can't be read. -

-
- )} + {type === "file" && ( +
+ + { + const next = e.target.files?.[0] ?? null; + setFile(next); + if (next && !name.trim()) setName(next.name.replace(/\.[a-z0-9]+$/i, "")); + }} + /> +

+ PDF, Word (.docx), Markdown, text, CSV or HTML, up to 4 MB. Scanned PDFs without a text layer can't be read. +

+
+ )} - {type === "article" && ( -
- -