Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=
GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=
# Optional GitHub App for connecting repositories to the knowledge base.
GITHUB_APP_ID=
GITHUB_APP_SLUG=
GITHUB_APP_CLIENT_ID=
GITHUB_APP_CLIENT_SECRET=
GITHUB_APP_PRIVATE_KEY=
GITHUB_APP_WEBHOOK_SECRET=

# Email
RESEND_API_KEY=
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,21 @@ All notable changes to Supo will be documented in this file.

## Unreleased

### GitHub App for the knowledge base

- **Connect GitHub:** a read-only GitHub App replaces pasting a public repository URL. Private repositories work, and the workspace picks repositories from a list.
- The installing GitHub user is verified through OAuth before an installation is linked, so an installation ID in the callback URL can't be forged.
- The connect `state` is signed, expires after 30 minutes and is bound to the Supo user who started it.
- **What is indexed:** README, changelog and documentation files at any depth (Markdown, MDX, reStructuredText, AsciiDoc), up to 400 files, using one git tree request instead of walking folders. Front matter titles are used, and MDX imports are dropped.
- Release notes (default on) and the 200 most recently updated issues (opt-in).
- Optional branch, include folders and skip folders.
- Source code, tests, dependencies and repository plumbing are skipped.
- **Updates:** a signed webhook (`/api/webhooks/github`) re-syncs on pushes to the tracked branch, releases and issue changes.
- Uninstalling the app, suspending it or removing repository access takes the source out of retrieval. Restoring access re-indexes it.
- Renames and visibility changes are followed.
- **Privacy:** answers from private repositories never link customers to the files.
- **Indexing:** unchanged documents reuse their stored embeddings. A sync only re-embeds the files or pages that changed. This applies to every source type.

### Review follow-ups (2)

- **Job queue:**
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,9 @@ drizzle/ SQL migrations

## Knowledge Base

**Sources:** articles (markdown), single pages, whole websites (crawled via `robots.txt` / `sitemap.xml`, sitemap indexes, or by following links under the start URL; up to 200 pages), GitHub repositories (README and `docs/`), and approved answers learned from conversations. URL, website and GitHub sources re-sync daily and are only re-embedded when their content changes.
**Sources:** articles (markdown), single pages, whole websites (crawled via `robots.txt` / `sitemap.xml`, sitemap indexes, or by following links under the start URL; up to 200 pages), GitHub repositories, and approved answers learned from conversations. URL, website and GitHub sources re-sync daily and are only re-embedded when their content changes; unchanged pages and files reuse their stored embeddings.

**GitHub App:** repositories connected through the Supo GitHub App (read-only, private repositories included) index README, changelog and documentation files anywhere in the repository (Markdown, MDX, reStructuredText, AsciiDoc), plus release notes and optionally recent issues. Source code, tests and dependencies are skipped. Pushes to the tracked branch, releases and issue changes re-index through the GitHub webhook, and only changed files are re-embedded. Answers from private repositories never link customers to the files. Public repositories can still be added by URL without the app.

**Pipeline** (`src/lib/knowledge/*`):

Expand Down Expand Up @@ -174,7 +176,8 @@ Paid WAF, CAPTCHA, bot-management products, and domain-based email hardening are
- Neon Postgres database
- Upstash Redis database
- Google OAuth app
- GitHub OAuth app
- GitHub OAuth app (sign-in)
- GitHub App (optional, connects repositories to the knowledge base)
- Google Gemini API key
- Resend API key
- Ably API key for realtime features
Expand All @@ -190,6 +193,7 @@ Notes:
- `BETTER_AUTH_ADMIN_USER_IDS` should contain only the trusted owner/admin Better Auth user id.
- Upstash Redis is required in production. Local development fails open when Redis env vars are absent.
- The email channel is optional. Set `INBOUND_EMAIL_DOMAIN` (a receiving domain in Resend) and `RESEND_WEBHOOK_SECRET` (the signing secret of a Resend webhook for `email.received` pointed at `/api/inbound/email`).
- The GitHub App is optional. Set `GITHUB_APP_ID`, `GITHUB_APP_SLUG`, `GITHUB_APP_CLIENT_ID`, `GITHUB_APP_CLIENT_SECRET`, `GITHUB_APP_PRIVATE_KEY` (the PEM, multi-line or with `\n` escapes) and `GITHUB_APP_WEBHOOK_SECRET`. Configure the app with callback URL `/api/integrations/github/callback` and "Request user authorization (OAuth) during installation" enabled, webhook URL `/api/webhooks/github`, read-only Contents, Metadata and Issues permissions, and the Push, Release, Repository and Issues events. It is separate from the GitHub OAuth app used for sign-in.
- `ABLY_API_KEY` enables realtime inbox/widget updates. The widget keeps polling as a fallback if Ably is absent.
- `CRON_SECRET` authorizes Vercel Cron and internal job routes. Vercel sends it as `Authorization: Bearer $CRON_SECRET` automatically when the variable is set on the project.
- `DATABASE_SSL=disable` (or `sslmode=disable` in the URL) turns off TLS for a local Postgres.
Expand Down
24 changes: 24 additions & 0 deletions drizzle/0028_github_app.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
-- GitHub App installations linked to workspaces, and GitHub App knowledge sources.
CREATE TABLE IF NOT EXISTS "github_installation" (
"id" text PRIMARY KEY NOT NULL,
"organization_id" text NOT NULL REFERENCES "organization"("id") ON DELETE cascade,
"installation_id" bigint NOT NULL,
"account_login" text NOT NULL,
"account_type" text NOT NULL,
"account_avatar_url" text,
"repository_selection" text,
"suspended_at" timestamp,
"created_by_id" text REFERENCES "user"("id") ON DELETE set null,
"created_at" timestamp NOT NULL,
"updated_at" timestamp NOT NULL
);
--> statement-breakpoint
CREATE UNIQUE INDEX IF NOT EXISTS "github_installation_org_installation_idx" ON "github_installation" ("organization_id", "installation_id");
--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "github_installation_installation_idx" ON "github_installation" ("installation_id");
--> statement-breakpoint
ALTER TABLE "knowledge_source" ADD COLUMN IF NOT EXISTS "github_installation_id" bigint;
--> statement-breakpoint
ALTER TABLE "knowledge_source" ADD COLUMN IF NOT EXISTS "github_repo_id" bigint;
--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "knowledge_source_github_repo_idx" ON "knowledge_source" ("github_repo_id") WHERE "github_repo_id" IS NOT NULL;
7 changes: 7 additions & 0 deletions drizzle/meta/_journal.json
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,13 @@
"when": 1778631600000,
"tag": "0027_add_widget_config_allowed_origins",
"breakpoints": true
},
{
"idx": 28,
"version": "7",
"when": 1778631660000,
"tag": "0028_github_app",
"breakpoints": true
}
]
}
227 changes: 128 additions & 99 deletions src/app/(app)/products/[id]/knowledge/AddSourceDialog.tsx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"use client";

import { useState, useTransition } from "react";
import { useRouter } from "next/navigation";
import { useEffect, useState, useTransition } from "react";
import { usePathname, useRouter, useSearchParams } from "next/navigation";
import { toast } from "sonner";

import { Button } from "@/components/ui/button";
Expand All @@ -13,6 +13,17 @@ import { Textarea } from "@/components/ui/textarea";
import type { AddSourceInput } from "@/lib/knowledge/sources";

import { addSource, uploadSourceFile } from "./actions";
import { GitHubSourceForm } from "./GitHubSourceForm";

// Results of the GitHub App connect round trip (see /api/integrations/github/callback).
const GITHUB_RESULTS: Record<string, { ok: boolean; message: string }> = {
connected: { ok: true, message: "GitHub connected — choose a repository to add" },
cancelled: { ok: false, message: "GitHub authorization was cancelled" },
unauthorized: { ok: false, message: "Your GitHub account can't access that installation" },
"no-installations": { ok: false, message: "Install the GitHub App first, then try again" },
"not-configured": { ok: false, message: "The GitHub App is not configured on this server" },
error: { ok: false, message: "Could not connect GitHub. Please try again." },
};

// "file" is parsed server-side and stored as an article source.
type SourceType = AddSourceInput["type"] | "file";
Expand All @@ -31,8 +42,21 @@ const segmentInactive = "text-[color:var(--text-secondary)] hover:text-foregroun

export function AddSourceDialog({ productId }: { productId: string }) {
const router = useRouter();
const [open, setOpen] = useState(false);
const [type, setType] = useState<SourceType>("article");
const pathname = usePathname();
const searchParams = useSearchParams();
const githubResult = searchParams.get("github");
// Back from GitHub: reopen the dialog on the GitHub tab to pick a repository.
const [returnedFromGitHub] = useState(() => githubResult === "connected" || githubResult === "no-installations");
const [open, setOpen] = useState(returnedFromGitHub);
const [type, setType] = useState<SourceType>(returnedFromGitHub ? "github" : "article");

useEffect(() => {
if (!githubResult) return;
const result = GITHUB_RESULTS[githubResult];
if (result?.ok) toast.success(result.message);
else if (result) toast.error(result.message);
router.replace(pathname);
}, [githubResult, pathname, router]);
const [name, setName] = useState("");
const [url, setUrl] = useState("");
const [content, setContent] = useState("");
Expand All @@ -49,7 +73,7 @@ export function AddSourceDialog({ productId }: { productId: string }) {
setFile(null);
}

const needsUrl = type !== "article" && type !== "file";
const needsUrl = type === "website" || type === "url";

function handleUpload() {
if (!file) return;
Expand Down Expand Up @@ -123,106 +147,111 @@ export function AddSourceDialog({ productId }: { productId: string }) {
))}
</div>

<div className="space-y-1.5">
<Label htmlFor="source-name" className="text-xs text-[color:var(--text-secondary)]">Name</Label>
<Input
id="source-name"
placeholder={type === "website" ? "e.g. Help center" : "e.g. Getting started guide"}
value={name}
maxLength={200}
onChange={(e) => setName(e.target.value)}
{type === "github" ? (
<GitHubSourceForm
productId={productId}
onAdded={() => {
setOpen(false);
reset();
router.refresh();
}}
/>
</div>
) : (
<>
<div className="space-y-1.5">
<Label htmlFor="source-name" className="text-xs text-[color:var(--text-secondary)]">Name</Label>
<Input
id="source-name"
placeholder={type === "website" ? "e.g. Help center" : "e.g. Getting started guide"}
value={name}
maxLength={200}
onChange={(e) => setName(e.target.value)}
/>
</div>

{type === "file" && (
<div className="space-y-1.5">
<Label htmlFor="source-file" className="text-xs text-[color:var(--text-secondary)]">Document</Label>
<Input
id="source-file"
type="file"
accept=".pdf,.docx,.md,.markdown,.txt,.csv,.html,.htm"
onChange={(e) => {
const next = e.target.files?.[0] ?? null;
setFile(next);
if (next && !name.trim()) setName(next.name.replace(/\.[a-z0-9]+$/i, ""));
}}
/>
<p className="text-xs text-[color:var(--text-tertiary)]">
PDF, Word (.docx), Markdown, text, CSV or HTML, up to 4 MB. Scanned PDFs without a text layer can&apos;t be read.
</p>
</div>
)}
{type === "file" && (
<div className="space-y-1.5">
<Label htmlFor="source-file" className="text-xs text-[color:var(--text-secondary)]">Document</Label>
<Input
id="source-file"
type="file"
accept=".pdf,.docx,.md,.markdown,.txt,.csv,.html,.htm"
onChange={(e) => {
const next = e.target.files?.[0] ?? null;
setFile(next);
if (next && !name.trim()) setName(next.name.replace(/\.[a-z0-9]+$/i, ""));
}}
/>
<p className="text-xs text-[color:var(--text-tertiary)]">
PDF, Word (.docx), Markdown, text, CSV or HTML, up to 4 MB. Scanned PDFs without a text layer can&apos;t be read.
</p>
</div>
)}

{type === "article" && (
<div className="space-y-1.5">
<Label htmlFor="source-content" className="text-xs text-[color:var(--text-secondary)]">Content</Label>
<Textarea
id="source-content"
placeholder="Paste documentation, FAQs or policies. Markdown headings (# Title) are used to split sections."
value={content}
onChange={(e) => setContent(e.target.value)}
rows={8}
className="resize-y text-sm"
/>
</div>
)}
{type === "article" && (
<div className="space-y-1.5">
<Label htmlFor="source-content" className="text-xs text-[color:var(--text-secondary)]">Content</Label>
<Textarea
id="source-content"
placeholder="Paste documentation, FAQs or policies. Markdown headings (# Title) are used to split sections."
value={content}
onChange={(e) => setContent(e.target.value)}
rows={8}
className="resize-y text-sm"
/>
</div>
)}

{needsUrl && (
<div className="space-y-1.5">
<Label htmlFor="source-url" className="text-xs text-[color:var(--text-secondary)]">
{type === "github" ? "Repository URL" : type === "website" ? "Website or docs URL" : "Page URL"}
</Label>
<Input
id="source-url"
type="url"
placeholder={
type === "github"
? "https://github.com/owner/repo"
: type === "website"
? "https://docs.your-site.com"
: "https://your-site.com/help/page"
}
value={url}
onChange={(e) => setUrl(e.target.value)}
/>
<p className="text-xs text-[color:var(--text-tertiary)]">
{type === "website" &&
"Pages are discovered from robots.txt / sitemap.xml, or by following links under this URL's path. Re-synced daily."}
{type === "url" && "Only this page is indexed. Re-synced daily."}
{type === "github" &&
"Indexes the README and markdown files in docs/, doc/ or guides/. Public repositories only."}
</p>
</div>
)}
{needsUrl && (
<div className="space-y-1.5">
<Label htmlFor="source-url" className="text-xs text-[color:var(--text-secondary)]">
{type === "website" ? "Website or docs URL" : "Page URL"}
</Label>
<Input
id="source-url"
type="url"
placeholder={type === "website" ? "https://docs.your-site.com" : "https://your-site.com/help/page"}
value={url}
onChange={(e) => setUrl(e.target.value)}
/>
<p className="text-xs text-[color:var(--text-tertiary)]">
{type === "website" &&
"Pages are discovered from robots.txt / sitemap.xml, or by following links under this URL's path. Re-synced daily."}
{type === "url" && "Only this page is indexed. Re-synced daily."}
</p>
</div>
)}

{type === "website" && (
<div className="space-y-1.5">
<Label htmlFor="source-max-pages" className="text-xs text-[color:var(--text-secondary)]">
Maximum pages
</Label>
<Input
id="source-max-pages"
type="number"
min={1}
max={200}
value={maxPages}
onChange={(e) => setMaxPages(Number(e.target.value) || 1)}
/>
</div>
)}
{type === "website" && (
<div className="space-y-1.5">
<Label htmlFor="source-max-pages" className="text-xs text-[color:var(--text-secondary)]">
Maximum pages
</Label>
<Input
id="source-max-pages"
type="number"
min={1}
max={200}
value={maxPages}
onChange={(e) => setMaxPages(Number(e.target.value) || 1)}
/>
</div>
)}

<Button
onClick={handleSubmit}
disabled={
isPending ||
(type === "file" ? !file : !name.trim()) ||
(type === "article" && !content.trim()) ||
(needsUrl && !url.trim())
}
className="w-full rounded-lg bg-foreground text-xs text-background hover:bg-foreground/90"
>
{isPending ? "Adding…" : "Add & index"}
</Button>
<Button
onClick={handleSubmit}
disabled={
isPending ||
(type === "file" ? !file : !name.trim()) ||
(type === "article" && !content.trim()) ||
(needsUrl && !url.trim())
}
className="w-full rounded-lg bg-foreground text-xs text-background hover:bg-foreground/90"
>
{isPending ? "Adding…" : "Add & index"}
</Button>
</>
)}
</div>
</DialogContent>
</Dialog>
Expand Down
Loading
Loading