Skip to content

docs: overhaul README into a presentational open-source landing doc #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Rayan9064 merged 1 commit into from
Aug 7, 2026
+122 −59
Merged

docs: overhaul README into a presentational open-source landing doc #134

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
181 changes: 122 additions & 59 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,84 @@
# DumpIt — Your AI Second Brain
<div align="center">

DumpIt is an AI-powered knowledge vault for saved links, plain-text notes, and PDF documents. Save useful resources, organize them into collections, and ask questions across your private vault plus public shared resources with cited source cards.
<img src="public/logo-with-text.png" alt="DumpIt" height="72" />

Built with **Next.js 14 (App Router)**, **TypeScript**, **Tailwind CSS**, **Firebase Auth & Firestore**, **Firebase Admin SDK**, **Google Gemini AI (RAG & Embeddings)**, **@sentry/nextjs**, and **@upstash/ratelimit**.
### Save anything. Ask anything. Get answers from your own knowledge — not a stranger's AI.

---
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
[![CI](https://github.com/Rayan9064/dumpit/actions/workflows/ci.yml/badge.svg)](https://github.com/Rayan9064/dumpit/actions/workflows/ci.yml)
[![Next.js 14](https://img.shields.io/badge/Next.js-14-black?logo=next.js)](https://nextjs.org)
[![TypeScript](https://img.shields.io/badge/TypeScript-strict-blue?logo=typescript)](https://www.typescriptlang.org)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)

## Features
[**Live app**](https://www.dumpit.page) · [Contributing](CONTRIBUTING.md) · [Code of Conduct](CODE_OF_CONDUCT.md) · [Docs](docs/)

- **Multi-Format Capture:**
- **Links:** Auto-enrichment of titles, descriptions, and tags via web scraping.
- **Notes:** Plain-text ideas, code snippets, and structured thoughts.
- **PDF Documents:** Fast in-memory text extraction for PDF uploads up to 10MB.
- **AI Search & RAG (Ask DumpIt):**
- `My Dump`: Query your private indexed vault.
- `Shared`: Discover public resources saved by the community.
- `All`: Search across your vault plus community shared resources.
- Answers include exact citations and source cards.
- **Organization & Curation:**
- Collections, tags, search filtering, and custom public profiles (`/u/[username]`).
- Cursor-based pagination on dashboard for high performance at scale.
- Skeleton shimmer card loading states.
- Duplicate resource detection.
- **Enterprise Infrastructure & Performance:**
- Sentry exception monitoring across Client, Server, and Edge runtimes.
- Upstash Redis API rate limiting (60 req/min auth, 20 req/min public).
- PostHog telemetry & product analytics.
- Dynamic SEO generation via Next.js `robots.ts` and dynamic `sitemap.ts`.
- Browser extension support (Chrome Extension).
</div>

---

## How RAG Works
DumpIt is an open-source, AI-powered knowledge vault. Save links, notes, and PDFs from wherever you already are — the web app, the browser extension, email, or your phone's share sheet — then ask questions in plain English and get cited answers grounded in what you actually saved, not hallucinated from general knowledge.

## Table of contents

- [Features](#features)
- [Platform](#platform)
- [How RAG works](#how-rag-works)
- [Tech stack](#tech-stack)
- [Repository structure](#repository-structure)
- [Quick start](#quick-start)
- [Environment variables](#environment-variables)
- [Firestore vector indexes](#firestore-vector-indexes)
- [Commands](#commands)
- [Documentation](#documentation)
- [Contributing](#contributing)

## Features

- **Capture from anywhere**
- Links — auto-enriched titles, descriptions, and tags via server-side scraping.
- Notes — plain-text ideas, code snippets, structured thoughts.
- PDFs — in-memory text extraction for uploads up to 10MB.
- Browser extension — one-click save, side-panel review, and text-selection capture (Chrome, Manifest V3).
- Mobile share sheet — share directly into DumpIt from any Android app (PWA, [Web Share Target](https://developer.mozilla.org/en-US/docs/Web/Manifest/Reference/share_target)).
- Email-to-save — forward anything to `save@dumpit.page` (built; pending DNS/Resend dashboard setup on our end before it's reachable).
- **AI search & cited Q&A**
- `My Dump` / `Shared` / `All` search scopes across your private vault and community-shared resources.
- Answers include inline citations and source cards you can verify.
- **Organization**
- Collections, tags, filtering, cursor-based pagination, duplicate detection.
- Public profiles at `/u/[username]` with per-resource visibility control.
- **Developer access**
- REST API secured by long-lived API keys (generate/revoke from Settings).
- [MCP server](dumpit-mcp/) — query and save to your vault directly from Claude Desktop or Cursor.
- **Infrastructure**
- Sentry error monitoring (client, server, edge).
- Upstash Redis rate limiting on authenticated, public, and AI-query routes.
- Dodo Payments (Merchant of Record — handles global tax compliance automatically).

## Platform

| Surface | What it's for |
|---|---|
| **Web app** | Full dashboard — search, capture, collections, profile, settings. |
| **[Browser extension](dumpit-extension/)** | One-click save from any tab without leaving the browser. |
| **REST API** | Programmatic add/search/ask, authenticated with an API key from Settings. |
| **[MCP server](dumpit-mcp/)** | `ask_vault`, `search_vault`, `save_to_vault` tools for Claude Desktop / Cursor. |
| **Mobile share sheet** | Share a link straight into your vault from Android's native share menu. |

## How RAG works

Saving a resource creates a `resources` document in Firestore. AI search relies on server-side background indexing:

1. **Extraction:**
- **For Links:** Fetches page content and extracts readable text.
- **For PDFs:** Parses PDF binary in memory via `pdf-parse` and extracts plain text into `captured_text`.
- **For Notes:** Uses the note content directly.
2. **Chunking & Embedding:**
- Splits text into contextual chunks.
- Generates 768-dimensional vector embeddings using Google's Gemini Embedding API.
3. **Storage & Search:**
- Stores vectorized chunks in Firestore `resource_chunks`.
- Executes vector similarity searches against user queries.
1. **Extraction**
- Links — fetches page content and extracts readable text.
- PDFs — parses the binary in memory via `pdf-parse`.
- Notes / emails — uses the content directly.
2. **Chunking & embedding** — splits text into contextual chunks, generates 768-dimensional vectors via Gemini's embedding model.
3. **Storage & search** — stores vectors in Firestore `resource_chunks`, runs vector similarity search against user queries, and returns an answer with inline citations.

```mermaid
flowchart TD
Input["Link / Note / PDF"] --> Extract["Extract Text (Fetch / pdf-parse)"]
Input["Link / Note / PDF / Email"] --> Extract["Extract Text (Fetch / pdf-parse)"]
Extract --> Chunk["Chunk Text"]
Chunk --> Embed["Gemini Embedding (768d)"]
Embed --> Store["Firestore resource_chunks"]
Expand All @@ -58,9 +88,40 @@ flowchart TD
Vector --> Answer["Gemini Answer with Citations"]
```

---
## Tech stack

## Quick Start
| Layer | What |
|---|---|
| Framework | Next.js 14 (App Router), TypeScript |
| Styling | Tailwind CSS — Zinc palette, Space Grotesk + Inter |
| Auth | Firebase Auth (Google sign-in only) |
| Database | Firebase Admin SDK + Firestore (incl. native vector search) |
| AI | Google Gemini — embeddings & generation |
| Rate limiting | Upstash Redis |
| Error tracking | Sentry (client, server, edge) |
| Email | Resend (inbound save-by-email) |
| Payments | Dodo Payments (Merchant of Record) |
| Deployment | Vercel (web app), Firebase (backend/db) |

## Repository structure

```
app/ Next.js App Router pages and API routes
api/ Server-side route handlers
resources/ CRUD for saved items (links, notes, PDFs)
ai/ Search + RAG "ask" endpoints
webhooks/ Dodo Payments + Resend inbound-email webhooks
settings/api-keys/ API key generate/list/revoke
share/ Mobile share-sheet capture page
u/[username]/ Public user profile pages
dumpit-extension/ Chrome extension (Manifest V3, separate package)
dumpit-mcp/ MCP server for Claude Desktop / Cursor (separate package)
docs/ Internal documentation
public/ Static assets, PWA manifest
types/ Shared TypeScript types
```

## Quick start

```bash
git clone https://github.com/Rayan9064/dumpit.git
Expand All @@ -72,20 +133,14 @@ npm run dev

Open [http://localhost:3000](http://localhost:3000).

---

## Chrome Extension
To try the browser extension or MCP server locally, see their own READMEs: [`dumpit-extension/README.md`](dumpit-extension/README.md) · [`dumpit-mcp/README.md`](dumpit-mcp/README.md).

DumpIt includes a Chrome extension for one-click link capturing, side-panel search, and text selection clipping. For setup instructions, see the [Extension README](dumpit-extension/README.md).
## Environment variables

---

## Environment Variables

Configure your environment variables in `.env.local` (and in Vercel for production deployments):
Configure in `.env.local` (and in Vercel for production):

```env
# Firebase Client SDK (browser-safe)
# Firebase client SDK (browser-safe)
NEXT_PUBLIC_FIREBASE_API_KEY=
NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN=
NEXT_PUBLIC_FIREBASE_PROJECT_ID=
Expand All @@ -105,19 +160,27 @@ GEMINI_MODEL=gemini-2.5-flash
GEMINI_EMBEDDING_MODEL=gemini-embedding-001

# App URL & SEO
NEXT_PUBLIC_APP_URL=https://dumpit-three.vercel.app
NEXT_PUBLIC_APP_URL=https://www.dumpit.page

# Monitoring & Rate Limiting (Optional)
NEXT_PUBLIC_SENTRY_DSN=
# Rate limiting
UPSTASH_REDIS_REST_URL=
UPSTASH_REDIS_REST_TOKEN=

# Monitoring & analytics (optional)
NEXT_PUBLIC_SENTRY_DSN=
NEXT_PUBLIC_POSTHOG_KEY=
NEXT_PUBLIC_POSTHOG_HOST=
```

---
# Resend (email-to-save)
RESEND_API_KEY=
RESEND_INBOUND_WEBHOOK_SECRET=

## Firestore Vector Indexes
# Dodo Payments
DODO_PAYMENTS_API_KEY=
DODO_WEBHOOK_SECRET=
```

## Firestore vector indexes

Ask DumpIt requires Firestore vector indexes for semantic search:

Expand All @@ -140,8 +203,6 @@ gcloud firestore indexes composite create \
--field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding
```

---

## Commands

```bash
Expand All @@ -151,8 +212,6 @@ npm run build # Build production bundle
npm test # Run Vitest unit tests
```

---

## Documentation

- [Deployment Guide](docs/deployment.md)
Expand All @@ -161,3 +220,7 @@ npm test # Run Vitest unit tests
- [API Spec](docs/api-spec.md)
- [Testing Guide](docs/testing.md)
- [Firebase Setup](FIREBASE_SETUP.md)

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for setup and workflow, and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for community guidelines. DumpIt is [MIT licensed](LICENSE.md).
Loading