feat: track public Discord invite counts

README.md

@@ -101,6 +101,12 @@ The preflight is read-only and reports the export scope before any snapshot is w
 
 ## Automation
 
+For public server-size observations without a bot token, use
+[`metrics collect`](docs/commands/metrics.md) with a separate metrics database.
+It records approximate members and online presence; online presence does not
+measure active posters. `metrics import` preserves historical observations and
+`metrics status` reports their freshness.
+
 Discrawl exposes stable JSON for launchers, agents, and CI:
 
 ```bash

docs/README.md

@@ -20,6 +20,7 @@ Mirror Discord guilds into local SQLite. Search server history without depending
 - **Need DM search?** [`wiretap`](commands/wiretap.html) imports local Discord Desktop cache.
 - **Need multilingual lexical search?** Configure [language analyzers](guides/search-modes.html) and run [`lexical rebuild`](commands/lexical.html).
 - **Want semantic search?** Configure [Embeddings](guides/embeddings.html), then run [`embed`](commands/embed.html).
+- **Need public server-size history without a bot?** [`metrics`](commands/metrics.html) records approximate membership and online presence in a separate database.
 - **Wiring an agent or launcher?** `discrawl metadata --json`, `discrawl status --json`, `discrawl diagnostics --json`, `discrawl coverage --json`, `discrawl failures --json`, `discrawl remote status`, and `discrawl doctor --json` expose the read-only crawlkit control surface.
 
 ## At a glance

docs/commands/metrics.md

@@ -0,0 +1,187 @@
+# `metrics`
+
+Record public Discord server size and online-presence observations without a bot
+token. Metrics use an explicitly configured, separate SQLite database; these
+commands do not open the message/member archive, import Desktop data, or generate
+embeddings.
+
+## Usage
+
+```bash
+discrawl metrics collect --config /absolute/path/metrics.json
+discrawl metrics import --config /absolute/path/metrics.json < history.ndjson
+discrawl metrics status --config /absolute/path/metrics.json
+discrawl help metrics
+```
+
+The metrics `--config` is a JSON file supplied after the subcommand. It is separate
+from Discrawl's normal TOML archive configuration. All three commands output JSON.
+`status` is read-only and reports observation/event counts, the latest observation
+sequence, and the most recent observation time (`null` for an empty store).
+
+## Configuration
+
+```json
+{
+  "database": "/absolute/path/discrawl-metrics.sqlite",
+  "targets": [
+    {"entity": "openclaw", "target": "clawd"}
+  ]
+}
+```
+
+`entity` is your series label; `target` is a public invite code, not an invite URL
+or guild ID. Codes must be unique in the configuration. Other servers can be
+configured with their own invite codes.
+
+The database path must be absolute. `collect` and `import` create a new metrics
+database if that path does not exist. Existing databases must identify their
+owner as `discrawl` and metrics version as `1`; unrelated, empty, or newer-version
+databases are refused before write access. `status` never creates a database.
+
+No token or cookie is needed. The shared configuration fields `cookieJar` and
+`tokenEnv` are accepted but unused by this collector.
+
+## Standalone metrics runtime
+
+Metrics can use a dedicated, versioned copy of the normal Discrawl executable.
+Keep its path distinct from the executable used by an existing message collector.
+For example, a private per-user installation can use:
+
+```text
+~/.local/libexec/discrawl-metrics/<source-commit>/discrawl
+```
+
+Keep the runtime directories and executable private to the owning user (`0700`
+on Unix). Copy the validated artifact into a new version directory, preserve its
+bytes and signature, then verify the installed SHA-256 against the artifact
+receipt. On macOS, also verify the copied executable against the approved signing
+identity and designated requirement. Record the source commit, hash, and
+verification result with the installation. A locally signed build remains a
+local build; it is not an official notarized release.
+
+Use the full versioned executable path and explicit metrics JSON configuration:
+
+```bash
+metrics_runtime="$HOME/.local/libexec/discrawl-metrics/<source-commit>/discrawl"
+"$metrics_runtime" metrics status --config "$HOME/.config/discrawl/metrics.json"
+```
+
+Replace `<source-commit>` with the installed build's commit. Read-only `status`
+verifies access to the existing metrics store before cutover. Installing this
+runtime does not register schedules or run collection/imports. Those operations
+belong to the metrics deployment owner, which selects the exact executable path
+for each job. Existing message-collector binaries, command symlinks, settings,
+and processes remain independently managed.
+
+### Hourly scheduling on macOS
+
+A user LaunchAgent can invoke the versioned executable directly. Keep the plist
+private (`0600`) under `~/Library/LaunchAgents`, with private (`0700`) log
+directories and pre-created log files (`0600`). Use absolute executable,
+configuration, and log paths; plist strings do not expand `$HOME` or `~`.
+
+For an hourly job at minute 4, configure these launchd keys:
+
+| Key | Value |
+| --- | --- |
+| `Label` | A dedicated label, such as `org.example.discrawl-metrics` |
+| `ProgramArguments` | Versioned executable, `metrics`, `collect`, `--config`, absolute metrics config path |
+| `StartCalendarInterval` | Dictionary with integer `Minute` set to `4` |
+| `RunAtLoad` | `true` for one collection when the job is bootstrapped |
+| `KeepAlive` | `false`; provider failures must not trigger a restart loop |
+| `Umask` | Integer `63` (octal `077`) |
+| `StandardOutPath`, `StandardErrorPath` | Absolute paths to the private log files |
+
+Set `PATH=/usr/bin:/bin:/usr/sbin:/sbin` in `EnvironmentVariables`. Include
+`DISCRAWL_NO_AUTO_UPDATE=1` and `DISCRAWL_NO_UPDATE_CHECK=1`; metrics commands already
+bypass archive update hooks, and the job should retain its verified executable.
+Do not put tokens or cookies in the plist. Public invite metrics need neither.
+
+Validate the plist with `plutil -lint`, then bootstrap it in the owning user's
+GUI domain. `RunAtLoad` supplies the first run, so a second kickstart is not
+needed. Verify `launchctl print` retains `Minute = 4`, the expected arguments,
+and a successful exit. Confirm a current `metric_runs` row with `status = 'ok'`
+and non-NULL observations for every configured target's required metrics.
+
+Launchd runs one instance of a label at a time. Native SQLite transactions
+serialize database writes across independent connections; a competing write
+must wait or fail without inserting a partial batch. This is transaction-level
+write exclusion, not a mutex covering the preceding HTTP requests. Avoid adding
+another scheduler for the same metrics job.
+
+If macOS requests access to the configured volume, report the permission gate
+before claiming successful collection. Do not change identities, broaden access,
+or repeatedly restart a blocked job. Database ownership and private permissions
+remain required for scheduled operation.
+
+After owner-granted consent, recheck the original invocation read-only. Acceptance
+requires exit code `0`, a corresponding successful `metric_runs` row, and
+non-NULL required measurements. The hourly calendar definition must remain
+loaded. A completed batch job normally shows `state = not running` while it
+waits for its next scheduled time. Leave the accepted job loaded; verifying this
+state does not require another collection or kickstart.
+
+## What is measured
+
+Each collection makes one public `GET /api/v10/invites/{code}?with_counts=true`
+request per target and records two observations:
+
+| Metric | Discord field | Meaning |
+| --- | --- | --- |
+| `members` | `approximate_member_count` | Approximate total server membership |
+| `online` | `approximate_presence_count` | Approximate online presence |
+
+Both use `kind: "counter"` and `provenance: "discord_invite_approximate"`.
+Here, counter means a timestamped snapshot: either value can decrease. Online
+presence does **not** count active posters, messages, daily active users, or
+engagement. See Discord's [Invite object and Get Invite documentation](https://docs.discord.com/developers/resources/invite#get-invite).
+
+Zero is a valid count. Missing or invalid values, expired invites, HTTP failures,
+and rate limits produce SQL `NULL`, never a fabricated zero. Valid fields and
+other successful targets are retained. Such a run records `partial`, outputs
+`"ok": false`, and exits nonzero. Responses are bounded in size, requests have a
+30-second timeout, and redirects are refused. There is no automatic retry loop
+or scheduler; invoke `collect` at the cadence appropriate for your application.
+
+## Importing history
+
+`import` reads one JSON object per line from stdin. Every row needs a stable,
+nonempty `id` and an exact configured `entity`/`target` pair. Times use RFC 3339
+(fractional seconds and offsets are accepted). Unknown fields are rejected.
+
+```json
+{"type":"metric","id":"historical-members-1","entity":"openclaw","target":"clawd","metric":"members","kind":"counter","ts":"2026-09-14T12:00:00Z","value":1200,"observed_at":"2026-09-14T12:00:00Z","provenance":"historical-import"}
+{"type":"metric","id":"historical-online-1","entity":"openclaw","target":"clawd","metric":"online","kind":"counter","ts":"2026-09-14T12:00:00Z","value":null,"observed_at":"2026-09-14T12:00:00Z","provenance":"historical-import"}
+{"type":"event","id":"historical-event-1","entity":"openclaw","target":"clawd","kind":"milestone","ts":"2026-09-14T12:00:00Z","label":"Example milestone","url":"https://example.org/milestone","observed_at":"2026-09-14T12:00:00Z","provenance":"historical-import"}
+```
+
+Metric rows use `kind: "counter"` or `"daily"` and a nonnegative numeric value or
+`null`. Daily rows describe a completed UTC day. Revisions require different IDs:
+consumers select the latest sequence for a daily series/day instead of summing
+revisions. Import preserves supplied timestamps, provenance, repeated values,
+decreases, zeroes, NULLs, and events. Only an already stored ID is ignored.
+
+Imports commit in batches of 500. If a later line is invalid, the JSON result
+reports the number already committed and the command exits nonzero. Correct the
+input and replay it: stored IDs make retries idempotent. A line is limited to 4 MiB.
+
+## Storage and delivery
+
+The metrics database contains:
+
+- `metric_meta(key, value)`: collector owner and schema version.
+- `metric_observations(sequence, id, entity, target, metric, kind, ts, value, observed_at, provenance)`.
+- `metric_events(sequence, id, entity, target, kind, ts, label, url, observed_at, provenance)`.
+- `metric_runs(sequence, ts, status, rows_written)`: collection attempts.
+
+Each table's sequence advances independently; gaps are allowed. Consumers can
+open this database read-only and keep separate delivery cursors for observations
+and events. Import IDs are preserved; native collection derives stable IDs from
+the complete observation, including observation time. Repeated collections keep
+their own observations even when counts do not change.
+
+## See also
+
+- [`analytics`](analytics.html) for activity calculated from archived messages.
+- [Data storage](../guides/data-storage.html).

internal/cli/cli.go

@@ -15,6 +15,7 @@ import (
 	"github.com/openclaw/crawlkit/embed"
 	"github.com/openclaw/discrawl/internal/config"
 	"github.com/openclaw/discrawl/internal/discord"
+	"github.com/openclaw/discrawl/internal/headlinemetrics"
 	"github.com/openclaw/discrawl/internal/share"
 	"github.com/openclaw/discrawl/internal/store"
 	"github.com/openclaw/discrawl/internal/syncer"
@@ -73,6 +74,11 @@ func Run(ctx context.Context, args []string, stdout, stderr io.Writer) (runErr e
 		_, _ = io.WriteString(stdout, currentVersion()+"\n")
 		return nil
 	}
+	if rest[0] == "metrics" {
+		// Metrics have their own explicit config and database; do not initialize
+		// archive services, resolve credentials, or run archive update checks.
+		return headlinemetrics.Run(ctx, rest[1:], "discrawl", headlinemetrics.CollectDiscord, os.Stdin, stdout, stderr)
+	}
 	level := slog.LevelInfo
 	if global.Quiet {
 		level = slog.LevelError
@@ -133,6 +139,7 @@ var discrawlCommandSpecs = []discrawlCommandSpec{
 	{name: "messages", description: "List archived messages."},
 	{name: "digest", description: "Summarize recent archive activity."},
 	{name: "analytics", description: "Analyze archive activity and trends."},
+	{name: "metrics", description: "Collect public invite counts in a separate metrics database."},
 	{name: "dms", description: "List local Discord Desktop conversations."},
 	{name: "mentions", description: "List archived mentions."},
 	{name: "attachments", description: "List or fetch archived attachments."},

internal/cli/metrics_test.go

@@ -0,0 +1,38 @@
+package cli
+
+import (
+	"bytes"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/openclaw/discrawl/internal/headlinemetrics"
+	"github.com/stretchr/testify/require"
+)
+
+func TestMetricsHelpAndIndependentStatus(t *testing.T) {
+	for _, args := range [][]string{{"help", "metrics"}, {"help", "metrics", "collect"}, {"metrics"}, {"metrics", "collect", "--help"}, {"--json", "metrics", "status", "--help"}} {
+		var out bytes.Buffer
+		require.NoError(t, Run(t.Context(), args, &out, &bytes.Buffer{}))
+		require.Contains(t, out.String(), "Usage: discrawl metrics")
+	}
+	var out bytes.Buffer
+	require.NoError(t, Run(t.Context(), []string{"--help"}, &out, &bytes.Buffer{}))
+	require.Contains(t, out.String(), "metrics")
+	root := t.TempDir()
+	database := filepath.Join(root, "metrics.sqlite")
+	s, err := headlinemetrics.Open(t.Context(), database, "discrawl")
+	require.NoError(t, err)
+	require.NoError(t, s.Close())
+	config := filepath.Join(root, "metrics.json")
+	b, err := json.Marshal(headlinemetrics.Config{Database: database, Targets: []headlinemetrics.Target{{Entity: "openclaw", Target: "clawd"}}})
+	require.NoError(t, err)
+	require.NoError(t, os.WriteFile(config, b, 0o600))
+	out.Reset()
+	// The invalid archive config must never be opened or initialized by metrics.
+	archiveConfig := filepath.Join(root, "absent-archive-config.toml")
+	require.NoError(t, Run(t.Context(), []string{"--config", archiveConfig, "--json", "metrics", "status", "--config", config}, &out, &bytes.Buffer{}))
+	require.JSONEq(t, `{"source":"discrawl","observations":0,"events":0,"sequence":0,"last_observed":null}`, out.String())
+	require.NoFileExists(t, archiveConfig)
+}

internal/cli/output.go

@@ -12,6 +12,7 @@ import (
 	"time"
 
 	"github.com/openclaw/discrawl/internal/discorddesktop"
+	"github.com/openclaw/discrawl/internal/headlinemetrics"
 	"github.com/openclaw/discrawl/internal/media"
 	"github.com/openclaw/discrawl/internal/report"
 	"github.com/openclaw/discrawl/internal/share"
@@ -113,6 +114,10 @@ func printCommandUsage(w io.Writer, args []string) error {
 }
 
 var commandUsage = map[string]string{
+	"metrics":         headlinemetrics.Usage,
+	"metrics collect": headlinemetrics.Usage,
+	"metrics import":  headlinemetrics.Usage,
+	"metrics status":  headlinemetrics.Usage,
 	"lexical":         "Usage: discrawl lexical rebuild\n\nRebuild configured language indexes locally, without contacting Discord.\n",
 	"lexical rebuild": "Usage: discrawl lexical rebuild\n\nRun after enabling languages or replacing helpers, dictionaries, or Kiwi models.\n",
 	"metadata": `Usage: discrawl metadata [--json]

internal/headlinemetrics/discord.go

@@ -0,0 +1,83 @@
+package headlinemetrics
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"net/http"
+	"net/url"
+	"regexp"
+	"time"
+)
+
+var inviteCode = regexp.MustCompile(`^[A-Za-z0-9_-]{1,100}$`)
+
+func CollectDiscord(ctx context.Context, c Config, ts string) ([]Row, error) {
+	client := &http.Client{
+		Timeout:       30 * time.Second,
+		CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
+	}
+	return collectDiscord(ctx, c, ts, client)
+}
+
+func collectDiscord(ctx context.Context, c Config, ts string, client *http.Client) ([]Row, error) {
+	rows := make([]Row, 0, len(c.Targets)*2)
+	failed := false
+	for _, t := range c.Targets {
+		members, online := inviteCounts(ctx, client, t.Target)
+		if members == nil || online == nil {
+			failed = true
+		}
+		rows = append(rows, Counter(t, "members", members, ts, "discord_invite_approximate"), Counter(t, "online", online, ts, "discord_invite_approximate"))
+	}
+	if failed {
+		return rows, errors.New("discord invite counts unavailable")
+	}
+	return rows, nil
+}
+
+func inviteCounts(ctx context.Context, client *http.Client, code string) (*float64, *float64) {
+	if !inviteCode.MatchString(code) {
+		return nil, nil
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://discord.com/api/v10/invites/"+url.PathEscape(code)+"?with_counts=true", nil)
+	if err != nil {
+		return nil, nil
+	}
+	req.Header.Set("Accept", "application/json")
+	res, err := client.Do(req)
+	if err != nil {
+		return nil, nil
+	}
+	defer func() { _ = res.Body.Close() }()
+	if res.StatusCode != http.StatusOK {
+		// A scheduler may try again later, including after 429; never convert
+		// permission, expiry, or transient HTTP errors into zero members.
+		return nil, nil
+	}
+	const maxBody = 2 * 1024 * 1024
+	body, err := io.ReadAll(io.LimitReader(res.Body, maxBody+1))
+	if err != nil || len(body) > maxBody {
+		return nil, nil
+	}
+	var data struct {
+		Guild *struct {
+			ID string `json:"id"`
+		} `json:"guild"`
+		Members *int64 `json:"approximate_member_count"`
+		Online  *int64 `json:"approximate_presence_count"`
+	}
+	if json.Unmarshal(body, &data) != nil || data.Guild == nil || data.Guild.ID == "" {
+		return nil, nil
+	}
+	return approximateCount(data.Members), approximateCount(data.Online)
+}
+
+func approximateCount(n *int64) *float64 {
+	if n == nil || *n < 0 || *n > 1<<53 {
+		return nil
+	}
+	value := float64(*n)
+	return &value
+}