diff --git a/CHANGELOG.md b/CHANGELOG.md index cd06165d..700f43e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,9 +16,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Local login routed through the auth provider registry** (#266, community contribution by @Justy116): `POST /auth/login` now authenticates via the same provider path used by OIDC and LDAP instead of the parallel `usersService.login` implementation, and registration auto-login rides that path too. If the auto-login after signup fails transiently, the endpoint returns 201 with the created user and no `session` (the account stays recoverable via a normal login) instead of propagating a 500 after the user row is already committed. The local provider also gained user-enumeration hardening: the password is verified before the disabled-account check, so a wrong password returns the same generic error whether or not the account is disabled, and only a correct password reveals the disabled state. New `docs/architecture/auth.md` documents the provider architecture. Follow-up fixes landed with the merge: the register page now handles the sessionless 201 (redirects to login with a "please sign in" toast instead of throwing), the disabled-account and SSO-required login rejections are now recorded as `auth.login_failed` audit events with a `reason` in the metadata (previously only wrong-password failures were audited; the reason never appears in the HTTP response, so anti-enumeration behavior is unchanged), and a failed post-registration auto-login is now logged through internal observability instead of being swallowed silently. Test-suite hardening that the registry routing made necessary: two auth test files deleted every `auth_providers` row including the migration-seeded `local` provider, which (now that login resolves through the registry) broke every login-based test that ran after them, masked intermittently by the registry's 5-minute cache; they now preserve the seeded row and the test setup re-seeds it defensively, since tracked migrations never re-run ### Fixed +- **A bad client-side telemetry key logged the user out on every page view** (UX audit): the global 401 interceptor treated a 401 from any `/api/v1/` request (other than `/auth/`) as an expired session and bounced the user to `/login`. But ingestion/telemetry endpoints (`/api/v1/ingest`, `/v1/otlp/*`, inbound `/api/v1/receivers/*`) authenticate with an API key or DSN, not the session, so a 401 there means a bad telemetry key, not an expired session. The client-side SDK (`hooks.client.ts`) captures a log on every navigation; if its DSN key is wrong or revoked, each page view returned 401 and the interceptor logged the user out in a loop. Those endpoints are now excluded from the session-expiry logout. +- **Silent PII rule toggles** (UX audit): toggling a PII masking rule on/off or changing its action (mask/redact/hash) updated the rule but showed no confirmation, unlike every other action in that page (create/update/delete all toast). Both now show a success toast, matching the rest of the app's consistent action feedback. +- **Inconsistent time-range selectors across pages** (UX audit): the preset time-range controls on Security, Usage and Metrics were each hand-rolled with different markup (button groups and a dropdown with different styling). They now share one `TimeRangeButtons` segmented component, so they look and behave identically. Logs and Traces keep the richer time-range popover (they support custom ranges); the preset-only pages are now visually consistent. +- **"Total Logs Today" and "Error Rate" collapsed to the last hour** (UX audit): the dashboard stats (project overview cards and custom-dashboard single-stat panels, both via `dashboardService.getStats`) read "today" from the hourly continuous aggregate for everything but the last hour, backfilling only the last hour from raw. When the aggregate is not kept warm (a refresh policy that is not running, seeded/backfilled data, or a fresh install), the older part of today is missing from it, so the total and the error rate silently collapsed to roughly the last hour. Verified live on the hosted demo: 24h = ~1,500 logs, last hour = ~67, yet "Total Logs Today" read ~60 and Error Rate 0.0%. Low-volume organizations now compute today/yesterday from an exact raw count (cheap over the bounded window and independent of aggregate health); high-volume instances keep the fast aggregate path, where live ingestion keeps the aggregate current. +- **The same error split into several error groups** (UX audit): Node.js runtime frames use the `node:` scheme (e.g. `node:internal/process/task_queues`), which was not in the exception detector's library-pattern list, so `isAppCode` classified them as application code. Async stack traces vary in which internal frames they carry, so those frames leaked into the stack fingerprint and the same logical error (observed live: one `TypeError: Cannot read properties of undefined (reading 'selectFrom')` at `products/routes.js:130` fanned out into 5+ groups of 60/48/46/1/1 occurrences). `node:` frames are now treated as library code: excluded from the fingerprint (so recurrences of one error collapse into a single group) and no longer shown under "Show App Code Only". Fingerprints for errors whose stacks included `node:` frames change, so existing groups re-consolidate going forward rather than retroactively. Regression tests added. +- **Sidebar flashed "Select organization" on a hard reload** (UX audit): the organization store only persisted the selected org's id and started with `currentOrganization: null`, so a full page load of any dashboard sub-route rendered "Select organization" in the sidebar until the org list finished fetching. The store now caches the whole selected organization object and hydrates its initial state from it, so the org name renders immediately and is reconciled with the authoritative list once it loads (falling back cleanly if the cached org no longer exists). `clear()` was also fixed to reset to an empty state instead of the module-load initial state, so logout no longer restores the cached org. +- **Project overview stat cards showed wrong trend units and a mislabeled window** (UX audit): the shared `StatsCard` hardcoded every trend as "N% from last period", so the "Active Services" card, whose trend is an absolute count delta (services today minus yesterday), rendered a plain count with a bogus "%" ("+1.00% from last period" for one new service). The card now takes an optional trend `unit` and `label`: Active Services shows "+1 vs yesterday", Error Rate shows its change in percentage points ("pp"), and Total Logs/Throughput keep percentages with accurate "from yesterday"/"from last hour" labels. Separately, the Error Rate card described itself as "last 24h" while the value it shows is today's rate (since local midnight); the description now reads "Error rate today" to match the number. +- **Error groups always showed "unknown" affected service on ClickHouse and MongoDB** (UX audit): the `error_groups` service attribution was computed by a Postgres trigger that resolved the service with `SELECT service FROM logs WHERE id = NEW.log_id`. On non-TimescaleDB reservoir backends the ingested logs never land in the Postgres `logs` table (there is deliberately no foreign key on `exceptions.log_id`, "no FK due to hypertable"), so the lookup returned NULL and every error group was attributed to `unknown`, regardless of the real emitting service. The ingestion path already knows the service (`log.service`), so it is now carried on the `exceptions` row (new nullable `service` column, migration 054) and the trigger prefers it (`NEW.service`), falling back to the logs lookup on TimescaleDB and only then to `unknown`. New occurrences also merge the real service into existing groups via the ON CONFLICT path, so groups previously stuck on `unknown` recover as they recur. Multi-engine regression tests added. +- **Monitoring page used unstyled native ``, which looked out of place next to the custom dropdowns used everywhere else in the app. All six now use the shared `Select` component, so styling, focus, and open/close behavior match the rest of LogTide. +- **Metrics Explorer showed duplicate project and time-range selectors** (UX audit): the Explorer tab's "Filters" card repeated the project and time-range selectors that already live in the always-visible page header (both bound to the same state), so the tab presented two of each control at once. The Filters card now keeps only the Explorer-specific metric selector; project and time range are driven solely by the header controls. +- **Projects page browser tab title said "Dashboard"** (UX audit): the `/dashboard/projects` list page hardcoded `Dashboard - LogTide`, so the browser tab and history entry read "Dashboard" instead of "Projects". Now titled "Projects - LogTide", matching every sibling page. +- **Members "Joined" column mixed absolute and relative dates** (UX audit): the settings members table used a local formatter that showed "Today"/"Yesterday"/"N days ago" for recent joins but flipped to an absolute `M/D/YYYY` date after a week, so the column mixed two formats side by side. It now shows a single consistent short date ("Apr 12, 2026") with the relative time in a hover tooltip, via a shared `formatDateShort` helper. The bespoke local formatter was removed. - **Ingestion health counters no longer depend on the metering toggle** (#279). `METERING_ENABLED=false` silently suppressed every `ingestion.*` counter, including `ingestion.pii_rejected`, the safety counter for records dropped because PII masking failed. That toggle governs usage and resource metering, while the `ingestion.*` counters are operational health signals already excluded from tenant usage breakdowns by a prefix filter, so they now bypass it. The buffer hard cap still applies to them: that is a memory valve, not a feature switch. ### Added +- **Custom time range now uses a calendar with explicit time inputs** (UX audit): the log/trace time-range picker's "Custom" option replaced the two native `datetime-local` inputs (an inconsistent, browser-styled widget) with a proper in-app range calendar (new shadcn-style `RangeCalendar` wrapping bits-ui, so it matches the rest of the UI) plus dedicated "From time" / "To time" inputs for choosing the hour. The stored value stays `YYYY-MM-DDTHH:mm`, so URL sync, the shared time-range store, and deep links are unchanged. Adds the `@internationalized/date` dependency (already used by bits-ui). +- **Auto-merge duplicate error groups at ingestion**: builds on the manual merge below so new occurrences stop creating duplicates in the first place. Alongside the exact stack fingerprint, each group now has a coarser `merge_key` (exception type + normalized message + the top application frame). When an incoming exception has no exact fingerprint match but shares a `merge_key` with an existing group in the same project, it folds into that group instead of spawning a duplicate; errors thrown from different sites, or with different messages once dynamic tokens (numbers, UUIDs, hex, emails, quoted strings) are masked, stay separate. The key is computed by a single Postgres function `logtide_merge_key` used by the trigger, the runtime fold lookup, and a one-time backfill of existing groups (migration 055), so there is no SQL/JS normalization to keep in sync; library-only exceptions (no app frame) get a null key and are never auto-merged. The manual "Merge N" banner remains for historical splits. DB-backed tests cover fold, throw-site separation, dynamic-message folding, and the library-only null case. +- **The time window is remembered across logs and traces** (UX audit): picking a range (or a custom window) on Logs now carries over to Traces and vice versa, via a shared `timeRange` store persisted in localStorage. Each page validates the stored preset against what it supports and falls back to its default otherwise; deep links with explicit from/to still win. Metrics/Security/Usage keep their own preset vocabularies. +- **The selected project is remembered across pages** (UX audit): Metrics, Traces and Monitoring each defaulted to "the first project" independently, so moving between them kept resetting which project you were looking at. A shared `currentProject` store now persists the last project you selected (localStorage) and those pages default to it when it is still valid, falling back to the first project otherwise. Deep links with an explicit project still win. +- **Recent searches in log search** (UX audit): the log search page remembers your recent query terms (in localStorage) and shows them as one-click chips under the search bar when the box is empty, with a Clear action. Capped at the last 8, deduplicated. +- **Dashboard auto-refresh** (UX audit): the custom dashboard header has an auto-refresh control (off / 30s / 1m / 5m) that periodically refreshes all panels of the active dashboard. The choice persists in localStorage and pauses automatically while editing the layout so a refresh cannot clobber unsaved edits. +- **Row density toggle in log search** (UX audit): a Cozy/Compact toggle in the table toolbar (styled as an outline button to match the neighbouring Columns control) tightens row padding for scanning more logs at once, persisted in localStorage. (Custom columns were already persisted per project.) +- **Error regression detection** (UX audit): when an error group that was marked resolved starts occurring again, it is now automatically reopened (status back to open, resolution cleared) and its notification is flagged as a regression, with distinct wording across the in-app notification ("Regression: ..."), the email subject ("[Regression] ..."), and the webhook envelope (high severity plus an `is_regression` field). Previously a resolved error could recur silently and stay hidden in the resolved bucket. Job + wording covered by tests. +- **Merge duplicate error groups** (UX audit): complements the fingerprint fix above for groups that already split before it landed. The error detail page detects other groups with the same exception type and message and offers a one-click "Merge N" that folds them into the current group: their exceptions are reassigned to the target fingerprint (so trend and logs stay complete), occurrence counts and affected services are summed/unioned, and first/last-seen are widened, all in one org-scoped transaction. New `GET /api/v1/error-groups/:id/duplicates` and `POST /api/v1/error-groups/:id/merge` (membership-checked, cross-org merges refused). Service + org-isolation tests added. +- **Shareable log search URLs** (UX audit): the log search page now writes its active filters back to the URL (query text, project, service, levels, trace/session id, and a custom time range), so a search can be bookmarked or sent to a teammate and reopened exactly as filtered. It reuses the existing (idempotent) deep-link reader, uses `replaceState` so it does not spam history, and skips the write when nothing changed. (The errors page already synced its filters; traces already accepts deep links.) +- **Command palette searches your data, not just navigation** (UX audit): typing in the palette now offers "Search logs for ...", "Search errors for ...", and, when the text looks like a 16/32-char hex trace id, "Open trace ...". The log search page learned a `q` query param and the traces page learned a `traceId` param so these deep links land pre-filled. +- **Duplicate monitors and alert rules** (UX audit): the monitoring list has a Duplicate action that opens the New Monitor form pre-filled from an existing monitor (name suffixed "(copy)"), and the alerts list has a Duplicate action that opens the builder pre-filled with the full rule (service, levels, threshold, time window, rate-of-change settings, metadata filters and notification channels). The alert builder's prefill was generalized into a single `AlertBuilderPrefill` object shared by duplicate and create-from-error. +- **"Create alert" from an error group** (UX audit): the error detail page has a Create alert button that opens the alert builder prefilled from the error, the name seeded from the exception type, levels set to error/critical, and the service filled in when the group is attributed to a real service. The builder accepts optional prefill props and the alerts page reads them from query params (then strips them so a refresh does not reopen the dialog). +- **Metrics empty state now shows copy-ready OTLP setup** (UX audit): the "No metrics found" state was a bare icon and one line. It now mirrors the traces empty state with a proper onboarding panel: Node.js/Python/Go OpenTelemetry metric-exporter snippets (pointing at `/v1/otlp/metrics`) with copy buttons, quick links to get an API key and the OTLP/OpenTelemetry docs, and a preview of what metrics unlock. +- **Theme "System" option** (UX audit): the theme control is now a Light / Dark / System menu instead of a two-state toggle. "System" follows the OS color scheme and tracks it live (a `prefers-color-scheme` change flips the app without a reload). The stored preference persists across sessions; existing consumers still read the resolved light/dark theme, and the previous `toggle()`/`set()` helpers keep working. +- **Search term highlighting and relative-time tooltips in log search** (UX audit): in full-text mode the matched query terms are now highlighted in the message column (rendered as plain text segments, never `{@html}`, so no markup injection), and each log's absolute timestamp carries a relative-time tooltip ("3 minutes ago") on hover. - **Clock skew detection for spans and metrics** (follows #279). Extends the log-side detection to OTLP span and metric ingestion: a span whose `end_time`, or a metric data point whose `time`, sits more than 24 hours in the past or more than 5 minutes ahead of the server clock is counted and surfaced on the project overview, never rejected. Spans are observed on `end_time` rather than `start_time` on purpose: a span that STARTED 27 hours ago is routinely legitimate, since spans are exported at end and batch jobs, long transactions and long-poll connections all produce old start times. Clock skew shifts both ends together while a long span moves only its start, so `end_time` is what separates a broken clock from a slow operation. Unlike the log threshold, which is exactly the maximum alert `time_window` and therefore an exact statement, the span and metric thresholds are a judgement call: no alert rules run over them, and the harm is that the data is invisible in the time windows of trace views and dashboards. Counters are `ingestion.span_timestamp_skew` and `ingestion.metric_timestamp_skew`, kept under the `ingestion.` prefix because both the tenant usage-breakdown exclusion and the metering-toggle bypass key off it literally. The project ingestion health endpoint now reports skew per signal, and the banner carries per-signal copy: the log wording ("cannot trigger an alert", "omit the time field") is false for spans and metrics. - **Ingestion clock skew detection** (#279). Logs whose client-supplied `time` is more than 24 hours in the past or more than 5 minutes ahead of the server clock are counted and surfaced, on the project overview page for tenants and in admin ingestion health for operators. Such logs are stored and searchable but can never be counted by a threshold alert rule, whose largest window is 24 hours, which previously made a misconfigured shipper look like an alerting bug: ingestion worked, rules were enabled, channels tested fine, and no alert ever fired. The 24 hour threshold is derived rather than chosen, being the maximum configurable alert `time_window`. The logs are never rejected, so backfilling historical data stays valid, and detection is always on with no configuration to get wrong. Detection runs inside the existing record-building pass in `ingestLogs`, the single choke point for every log path (batch, Fluent Bit, OTLP, receivers), at a cost of one subtraction per record. - **Scheduled email digest reports, completed and enabled** (#154): the digest feature whose foundation landed in #209 (and shipped disabled since 1.0.0-beta) is now finished and active. The report grew from log volume only to five sections, each with a quiet empty state: log volume with period-over-period trend, top 5 services by error+critical count with delta vs the previous period (engine-agnostic via `reservoir.topValues`, so TimescaleDB/ClickHouse/MongoDB all work), new error groups first seen in the period (`error_groups.first_seen`), a security summary (windowed detection totals and top triggered Sigma rules read from raw `detection_events` to avoid continuous-aggregate staleness, plus a point-in-time open/investigating incident count), and monitor uptime (aggregated from `monitor_uptime_daily`, section omitted for orgs without monitors). Emails are now real HTML built on the shared `email-templates.ts` component library (inline CSS, escaped user-controlled strings, en-US number formatting) with the full report duplicated in the plaintext fallback, and the base template footer learned an `unsubscribeUrl` variant so digest recipients (who may have no account) get a one-click unsubscribe instead of the channel-settings link. Scheduling was redesigned: instead of one cron per organization registered at worker boot (which could never pick up config changes at runtime, because graphile-worker cron items are fixed once the runner starts), a single static hourly `digest-dispatch` cron sweeps enabled `digest_configs`, matches `delivery_hour`/`delivery_day_of_week` against the current UTC hour, and enqueues `digest-generation` jobs with hour-keyed dedup keys and a `last_sent_at` double-fire guard (migration 053), so config CRUD is live on both queue backends with no restart. New management surface: session-auth CRUD under `/api/v1/digests` (config upsert/delete, recipient add/remove/resubscribe with token rotation on resubscribe, membership for reads and owner/admin for writes, audit-logged as `digest.*`, recipients capped by a new `digests.max_recipients` capability with the canonical 4-case tests), a public `POST /api/v1/digests/unsubscribe` where the 32-byte random token is the credential (idempotent, returns a masked email), an "Email Digests" settings page (schedule card with UTC delivery time and recipient management) and a public `/unsubscribe` page consumed by the email footer link. The worker-side disable comment from the beta is gone: the dispatch cron registers at boot diff --git a/packages/backend/migrations/054_exception_service_attribution.sql b/packages/backend/migrations/054_exception_service_attribution.sql new file mode 100644 index 00000000..d0467439 --- /dev/null +++ b/packages/backend/migrations/054_exception_service_attribution.sql @@ -0,0 +1,71 @@ +-- ============================================================================ +-- Migration 054: Engine-independent service attribution for error groups +-- The error-group trigger resolved a group's affected service with +-- `SELECT service FROM logs WHERE id = NEW.log_id`, but on ClickHouse and +-- MongoDB reservoir backends the ingested logs never land in the Postgres +-- `logs` table, so the lookup always returned NULL and every error group was +-- attributed to 'unknown'. The ingestion path already knows the service, so we +-- carry it on the exception row and have the trigger prefer it, falling back to +-- the logs lookup (TimescaleDB) and only then to 'unknown'. +-- ============================================================================ + +ALTER TABLE exceptions ADD COLUMN IF NOT EXISTS service TEXT; + +CREATE OR REPLACE FUNCTION update_error_group_on_exception() +RETURNS TRIGGER AS $$ +DECLARE + v_service TEXT; +BEGIN + -- Prefer the service carried on the exception (known at ingestion, works on + -- every storage engine); fall back to the Postgres logs table (TimescaleDB) + -- and finally to 'unknown'. + v_service := NEW.service; + + IF v_service IS NULL THEN + SELECT service INTO v_service + FROM logs + WHERE id = NEW.log_id + LIMIT 1; + END IF; + + v_service := COALESCE(v_service, 'unknown'); + + -- Insert or update error group + INSERT INTO error_groups ( + organization_id, + project_id, + fingerprint, + exception_type, + exception_message, + language, + occurrence_count, + first_seen, + last_seen, + affected_services, + sample_log_id + ) + VALUES ( + NEW.organization_id, + NEW.project_id, + NEW.fingerprint, + NEW.exception_type, + NEW.exception_message, + NEW.language, + 1, + NEW.created_at, + NEW.created_at, + ARRAY[v_service], + NEW.log_id + ) + ON CONFLICT (organization_id, COALESCE(project_id, '00000000-0000-0000-0000-000000000000'::UUID), fingerprint) + DO UPDATE SET + occurrence_count = error_groups.occurrence_count + 1, + last_seen = NEW.created_at, + affected_services = ( + SELECT ARRAY(SELECT DISTINCT unnest(array_cat(error_groups.affected_services, ARRAY[v_service]))) + ), + updated_at = NOW(); + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; diff --git a/packages/backend/migrations/055_error_group_auto_merge.sql b/packages/backend/migrations/055_error_group_auto_merge.sql new file mode 100644 index 00000000..bc7f1273 --- /dev/null +++ b/packages/backend/migrations/055_error_group_auto_merge.sql @@ -0,0 +1,123 @@ +-- ============================================================================ +-- Migration 055: Auto-merge error groups at ingestion +-- Stack fingerprints split one logical error into several groups when the deep +-- stack varies (source maps, async internal frames, different callers). We add a +-- coarser "merge key" (exception type + normalized message + the top application +-- frame) so a new occurrence folds into an existing group instead of creating a +-- duplicate. The key is computed by ONE Postgres function used by the backfill, +-- the trigger, and the runtime fold lookup, so there is no SQL/JS normalization +-- to keep in sync. See docs/superpowers/specs/2026-07-21-error-group-auto-merge-design.md +-- ============================================================================ + +-- Single source of truth for the merge key. IMMUTABLE so it can be used in a +-- WHERE clause against the merge_key index. Returns NULL when there is no app +-- frame, so library-only errors are never auto-merged. md5 (not a security +-- digest, just an internal grouping key) avoids a pgcrypto dependency. +CREATE OR REPLACE FUNCTION logtide_merge_key(p_type TEXT, p_message TEXT, p_top_frame TEXT) +RETURNS TEXT AS $$ +DECLARE + m TEXT; +BEGIN + IF p_top_frame IS NULL THEN + RETURN NULL; + END IF; + + m := COALESCE(p_message, ''); + -- UUIDs first (they contain hex runs the next step would otherwise match). + m := regexp_replace(m, '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '', 'gi'); + -- Long hex runs (request ids, hashes, addresses). + m := regexp_replace(m, '\y[0-9a-f]{8,}\y', '', 'gi'); + m := regexp_replace(m, '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}', '', 'gi'); + m := regexp_replace(m, '''[^'']*''|"[^"]*"', '', 'g'); + m := regexp_replace(m, '\d+(\.\d+)?', '', 'g'); + m := btrim(regexp_replace(m, '\s+', ' ', 'g')); + + RETURN md5(p_type || E'\n' || m || E'\n' || p_top_frame); +END; +$$ LANGUAGE plpgsql IMMUTABLE; + +ALTER TABLE error_groups ADD COLUMN IF NOT EXISTS merge_key TEXT; +CREATE INDEX IF NOT EXISTS idx_error_groups_merge_key ON error_groups (organization_id, merge_key); + +-- Raw "file:function" of the first app frame, carried from the app (which has the +-- parsed frames) so the trigger can compute the group's merge key. +ALTER TABLE exceptions ADD COLUMN IF NOT EXISTS top_frame TEXT; + +CREATE OR REPLACE FUNCTION update_error_group_on_exception() +RETURNS TRIGGER AS $$ +DECLARE + v_service TEXT; + v_merge_key TEXT; +BEGIN + -- Service: prefer the value carried on the exception (works on every storage + -- engine), fall back to the Postgres logs table (TimescaleDB), then 'unknown'. + v_service := NEW.service; + IF v_service IS NULL THEN + SELECT service INTO v_service + FROM logs + WHERE id = NEW.log_id + LIMIT 1; + END IF; + v_service := COALESCE(v_service, 'unknown'); + + v_merge_key := logtide_merge_key(NEW.exception_type, NEW.exception_message, NEW.top_frame); + + INSERT INTO error_groups ( + organization_id, + project_id, + fingerprint, + exception_type, + exception_message, + language, + occurrence_count, + first_seen, + last_seen, + affected_services, + sample_log_id, + merge_key + ) + VALUES ( + NEW.organization_id, + NEW.project_id, + NEW.fingerprint, + NEW.exception_type, + NEW.exception_message, + NEW.language, + 1, + NEW.created_at, + NEW.created_at, + ARRAY[v_service], + NEW.log_id, + v_merge_key + ) + ON CONFLICT (organization_id, COALESCE(project_id, '00000000-0000-0000-0000-000000000000'::UUID), fingerprint) + DO UPDATE SET + occurrence_count = error_groups.occurrence_count + 1, + last_seen = NEW.created_at, + affected_services = ( + SELECT ARRAY(SELECT DISTINCT unnest(array_cat(error_groups.affected_services, ARRAY[v_service]))) + ), + merge_key = COALESCE(error_groups.merge_key, v_merge_key), + updated_at = NOW(); + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +-- Backfill existing groups so historical duplicates can fold forward. +UPDATE error_groups g +SET merge_key = logtide_merge_key(g.exception_type, g.exception_message, tf.top_frame) +FROM ( + SELECT + eg.id AS group_id, + ( + SELECT sf.file_path || ':' || COALESCE(sf.function_name, '') + FROM exceptions e + JOIN stack_frames sf ON sf.exception_id = e.id + WHERE e.log_id = eg.sample_log_id AND sf.is_app_code = TRUE + ORDER BY sf.frame_index ASC + LIMIT 1 + ) AS top_frame + FROM error_groups eg +) tf +WHERE g.id = tf.group_id; diff --git a/packages/backend/src/database/types.ts b/packages/backend/src/database/types.ts index d6c65c65..ba9cbc95 100644 --- a/packages/backend/src/database/types.ts +++ b/packages/backend/src/database/types.ts @@ -641,6 +641,8 @@ export interface ExceptionsTable { fingerprint: string; raw_stack_trace: string; frame_count: number; + service: string | null; + top_frame: string | null; created_at: Generated; } @@ -689,6 +691,7 @@ export interface ErrorGroupsTable { resolved_by: string | null; affected_services: string[] | null; sample_log_id: string | null; + merge_key: string | null; last_notified_at: Timestamp | null; created_at: Generated; updated_at: Generated; diff --git a/packages/backend/src/modules/dashboard/service.ts b/packages/backend/src/modules/dashboard/service.ts index 938a28e2..9f64d67f 100644 --- a/packages/backend/src/modules/dashboard/service.ts +++ b/packages/backend/src/modules/dashboard/service.ts @@ -49,6 +49,20 @@ export interface TimelineEvent { detectionsBySeverity: { critical: number; high: number; medium: number; low: number }; } +/** + * Below this recent (last-hour) log volume, dashboard "today" stats are computed + * from raw logs instead of the hourly continuous aggregate. The aggregate only + * reflects data below its materialization watermark that its refresh policy has + * actually processed (start_offset = 3h); on instances where the policy is not + * keeping the aggregate warm, "today" collapses to roughly the last hour (the + * only part read from raw), which is what makes "Total Logs Today" and the error + * rate read far too low. Verified live on the hosted demo: 24h = ~1,500 logs, + * last hour = ~67, yet "Total Logs Today" showed ~60. A raw count over the + * bounded today/yesterday window is exact and cheap at low volume; higher-volume + * instances keep the fast aggregate path (their live ingestion keeps it warm). + */ +const RAW_STATS_MAX_RECENT_VOLUME = 5000; + class DashboardService { /** * Resolve project IDs from org or single project. @@ -163,6 +177,13 @@ class DashboardService { }); const recentVolume = recentTotalResult.count; + // Low-volume instances (fresh installs, backfilled/seeded data, or a stale + // continuous aggregate) are exactly where the aggregate undercounts "today", + // so count today/yesterday from raw for an exact, cheap result. + if (recentVolume <= RAW_STATS_MAX_RECENT_VOLUME) { + return this.getStatsFromRawLogs(projectIds, todayStart, yesterdayStart, lastHourStart, prevHourStart); + } + // 2. Query aggregate for historical data and reservoir for recent data in parallel const [todayAggregateStats, recentTotal, recentErrors, recentServices, yesterdayAggregateStats, prevHourCount] = await Promise.all([ // Today's historical stats from aggregate (today start to 1 hour ago) diff --git a/packages/backend/src/modules/exceptions/detection.ts b/packages/backend/src/modules/exceptions/detection.ts index da2f3405..1f7b70c9 100644 --- a/packages/backend/src/modules/exceptions/detection.ts +++ b/packages/backend/src/modules/exceptions/detection.ts @@ -13,6 +13,11 @@ import type { ExceptionLanguage, StructuredException } from '@logtide/shared'; // Library patterns for detecting vendor/library code const LIBRARY_PATTERNS = [ /node_modules/, + // Node.js runtime frames (node:internal/..., node:events, ...). These are not + // application code, and because async stack traces vary in which internal + // frames they include, treating them as app code makes the same logical error + // fingerprint differently and split into several error groups. + /^node:/, /vendor\//, /site-packages/, /\.cargo/, diff --git a/packages/backend/src/modules/exceptions/fingerprint-service.ts b/packages/backend/src/modules/exceptions/fingerprint-service.ts index aafd6603..e60cf493 100644 --- a/packages/backend/src/modules/exceptions/fingerprint-service.ts +++ b/packages/backend/src/modules/exceptions/fingerprint-service.ts @@ -43,6 +43,20 @@ export class FingerprintService { return crypto.createHash('sha256').update(input).digest('hex'); } + /** + * The throw site: raw "file:function" of the first application frame, or null + * when there is none. Used as the coarse-grouping key's frame component (see + * migration 055's logtide_merge_key). Kept raw (not path-normalized) so the app + * and the SQL function agree without duplicating normalization. + */ + static topAppFrame(parsedException: ParsedException): string | null { + const frame = parsedException.frames.find((f) => f.isAppCode); + if (!frame) return null; + const file = frame.originalFile || frame.filePath; + const func = frame.originalFunction || frame.functionName || ''; + return `${file}:${func}`; + } + /** * Default normalization when parser is not available */ diff --git a/packages/backend/src/modules/exceptions/routes.ts b/packages/backend/src/modules/exceptions/routes.ts index a11a2fed..e7a78ff8 100644 --- a/packages/backend/src/modules/exceptions/routes.ts +++ b/packages/backend/src/modules/exceptions/routes.ts @@ -567,6 +567,85 @@ export async function exceptionsRoutes(fastify: FastifyInstance) { } ); + /** + * GET /api/v1/error-groups/:id/duplicates + * Other groups with the same exception type and message (merge candidates). + */ + fastify.get( + '/api/v1/error-groups/:id/duplicates', + async (request: any, reply) => { + try { + const { id } = z.object({ id: z.string().uuid() }).parse(request.params); + const { organizationId } = z + .object({ organizationId: z.string().uuid() }) + .parse(request.query); + + const isMember = await checkOrganizationMembership(request.user.id, organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'You are not a member of this organization' }); + } + + const group = await exceptionService.getErrorGroupById(id); + if (!group || group.organizationId !== organizationId) { + return reply.status(404).send({ error: 'Error group not found' }); + } + + const duplicates = await exceptionService.findDuplicateErrorGroups(id, organizationId); + return reply.send({ duplicates }); + } catch (error: any) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error('Error finding duplicate error groups:', error); + return reply.status(500).send({ error: 'Failed to find duplicate error groups' }); + } + } + ); + + /** + * POST /api/v1/error-groups/:id/merge + * Merge the given source groups into this group. + */ + fastify.post( + '/api/v1/error-groups/:id/merge', + { + config: { rateLimit: { max: 20, timeWindow: '1 minute' } }, + }, + async (request: any, reply) => { + try { + const { id } = z.object({ id: z.string().uuid() }).parse(request.params); + const body = z + .object({ + organizationId: z.string().uuid(), + sourceIds: z.array(z.string().uuid()).min(1).max(100), + }) + .parse(request.body); + + const isMember = await checkOrganizationMembership(request.user.id, body.organizationId); + if (!isMember) { + return reply.status(403).send({ error: 'You are not a member of this organization' }); + } + + const group = await exceptionService.getErrorGroupById(id); + if (!group || group.organizationId !== body.organizationId) { + return reply.status(404).send({ error: 'Error group not found' }); + } + + const merged = await exceptionService.mergeErrorGroups(id, body.sourceIds, body.organizationId); + if (!merged) { + return reply.status(404).send({ error: 'Error group not found' }); + } + return reply.send(merged); + } catch (error: any) { + if (error instanceof z.ZodError) { + return reply.status(400).send({ error: 'Validation error', details: error.errors }); + } + console.error('Error merging error groups:', error); + return reply.status(500).send({ error: 'Failed to merge error groups' }); + } + } + ); + /** * GET /api/v1/error-groups/:id/trend * Get error group occurrence trend (time-series) diff --git a/packages/backend/src/modules/exceptions/service.ts b/packages/backend/src/modules/exceptions/service.ts index 8dd693c2..09848b6b 100644 --- a/packages/backend/src/modules/exceptions/service.ts +++ b/packages/backend/src/modules/exceptions/service.ts @@ -26,7 +26,7 @@ export class ExceptionService { * The trigger will automatically update/create the error group */ async createException(params: CreateExceptionParams): Promise { - const { organizationId, projectId, logId, parsedData, fingerprint } = params; + const { organizationId, projectId, logId, parsedData, fingerprint, service, topFrame } = params; return await this.db.transaction().execute(async (trx) => { const exception = await trx @@ -41,6 +41,8 @@ export class ExceptionService { fingerprint, raw_stack_trace: parsedData.rawStackTrace, frame_count: parsedData.frames.length, + service: service ?? null, + top_frame: topFrame ?? null, }) .returning('id') .executeTakeFirstOrThrow(); @@ -384,6 +386,125 @@ export class ExceptionService { return this.getErrorGroupById(groupId); } + /** + * Find other error groups in the same org that share this group's exception + * type and message (the shape a fingerprint split produces). Candidates to + * merge into this group. + */ + async findDuplicateErrorGroups( + groupId: string, + organizationId: string + ): Promise> { + const target = await this.db + .selectFrom('error_groups') + .select(['exception_type', 'exception_message']) + .where('id', '=', groupId) + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + + if (!target) return []; + + let query = this.db + .selectFrom('error_groups') + .select(['id', 'occurrence_count', 'first_seen', 'last_seen']) + .where('organization_id', '=', organizationId) + .where('exception_type', '=', target.exception_type) + .where('id', '!=', groupId); + + query = + target.exception_message === null + ? query.where('exception_message', 'is', null) + : query.where('exception_message', '=', target.exception_message); + + const rows = await query.execute(); + return rows.map((r) => ({ + id: r.id, + occurrenceCount: Number(r.occurrence_count), + firstSeen: r.first_seen as Date, + lastSeen: r.last_seen as Date, + })); + } + + /** + * Merge source error groups into a target group: reassign their exceptions to + * the target fingerprint, fold their counts / services / first-last seen into + * the target, then delete the now-empty source groups. All org-scoped and + * transactional. + */ + async mergeErrorGroups( + targetId: string, + sourceIds: string[], + organizationId: string + ): Promise { + await this.db.transaction().execute(async (trx) => { + const target = await trx + .selectFrom('error_groups') + .select(['id', 'fingerprint', 'occurrence_count', 'affected_services', 'first_seen', 'last_seen']) + .where('id', '=', targetId) + .where('organization_id', '=', organizationId) + .executeTakeFirst(); + + if (!target) return; + + const sources = await trx + .selectFrom('error_groups') + .select(['id', 'fingerprint', 'occurrence_count', 'affected_services', 'first_seen', 'last_seen']) + .where('organization_id', '=', organizationId) + .where('id', 'in', sourceIds) + .where('id', '!=', targetId) + .execute(); + + if (sources.length === 0) return; + + const sourceFingerprints = sources.map((s) => s.fingerprint); + + // Point the merged-in exceptions at the target's fingerprint so trend/logs + // queries for the target include them. + await trx + .updateTable('exceptions') + .set({ fingerprint: target.fingerprint }) + .where('organization_id', '=', organizationId) + .where('fingerprint', 'in', sourceFingerprints) + .execute(); + + const addOccurrences = sources.reduce((sum, s) => sum + Number(s.occurrence_count), 0); + const mergedServices = Array.from( + new Set([ + ...((target.affected_services as string[] | null) || []), + ...sources.flatMap((s) => (s.affected_services as string[] | null) || []), + ]) + ); + const times = [target, ...sources]; + const firstSeen = times + .map((g) => new Date(g.first_seen as unknown as string)) + .reduce((a, b) => (a < b ? a : b)); + const lastSeen = times + .map((g) => new Date(g.last_seen as unknown as string)) + .reduce((a, b) => (a > b ? a : b)); + + await trx + .updateTable('error_groups') + .set({ + occurrence_count: Number(target.occurrence_count) + addOccurrences, + affected_services: mergedServices, + first_seen: firstSeen, + last_seen: lastSeen, + updated_at: new Date(), + }) + .where('id', '=', targetId) + .where('organization_id', '=', organizationId) + .execute(); + + await trx + .deleteFrom('error_groups') + .where('organization_id', '=', organizationId) + .where('id', 'in', sources.map((s) => s.id)) + .execute(); + }); + + return this.getErrorGroupById(targetId); + } + /** * Get error group trend (time-series data) */ diff --git a/packages/backend/src/modules/exceptions/types.ts b/packages/backend/src/modules/exceptions/types.ts index 05569a9c..ffdb487b 100644 --- a/packages/backend/src/modules/exceptions/types.ts +++ b/packages/backend/src/modules/exceptions/types.ts @@ -47,4 +47,15 @@ export interface CreateExceptionParams { logId: string; parsedData: ParsedException; fingerprint: string; + /** + * Service that emitted the error log. Carried on the exception row so the + * error-group trigger can attribute the service on every storage engine, + * not just TimescaleDB (where logs live in Postgres). + */ + service?: string | null; + /** + * Raw "file:function" of the first app frame (throw site). Carried so the + * trigger can compute the group's coarse merge key for auto-merge. + */ + topFrame?: string | null; } diff --git a/packages/backend/src/queue/jobs/error-notification.ts b/packages/backend/src/queue/jobs/error-notification.ts index abd10dd3..12336fd1 100644 --- a/packages/backend/src/queue/jobs/error-notification.ts +++ b/packages/backend/src/queue/jobs/error-notification.ts @@ -28,6 +28,8 @@ export interface ErrorNotificationJobData { language: ExceptionLanguage; service: string; isNewErrorGroup: boolean; + /** True when a previously resolved error group recurred (a regression). */ + isRegression?: boolean; } // Create the queue @@ -72,9 +74,9 @@ async function sendErrorWebhook( organizationId: data.organizationId, projectId: data.projectId ?? null, data: { - title: `${data.isNewErrorGroup ? 'New Error' : 'Error'}: ${data.exceptionType}`, + title: `${data.isRegression ? 'Regression' : data.isNewErrorGroup ? 'New Error' : 'Error'}: ${data.exceptionType}`, message: data.exceptionMessage || `An error occurred in ${data.service}`, - severity: data.isNewErrorGroup ? 'high' : 'medium', + severity: data.isRegression || data.isNewErrorGroup ? 'high' : 'medium', organization: { id: data.organizationId, name: orgName, @@ -88,6 +90,7 @@ async function sendErrorWebhook( language: data.language, service: data.service, is_new: data.isNewErrorGroup, + is_regression: data.isRegression ?? false, link: `${frontendUrl}/dashboard/errors/${errorGroupId}`, }, }); @@ -220,7 +223,9 @@ export async function processErrorNotification(job: IJob } const fingerprint = FingerprintService.generate(parsed); + const topFrame = FingerprintService.topAppFrame(parsed); // Check if this is a new error group (first occurrence with this fingerprint) - const existingGroup = await db + let existingGroup = await db .selectFrom('error_groups') .select(['id', 'occurrence_count', 'status']) .where('fingerprint', '=', fingerprint) .where('organization_id', '=', organizationId) .executeTakeFirst(); + // No exact fingerprint match: auto-merge into an existing group that shares + // the coarse key (same type + normalized message + top app frame), so a + // stack that differs only in its deep frames does not spawn a duplicate + // group. The key is computed by the same Postgres function on both sides. + let effectiveFingerprint = fingerprint; + if (!existingGroup && topFrame) { + const mergeMatch = await db + .selectFrom('error_groups') + .select(['id', 'fingerprint', 'occurrence_count', 'status']) + .where('organization_id', '=', organizationId) + .where('project_id', '=', projectId) + .where( + 'merge_key', + '=', + sql`logtide_merge_key(${parsed.exceptionType}, ${parsed.exceptionMessage}, ${topFrame})` + ) + .orderBy('first_seen', 'asc') + .executeTakeFirst(); + + if (mergeMatch) { + effectiveFingerprint = mergeMatch.fingerprint; + existingGroup = { + id: mergeMatch.id, + occurrence_count: mergeMatch.occurrence_count, + status: mergeMatch.status, + }; + } + } + const isNewErrorGroup = !existingGroup; + // A previously resolved error that recurs is a regression. + const isRegression = existingGroup?.status === 'resolved'; const exceptionId = await exceptionService.createException({ organizationId, projectId, logId: log.id, parsedData: parsed, - fingerprint, + fingerprint: effectiveFingerprint, + service: log.service, + topFrame, }); + // Reopen a resolved group so the regression is visible and not silently + // folded into a "resolved" bucket. + if (isRegression && existingGroup) { + await db + .updateTable('error_groups') + .set({ status: 'open', resolved_at: null, resolved_by: null, updated_at: new Date() }) + .where('id', '=', existingGroup.id) + .where('organization_id', '=', organizationId) + .execute(); + } + stats.parsed++; console.log( @@ -104,12 +150,13 @@ export async function processExceptionParsing(job: IJob exceptionId, organizationId, projectId, - fingerprint, + fingerprint: effectiveFingerprint, exceptionType: parsed.exceptionType, exceptionMessage: parsed.exceptionMessage, language: parsed.language, service: log.service, isNewErrorGroup, + isRegression, }; await errorNotificationQueue.add('error-notification', notificationData, { diff --git a/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts b/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts index 8ae9b969..cbe829f4 100644 --- a/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts +++ b/packages/backend/src/tests/modules/dashboard/dashboard-service.test.ts @@ -76,6 +76,31 @@ describe('DashboardService', () => { expect(stats.errorRate.value).toBe(20); // 2/10 = 20% }); + it('counts logs from earlier today, not just the last hour', async () => { + // Low-volume orgs take the raw-count path, which must count the whole + // day and not collapse to the last hour the way a stale continuous + // aggregate does (see RAW_STATS_MAX_RECENT_VOLUME). The logs below sit + // earlier today (older than the last hour) yet must all be counted. + const { organization, project } = await createTestContext(); + const now = new Date(); + const todayStart = new Date(now.getFullYear(), now.getMonth(), now.getDate()); + const earlierToday = new Date( + Math.max(todayStart.getTime() + 60_000, now.getTime() - 3 * 60 * 60 * 1000) + ); + + for (let i = 0; i < 6; i++) { + await createTestLog({ projectId: project.id, service: 'api', level: 'info', time: earlierToday }); + } + for (let i = 0; i < 2; i++) { + await createTestLog({ projectId: project.id, service: 'api', level: 'error', time: earlierToday }); + } + + const stats = await dashboardService.getStats(organization.id); + + expect(stats.totalLogsToday.value).toBe(8); + expect(stats.errorRate.value).toBeCloseTo(25, 1); // 2 errors / 8 logs + }); + it('should count distinct active services', async () => { const { organization, project } = await createTestContext(); diff --git a/packages/backend/src/tests/modules/exceptions/detection.test.ts b/packages/backend/src/tests/modules/exceptions/detection.test.ts index 5149af5a..2649e2d0 100644 --- a/packages/backend/src/tests/modules/exceptions/detection.test.ts +++ b/packages/backend/src/tests/modules/exceptions/detection.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from 'vitest'; import { ExceptionDetectionService } from '../../../modules/exceptions/detection.js'; +import { FingerprintService } from '../../../modules/exceptions/fingerprint-service.js'; describe('ExceptionDetectionService', () => { describe('detectException', () => { @@ -531,6 +532,52 @@ describe('ExceptionDetectionService', () => { expect(result?.frames[1].isAppCode).toBe(true); expect(result?.frames[2].isAppCode).toBe(true); }); + + it('should detect node: runtime frames as library code', () => { + const metadata = { + exception: { + type: 'Error', + message: 'Test', + stacktrace: [ + { file: 'node:internal/process/task_queues', function: 'process.processTicksAndRejections', line: 95 }, + { file: 'node:events', function: 'EventEmitter.emit', line: 1 }, + ], + }, + }; + + const result = ExceptionDetectionService.detectException('Error', metadata); + + expect(result?.frames[0].isAppCode).toBe(false); + expect(result?.frames[1].isAppCode).toBe(false); + }); + + it('should not split the same error across groups when only node: internal frames differ', () => { + // Regression: async Node stacks include varying internal frames + // (node:internal/...). When those were treated as app code they + // entered the fingerprint, so the same logical error fingerprinted + // differently and split into several error groups. + const appFrame = { file: '/app/apps/api/dist/modules/products/routes.js', function: 'Object.', line: 130, column: 14 }; + + const occurrenceA = ExceptionDetectionService.detectException('TypeError', { + exception: { + type: 'TypeError', + message: "Cannot read properties of undefined (reading 'selectFrom')", + stacktrace: [appFrame, { file: 'node:internal/process/task_queues', function: 'process.processTicksAndRejections', line: 95 }], + }, + }); + + const occurrenceB = ExceptionDetectionService.detectException('TypeError', { + exception: { + type: 'TypeError', + message: "Cannot read properties of undefined (reading 'selectFrom')", + stacktrace: [appFrame, { file: 'node:internal/async_hooks', function: 'AsyncResource.runInAsyncScope', line: 206 }], + }, + }); + + expect(occurrenceA).not.toBeNull(); + expect(occurrenceB).not.toBeNull(); + expect(FingerprintService.generate(occurrenceA!)).toBe(FingerprintService.generate(occurrenceB!)); + }); }); describe('raw trace reconstruction', () => { diff --git a/packages/backend/src/tests/modules/exceptions/routes.test.ts b/packages/backend/src/tests/modules/exceptions/routes.test.ts index c46ecaaf..f24ddb2e 100644 --- a/packages/backend/src/tests/modules/exceptions/routes.test.ts +++ b/packages/backend/src/tests/modules/exceptions/routes.test.ts @@ -842,4 +842,133 @@ describe('Exceptions Routes', () => { expect(response.statusCode).toBe(200); }); }); + + describe('error-group duplicates and merge', () => { + it('GET /:id/duplicates returns same type+message groups', async () => { + const target = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `t-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + }); + const dup = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `d-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + }); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/error-groups/${target.id}/duplicates`, + headers: { Authorization: `Bearer ${authToken}` }, + query: { organizationId: testOrganization.id }, + }); + + expect(response.statusCode).toBe(200); + expect(response.json().duplicates.map((d: any) => d.id)).toContain(dup.id); + }); + + it('GET /:id/duplicates returns 403 for a non-member organization', async () => { + const other = await createTestContext(); + const group = await createTestErrorGroup({ + organizationId: other.organization.id, + projectId: other.project.id, + }); + + const response = await app.inject({ + method: 'GET', + url: `/api/v1/error-groups/${group.id}/duplicates`, + headers: { Authorization: `Bearer ${authToken}` }, + query: { organizationId: other.organization.id }, + }); + + expect(response.statusCode).toBe(403); + }); + + it('GET /:id/duplicates returns 404 for a missing group', async () => { + const response = await app.inject({ + method: 'GET', + url: `/api/v1/error-groups/00000000-0000-0000-0000-000000000000/duplicates`, + headers: { Authorization: `Bearer ${authToken}` }, + query: { organizationId: testOrganization.id }, + }); + + expect(response.statusCode).toBe(404); + }); + + it('POST /:id/merge folds source groups into the target', async () => { + const target = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `mt-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + occurrenceCount: 1, + }); + const source = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + fingerprint: `ms-${Date.now()}`, + exceptionType: 'TypeError', + exceptionMessage: 'boom', + occurrenceCount: 3, + }); + + const response = await app.inject({ + method: 'POST', + url: `/api/v1/error-groups/${target.id}/merge`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { organizationId: testOrganization.id, sourceIds: [source.id] }, + }); + + expect(response.statusCode).toBe(200); + expect(response.json().occurrenceCount).toBe(4); + + const gone = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', source.id) + .executeTakeFirst(); + expect(gone).toBeUndefined(); + }); + + it('POST /:id/merge returns 400 for empty sourceIds', async () => { + const target = await createTestErrorGroup({ + organizationId: testOrganization.id, + projectId: testProject.id, + }); + + const response = await app.inject({ + method: 'POST', + url: `/api/v1/error-groups/${target.id}/merge`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { organizationId: testOrganization.id, sourceIds: [] }, + }); + + expect(response.statusCode).toBe(400); + }); + + it('POST /:id/merge returns 403 for a non-member organization', async () => { + const other = await createTestContext(); + const target = await createTestErrorGroup({ + organizationId: other.organization.id, + projectId: other.project.id, + }); + + const response = await app.inject({ + method: 'POST', + url: `/api/v1/error-groups/${target.id}/merge`, + headers: { Authorization: `Bearer ${authToken}` }, + payload: { + organizationId: other.organization.id, + sourceIds: ['00000000-0000-0000-0000-000000000000'], + }, + }); + + expect(response.statusCode).toBe(403); + }); + }); }); diff --git a/packages/backend/src/tests/modules/exceptions/service.test.ts b/packages/backend/src/tests/modules/exceptions/service.test.ts index 229010e6..71cd550e 100644 --- a/packages/backend/src/tests/modules/exceptions/service.test.ts +++ b/packages/backend/src/tests/modules/exceptions/service.test.ts @@ -1,4 +1,5 @@ import { describe, it, expect, beforeEach } from 'vitest'; +import { randomUUID } from 'node:crypto'; import { ExceptionService } from '../../../modules/exceptions/service.js'; import { db } from '../../../database/index.js'; import { @@ -93,6 +94,199 @@ describe('ExceptionService', () => { }); }); + describe('error group service attribution', () => { + const parsedData = { + exceptionType: 'TypeError', + exceptionMessage: "Cannot read properties of undefined (reading 'x')", + language: 'nodejs' as const, + rawStackTrace: 'TypeError: x\n at h (/app/h.js:1:1)', + frames: [], + }; + + async function affectedServices(orgId: string, fingerprint: string): Promise { + const group = await db + .selectFrom('error_groups') + .select('affected_services') + .where('organization_id', '=', orgId) + .where('fingerprint', '=', fingerprint) + .executeTakeFirst(); + return (group?.affected_services as string[] | undefined) ?? []; + } + + it('attributes the service carried on the exception, not the Postgres logs lookup', async () => { + // Simulate a non-TimescaleDB reservoir (ClickHouse / MongoDB): the log is + // NOT in the Postgres logs table, so the trigger's logs lookup finds + // nothing. The service must still come through from the exception row. + const ctx = await createTestContext(); + const fingerprint = `svc-carry-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), // no matching row in `logs` + fingerprint, + service: 'checkout-api', + parsedData, + }); + + expect(await affectedServices(ctx.organization.id, fingerprint)).toEqual(['checkout-api']); + }); + + it('falls back to the logs table when no service is carried (TimescaleDB path)', async () => { + const ctx = await createTestContext(); + const log = await createTestLog({ + projectId: ctx.project.id, + level: 'error', + service: 'billing-worker', + }); + const fingerprint = `svc-fallback-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: log.id, + fingerprint, + parsedData, + }); + + expect(await affectedServices(ctx.organization.id, fingerprint)).toEqual(['billing-worker']); + }); + + it('never leaves a group as unknown when the service is known at ingestion', async () => { + const ctx = await createTestContext(); + const fingerprint = `svc-known-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint, + service: 'api', + parsedData, + }); + + expect(await affectedServices(ctx.organization.id, fingerprint)).not.toContain('unknown'); + }); + }); + + describe('mergeErrorGroups', () => { + const parsed = { + exceptionType: 'TypeError', + exceptionMessage: "Cannot read properties of undefined (reading 'x')", + language: 'nodejs' as const, + rawStackTrace: 'TypeError: x\n at h (/app/h.js:1:1)', + frames: [], + }; + + it('folds duplicate groups into one and reassigns their exceptions', async () => { + const ctx = await createTestContext(); + const fpTarget = `merge-target-${randomUUID().slice(0, 8)}`; + const fpSource = `merge-source-${randomUUID().slice(0, 8)}`; + + // Target: 1 occurrence on service "api" + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpTarget, + service: 'api', + parsedData: parsed, + }); + // Source: 2 occurrences on service "worker", same type+message, different fingerprint + for (let i = 0; i < 2; i++) { + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpSource, + service: 'worker', + parsedData: parsed, + }); + } + + const groups = await db + .selectFrom('error_groups') + .select(['id', 'fingerprint']) + .where('organization_id', '=', ctx.organization.id) + .execute(); + const target = groups.find((g) => g.fingerprint === fpTarget)!; + const source = groups.find((g) => g.fingerprint === fpSource)!; + + // Same type+message groups surface as duplicates of each other. + const dups = await service.findDuplicateErrorGroups(target.id, ctx.organization.id); + expect(dups.map((d) => d.id)).toContain(source.id); + + const merged = await service.mergeErrorGroups(target.id, [source.id], ctx.organization.id); + expect(merged).not.toBeNull(); + expect(merged!.occurrenceCount).toBe(3); // 1 + 2 + expect([...merged!.affectedServices].sort()).toEqual(['api', 'worker']); + + // Source group is gone + const remaining = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', source.id) + .executeTakeFirst(); + expect(remaining).toBeUndefined(); + + // Its exceptions now point at the target fingerprint + const stillSource = await db + .selectFrom('exceptions') + .select('id') + .where('organization_id', '=', ctx.organization.id) + .where('fingerprint', '=', fpSource) + .execute(); + expect(stillSource.length).toBe(0); + }); + + it('does not merge groups from another organization', async () => { + const ctx = await createTestContext(); + const other = await createTestContext(); + const fpTarget = `merge-iso-t-${randomUUID().slice(0, 8)}`; + const fpOther = `merge-iso-o-${randomUUID().slice(0, 8)}`; + + await service.createException({ + organizationId: ctx.organization.id, + projectId: ctx.project.id, + logId: randomUUID(), + fingerprint: fpTarget, + service: 'api', + parsedData: parsed, + }); + await service.createException({ + organizationId: other.organization.id, + projectId: other.project.id, + logId: randomUUID(), + fingerprint: fpOther, + service: 'api', + parsedData: parsed, + }); + + const target = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', ctx.organization.id) + .where('fingerprint', '=', fpTarget) + .executeTakeFirstOrThrow(); + const otherGroup = await db + .selectFrom('error_groups') + .select(['id']) + .where('organization_id', '=', other.organization.id) + .where('fingerprint', '=', fpOther) + .executeTakeFirstOrThrow(); + + // Attempting to merge another org's group is a no-op; it stays put. + const merged = await service.mergeErrorGroups(target.id, [otherGroup.id], ctx.organization.id); + expect(merged!.occurrenceCount).toBe(1); + const survives = await db + .selectFrom('error_groups') + .select('id') + .where('id', '=', otherGroup.id) + .executeTakeFirst(); + expect(survives).not.toBeUndefined(); + }); + }); + describe('getExceptionByLogId', () => { it('should return exception with frames for valid log', async () => { const ctx = await createTestContext(); diff --git a/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts b/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts index 8eaae7d6..905f6591 100644 --- a/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts +++ b/packages/backend/src/tests/queue/jobs/exception-parsing.test.ts @@ -409,6 +409,160 @@ ValueError: Repeated error`; ); }); + it('reopens a resolved error group and flags a regression when it recurs', async () => { + const owner = await createTestUser({ name: 'Owner User' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Test Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + const pythonStackTrace = `Traceback (most recent call last): + File "/app/regress.py", line 10, in + do_something() +ValueError: Regressed error`; + + const log1 = await createTestLog({ projectId: project.id, level: 'error', message: pythonStackTrace, service: 'test-service' }); + const log2 = await createTestLog({ projectId: project.id, level: 'error', message: pythonStackTrace, service: 'test-service' }); + + const mkJob = (logId: string, message: string) => + ({ + data: { + logs: [{ id: logId, message, level: 'error' as const, service: 'test-service' }], + organizationId: org.id, + projectId: project.id, + } as ExceptionParsingJobData, + } as Job); + + await processExceptionParsing(mkJob(log1.id, log1.message)); + + // Operator resolves the group + await db.updateTable('error_groups').set({ status: 'resolved' }).where('organization_id', '=', org.id).execute(); + + vi.clearAllMocks(); + mockState.queueAdd.mockResolvedValue({ id: 'job-id' }); + + // The same error recurs + await processExceptionParsing(mkJob(log2.id, log2.message)); + + expect(mockState.queueAdd).toHaveBeenCalledWith( + 'error-notification', + expect.objectContaining({ isRegression: true }), + expect.any(Object) + ); + + const group = await db + .selectFrom('error_groups') + .select('status') + .where('organization_id', '=', org.id) + .executeTakeFirstOrThrow(); + expect(group.status).toBe('open'); + }); + + describe('auto-merge at ingestion', () => { + async function ingest( + org: string, + project: string, + frames: Array<{ file: string; function: string; line: number }>, + message = 'boom', + type = 'TypeError' + ) { + const log = await createTestLog({ projectId: project, level: 'error', message: `${type}: ${message}`, service: 'svc' }); + const job = { + data: { + logs: [ + { + id: log.id, + message: log.message, + level: 'error' as const, + service: 'svc', + metadata: { exception: { type, message, language: 'nodejs', stacktrace: frames } }, + }, + ], + organizationId: org, + projectId: project, + } as ExceptionParsingJobData, + } as Job; + await processExceptionParsing(job); + } + + async function groupCount(org: string): Promise { + const rows = await db.selectFrom('error_groups').select('id').where('organization_id', '=', org).execute(); + return rows.length; + } + + it('folds a stack that differs only in deeper app frames into one group', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + // Same top app frame + message, different SECOND app frame -> different + // fingerprint (would split) but same merge key (auto-folds). + await ingest(org.id, project.id, [ + { file: '/app/src/charge.js', function: 'charge', line: 10 }, + { file: '/app/src/handlerA.js', function: 'handleA', line: 5 }, + ]); + await ingest(org.id, project.id, [ + { file: '/app/src/charge.js', function: 'charge', line: 10 }, + { file: '/app/src/handlerB.js', function: 'handleB', line: 7 }, + ]); + + expect(await groupCount(org.id)).toBe(1); + }); + + it('keeps errors from different throw sites in separate groups', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + await ingest(org.id, project.id, [{ file: '/app/src/charge.js', function: 'charge', line: 10 }]); + await ingest(org.id, project.id, [{ file: '/app/src/refund.js', function: 'refund', line: 20 }]); + + expect(await groupCount(org.id)).toBe(2); + }); + + it('folds messages that differ only by dynamic values', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + await ingest( + org.id, + project.id, + [ + { file: '/app/src/net.js', function: 'call', line: 3 }, + { file: '/app/src/a.js', function: 'a', line: 1 }, + ], + 'Timeout after 5023ms' + ); + await ingest( + org.id, + project.id, + [ + { file: '/app/src/net.js', function: 'call', line: 3 }, + { file: '/app/src/b.js', function: 'b', line: 2 }, + ], + 'Timeout after 812ms' + ); + + expect(await groupCount(org.id)).toBe(1); + }); + + it('does not compute a merge key for a library-only exception', async () => { + const owner = await createTestUser({ name: 'Owner' }); + const org = await createTestOrganization({ ownerId: owner.id, name: 'Org' }); + const project = await createTestProject({ organizationId: org.id, userId: owner.id }); + + await ingest(org.id, project.id, [ + { file: '/app/node_modules/pg/lib/client.js', function: 'query', line: 1 }, + ]); + + const group = await db + .selectFrom('error_groups') + .select('merge_key') + .where('organization_id', '=', org.id) + .executeTakeFirstOrThrow(); + expect(group.merge_key).toBeNull(); + }); + }); + it('should handle empty logs array', async () => { const owner = await createTestUser({ name: 'Owner User' }); const org = await createTestOrganization({ ownerId: owner.id, name: 'Test Org' }); diff --git a/packages/frontend/package.json b/packages/frontend/package.json index 529da4b2..eee96373 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -20,6 +20,7 @@ "sourcemaps:upload": "logtide sourcemaps upload build/client --release $npm_package_version" }, "dependencies": { + "@internationalized/date": "^3.12.2", "@logtide/core": "0.7.0", "@logtide/shared": "workspace:*", "@logtide/sveltekit": "0.7.0", diff --git a/packages/frontend/src/lib/api/alerts.ts b/packages/frontend/src/lib/api/alerts.ts index 6bf4e48e..1438f469 100644 --- a/packages/frontend/src/lib/api/alerts.ts +++ b/packages/frontend/src/lib/api/alerts.ts @@ -5,6 +5,23 @@ import type { LogLevel, MetadataFilter, MetadataFilterInput } from '@logtide/sha export type AlertType = 'threshold' | 'rate_of_change'; export type BaselineType = 'same_time_yesterday' | 'same_day_last_week' | 'rolling_7d_avg' | 'percentile_p95'; +/** Optional values to prefill the alert builder (create-from-error, duplicate). */ +export interface AlertBuilderPrefill { + name?: string; + service?: string | null; + levels?: string[]; + threshold?: number; + timeWindow?: number; + alertType?: AlertType; + baselineType?: BaselineType | null; + deviationMultiplier?: number | null; + minBaselineValue?: number | null; + cooldownMinutes?: number | null; + sustainedMinutes?: number | null; + metadataFilters?: MetadataFilterInput[]; + channelIds?: string[]; +} + export interface AlertRule { id: string; organizationId: string; diff --git a/packages/frontend/src/lib/api/exceptions.ts b/packages/frontend/src/lib/api/exceptions.ts index 82319db7..db2f4d7e 100644 --- a/packages/frontend/src/lib/api/exceptions.ts +++ b/packages/frontend/src/lib/api/exceptions.ts @@ -204,6 +204,58 @@ export async function updateErrorGroupStatus( return response.json(); } +export interface DuplicateErrorGroup { + id: string; + occurrenceCount: number; + firstSeen: string; + lastSeen: string; +} + +export async function getDuplicateErrorGroups( + groupId: string, + organizationId: string +): Promise { + const token = getAuthToken(); + const searchParams = new URLSearchParams({ organizationId }); + + const response = await fetch( + `${getApiUrl()}/api/v1/error-groups/${groupId}/duplicates?${searchParams}`, + { headers: { Authorization: `Bearer ${token}` } } + ); + + if (!response.ok) { + const error = await response.json().catch(() => ({})); + throw new Error(error.error || `Failed to load duplicate groups (HTTP ${response.status})`); + } + + const data = await response.json(); + return data.duplicates ?? []; +} + +export async function mergeErrorGroups( + groupId: string, + sourceIds: string[], + organizationId: string +): Promise { + const token = getAuthToken(); + + const response = await fetch(`${getApiUrl()}/api/v1/error-groups/${groupId}/merge`, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${token}`, + }, + body: JSON.stringify({ organizationId, sourceIds }), + }); + + if (!response.ok) { + const error = await response.json().catch(() => ({})); + throw new Error(error.error || `Failed to merge error groups (HTTP ${response.status})`); + } + + return response.json(); +} + export async function getErrorGroupTrend(params: { groupId: string; organizationId: string; diff --git a/packages/frontend/src/lib/api/fetch-interceptor.ts b/packages/frontend/src/lib/api/fetch-interceptor.ts index 32dd038a..6dc0e0ed 100644 --- a/packages/frontend/src/lib/api/fetch-interceptor.ts +++ b/packages/frontend/src/lib/api/fetch-interceptor.ts @@ -38,6 +38,19 @@ function isAuthEndpoint(url: string): boolean { return url.includes('/auth/'); } +// Ingestion / telemetry endpoints authenticate with an API key or DSN, not the +// session. A 401 here means a bad or expired telemetry key (e.g. the client-side +// SDK in hooks.client.ts, or an inbound receiver), NOT an expired user session, +// so it must never log the user out. Otherwise a misconfigured browser SDK key +// bounces the user to /login on every page view. +function isTelemetryEndpoint(url: string): boolean { + return ( + url.includes('/api/v1/ingest') || + url.includes('/v1/otlp/') || + url.includes('/api/v1/receivers/') + ); +} + function handleUnauthorized(): void { if (handling) return; handling = true; @@ -74,7 +87,8 @@ export function installAuthFetchInterceptor(): void { response.status === 401 && getAuthToken() && isApiRequest(urlOf(input)) && - !isAuthEndpoint(urlOf(input)) + !isAuthEndpoint(urlOf(input)) && + !isTelemetryEndpoint(urlOf(input)) ) { handleUnauthorized(); } diff --git a/packages/frontend/src/lib/components/CommandPalette.svelte b/packages/frontend/src/lib/components/CommandPalette.svelte index ad4a63cd..45a56c4f 100644 --- a/packages/frontend/src/lib/components/CommandPalette.svelte +++ b/packages/frontend/src/lib/components/CommandPalette.svelte @@ -20,6 +20,11 @@ import Moon from '@lucide/svelte/icons/moon'; let open = $state(false); + let query = $state(''); + + let trimmedQuery = $derived(query.trim()); + // Trace/span IDs are 16 or 32 lowercase hex characters. + let looksLikeTraceId = $derived(/^[0-9a-f]{16}$|^[0-9a-f]{32}$/i.test(trimmedQuery)); $effect(() => { const unsubscribe = shortcutsStore.subscribe((s) => { @@ -32,6 +37,7 @@ if (newOpen) { shortcutsStore.openCommandPalette(); } else { + query = ''; shortcutsStore.closeCommandPalette(); } } @@ -51,6 +57,7 @@ function navigate(href: string) { goto(href); + query = ''; shortcutsStore.closeCommandPalette(); } @@ -61,10 +68,31 @@ - + No results found. + {#if trimmedQuery} + + {#if looksLikeTraceId} + navigate(`/dashboard/traces?traceId=${encodeURIComponent(trimmedQuery)}`)}> + + Open trace {trimmedQuery} + + {/if} + navigate(`/dashboard/search?q=${encodeURIComponent(trimmedQuery)}`)}> + + Search logs for "{trimmedQuery}" + + navigate(`/dashboard/errors?search=${encodeURIComponent(trimmedQuery)}`)}> + + Search errors for "{trimmedQuery}" + + + + + {/if} + {#each navItems as item} {@const Icon = item.icon} diff --git a/packages/frontend/src/lib/components/CreateAlertDialog.svelte b/packages/frontend/src/lib/components/CreateAlertDialog.svelte index 1082f4ec..1c2000c2 100644 --- a/packages/frontend/src/lib/components/CreateAlertDialog.svelte +++ b/packages/frontend/src/lib/components/CreateAlertDialog.svelte @@ -1,5 +1,5 @@ diff --git a/packages/frontend/src/lib/components/EmptyMetrics.svelte b/packages/frontend/src/lib/components/EmptyMetrics.svelte new file mode 100644 index 00000000..3cc77624 --- /dev/null +++ b/packages/frontend/src/lib/components/EmptyMetrics.svelte @@ -0,0 +1,201 @@ + + +
+ +
+
+ +
+

No Metrics Yet

+

+ Send OTLP metrics from your application to explore counters, gauges and histograms here. +

+
+ + +
+ + + + +

Get API Key

+

Generate credentials

+ + + + + + + +

OTLP Docs

+

Setup guide

+ + + + + + + +

OpenTelemetry

+

Official docs

+ + + +
+ + + + +
+ + Send Metrics with OpenTelemetry +
+ + Configure your application to export OTLP metrics to LogTide + + + + + + Node.js + Python + Go + + + {#each Object.entries(codeExamples) as [key, code]} + +
+ 
{code}
+ +
+ + {/each} + + + + + +
+ + +

Once metrics arrive, you can:

+
    +
  • + + Explore metrics by name and service +
    • +
  • + + Track golden signals at a glance +
    • +
  • + + Visualize time series over any range +
    • +
  • + + Spot trends and anomalies early +
    • +
+ + +
+
diff --git a/packages/frontend/src/lib/components/EmptyTraces.svelte b/packages/frontend/src/lib/components/EmptyTraces.svelte index f9eea66f..7c642901 100644 --- a/packages/frontend/src/lib/components/EmptyTraces.svelte +++ b/packages/frontend/src/lib/components/EmptyTraces.svelte @@ -65,9 +65,10 @@ import ( "go.opentelemetry.io/otel/sdk/trace" ) -exporter, _ := otlptracehttp.New(ctx, +exporter, _ := otlptracehttp.New(context.Background(), otlptracehttp.WithEndpoint("${apiUrlValue.replace('https://', '').replace('http://', '')}"), otlptracehttp.WithURLPath("/v1/otlp/traces"), + otlptracehttp.WithInsecure(), // remove for HTTPS otlptracehttp.WithHeaders(map[string]string{ "X-API-Key": "YOUR_API_KEY", }), diff --git a/packages/frontend/src/lib/components/ThemeToggle.svelte b/packages/frontend/src/lib/components/ThemeToggle.svelte index 2b6483c6..df816a52 100644 --- a/packages/frontend/src/lib/components/ThemeToggle.svelte +++ b/packages/frontend/src/lib/components/ThemeToggle.svelte @@ -1,27 +1,49 @@ - + + + + + + {#each options as opt} + themeStore.setPreference(opt.value)}> + + {opt.label} + {#if current === opt.value} + + {/if} + + {/each} + + diff --git a/packages/frontend/src/lib/components/TimeRangeButtons.svelte b/packages/frontend/src/lib/components/TimeRangeButtons.svelte new file mode 100644 index 00000000..319b3219 --- /dev/null +++ b/packages/frontend/src/lib/components/TimeRangeButtons.svelte @@ -0,0 +1,36 @@ + + +
+ {#each options as opt (opt.value)} + + {/each} +
diff --git a/packages/frontend/src/lib/components/TimeRangePicker.svelte b/packages/frontend/src/lib/components/TimeRangePicker.svelte index d7ac9855..5811026e 100644 --- a/packages/frontend/src/lib/components/TimeRangePicker.svelte +++ b/packages/frontend/src/lib/components/TimeRangePicker.svelte @@ -3,6 +3,9 @@ import Input from "$lib/components/ui/input/input.svelte"; import Label from "$lib/components/ui/label/label.svelte"; import Clock from "@lucide/svelte/icons/clock"; + import { RangeCalendar } from "$lib/components/ui/range-calendar"; + import { CalendarDate, type DateValue } from "@internationalized/date"; + import type { DateRange } from "bits-ui"; export type TimeRangeType = "last_hour" | "last_24h" | "last_7d" | "custom"; @@ -144,12 +147,49 @@ emitChange(); } - function handleCustomTimeChange() { + function emitChange() { + onchange?.(getTimeRange()); + } + + // ── Custom range as calendar (date) + time input (hour) ────────────────── + // customFromTime / customToTime stay the source of truth ("YYYY-MM-DDTHH:mm"); + // the calendar drives the date part and the time inputs the hour part. + const pad = (n: number) => String(n).padStart(2, "0"); + const datePart = (s: string) => (s ? s.slice(0, 10) : ""); + const timePart = (s: string) => (s && s.length >= 16 ? s.slice(11, 16) : "00:00"); + const combine = (d: string, t: string) => (d ? `${d}T${t || "00:00"}` : ""); + + function toCalendarDate(s: string): DateValue | undefined { + const d = datePart(s); + if (!d) return undefined; + const [y, m, day] = d.split("-").map(Number); + if (!y || !m || !day) return undefined; + return new CalendarDate(y, m, day); + } + + function dateStr(d: DateValue): string { + return `${d.year}-${pad(d.month)}-${pad(d.day)}`; + } + + let calendarValue = $derived({ + start: toCalendarDate(customFromTime), + end: toCalendarDate(customToTime), + }); + + function handleRangeChange(range: DateRange | undefined) { + if (!range) return; + if (range.start) customFromTime = combine(dateStr(range.start), timePart(customFromTime)); + if (range.end) customToTime = combine(dateStr(range.end), timePart(customToTime)); emitChange(); } - function emitChange() { - onchange?.(getTimeRange()); + function handleTimeInput(which: "from" | "to", value: string) { + if (which === "from") { + customFromTime = combine(datePart(customFromTime), value); + } else { + customToTime = combine(datePart(customToTime), value); + } + emitChange(); } @@ -190,24 +230,33 @@ {#if timeRangeType === "custom"} -
-
- - +
+
-
- - +
+
+ + handleTimeInput("from", (e.currentTarget as HTMLInputElement).value)} + /> +
+
+ + handleTimeInput("to", (e.currentTarget as HTMLInputElement).value)} + /> +
{/if} diff --git a/packages/frontend/src/lib/components/dashboard/StatsCard.svelte b/packages/frontend/src/lib/components/dashboard/StatsCard.svelte index d0ef9850..a09f32bc 100644 --- a/packages/frontend/src/lib/components/dashboard/StatsCard.svelte +++ b/packages/frontend/src/lib/components/dashboard/StatsCard.svelte @@ -10,11 +10,22 @@ trend?: { value: number; isPositive: boolean; + /** Suffix after the number. Defaults to '%'. Pass '' for a plain count, ' pp' for percentage points. */ + unit?: string; + /** Trailing label. Defaults to 'from last period'. */ + label?: string; }; onclick?: () => void; } let { title, value, icon: Icon, description, trend, onclick }: Props = $props(); + + function formatTrend(t: NonNullable): string { + const unit = t.unit ?? '%'; + const decimals = unit === '' ? 0 : 2; + const sign = t.isPositive ? '+' : ''; + return `${sign}${t.value.toFixed(decimals)}${unit} ${t.label ?? 'from last period'}`; + } {#if onclick} @@ -36,7 +47,7 @@ {/if} {#if trend}

- {trend.isPositive ? '+' : ''}{trend.value.toFixed(2)}% from last period + {formatTrend(trend)}

{/if} diff --git a/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte b/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte index ec49c956..d88c1050 100644 --- a/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte +++ b/packages/frontend/src/lib/components/metrics/OverviewPanel.svelte @@ -1,9 +1,9 @@ {#if displayServices.length === 0} -
- -

No metrics found

-

Start sending OTLP metrics to see them here

-
+ {:else} {#each displayServices as service, si}
diff --git a/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte b/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte index 02065a6c..f80c3fa8 100644 --- a/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte +++ b/packages/frontend/src/lib/components/metrics/ServiceSelector.svelte @@ -1,6 +1,6 @@ + + + {#snippet children({ months, weekdays })} + + + + + + + + + + +
+ {#each months as month (month.value)} + + + + {#each weekdays as weekday (weekday)} + + {weekday.slice(0, 2)} + + {/each} + + + + {#each month.weeks as weekDates (weekDates)} + + {#each weekDates as date (date)} + + + + {/each} + + {/each} + + + {/each} +
+ {/snippet} + diff --git a/packages/frontend/src/lib/stores/current-project.ts b/packages/frontend/src/lib/stores/current-project.ts new file mode 100644 index 00000000..86207343 --- /dev/null +++ b/packages/frontend/src/lib/stores/current-project.ts @@ -0,0 +1,39 @@ +import { writable } from 'svelte/store'; +import { browser } from '$app/environment'; + +// Remembers the last project the user looked at, so pages that have a project +// selector default to the same one across navigations and reloads instead of +// each falling back to "the first project". +const STORAGE_KEY = 'logtide_current_project'; + +function load(): string | null { + if (!browser) return null; + try { + return localStorage.getItem(STORAGE_KEY); + } catch { + return null; + } +} + +function createCurrentProjectStore() { + const { subscribe, set } = writable(load()); + + return { + subscribe, + set: (projectId: string | null) => { + if (browser) { + try { + if (projectId) localStorage.setItem(STORAGE_KEY, projectId); + else localStorage.removeItem(STORAGE_KEY); + } catch { + // localStorage may be unavailable + } + } + set(projectId); + }, + /** Current value without subscribing. */ + get: (): string | null => load(), + }; +} + +export const currentProjectStore = createCurrentProjectStore(); diff --git a/packages/frontend/src/lib/stores/organization.ts b/packages/frontend/src/lib/stores/organization.ts index b1850e12..7ab11b80 100644 --- a/packages/frontend/src/lib/stores/organization.ts +++ b/packages/frontend/src/lib/stores/organization.ts @@ -8,9 +8,39 @@ interface OrganizationState { loading: boolean; } +const ORG_ID_KEY = 'currentOrganizationId'; +const ORG_OBJECT_KEY = 'currentOrganization'; + +/** + * Restore the last selected organization object from localStorage. + * Persisting the whole object (not just its id) lets a hard reload of a + * sub-route render the org name immediately instead of flashing + * "Select organization" while the org list is still being fetched. + */ +function loadCachedOrganization(): OrganizationWithRole | null { + if (!browser) return null; + try { + const raw = localStorage.getItem(ORG_OBJECT_KEY); + return raw ? (JSON.parse(raw) as OrganizationWithRole) : null; + } catch { + return null; + } +} + +function persistCurrentOrganization(org: OrganizationWithRole | null): void { + if (!browser) return; + if (org) { + localStorage.setItem(ORG_ID_KEY, org.id); + localStorage.setItem(ORG_OBJECT_KEY, JSON.stringify(org)); + } else { + localStorage.removeItem(ORG_ID_KEY); + localStorage.removeItem(ORG_OBJECT_KEY); + } +} + const initialState: OrganizationState = { organizations: [], - currentOrganization: null, + currentOrganization: loadCachedOrganization(), loading: false, }; @@ -25,15 +55,18 @@ function createOrganizationStore() { update((state) => { const currentOrganization = state.currentOrganization || organizations[0] || null; - const savedOrgId = browser ? localStorage.getItem('currentOrganizationId') : null; + const savedOrgId = browser ? localStorage.getItem(ORG_ID_KEY) : null; const restoredOrg = savedOrgId ? organizations.find((org) => org.id === savedOrgId) : null; + const resolved = restoredOrg || currentOrganization; + persistCurrentOrganization(resolved); + return { ...state, organizations, - currentOrganization: restoredOrg || currentOrganization, + currentOrganization: resolved, }; }); }, @@ -49,13 +82,7 @@ function createOrganizationStore() { organization = organizationOrId; } - if (browser) { - if (organization) { - localStorage.setItem('currentOrganizationId', organization.id); - } else { - localStorage.removeItem('currentOrganizationId'); - } - } + persistCurrentOrganization(organization); return { ...state, @@ -66,11 +93,14 @@ function createOrganizationStore() { addOrganization: (organization: OrganizationWithRole) => { - update((state) => ({ - ...state, - organizations: [organization, ...state.organizations], - currentOrganization: organization, - })); + update((state) => { + persistCurrentOrganization(organization); + return { + ...state, + organizations: [organization, ...state.organizations], + currentOrganization: organization, + }; + }); }, updateOrganization: (id: string, updates: Partial) => { @@ -84,6 +114,11 @@ function createOrganizationStore() { ? { ...state.currentOrganization, ...updates } : state.currentOrganization; + // Keep the cached object fresh so a reload does not show a stale name. + if (state.currentOrganization?.id === id) { + persistCurrentOrganization(currentOrganization); + } + return { ...state, organizations, @@ -101,6 +136,10 @@ function createOrganizationStore() { ? organizations[0] || null : state.currentOrganization; + if (state.currentOrganization?.id === id) { + persistCurrentOrganization(currentOrganization); + } + return { ...state, organizations, @@ -122,7 +161,7 @@ function createOrganizationStore() { const newOrg = await apiCall(); update((state) => { - if (browser) localStorage.setItem('currentOrganizationId', newOrg.id); + persistCurrentOrganization(newOrg); return { ...state, @@ -147,16 +186,16 @@ function createOrganizationStore() { const orgs = await apiCall(); update((state) => { - const savedOrgId = browser ? localStorage.getItem('currentOrganizationId') : null; + const savedOrgId = browser ? localStorage.getItem(ORG_ID_KEY) : null; const restoredOrg = savedOrgId ? orgs.find((org) => org.id === savedOrgId) : null; const currentOrganization = restoredOrg || orgs[0] || null; - if (browser && currentOrganization && !savedOrgId) { - localStorage.setItem('currentOrganizationId', currentOrganization.id); - } + // Refresh the cache with the authoritative fetched object (fresh name/role), + // or drop it if the previously selected org no longer exists. + persistCurrentOrganization(currentOrganization); return { ...state, @@ -181,8 +220,8 @@ function createOrganizationStore() { clear: () => { - if (browser) localStorage.removeItem('currentOrganizationId'); - set(initialState); + persistCurrentOrganization(null); + set({ organizations: [], currentOrganization: null, loading: false }); }, }; } diff --git a/packages/frontend/src/lib/stores/theme.ts b/packages/frontend/src/lib/stores/theme.ts index 15d5ce04..f532f928 100644 --- a/packages/frontend/src/lib/stores/theme.ts +++ b/packages/frontend/src/lib/stores/theme.ts @@ -1,68 +1,86 @@ -import { writable } from 'svelte/store'; +import { writable, get } from 'svelte/store'; import { browser } from '$app/environment'; +// The theme actually applied to the DOM. export type Theme = 'light' | 'dark'; +// What the user picked. "system" follows the OS and tracks it live. +export type ThemePreference = 'light' | 'dark' | 'system'; const STORAGE_KEY = 'logtide-theme'; -function getInitialTheme(): Theme { +function systemTheme(): Theme { if (!browser) return 'dark'; + return window.matchMedia('(prefers-color-scheme: light)').matches ? 'light' : 'dark'; +} +function getStoredPreference(): ThemePreference { + if (!browser) return 'dark'; try { const stored = localStorage.getItem(STORAGE_KEY); - if (stored === 'light' || stored === 'dark') { + if (stored === 'light' || stored === 'dark' || stored === 'system') { return stored; } } catch { // localStorage may be unavailable (incognito, etc.) } + // No explicit choice yet: follow the OS. + return 'system'; +} - // Check OS preference - if (window.matchMedia('(prefers-color-scheme: light)').matches) { - return 'light'; - } - - return 'dark'; +function resolve(pref: ThemePreference): Theme { + return pref === 'system' ? systemTheme() : pref; } function createThemeStore() { - const initialTheme = getInitialTheme(); - const { subscribe, set, update } = writable(initialTheme); + const initialPref = getStoredPreference(); + const preference = writable(initialPref); + // Public store yields the RESOLVED theme, so existing consumers keep working. + const resolved = writable(resolve(initialPref)); - function applyTheme(theme: Theme) { - if (browser) { - const root = document.documentElement; - if (theme === 'dark') { - root.classList.add('dark'); - } else { - root.classList.remove('dark'); - } + function apply(theme: Theme) { + if (!browser) return; + document.documentElement.classList.toggle('dark', theme === 'dark'); + } - try { - localStorage.setItem(STORAGE_KEY, theme); - } catch { - // localStorage may be unavailable - } + function persist(pref: ThemePreference) { + if (!browser) return; + try { + localStorage.setItem(STORAGE_KEY, pref); + } catch { + // localStorage may be unavailable } } - // Apply initial theme if (browser) { - applyTheme(initialTheme); + apply(resolve(initialPref)); + // Track OS changes while the preference follows the system. + window + .matchMedia('(prefers-color-scheme: dark)') + .addEventListener('change', () => { + if (get(preference) === 'system') { + const t = systemTheme(); + resolved.set(t); + apply(t); + } + }); + } + + function setPreference(pref: ThemePreference) { + preference.set(pref); + persist(pref); + const t = resolve(pref); + resolved.set(t); + apply(t); } return { - subscribe, - set: (theme: Theme) => { - set(theme); - applyTheme(theme); - }, + subscribe: resolved.subscribe, + preference: { subscribe: preference.subscribe }, + setPreference, + // Backward-compatible helpers. + set: (theme: Theme) => setPreference(theme), toggle: () => { - update((current) => { - const newTheme = current === 'dark' ? 'light' : 'dark'; - applyTheme(newTheme); - return newTheme; - }); + setPreference(get(resolved) === 'dark' ? 'light' : 'dark'); }, }; } diff --git a/packages/frontend/src/lib/stores/time-range.ts b/packages/frontend/src/lib/stores/time-range.ts new file mode 100644 index 00000000..793874aa --- /dev/null +++ b/packages/frontend/src/lib/stores/time-range.ts @@ -0,0 +1,49 @@ +import { writable } from 'svelte/store'; +import { browser } from '$app/environment'; + +// Remembers the last time window the user picked, so it carries across the +// time-driven pages (logs, traces) instead of each resetting to its own default. +// Consumers validate `type` against what they support and fall back otherwise, +// since not every page offers the same presets. +const STORAGE_KEY = 'logtide_time_range'; + +export interface StoredTimeRange { + type: string; + from: string; + to: string; +} + +function load(): StoredTimeRange | null { + if (!browser) return null; + try { + const raw = localStorage.getItem(STORAGE_KEY); + if (!raw) return null; + const parsed = JSON.parse(raw); + if (parsed && typeof parsed.type === 'string') return parsed as StoredTimeRange; + return null; + } catch { + return null; + } +} + +function createTimeRangeStore() { + const { subscribe, set } = writable(load()); + + return { + subscribe, + set: (value: StoredTimeRange | null) => { + if (browser) { + try { + if (value) localStorage.setItem(STORAGE_KEY, JSON.stringify(value)); + else localStorage.removeItem(STORAGE_KEY); + } catch { + // localStorage may be unavailable + } + } + set(value); + }, + get: (): StoredTimeRange | null => load(), + }; +} + +export const timeRangeStore = createTimeRangeStore(); diff --git a/packages/frontend/src/lib/utils/datetime.ts b/packages/frontend/src/lib/utils/datetime.ts index 9b39f4e7..52403e28 100644 --- a/packages/frontend/src/lib/utils/datetime.ts +++ b/packages/frontend/src/lib/utils/datetime.ts @@ -64,6 +64,25 @@ export function formatTimeAgo(date: Date | string, now: Date = new Date()): stri return `${diffDays} day${diffDays > 1 ? 's' : ''} ago`; } +/** + * Format a date as a short, consistent absolute date (e.g. "Apr 12, 2026"). + * Use for record dates (joined, created) where an absolute value reads better + * than a relative one, pairing it with formatTimeAgo() in a tooltip. + */ +export function formatDateShort(date: Date | string): string { + const dateObj = typeof date === 'string' ? new Date(date) : date; + + if (isNaN(dateObj.getTime())) { + return 'Invalid date'; + } + + return dateObj.toLocaleDateString('en-US', { + year: 'numeric', + month: 'short', + day: 'numeric' + }); +} + /** * Get the browser's timezone */ diff --git a/packages/frontend/src/routes/dashboard/+page.svelte b/packages/frontend/src/routes/dashboard/+page.svelte index 24fe5287..e5fc2278 100644 --- a/packages/frontend/src/routes/dashboard/+page.svelte +++ b/packages/frontend/src/routes/dashboard/+page.svelte @@ -19,6 +19,8 @@ } from '$lib/stores/custom-dashboards'; import DashboardContainer from '$lib/components/custom-dashboards/DashboardContainer.svelte'; import DashboardSwitcher from '$lib/components/custom-dashboards/DashboardSwitcher.svelte'; + import * as Select from '$lib/components/ui/select'; + import RefreshCw from '@lucide/svelte/icons/refresh-cw'; import PanelConfigDialog from '$lib/components/custom-dashboards/PanelConfigDialog.svelte'; import AddPanelDialog from '$lib/components/custom-dashboards/AddPanelDialog.svelte'; import CreateDashboardDialog from '$lib/components/custom-dashboards/CreateDashboardDialog.svelte'; @@ -168,6 +170,34 @@ // ─── Edit mode ──────────────────────────────────────────────────────── + // Auto-refresh (global, for the active dashboard's panels) + let autoRefreshMs = $state(0); + const autoRefreshOptions = [ + { value: '0', label: 'Auto-refresh off', short: 'Off' }, + { value: '30000', label: 'Every 30s', short: '30s' }, + { value: '60000', label: 'Every 1m', short: '1m' }, + { value: '300000', label: 'Every 5m', short: '5m' }, + ]; + const autoRefreshShort = $derived( + autoRefreshOptions.find((o) => o.value === String(autoRefreshMs))?.short ?? 'Off' + ); + + function setAutoRefresh(v: string | undefined) { + autoRefreshMs = v ? parseInt(v, 10) || 0 : 0; + if (browser) localStorage.setItem('logtide_dashboard_autorefresh', String(autoRefreshMs)); + } + + $effect(() => { + if (!browser) return; + const ms = autoRefreshMs; + // Pause while editing so a refresh does not clobber unsaved layout edits. + if (ms <= 0 || $editMode) return; + const id = setInterval(() => { + void customDashboardsStore.fetchAllPanelData(); + }, ms); + return () => clearInterval(id); + }); + function startEdit() { customDashboardsStore.enterEditMode(); } @@ -235,6 +265,11 @@ // ─── Shortcuts ──────────────────────────────────────────────────────── onMount(() => { + const savedAutoRefresh = localStorage.getItem('logtide_dashboard_autorefresh'); + if (savedAutoRefresh) { + const parsed = parseInt(savedAutoRefresh, 10); + if (!isNaN(parsed) && parsed >= 0) autoRefreshMs = parsed; + } shortcutsStore.setScope('dashboard'); shortcutsStore.register([ { @@ -307,6 +342,21 @@ {$dashboardSaving ? 'Saving…' : 'Save'} {:else if $activeDashboard} + + + + {autoRefreshShort} + + + {#each autoRefreshOptions as opt} + {opt.label} + {/each} + +
+ {#if recentSearches.length > 0 && !searchQuery.trim()} +
+ Recent: + {#each recentSearches as term (term)} + + {/each} + +
+ {/if} +
@@ -1762,6 +1898,17 @@ {/if} {#if viewMode === "table"} + { columnStore?.set(cols); }} @@ -1805,7 +1952,7 @@ {:else}
- +
Time @@ -1824,7 +1971,7 @@ {@const globalIndex = i} - {formatDateTime(log.time)} + {formatDateTime(log.time)} - {log.message} + + {#each highlightSegments(log.message, searchQuery) as seg} + {#if seg.match}{seg.text}{:else}{seg.text}{/if} + {/each} + {#each customColumns as col (col)} {@const cellValue = resolveMetadataPath(log.metadata, col)}
-
- - - -
+ handleTimeRangeChange(v as typeof timeRange)} + />
- {formatTimeAgo(member.createdAt)} + {formatDateShort(member.createdAt)} {#if canManage} diff --git a/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte b/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte index e46109a8..3cc2e1b9 100644 --- a/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte +++ b/packages/frontend/src/routes/dashboard/settings/pii-masking/+page.svelte @@ -282,6 +282,7 @@ // Update locally to avoid full page refresh rules = rules.map((r) => r.id === rule.id ? { ...r, enabled: !r.enabled } : r); } + toastStore.success(!rule.enabled ? `${rule.displayName} enabled` : `${rule.displayName} disabled`); } catch (e) { const msg = e instanceof Error ? e.message : 'Failed to update rule'; toastStore.error(msg); @@ -308,6 +309,7 @@ // Update locally to avoid full page refresh rules = rules.map((r) => r.id === rule.id ? { ...r, action: newAction } : r); } + toastStore.success(`${rule.displayName} set to ${newAction}`); } catch (e) { const msg = e instanceof Error ? e.message : 'Failed to update action'; toastStore.error(msg); diff --git a/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte b/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte index 42714dae..fc5f181b 100644 --- a/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte +++ b/packages/frontend/src/routes/dashboard/settings/usage/+page.svelte @@ -19,6 +19,7 @@ TableRow, } from '$lib/components/ui/table'; import { Button } from '$lib/components/ui/button'; + import TimeRangeButtons from '$lib/components/TimeRangeButtons.svelte'; import Spinner from '$lib/components/Spinner.svelte'; import type { OrganizationWithRole } from '@logtide/shared'; import BarChart3 from '@lucide/svelte/icons/bar-chart-3'; @@ -208,19 +209,11 @@
-
- {#each RANGES as range} - - {/each} -
+ ({ value: String(r.days), label: r.label }))} + value={String(selectedDays)} + onChange={(v) => { selectedDays = parseInt(v, 10) as RangeDays; }} + />