Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/components/Performance.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ describe('Performance - Property 6: Below-fold images are lazy-loaded', () => {
fc.array(
fc.record({
name: fc.string({ minLength: 3, maxLength: 30 })
.map(s => s.replace(/[<>"'&]/g, '')),
.map(s => s.replace(/[<>"'&]/g, '') || 'company name'), // Sanitize with fallback
logo: fc.oneof(
fc.constant('/logos/company1.svg'),
fc.constant('/logos/company2.svg'),
Expand Down
34 changes: 25 additions & 9 deletions docs/starlight-docs/astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,31 @@ export default defineConfig({
},
],
},
{
label: 'Alerting',
collapsed: true,
items: [
{ label: 'Alerting', link: '/alerting/' },
{
label: 'Unified Alerts View',
collapsed: true,
items: [
{ label: 'Overview', link: '/alerting/unified-alerts/' },
],
},
{
label: 'SLOs',
collapsed: true,
items: [
{ label: 'Overview', link: '/slo/' },
{ label: 'Create an SLO', link: '/slo/create/' },
{ label: 'Explore an SLO', link: '/slo/detail/' },
],
},
{ label: 'Anomaly Detection', link: '/anomaly-detection/' },
{ label: 'Forecasting', link: '/forecasting/' },
],
},
{
label: 'PPL - Query Language',
collapsed: true,
Expand Down Expand Up @@ -299,15 +324,6 @@ export default defineConfig({
},
],
},
{
label: 'Alerting',
collapsed: true,
items: [
{ label: 'Alerting', link: '/alerting/' },
{ label: 'Anomaly Detection', link: '/anomaly-detection/' },
{ label: 'Forecasting', link: '/forecasting/' },
],
},
{
label: 'Agent Health',
collapsed: true,
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
53 changes: 5 additions & 48 deletions docs/starlight-docs/src/content/docs/alerting/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,60 +52,17 @@ Set the trigger to fire when the document count exceeds your threshold, and conf

For the full alerting reference - including API operations, composite monitors, alert acknowledgment, and notification channel configuration - see the [Alerting documentation](https://docs.opensearch.org/latest/observing-your-data/alerting/index/) in the official OpenSearch docs.

## Prometheus/Cortex alerting
## Two alerting surfaces

OpenSearch Alerting is one of two alerting surfaces in the stack. The other is a Cortex-side PromQL ruler that evaluates alert rules against time-series metrics and routes firing alerts through Alertmanager. Both surface in the same **Alert Manager** UI in OpenSearch Dashboards, so responders don't need to know which side produced an alert.
OpenSearch Alerting is one of two alerting surfaces. The other is a Prometheus-side PromQL ruler that evaluates alert rules against time-series metrics and routes firing alerts through an alert manager. Both surface in the same **Alert Manager** UI in OpenSearch Dashboards, so responders don't need to know which side produced an alert.

**When to use which:**

| Signal | Use |
|---|---|
| Log-volume thresholds, trace counts, OpenSearch cluster state | OpenSearch Alerting monitors |
| Metric thresholds, rate-based SLO burn, RED-method alerts | Cortex PromQL rules |
| Metric thresholds, rate-based SLO burn, RED-method alerts | Prometheus PromQL rules |

### Rule file locations
Prometheus rules are grouped into namespaces and loaded into the ruler, which evaluates them and forwards firing alerts to the alert manager for grouping, deduplication, and routing to receivers (Slack, email, PagerDuty, webhook, and so on).

Cortex rules are shipped as YAML files mounted into the `alerting-rules-monitors-init` container on startup (and `otel-demo-alerting-rules-monitors-init` when the otel-demo overlay is enabled). Two namespaces are loaded:

- **`stack`** — watches the observability stack itself. Loaded always.
- File: `docker-compose/prometheus/rules-stack/stack-alerts.yml`
- Alerts: `PrometheusTargetDown`, `OtelCollectorExportFailures`, `OtelCollectorHighMemory`, `OtelCollectorQueueNearCapacity`
- **`otel_demo`** — RED-method alerts against the OpenTelemetry demo services. Loaded only when `INCLUDE_COMPOSE_OTEL_DEMO` is enabled in `.env`.
- File: `docker-compose/prometheus/rules-otel-demo/otel-demo-alerts.yml`
- Alerts: `OtelDemoFrontendHighErrorRate`, `OtelDemoFrontendHighLatency`, `OtelDemoFrontendProxyErrors`, `OtelDemoCheckoutErrors`, `OtelDemoPaymentFailures`, `OtelDemoCartErrors`, `OtelDemoServiceHighErrorRate`, `OtelDemoServiceHighLatency`, `OtelDemoAdServiceHighCpu`

To add or edit rules, change the YAML file and re-run the loader:

```bash
docker compose up -d --force-recreate alerting-rules-monitors-init
```

The loader upserts via `POST /api/v1/rules/{namespace}`, so re-runs are idempotent and edits take effect immediately. Inspect loaded groups at `http://localhost:9090/api/v1/rules/stack` or `http://localhost:9090/api/v1/rules/otel_demo` (Cortex returns YAML from this Ruler API endpoint).

### Alertmanager routing

Alertmanager runs on `localhost:9093` and is configured via `docker-compose/alertmanager/alertmanager.template.yml` (credentials are injected at container start). The default routing tree sends:

- `component=observability-stack` alerts → `opensearch-webhook` receiver (posts to the stack's own OpenSearch indices for correlation with traces/logs).
- otel-demo critical alerts → `otel-demo-critical` receiver.
- otel-demo warnings → `otel-demo-warning` receiver.
- Everything else → `null` receiver (dropped).

Placeholder receivers for Slack, email, and PagerDuty are included as examples — replace the dummy URLs with your real endpoints before wiring alerts to production channels. `amtool check-config` validates the template, and `curl http://localhost:9093/api/v2/alerts` lists currently firing alerts.

### The Alert Manager UI

In OpenSearch Dashboards, **Alert Manager** (under the main menu) renders both OpenSearch monitors and Cortex alerts in one list. It reads from two datasources:

- **Local cluster** — OpenSearch Alerting monitors (the ones described earlier on this page).
- **`ObservabilityStack_Prometheus`** — the Cortex datasource configured with `prometheus.uri`, `prometheus.ruler.uri`, and `alertmanager.uri`. The UI pulls firing alerts from Alertmanager, rule definitions from the Cortex Ruler API, and query results from Cortex's PromQL endpoint.

Filter by datasource in the UI's top-right to scope to just one source when investigating.

If the UI shows zero Cortex alerts even though they are firing in Cortex (check `curl http://localhost:9090/prometheus/api/v1/alerts`), confirm the datasource has all three URI properties set:

```bash
curl -u admin:PASSWORD http://localhost:5601/api/dataconnections | jq '.[] | select(.name=="ObservabilityStack_Prometheus") | .properties'
```

The stack's init container reconciles these properties automatically on every run; if they are still missing after a rerun, re-create the datasource with `docker compose down -v && docker compose up -d`.
The **Alert Manager** UI (under the main menu in OpenSearch Dashboards) renders OpenSearch monitors and Prometheus alerts in one list, reading from two datasources: the local OpenSearch cluster for monitors, and a Prometheus datasource for PromQL rules and firing alerts. Filter by datasource to scope to just one source when investigating. For a full walkthrough of this screen, see [Unified Alerts View](/docs/alerting/unified-alerts/).
58 changes: 58 additions & 0 deletions docs/starlight-docs/src/content/docs/alerting/unified-alerts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
title: Unified Alerts View
description: One list for OpenSearch monitors and Prometheus alerts, with rules and routing in the same app
---

The **Unified Alerts View** (labeled **Alerts** in the OpenSearch Dashboards side navigation, and **Alert Manager** in the menu) brings every alert in the stack into a single screen. Whether an alert came from an OpenSearch Alerting monitor or from a Prometheus rule, responders see it in one list, filter it the same way, and never have to know which engine produced it.

![Walkthrough of the Unified Alerts view cycling through the Alerts, Rules, and Routing tabs with both datasources selected](/docs/images/alerting/unified-alerts-walkthrough.gif)

*Cycling through the **Alerts**, **Rules**, and **Routing** tabs with both datasources selected. SLO burn-rate alerts appear in the same queue.*

## Layout

The view has three tabs:

| Tab | What it shows |
|---|---|
| **Alerts** | Every currently firing (and, where available, historical) alert across the selected datasources. |
| **Rules** | Every alerting rule / monitor definition, with status, severity, type, and health. |
| **Routing** | The alert manager routing tree — which receiver each alert is sent to. |

### Filters

The left rail scopes the list without editing any query:

- **Datasource** — pick the OpenSearch cluster (its monitors), the Prometheus datasource (its alerts), or both.
- **Severity** — `critical`, `medium`, and so on, with live counts.
- **State** — `active`, `pending`, `resolved`.
- **Labels** — every label present on the current alert set (`alertname`, `component`, `service`, `job`, …) becomes a facet. This is where Prometheus label cardinality pays off: filter to one service, one exporter, or one component in a click.

### Alert timeline

The histogram at the top of the **Alerts** tab buckets firing alerts over the selected time range (default **Last 24 hours**), colored by severity. Use it to spot bursts — a spike of `critical` bars usually lines up with an incident.

### Rules tab

The **Rules** tab lists every rule/monitor from both engines side by side, with status, severity, type, health, and the owning datasource. Filter by any of those facets.

### Routing tab

The **Routing** tab is a read-only view of the alert manager route tree: which receiver gets which alerts, the grouping and timing settings, and the configured receivers (webhook, Slack, email, PagerDuty). Routing is managed in the alert manager's own configuration.

## Selecting datasources

By default only the OpenSearch cluster is selected. To see Prometheus alerts, tick the **Prometheus datasource** in the **Datasource** filter. The tab counts update immediately to reflect both engines. Because [SLOs](/docs/slo/) deploy their burn-rate alerts as Prometheus rules, SLO breaches appear in this list too — the unified queue is where they surface.

:::note[Prometheus shows current alerts only]
Prometheus does not retain historical alert instances the way OpenSearch does. When the Prometheus datasource is selected you'll see a `Showing current alerts only` banner — the list reflects what is firing *right now*, evaluated every minute, rather than a historical record.
:::

## Empty tabs

If the **Alerts** and **Rules** tabs are empty, there are no monitors or rules for the selected datasources yet, or nothing is currently firing. Confirm the right datasources are selected in the filter, create a monitor or rule, or generate some load so alerts have something to fire on.

## Related

- [Alerting](/docs/alerting/) — the two alerting surfaces and how they route.
- [SLOs](/docs/slo/) — SLO burn-rate and error-budget alerts surface here too.
73 changes: 73 additions & 0 deletions docs/starlight-docs/src/content/docs/slo/create.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
title: Create an SLO
description: Walk through the template-first SLO wizard — pick a template, define the SLI, set objectives, and preview the generated Prometheus rules
---

Creating an SLO starts from a **template** and ends with a deployed Prometheus rule group. The wizard fills in the PromQL for you based on the metric family you pick, and shows a live preview of exactly what will be deployed before you commit.

Open the SLO app (**SLOs** in the side navigation, under Application Performance) and click **Create SLO**.

![Walkthrough of the Create SLO wizard — picking a template, filling identity/service/objective fields, and the live rule-group preview](/docs/images/slo/slo-create-walkthrough.gif)

*Pick a template, fill in identity and objectives, and watch the Prometheus rule group preview build in real time.*

## Step 1 — Pick a template

![SLO template picker with three groups: APM span-derived, OTel semconv metrics, and Custom PromQL](/docs/images/slo/slo-create-templates.png)

Templates are grouped by the metric family they read:

**APM service SLOs (span-derived)** — built from the RED metrics Data Prepper derives from spans for every traced service (`request` / `fault` / `latency_seconds_bucket` with `namespace="span_derived"`).

- **APM service availability** — non-fault request ratio for a service.
- **APM service latency** — fraction of requests under a latency bound (default 500 ms).
- **APM dependency availability** — non-fault ratio for calls a service makes to a downstream dependency.
- **APM dependency latency** — fraction of those dependency calls under a latency bound.

**OTel semconv metrics** — target OpenTelemetry semantic-convention metrics directly.

- **HTTP server availability / latency** — from `http_server_request_duration_seconds_*` (semconv v1.23+).
- **RPC / gRPC availability / latency** — from `rpc_server_duration_seconds_*`.
- **Database client latency** — from `db_client_operation_duration_seconds_bucket`.
- **Messaging processing latency** — from `messaging_process_duration_seconds_bucket`.
- **GenAI invocation availability** — from `gen_ai_client_operation_duration_seconds_count`; good events have `error_type=""`.

**Custom** — **Custom PromQL** starts from a blank slate: supply your own good + total queries, or a single pre-computed error-ratio query.

## Step 2 — Fill in the wizard

The wizard is a single scrollable form with a section jump-nav on the left.

![SLO create wizard sections: identity, window and mode, service and owner, SLI, objectives, advanced, and rule preview](/docs/images/slo/slo-create-wizard.png)

| Section | What to set |
|---|---|
| **Identity** | The Prometheus datasource to target, a **Name**, and an optional description. |
| **Window & mode** | Rolling window — **7 / 14 / 28 (recommended) / 30 days**. Optionally enable **Shadow mode** to deploy recording rules only and suppress alerts while you validate. |
| **Service & owner** | Service name, primary team, optional primary user and tier. These become filter facets in the catalog and labels on the rules. |
| **SLI** | Template-specific. For availability templates, a **good-events filter** (e.g. `error_type=""`) and optional **dimensions** (label selectors like `service_name="weather-agent"`). |
| **Objectives** | One or more targets, each producing its own rule set. The field shows the equivalent decimal and the resulting error budget as a duration (e.g. `99.9% over 28d → 40m 19s`). |
| **Advanced** | Burn-rate multipliers, budget-warning thresholds, and supplemental alarm severities. |
| **Exclusion windows** | Maintenance / deploy-freeze windows to exclude from budget accounting. |
| **Labels & annotations** | Labels propagate to rules as `slo_label_<key>`; annotations (e.g. runbook URLs) stay on the SLO document. |

### Probe the SLI before you commit

The **Probe SLI** button runs the SLI's good and total queries against the target Prometheus backend over a 1h / 24h / 7d lookback and reports the **Good**, **Total**, and **SLI ratio** it found. If it returns zero, the SLO would show `no_data` — a signal to re-check your metric name, filter, or dimensions before creating it.

## Step 3 — Review the rule preview

Before you commit, the **Rule preview** shows the exact Prometheus rule group that will be deployed — its name, the `slo-generated` namespace, the 60s evaluation interval, and the rule count. Click **Show rule-group YAML** if you want to read the generated recording and burn-rate rules line by line; you don't need to, but nothing is hidden.

## Step 4 — Create

Click **Create SLO**. The rule group deploys to Prometheus and you land on the [SLO detail page](/docs/slo/detail/). Recording rules evaluate on a short interval, so charts populate within a minute or two; until the first samples arrive the SLO shows `no_data`.

:::tip[Span-derived SLIs and burn rate]
Span-derived samples are gauge-style — all recording windows record the same instantaneous ratio, so burn-rate alerts are less meaningful. For those templates the wizard suggests relying on attainment-breach alarms instead. OTel semconv counter metrics (HTTP, RPC, GenAI, …) support true multi-window burn rate.
:::

## Related

- [SLOs overview](/docs/slo/)
- [Explore an SLO](/docs/slo/detail/)
Loading
Loading