diff --git a/README.md b/README.md index 44de3b96..21057bb6 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,11 @@ # ASAPQuery **ASAPQuery** is a drop-in query accelerator for queries in various languages like PromQL and SQL. ASAPQuery delivers: -- **100x latency reduction** for various complex aggregate queries such as quantiles -- **Configurable query accuracy** -- **Ease-of-use with your existing tech stack** +- 100x latency reduction for various complex aggregate queries such as quantiles +- Lower CPU and memory usage during querying +- Ease-of-use with your existing tech stack + +The current version of ASAPQuery works with a Prometheus-Grafana stack. We are extending it to work seamlessly with a Clickhouse-Grafana stack. ![ASAPQuery intercepts queries between Grafana and Prometheus, and accelerates them. It ingests data using Prometheus' remote_write interface](assets/img/asapquery_intro_figure.jpg) @@ -14,9 +16,9 @@ ASAPQuery sits between Prometheus and Grafana. It intercepts queries from Grafan It ingests data using Prometheus' remote_write interface. ASAPQuery also targets other observability systems (e.g. VictoriaMetrics) and time-series databases (e.g. Clickhouse, Elastic). -## Quick Start +## Try ASAPQuery with our self-contained demo -**Try ASAPQuery in 5 minutes** with our self-contained demo: +Try ASAPQuery in 5 minutes with our self-contained demo: ```bash cd asap-quickstart @@ -27,7 +29,9 @@ Open http://localhost:3000 and see ASAPQuery vs Prometheus side-by-side! Full quickstart instructions at [**Quickstart Guide**](asap-quickstart/README.md) -**Already have Prometheus and Grafana running?** Use the drop-in instead: +If you already have Prometheus and Grafana running, follow the instruction below for asap-dropin + +## Try ASAPQuery with your Prometheus-Grafana stack ```bash cd asap-dropin @@ -37,58 +41,48 @@ docker compose up -d Full drop-in instructions at [**Drop-in Guide**](asap-dropin/README.md) +## Supported Queries + +ASAPQuery accelerates PromQL queries that match the following patterns. Queries that don't match are transparently forwarded to Prometheus. + +**Temporal range functions** (no label selectors): +- `rate(metric[range])` +- `increase(metric[range])` +- `sum_over_time(metric[range])`, `count_over_time(metric[range])`, `avg_over_time(metric[range])`, `min_over_time(metric[range])`, `max_over_time(metric[range])` +- `quantile_over_time(φ, metric[range])` + +**Aggregation operators** (with optional `by (label, ...)` clause, no label selectors): +- `sum(...)`, `count(...)`, `avg(...)`, `min(...)`, `max(...)`, `quantile(φ, ...)`, `topk(k, ...)` + +**Binary arithmetic** — arithmetic combinations of the above patterns: +- e.g. `rate(errors_total[5m]) / rate(requests_total[5m])` + ## Why ASAPQuery? ### The Problem Prometheus (and most time-series analytics) struggle with: -- **High-cardinality metrics** that slow down queries -- **Complex aggregations** such as percentiles -- **Long time windows** - `quantile_over_time(...[1h])` can take seconds or fail -- **Memory pressure** - Loading all raw timeseries required for query computation +- High-cardinality metrics that slow down queries +- Complex aggregations such as percentiles +- Long time windows - `quantile_over_time(...[1h])` can take seconds or fail +- Memory pressure - Loading all raw timeseries required for query computation ### The Solution -ASAPQuery uses **streaming sketches** to: -1. **Pre-compute approximate summaries** as data arrives -2. **Answer queries in milliseconds** using compact sketches instead of raw data -3. **Bound memory usage** - sketches are fixed-size regardless of data volume -4. **Maintain high accuracy** - configurable error bounds (typically <1% error) +ASAPQuery uses streaming sketches to: +1. Pre-compute approximate summaries as data arrives +2. Answer queries in milliseconds using compact sketches instead of raw data +3. Bound memory usage - sketches are fixed-size regardless of data volume ## Architecture -ASAPQuery has two main components: **asap-planner-rs** analyzes your PromQL query workload and generates sketch configurations, and **asap-query-engine** intercepts PromQL queries and serves them from pre-computed sketches. The query engine includes a built-in **precompute engine** that continuously builds sketches directly from live metrics via Prometheus `remote_write` — no external streaming infrastructure required. +ASAPQuery has two main components: **asap-planner-rs** analyzes your PromQL query workload and generates sketch configurations, and **asap-query-engine** intercepts PromQL queries and serves them from pre-computed sketches. The query engine includes a built-in streaming engine that continuously builds sketches directly from live metrics via Prometheus `remote_write`. ### Components - **[asap-planner-rs](asap-planner-rs/)** - Analyzes a PromQL query workload and auto-generates sketch configurations for asap-query-engine - **[asap-query-engine](asap-query-engine/)** - Intercepts incoming PromQL queries and serves them from pre-computed sketches, falling back to Prometheus for unsupported queries; includes a built-in precompute engine that continuously builds sketches from live metrics via Prometheus `remote_write` -### Repository Structure - -``` -├── asap-quickstart/ # Self-contained demo (start here!) -├── asap-dropin/ # Drop-in for existing Prometheus-Grafana stacks -├── asap-planner-rs/ # Auto-configuration service -└── asap-query-engine/ # Query serving and sketch precomputation engine -``` - -## Supported Queries - -ASAPQuery accelerates PromQL queries that match the following patterns. Queries that don't match are transparently forwarded to Prometheus. - -**Temporal range functions** (no label selectors): -- `rate(metric[range])` -- `increase(metric[range])` -- `sum_over_time(metric[range])`, `count_over_time(metric[range])`, `avg_over_time(metric[range])`, `min_over_time(metric[range])`, `max_over_time(metric[range])` -- `quantile_over_time(φ, metric[range])` - -**Aggregation operators** (with optional `by (label, ...)` clause, no label selectors): -- `sum(...)`, `count(...)`, `avg(...)`, `min(...)`, `max(...)`, `quantile(φ, ...)`, `topk(k, ...)` - -**Binary arithmetic** — arithmetic combinations of the above patterns: -- e.g. `rate(errors_total[5m]) / rate(requests_total[5m])` - ## Coming soon 1. Drop-in ASAPQuery artifact that accelerates Clickhouse queries @@ -97,20 +91,17 @@ ASAPQuery accelerates PromQL queries that match the following patterns. Queries ASAPQuery is currently alpha. There are missing features, known bugs, and possible performance issues. We will continue to work on these and create a more mature artifact. +## Known limitations + +- Only supports a single Grafana dashboard at a time, configured to auto-refresh at a fixed interval +- ASAPQuery ingests data via Prometheus `remote_write`, so the rightmost timestamps in Grafana may show no data even when data exists in Prometheus (data hasn't been ingested into ASAPQuery yet). + ## Research ASAPQuery is part of [ProjectASAP](https://projectasap.github.io/), a joint effort by researchers at Carnegie Mellon University and University of Maryland. ASAPQuery is based on academic research on query processing and sketching algorithms. If you are a researcher interested in using or contributing to ASAPQuery, please [contact us](README.md#contact-us). We are happy to help you. -## Development - - - -## Contributing - - - ## License ASAPQuery is licensed under the MIT License. diff --git a/asap-quickstart/README.md b/asap-quickstart/README.md index 645f706d..1cd692ea 100644 --- a/asap-quickstart/README.md +++ b/asap-quickstart/README.md @@ -16,7 +16,7 @@ Then it adds ASAPQuery's components on top: - **Query Engine** - Prometheus-compatible API with sketch-based acceleration and built-in precompute engine - **asap-planner-rs** - Automatically configures sketches from PromQL queries -Once you run the quickstart, you will see a pre-configured Grafana dashboard that compares Prometheues and ASAPQuery side-by-side. You will see **visually indistinguishable** results from Prometheus and ASAPQuery, with ASAPQuery being 100x faster +Once you run the quickstart, you will see a pre-configured Grafana dashboard that compares Prometheues and ASAPQuery side-by-side. You will see almost indistinguishable results from Prometheus and ASAPQuery, with ASAPQuery's dashboard refreshing much faster. ## Prerequisites