From 3e1e75d123d7bc6bd71f6cca6d69c49fb8d18e14 Mon Sep 17 00:00:00 2001 From: Barry Pollard Date: Tue, 21 Jul 2026 14:00:03 +0100 Subject: [PATCH 1/5] Encourage custom metrics over response_body --- .../docs/reference/blobs/page-metadata.mdx | 20 +++++++++++++++++++ .../docs/reference/tables/requests.mdx | 20 +++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/src/content/docs/reference/blobs/page-metadata.mdx b/src/content/docs/reference/blobs/page-metadata.mdx index e0f871e..9026675 100644 --- a/src/content/docs/reference/blobs/page-metadata.mdx +++ b/src/content/docs/reference/blobs/page-metadata.mdx @@ -30,6 +30,26 @@ JSON-encoded HTTP Archive metadata about the page that was tested. ``` +:::note +There is lots of other metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. + +Always check that before reverting to `response_body` in `requests` table which is slow, does not include the rendered DOM after JavaScript runs, and is expensive to query. + +For example, to get the `title` and `meta_description` you can use: +```sql +SELECT + page, + custom_metrics.wpt_bodies.title.rendered.primary.text AS title, + custom_metrics.wpt_bodies.meta_description.rendered.primary.text AS meta_description. +FROM + `httparchive.crawl.pages` +WHERE + ... +``` + +The source of the [custom metrics](https://github.com/HTTPArchive/custom-metrics/) can be useful to understand exactly what is collected. +::: + ## Schema ### `crawl_depth` diff --git a/src/content/docs/reference/tables/requests.mdx b/src/content/docs/reference/tables/requests.mdx index f9ed34d..119e75e 100644 --- a/src/content/docs/reference/tables/requests.mdx +++ b/src/content/docs/reference/tables/requests.mdx @@ -98,6 +98,26 @@ See the [Header](/reference/structs/header/) reference for more details. Text-based response body +:::note +There is lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. + +Always check that before reverting to `response_body` which is slow, does not include the rendered DOM after JavaScript runs, and is expensive to query. + +For example, to get the `title` and `meta_description` you can use: +```sql +SELECT + page, + custom_metrics.wpt_bodies.title.rendered.primary.text AS title, + custom_metrics.wpt_bodies.meta_description.rendered.primary.text AS meta_description. +FROM + `httparchive.crawl.pages` +WHERE + ... +``` + +The source of the [custom metrics](https://github.com/HTTPArchive/custom-metrics/) can be useful to understand exactly what is collected. +::: + ## Example queries Here are some common operations you can perform with the `requests` table. From 4821f4341234aecd142ec41523bbc7c59303d21e Mon Sep 17 00:00:00 2001 From: Barry Pollard Date: Tue, 21 Jul 2026 15:55:42 +0100 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com> --- src/content/docs/reference/blobs/page-metadata.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/docs/reference/blobs/page-metadata.mdx b/src/content/docs/reference/blobs/page-metadata.mdx index 9026675..51fc55c 100644 --- a/src/content/docs/reference/blobs/page-metadata.mdx +++ b/src/content/docs/reference/blobs/page-metadata.mdx @@ -31,7 +31,7 @@ JSON-encoded HTTP Archive metadata about the page that was tested. :::note -There is lots of other metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. +There is lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. Always check that before reverting to `response_body` in `requests` table which is slow, does not include the rendered DOM after JavaScript runs, and is expensive to query. @@ -47,7 +47,7 @@ WHERE ... ``` -The source of the [custom metrics](https://github.com/HTTPArchive/custom-metrics/) can be useful to understand exactly what is collected. +The [source code](https://github.com/HTTPArchive/custom-metrics/) of the custom metrics can be useful to understand exactly how they are collected. ::: ## Schema From 9d32ab9047d88fb2a8206fd5e075c00b84e54828 Mon Sep 17 00:00:00 2001 From: Barry Pollard Date: Tue, 21 Jul 2026 16:01:20 +0100 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: Barry Pollard --- src/content/docs/reference/blobs/page-metadata.mdx | 4 ++-- src/content/docs/reference/tables/requests.mdx | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/content/docs/reference/blobs/page-metadata.mdx b/src/content/docs/reference/blobs/page-metadata.mdx index 51fc55c..0fc7752 100644 --- a/src/content/docs/reference/blobs/page-metadata.mdx +++ b/src/content/docs/reference/blobs/page-metadata.mdx @@ -31,9 +31,9 @@ JSON-encoded HTTP Archive metadata about the page that was tested. :::note -There is lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. +There are lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. -Always check that before reverting to `response_body` in `requests` table which is slow, does not include the rendered DOM after JavaScript runs, and is expensive to query. +Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results. For example, to get the `title` and `meta_description` you can use: ```sql diff --git a/src/content/docs/reference/tables/requests.mdx b/src/content/docs/reference/tables/requests.mdx index 119e75e..11c69e5 100644 --- a/src/content/docs/reference/tables/requests.mdx +++ b/src/content/docs/reference/tables/requests.mdx @@ -99,9 +99,9 @@ See the [Header](/reference/structs/header/) reference for more details. Text-based response body :::note -There is lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. +There are lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. -Always check that before reverting to `response_body` which is slow, does not include the rendered DOM after JavaScript runs, and is expensive to query. +Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results. For example, to get the `title` and `meta_description` you can use: ```sql From b6a817ef3ef1809d75dc6c0d7cec000481381b06 Mon Sep 17 00:00:00 2001 From: Barry Pollard Date: Tue, 21 Jul 2026 16:02:31 +0100 Subject: [PATCH 4/5] Apply suggestions from code review Co-authored-by: Barry Pollard --- src/content/docs/reference/blobs/page-metadata.mdx | 2 +- src/content/docs/reference/tables/requests.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/docs/reference/blobs/page-metadata.mdx b/src/content/docs/reference/blobs/page-metadata.mdx index 0fc7752..ac37e23 100644 --- a/src/content/docs/reference/blobs/page-metadata.mdx +++ b/src/content/docs/reference/blobs/page-metadata.mdx @@ -31,7 +31,7 @@ JSON-encoded HTTP Archive metadata about the page that was tested. :::note -There are lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. +There is lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results. diff --git a/src/content/docs/reference/tables/requests.mdx b/src/content/docs/reference/tables/requests.mdx index 11c69e5..b09a06c 100644 --- a/src/content/docs/reference/tables/requests.mdx +++ b/src/content/docs/reference/tables/requests.mdx @@ -99,7 +99,7 @@ See the [Header](/reference/structs/header/) reference for more details. Text-based response body :::note -There are lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. +There is lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results. From a80800f2ac5cad9261542714939b68ab79b80605 Mon Sep 17 00:00:00 2001 From: Barry Pollard Date: Tue, 21 Jul 2026 20:17:48 +0100 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Barry Pollard --- src/content/docs/reference/blobs/page-metadata.mdx | 4 ++-- src/content/docs/reference/tables/requests.mdx | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/content/docs/reference/blobs/page-metadata.mdx b/src/content/docs/reference/blobs/page-metadata.mdx index ac37e23..76d1a03 100644 --- a/src/content/docs/reference/blobs/page-metadata.mdx +++ b/src/content/docs/reference/blobs/page-metadata.mdx @@ -31,9 +31,9 @@ JSON-encoded HTTP Archive metadata about the page that was tested. :::note -There is lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. +There is a lot of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript (see [source code](https://github.com/HTTPArchive/custom-metrics/)) run during the crawl to extract information from the page. -Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results. +Check the custom metrics before falling back to `response_body`. Custom metrics include the post-JavaScript DOM, are far less brittle than regexing HTML, and are significantly faster and cheaper to query. For example, to get the `title` and `meta_description` you can use: ```sql diff --git a/src/content/docs/reference/tables/requests.mdx b/src/content/docs/reference/tables/requests.mdx index b09a06c..2b485fe 100644 --- a/src/content/docs/reference/tables/requests.mdx +++ b/src/content/docs/reference/tables/requests.mdx @@ -101,7 +101,7 @@ Text-based response body :::note There is lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page. -Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results. +Check the custom metrics before falling back to `response_body`. Custom metrics include the post-JavaScript DOM, are far less brittle than regexing HTML, and are significantly faster and cheaper to query. For example, to get the `title` and `meta_description` you can use: ```sql