Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions src/content/docs/reference/blobs/page-metadata.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,26 @@ JSON-encoded HTTP Archive metadata about the page that was tested.
```
</details>

:::note
There is a lot of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript (see [source code](https://github.com/HTTPArchive/custom-metrics/)) run during the crawl to extract information from the page.

Check the custom metrics before falling back to `response_body`. Custom metrics include the post-JavaScript DOM, are far less brittle than regexing HTML, and are significantly faster and cheaper to query.

For example, to get the `title` and `meta_description` you can use:
```sql
SELECT
page,
custom_metrics.wpt_bodies.title.rendered.primary.text AS title,
custom_metrics.wpt_bodies.meta_description.rendered.primary.text AS meta_description.
FROM
`httparchive.crawl.pages`
WHERE
...
```

The [source code](https://github.com/HTTPArchive/custom-metrics/) of the custom metrics can be useful to understand exactly how they are collected.
:::

## Schema

### `crawl_depth`
Expand Down
20 changes: 20 additions & 0 deletions src/content/docs/reference/tables/requests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,26 @@ See the [Header](/reference/structs/header/) reference for more details.

Text-based response body

:::note
There is lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page.

Check the custom metrics before falling back to `response_body`. Custom metrics include the post-JavaScript DOM, are far less brittle than regexing HTML, and are significantly faster and cheaper to query.

For example, to get the `title` and `meta_description` you can use:
```sql
SELECT
page,
custom_metrics.wpt_bodies.title.rendered.primary.text AS title,
custom_metrics.wpt_bodies.meta_description.rendered.primary.text AS meta_description.
FROM
`httparchive.crawl.pages`
WHERE
...
```

The source of the [custom metrics](https://github.com/HTTPArchive/custom-metrics/) can be useful to understand exactly what is collected.
:::

## Example queries

Here are some common operations you can perform with the `requests` table.
Expand Down