Skip to content

Setting-up-Search.md

Latest commit

 

History

History
226 lines (151 loc) · 10.7 KB

Setting-up-Search.md

File metadata and controls

226 lines (151 loc) · 10.7 KB
Raw

Setting Up Search

How to Search in ArchiveBox

You can search your ArchiveBox data in a number of ways:

  • using the CLI: archivebox search 'text to search' (archivebox search --help for more)
  • using the Web UI: both the /public index and /admin/core/snapshot pages provide a search box
  • using the REST API: /api/v1/core/snapshots?search=text+to+search&search_mode=contents
  • by searching the archive data folder directly with external tools (e.g. macOS Spotlight, Cerebro, ag, Yacy, etc.)

image


How Search Works

ArchiveBox search works by doing substring matches in Snapshot metadata fields (url, title, timestamp, tags), and by searching the full archived content within each Snapshot (using the selected search backend below). You can find the search implementation source code here: archivebox/core/views.py: PublicIndex.get_queryset().

Note: ArchiveBox currently only returns the bare list of snapshots that match when performing a search.

This will be improved in the future to highlight the specific paragraph/line/area that matched within a Snapshot.
For now we recommend using Ctl+F in the browser or one of the external tools listed above to further filter for a term within a Snapshot's contents.


ArchiveBox Search Backends

ArchiveBox provides a number of "Search Backend Engines" to tune its performance & behavior for different use-cases.

# this setting controls which search backend ArchiveBox uses
archivebox config --set SEARCH_BACKEND_ENGINE=sonic

# to see information about the backend you are currently using, run:
archivebox version
archivebox config --get SEARCH_BACKEND_ENGINE

ArchiveBox installs and enables both Sonic and ripgrep. Sonic is selected by default for fast indexed search, while ripgrep remains available as the filesystem fallback when Sonic is unavailable or explicitly selected.

Tip

You should consider selecting ripgrep instead of Sonic if:

  • Sonic is unavailable on your platform
  • you prefer filesystem scanning without a background daemon
  • you need ripgrep-compatible regular expressions

ripgrep (fallback)

ArchiveBox resolves ripgrep through abxpkg: a compatible host installation is used first, otherwise a managed copy is installed.

archivebox install ripgrep
archivebox config --set SEARCH_BACKEND_ENGINE=ripgrep

# check the resolved provider, version, and projected binary:
archivebox version

# then try it out by searching via the Web UI or CLI:
archivebox search 'text to search for'

Pros

  • supports advanced searching with regex patterns
  • simple, few moving parts, and broadly available for all OSs and CPU architectures
  • 0 idle resource use as there is no background indexer process running
  • 0 additional disk storage needed as it searches the original data instead of maintaining a separate index
  • reasonably fast on NVMe and SSD drives for small collections

Cons

  • very slow as archive collection size increases (doesn't scale well beyond 500~1,000 Snapshots)
  • very slow if underlying filesystem is slow (e.g. HDDs or network mounts)
  • doesn't support stemming, boolean operators, or other advanced full-text search features

ripgrep-all (aka rga)

ripgrep-all supports more binary file types such as PDFs, eBooks, Office documents, zip, and tar files. It is useful as an external companion tool, but it is not currently a supported drop-in binary for ArchiveBox's ripgrep backend. The backend relies on rg's command and output contract.


ugrep

ugrep is another capable external search tool, but it is not a supported drop-in binary for ArchiveBox's ripgrep backend. Contributions adding a dedicated integration are welcome.

ugrep is similar to ripgrep and ripgrep-all in that it's an indexless disk-search tool, but it provides some more of the full-text search features without the performance overhead of maintaining a separate search backend worker with an independent index.

https://github.com/Genivia/ugrep

Pros

  • supports boolean operators in search queries
  • supports binary formats like compressed archives, PDFs, eBooks, etc.
  • better support for Unicode, special characters, and searching across multiple lines of text
  • supports fuzzy search

Cons

  • not as fast as sonic and but also not as simple as ripgrep
  • not all of its features are fully integrated with ArchiveBox yet



sonic ⭐️ (the recommended upgrade path for most people)

Sonic is a fast, lightweight, rust-based alternative to super-heavy traditional search backends like Elasticsearch. It is capable of normalizing natural language search queries, fuzzy matching, and searching Unicode, without needing to maintain a duplicate document store index of all the searchable text.

Internally it functions as an index store, storing only the original IDs of the Snapshots with a super-compressed representation of the text. This allows it to scale to searching terabytes of archive data while maintaining an index only a fraction of that size.

ArchiveBox has supported Sonic for years, and it is the most thoroughly tested and recommended backend for ArchiveBox users that need to scale beyond ripgrep.

ArchiveBox resolves and starts Sonic through the same abxpkg lifecycle on both Docker and bare-metal installations.

archivebox config --set SEARCH_BACKEND_ENGINE=sonic
archivebox install sonic
archivebox update --index-only
archivebox search 'some text to search'

Run the same commands as docker compose run --rm archivebox ... when using Docker Compose.

Fore more detailed instructions see here...

Pros

  • extremely fast, most queries complete in microseconds even with 100k+ snapshots
  • maintains lightweight, compressed search index that is minuscule compared to original data
  • all-in-one binary written in rust, available cross-platform and easy to deploy
  • supports advanced full-text search features like normalization, stemming, etc.
  • supports indexing and querying on a remote server (many ArchiveBox instances can share a single sonic instance)

Cons

  • one extra dependency to install and background worker to keep running (Docker Compose makes this easy though)
  • does not support searching binary files like PDFs, eBooks, compressed archives, etc.

SQLite FTS5

This is a recently added experimental option that uses a separate SQLite3 Database (similar to the one ArchiveBox already uses for Snapshot metadata) to provide full-text search.

archivebox config --set SEARCH_BACKEND_ENGINE=sqlite

# add existing data to index by running update:
archivebox update --index-only

# test it out using the archivebox Web UI or CLI:
archivebox search 'some text to search'

You can also inspect the separate FTS database directly:

sqlite3 ./search.sqlite3

> SELECT snapshot_id, url FROM search_index
      WHERE search_index MATCH 'some text to search';
# optional advanced tuning:
archivebox config --set FTS_SEPARATE_DATABASE=True
archivebox config --set FTS_TOKENIZERS="porter unicode61 remove_diacritics 2"

Pros

  • No additional dependencies needed to install, SQLite3 is already available and used by ArchiveBox
  • No long-running background search worker process needed, 0 idle resource use
  • Supports advanced full-text search features like boolean operators, stemming, phrases, etc.
  • Comparable speed and efficiency to sonic for most use-cases (much faster than ripgrep/ugrep)
  • Durability and portability, SQLite is widely used and supported by every major platform on earth

Cons

  • Not as thoroughly-tested by ArchiveBox team as our sonic or ripgrep backends
  • Maintains a (compressed, but still potentially large) duplicate copy of all searchable text in search.sqlite3 db
  • Does not support searching binary files PDFs, eBooks, compressed archives, etc.
  • Search indexing and querying must be performed on same server as ArchiveBox data (we don't yet support sending FTS5 queries to a remote server)


Further Reading