diff --git a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md index 507dfd7602..b02233b9c0 100644 --- a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md +++ b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md @@ -49,7 +49,7 @@ This is a high-risk, manual disaster recovery process that will require cluster 2. **Attempt Data Recovery:** If the failed disk is still partially readable, make a best-effort attempt to copy the contents of the `ozone.om.db.dirs` directory to a safe, temporary location. -3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, you must restore from your most recent backup. This document does not cover the backup process itself, but it is the only path to recovery in this scenario. +3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, restore from your most recent backup. When another OM is reachable, use [backup and restore from a live OM](../backup-and-recovery/om-metadata-backup#backup-and-restore-from-a-live-om) (`ozone repair om download`). Otherwise extract and install a saved tarball (same page). 4. **Replace and Configure Disk:** Physically replace the hardware and ensure the new, empty disk is mounted at the correct path defined in `ozone.om.db.dirs`. diff --git a/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md b/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md index b687ef2103..c4b9cc65a8 100644 --- a/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md +++ b/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md @@ -1,6 +1,6 @@ # Ozone Repair -Ozone Repair (`ozone repair`) is an advanced tool to repair Ozone. The nodes being repaired must be stopped before the tool is run. +Ozone Repair (`ozone repair`) is an advanced tool to repair Ozone. Most subcommands require the node being repaired to be stopped before the tool is run. The `ozone repair om download` subcommand is an exception: it reads from a **running** OM over HTTP(S) while you restore metadata onto a stopped node. :::note All repair commands support a `--dry-run` option which allows a user to see what repair the command will be performing without actually making any changes to the cluster. @@ -69,6 +69,7 @@ Operational tool to repair OM. - quota - compact - skip-ratis-transaction +- download ### FSO-tree @@ -229,6 +230,21 @@ from one of the good OMs to the crashing OM instead. Path of the input segment file ``` +#### download + +Download OM metadata from a running OM (same v2 checkpoint flow as follower bootstrap). The source cluster must be **running**; stop the target OM before installing the output. Requires Ozone 2.2+ (HDDS-16171). See [OM metadata backup](../backup-and-recovery/om-metadata-backup#backup-and-restore-from-a-live-om). + +```bash +Usage: ozone repair om download [-hV] [--dry-run] [--overwrite] [--verbose] + [--node-id=] + [--om-service-id=] + --output-dir= + --node-id= OM node to download from (recommended: leader) + --om-service-id, --service-id Ozone Manager Service ID + --output-dir= Output directory (om.db and db.snapshots) + --overwrite Overwrite output directory if it exists +``` + ## ozone repair SCM Operational tool to repair SCM. diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md new file mode 100644 index 0000000000..b289b7535d --- /dev/null +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md @@ -0,0 +1,141 @@ +--- +sidebar_label: OM metadata backup +--- + +# OM metadata backup (including bucket snapshots) + +Back up **Ozone Manager (OM) namespace metadata**, including **bucket snapshot** RocksDB state. This is not a substitute for Datanode block or SCM metadata backups. + +The transfer uses the same v2 checkpoint mechanism as OM HA follower bootstrap ([design doc](../../../system-internals/features/om-bootstrapping-with-snapshots)). + +## What is included + +- `om.db` — volumes, buckets, keys, and related AOS metadata +- `db.snapshots` — bucket snapshot RocksDB state (when snapshots exist on the leader) +- Compaction backup and log directories needed to read snapshot SST files + +Requires Ozone **2.2+** with inode-based checkpoint transfer (default: `ozone.om.db.checkpoint.use.inode.based.transfer=true`). + +## Backup and restore (recommended) + +Use `ozone repair om download` ([HDDS-16171](https://issues.apache.org/jira/browse/HDDS-16171)) when a live OM is reachable. It handles multi-batch transfer, hard link reconstruction, and writes a **ready-to-install** copy of everything under `--output-dir` (including `db.snapshots` when snapshots exist on the leader). No separate snapshot steps. + +**Requirements:** `ozone-site.xml` on the host; `kinit` in secure clusters (`ozone.administrators`); stop the **target** OM before installing downloaded metadata. + +```shell +# Download +ozone repair om download --output-dir /backup/om-metadata --overwrite + +# OM HA — add --service-id; --node-id targets a specific OM (leader recommended) +ozone repair om download \ + --service-id \ + --node-id \ + --output-dir /backup/om-metadata \ + --overwrite + +# Archive off-cluster (top-level om-metadata/ directory in the tarball) +tar -czf om-metadata-$(date +%Y%m%d).tar.gz -C /backup om-metadata +``` + +**Restore** — stop the target OM first. If a live OM is still up, re-run `ozone repair om download`. Otherwise extract a CLI backup archive and install: + +```shell +mkdir -p /restore +tar -xzf om-metadata-YYYYMMDD.tar.gz -C /restore # yields /restore/om-metadata/ + +OM_DB_DIRS=/var/lib/ozone/om/metadata # ozone.om.db.dirs +rm -rf "$OM_DB_DIRS/om.db" "$OM_DB_DIRS/db.snapshots" +rsync -a /restore/om-metadata/ "$OM_DB_DIRS/" + +# Start OM; verify with: ozone sh volume list +``` + +For HA with an intact Ratis ring, prefer `ozone om --bootstrap` over manual install. See [Replacing Ozone Manager disks](../disk-replacement/ozone-manager) and [Ozone Repair](../tools/ozone-repair#download). + +## Alternative: HTTP checkpoint download + +Use `curl` or `om-metadata-backup.sh` only when the backup host has no Ozone CLI. This runs the same `/v2/dbCheckpoint` transfer, but the saved tarball stays in inode-based form and needs hard link reconstruction before install. + +Target the **leader** OM. Plan for at least **2×** the `X-Ozone-Om-Checkpoint-Estimated-Sst-Bytes` response header value (see [OM HA](../../../system-internals/components/ozone-manager/high-availability)). + +Always pass `includeSnapshotData=true&flushBeforeCheckpoint=true`. + +**Single batch** (snapshot SST below `ozone.om.ratis.snapshot.max.total.sst.size`, default 10 GB): + +```shell +# Non-secure +curl -f -X POST \ + "http://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ + -F "toExcludeSST[]=" \ + -o "om-metadata-$(date +%Y%m%d).tar" + +# Secure (run kinit first) +curl -f --negotiate -u : -X POST \ + "https://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ + -F "toExcludeSST[]=" \ + -o "om-metadata-$(date +%Y%m%d).tar" +``` + +**Multiple batches** — `om-metadata-backup.sh` loops until `OZONE_RATIS_SNAPSHOT_COMPLETE`: + +```shell +chmod +x om-metadata-backup.sh +./om-metadata-backup.sh --base-url "http://:" \ + --archive "om-metadata-$(date +%Y%m%d).tar.gz" +# Add --kerberos for secure clusters (run kinit first) +``` + +HTTP archives contain a **flat** inode-based tree (not the `om-metadata/` wrapper used by the CLI). Stop the target OM, then reconstruct hard links and install: + +```shell +mkdir -p /restore/om-staging +OM_DB_DIRS=/var/lib/ozone/om/metadata + +tar -xf om-metadata-YYYYMMDD.tar -C /restore/om-staging # .tar from curl +# tar -xzf om-metadata-YYYYMMDD.tar.gz -C /restore/om-staging # .tar.gz from om-metadata-backup.sh --archive + +while IFS=$'\t' read -r dest src; do + [[ -z "$dest" || -z "$src" ]] && continue + install -d "$(dirname "/restore/om-staging/$dest")" + ln "/restore/om-staging/$src" "/restore/om-staging/$dest" +done < /restore/om-staging/hardLinkFile + +while IFS=$'\t' read -r dest src; do + [[ -z "$src" ]] && continue + if [[ -d "/restore/om-staging/$src" ]]; then + rm -rf "/restore/om-staging/$src" + else + rm -f "/restore/om-staging/$src" + fi +done < /restore/om-staging/hardLinkFile + +rm -f /restore/om-staging/hardLinkFile /restore/om-staging/OZONE_RATIS_SNAPSHOT_COMPLETE + +rm -rf "$OM_DB_DIRS/om.db" "$OM_DB_DIRS/db.snapshots" +rsync -a /restore/om-staging/ "$OM_DB_DIRS/" +``` + +The legacy v1 `/dbCheckpoint` endpoint omits bucket snapshot data even with `includeSnapshotData=true`. Use v2 only. + +## Operational notes + +- Run checkpoints against the **leader** OM. +- Schedule during low snapshot activity; including snapshot data takes a short-lived cache lock. +- Do **not** use Recon as a backup source (`includeSnapshotData=false` by design). +- User-visible **Ozone Snapshots** are included. **Ratis snapshots** (OM HA replication) are a separate concept — see [OM high availability](../../../system-internals/components/ozone-manager/high-availability). + +## Related configuration + +| Property | Default | Relevance | +| ---------- | --------- | --------- | +| `ozone.om.db.checkpoint.use.inode.based.transfer` | `true` | Required for snapshot-inclusive backup | +| `ozone.om.ratis.snapshot.max.total.sst.size` | `10GB` | HTTP multi-batch threshold | +| `ozone.administrators` | (none) | HTTP/CLI access in secure mode | + +## See also + +- [OM bootstrapping with snapshots (design)](../../../system-internals/features/om-bootstrapping-with-snapshots) +- [OM HA configuration](../../configuration/high-availability/om-ha) +- [Replacing Ozone Manager disks](../disk-replacement/ozone-manager) +- [Troubleshooting OM HA snapshot installation](../../../troubleshooting/om-ha-snapshot-installation-issues) +- [Snapshots overview](../snapshots/overview) diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx new file mode 100644 index 0000000000..c518ac0767 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx @@ -0,0 +1,11 @@ +--- +sidebar_label: Backup and recovery +--- + +# Backup and recovery + +import DocCardList from '@theme/DocCardList'; + +Operator procedures for backing up Ozone metadata and related disaster-recovery planning. + + diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh new file mode 100755 index 0000000000..0e3354bd63 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh @@ -0,0 +1,236 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Download a consistent OM metadata checkpoint tarball from the leader OM HTTP(S) +# endpoint, including bucket snapshot RocksDB state. Handles multi-batch transfers +# when snapshot SST data exceeds ozone.om.ratis.snapshot.max.total.sst.size. +# +# See om-metadata-backup.md in this directory for prerequisites and restore notes. + +set -euo pipefail + +readonly COMPLETE_SENTINEL='OZONE_RATIS_SNAPSHOT_COMPLETE' +readonly HARDLINK_FILE='hardLinkFile' +readonly DEFAULT_ENDPOINT='v2/dbCheckpoint' +readonly QUERY='includeSnapshotData=true&flushBeforeCheckpoint=true' + +usage() { + cat <<'EOF' +Usage: om-metadata-backup.sh [options] --base-url URL + +Download OM metadata (AOS + bucket snapshots) from the leader OM checkpoint API. +Extracts all batches into a staging directory until OZONE_RATIS_SNAPSHOT_COMPLETE +is present. + +Required: + --base-url URL Leader OM base URL, e.g. https://om1.example.com:9874 + (no trailing slash; do not include /v2/dbCheckpoint) + +Options: + --staging-dir DIR Directory to accumulate extracted checkpoint files + (default: temporary directory) + --batch-dir DIR Directory for per-batch .tar downloads (default: staging-dir/batches) + --archive FILE After success, create a gzip tarball of the staging directory + --kerberos Use curl SPNEGO (--negotiate -u :); run kinit first + --endpoint PATH Checkpoint path (default: v2/dbCheckpoint; v1 omits bucket snapshots) + --max-batches N Stop after N batches (default: 1000) + --keep-batches Do not delete per-batch .tar files after successful extract + -h, --help Show this help + +Examples: + kinit -k -t /path/to/admin.keytab admin@REALM + ./om-metadata-backup.sh --kerberos --base-url https://om-leader:9874 \\ + --archive om-metadata-$(date +%Y%m%d-%H%M%S).tar.gz + + ./om-metadata-backup.sh --base-url http://om-leader:9874 --staging-dir /backup/om-staging +EOF +} + +log() { + printf '[om-metadata-backup] %s\n' "$*" +} + +die() { + log "ERROR: $*" >&2 + exit 1 +} + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" +} + +collect_exclude_ids() { + local root="$1" + if [[ ! -d "$root" ]]; then + return 0 + fi + find "$root" -type f -exec basename {} \; | sort -u +} + +is_complete() { + [[ -f "$1/$COMPLETE_SENTINEL" ]] +} + +download_batch() { + local url="$1" + local output="$2" + shift 2 + local -a curl_auth=("$@") + local -a curl_form=() + local inode_id + + if [[ ${#EXCLUDE_IDS[@]} -eq 0 ]]; then + curl_form+=(-F 'toExcludeSST[]=') + else + for inode_id in "${EXCLUDE_IDS[@]}"; do + curl_form+=(-F "toExcludeSST[]=${inode_id}") + done + fi + + log "POST ${url} (excluding ${#EXCLUDE_IDS[@]} file(s))" + curl -f -sS "${curl_auth[@]}" \ + -X POST \ + "${url}?${QUERY}" \ + "${curl_form[@]}" \ + -o "$output" +} + +BASE_URL='' +STAGING_DIR='' +BATCH_DIR='' +ARCHIVE_FILE='' +ENDPOINT="$DEFAULT_ENDPOINT" +MAX_BATCHES=1000 +KEEP_BATCHES=false +KERBEROS=false +EXCLUDE_IDS=() + +while [[ $# -gt 0 ]]; do + case "$1" in + --base-url) + BASE_URL="${2:-}" + shift 2 + ;; + --staging-dir) + STAGING_DIR="${2:-}" + shift 2 + ;; + --batch-dir) + BATCH_DIR="${2:-}" + shift 2 + ;; + --archive) + ARCHIVE_FILE="${2:-}" + shift 2 + ;; + --endpoint) + ENDPOINT="${2:-}" + shift 2 + ;; + --max-batches) + MAX_BATCHES="${2:-}" + shift 2 + ;; + --keep-batches) + KEEP_BATCHES=true + shift + ;; + --kerberos) + KERBEROS=true + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + die "Unknown argument: $1 (use --help)" + ;; + esac +done + +[[ -n "$BASE_URL" ]] || die "--base-url is required" +require_cmd curl +require_cmd tar + +BASE_URL="${BASE_URL%/}" +CHECKPOINT_URL="${BASE_URL}/${ENDPOINT#/}" + +if [[ -z "$STAGING_DIR" ]]; then + STAGING_DIR="$(mktemp -d "${TMPDIR:-/tmp}/om-metadata-backup.XXXXXX")" + log "Using temporary staging directory: ${STAGING_DIR}" +else + mkdir -p "$STAGING_DIR" +fi + +if [[ -z "$BATCH_DIR" ]]; then + BATCH_DIR="${STAGING_DIR}/batches" +fi +mkdir -p "$BATCH_DIR" "$STAGING_DIR" + +CURL_AUTH=() +if [[ "$KERBEROS" == true ]]; then + CURL_AUTH=(--negotiate -u :) +fi + +batch=1 +while [[ "$batch" -le "$MAX_BATCHES" ]]; do + batch_tar="${BATCH_DIR}/batch-$(printf '%04d' "$batch").tar" + EXCLUDE_IDS=() + while IFS= read -r inode_id; do + [[ -n "$inode_id" ]] && EXCLUDE_IDS+=("$inode_id") + done < <(collect_exclude_ids "$STAGING_DIR") + + download_batch "$CHECKPOINT_URL" "$batch_tar" "${CURL_AUTH[@]}" + + if [[ ! -s "$batch_tar" ]]; then + die "Batch ${batch} download is empty: ${batch_tar}" + fi + + log "Extracting batch ${batch} into ${STAGING_DIR}" + tar -xf "$batch_tar" -C "$STAGING_DIR" + + if [[ "$KEEP_BATCHES" != true ]]; then + rm -f "$batch_tar" + fi + + if is_complete "$STAGING_DIR"; then + log "Found ${COMPLETE_SENTINEL} after batch ${batch}" + break + fi + + log "Batch ${batch} incomplete; requesting next batch" + batch=$((batch + 1)) +done + +if ! is_complete "$STAGING_DIR"; then + die "Checkpoint incomplete after ${MAX_BATCHES} batch(es); ${COMPLETE_SENTINEL} not found in ${STAGING_DIR}" +fi + +if [[ ! -f "${STAGING_DIR}/${HARDLINK_FILE}" ]]; then + log "WARNING: ${HARDLINK_FILE} not found in staging directory (expected on final batch)" +fi + +if [[ -n "$ARCHIVE_FILE" ]]; then + log "Creating archive ${ARCHIVE_FILE}" + tar -czf "$ARCHIVE_FILE" -C "$STAGING_DIR" . +fi + +log "Backup complete. Staging directory: ${STAGING_DIR}" +if [[ -n "$ARCHIVE_FILE" ]]; then + log "Archive: ${ARCHIVE_FILE}" +fi