From f6eca6669827ca256555e1550c5ec5fea663380b Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Tue, 11 Aug 2026 09:04:26 -0700 Subject: [PATCH 1/3] Document OM metadata backup and cold restore procedure. Add Administrator Guide backup-and-recovery section with v2 checkpoint backup steps, multi-batch helper script, and prescriptive cold restore from offline tarballs; cross-link from OM disk replacement. Co-authored-by: Cursor --- .../04-disk-replacement/01-ozone-manager.md | 2 +- .../01-om-metadata-backup.md | 234 +++++++++++++++++ .../16-backup-and-recovery/README.mdx | 11 + .../om-metadata-backup.sh | 236 ++++++++++++++++++ 4 files changed, 482 insertions(+), 1 deletion(-) create mode 100644 docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md create mode 100644 docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx create mode 100755 docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh diff --git a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md index 507dfd7602..17a59f92cf 100644 --- a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md +++ b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md @@ -49,7 +49,7 @@ This is a high-risk, manual disaster recovery process that will require cluster 2. **Attempt Data Recovery:** If the failed disk is still partially readable, make a best-effort attempt to copy the contents of the `ozone.om.db.dirs` directory to a safe, temporary location. -3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, you must restore from your most recent backup. This document does not cover the backup process itself, but it is the only path to recovery in this scenario. +3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, restore from your most recent backup. See [OM metadata backup](../backup-and-recovery/om-metadata-backup) for backup creation and [cold restore from an offline tarball](../backup-and-recovery/om-metadata-backup#cold-restore-from-an-offline-tarball-without-a-live-leader) when no live OM leader is available. 4. **Replace and Configure Disk:** Physically replace the hardware and ensure the new, empty disk is mounted at the correct path defined in `ozone.om.db.dirs`. diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md new file mode 100644 index 0000000000..de0e8d43c2 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md @@ -0,0 +1,234 @@ +--- +sidebar_label: OM metadata backup +--- + +# OM metadata backup (including bucket snapshots) + +This procedure describes how to take a **consistent tarball backup of Ozone Manager (OM) metadata**, including **bucket snapshot** RocksDB state. It uses the same checkpoint mechanism that OM HA bootstrap and Ratis snapshot installation rely on internally ([OM bootstrapping with snapshots](../../../system-internals/features/om-bootstrapping-with-snapshots)). + +## What is backed up + +When `includeSnapshotData=true`, the leader OM streams a tar archive containing: + +- The Active Object Store (AOS) RocksDB checkpoint (`om.db`). +- All bucket snapshot RocksDB directories registered in the snapshot metadata table. +- Snapshot compaction backup and compaction log directories required to interpret snapshot SST files. +- Snapshot-local property files used by snapshot defragmentation. +- A `hardLinkFile` mapping so hard-linked SST files can be reconstructed on restore. +- An `OZONE_RATIS_SNAPSHOT_COMPLETE` sentinel on the **final** batch when the transfer finishes. + +This is **OM namespace metadata only** (volumes, buckets, keys, snapshot definitions, and related RocksDB state). It does **not** replace Datanode block backups or SCM metadata backups. + +## Prerequisites + +1. An OM HA cluster with a reachable **leader** OM HTTP(S) endpoint (default port `9874`). +2. A caller identity in `ozone.administrators` when `ozone.security.enabled=true` (Kerberos SPNEGO or equivalent HTTP auth). +3. Sufficient disk space on the backup host for the tarball(s). The leader sets response header `X-Ozone-Om-Checkpoint-Estimated-Sst-Bytes` with an uncompressed SST size estimate; plan for at least **2×** that value to allow for tar overhead and unpack headroom (same guidance as [OM HA](../../../system-internals/components/ozone-manager/high-availability)). +4. Ozone **2.2+** with inode-based checkpoint transfer enabled (default): + +```xml + + ozone.om.db.checkpoint.use.inode.based.transfer + true + +``` + +## Important: include snapshot data explicitly + +A checkpoint request **without** `includeSnapshotData=true` returns **AOS metadata only** and omits bucket snapshots. This was the limitation of older backup procedures. + +Always pass: + +| Query parameter | Required value | Purpose | +| ----------------------- | -------------- | ------------------------------------------------------------- | +| `includeSnapshotData` | `true` | Include bucket snapshot RocksDB directories and related files | +| `flushBeforeCheckpoint` | `true` | Flush memtables before taking the AOS checkpoint | + +## Single-batch backup (small to medium metadata) + +Use when total snapshot SST size fits in one batch (typically when snapshot SST footprint is below `ozone.om.ratis.snapshot.max.total.sst.size`, default **10 GB**). + +Replace ``, ``, and auth flags for your environment. + +**Secure cluster (Kerberos):** + +```shell +kinit -k -t /path/to/admin.keytab admin@REALM +curl -f --negotiate -u : \ + -X POST \ + "https://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ + -F "toExcludeSST[]=" \ + -o "om-metadata-$(date +%Y%m%d-%H%M%S).tar" +``` + +**Non-secure cluster:** + +```shell +curl -f \ + -X POST \ + "http://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ + -F "toExcludeSST[]=" \ + -o "om-metadata-$(date +%Y%m%d-%H%M%S).tar" +``` + +### Verify the archive + +The tarball uses inode-based file names during transfer. On the **final** batch, expect: + +- `hardLinkFile` — tab-separated mapping of relative paths to inode identifiers +- `OZONE_RATIS_SNAPSHOT_COMPLETE` — completion sentinel + +If your backup stopped after one batch but lacks the completion sentinel, the snapshot set is **incomplete** (see multi-batch section below). + +## Multi-batch backup (large snapshot footprint) + +When snapshot SST data exceeds `ozone.om.ratis.snapshot.max.total.sst.size`, the leader splits the transfer across multiple POST responses. Each batch excludes inode IDs already received via the `toExcludeSST[]` multipart form field. + +Use the helper script in this directory (same logic as OM bootstrap's `OmRatisSnapshotProvider` / `HAUtils.getExistingFiles`): + +```shell +chmod +x om-metadata-backup.sh + +# Secure cluster (run kinit first) +./om-metadata-backup.sh --kerberos \ + --base-url "https://:" \ + --archive "om-metadata-$(date +%Y%m%d-%H%M%S).tar.gz" + +# Non-secure cluster +./om-metadata-backup.sh \ + --base-url "http://:" \ + --staging-dir "/backup/om-metadata-staging" +``` + +The script: + +1. POSTs to `/v2/dbCheckpoint` with `includeSnapshotData=true` and `flushBeforeCheckpoint=true`. +2. Saves each response tarball, extracts it into a staging directory, and collects file basenames (inode IDs) already on disk. +3. Repeats with `toExcludeSST[]=` for each collected file until `OZONE_RATIS_SNAPSHOT_COMPLETE` appears in the staging directory. + +Optional flags: `--staging-dir`, `--archive` (gzip tarball of the final staging tree), and `--max-batches` as a safety limit. Run `./om-metadata-backup.sh --help` for the full list. + +Verify the staging directory contains `hardLinkFile` and `OZONE_RATIS_SNAPSHOT_COMPLETE` before treating the backup as complete. + +## Legacy v1 endpoint + +The v1 `/dbCheckpoint` endpoint (selected when `ozone.om.db.checkpoint.use.inode.based.transfer=false`) transfers **AOS (`om.db`) metadata only**. It does **not** include bucket snapshot RocksDB state in the checkpoint, even when `includeSnapshotData=true` is passed. + +**New deployments should run Ozone 2.2+** with inode-based transfer enabled (the default) and use `/v2/dbCheckpoint` for backups that include bucket snapshots. Reserve v1 only for legacy clusters that have not yet migrated to inode-based transfer. + +## Restore scenarios + +### Add or replace an OM node in an existing HA cluster + +Use `ozone om --bootstrap` (downloads a fresh checkpoint from the live leader, including snapshots). See [Replacing Ozone Manager disks](../disk-replacement/ozone-manager). + +### Cold restore from an offline tarball without a live leader + +Use when **no OM leader is available** to serve bootstrap—for example, total metadata loss on a standalone OM, or every HA OM lost its metadata volume. This installs a previously saved **v2** checkpoint tarball manually, following the same hardlink reconstruction and directory layout that `ozone om --bootstrap` applies after download. + +**Requirements** + +- Backup taken from `/v2/dbCheckpoint` with `includeSnapshotData=true` on Ozone **2.2+**. +- Extracted staging tree contains `hardLinkFile` and `OZONE_RATIS_SNAPSHOT_COMPLETE`. +- Plan for **cluster downtime** while OM metadata is offline. + +**Procedure** + +1. **Stop writers.** Stop all Ozone Manager processes, clients, and services that write to the cluster. + +2. **Verify the backup.** For multi-batch backups, confirm the staging directory contains `hardLinkFile` and `OZONE_RATIS_SNAPSHOT_COMPLETE`. Do not proceed with an incomplete transfer. + +3. **Extract the archive** into a clean staging directory: + +```shell +mkdir -p /restore/om-checkpoint-staging +tar -xf om-metadata-YYYYMMDD.tar -C /restore/om-checkpoint-staging +# If you used om-metadata-backup.sh --archive: +tar -xzf om-metadata-YYYYMMDD.tar.gz -C /restore/om-checkpoint-staging +``` + +4. **Reconstruct hard links.** v2 checkpoints store many files under inode IDs; `hardLinkFile` maps logical paths (for example `om.db/000012.sst`) to those IDs: + +```shell +STAGING=/restore/om-checkpoint-staging + +while IFS=$'\t' read -r dest src; do + [[ -z "$dest" ]] && continue + install -d "$(dirname "$STAGING/$dest")" + ln "$STAGING/$src" "$STAGING/$dest" +done < "$STAGING/hardLinkFile" + +while IFS=$'\t' read -r dest src; do + [[ -z "$src" ]] && continue + if [[ -d "$STAGING/$src" ]]; then + rm -rf "$STAGING/$src" + else + rm -f "$STAGING/$src" + fi +done < "$STAGING/hardLinkFile" + +rm -f "$STAGING/hardLinkFile" "$STAGING/OZONE_RATIS_SNAPSHOT_COMPLETE" +``` + +5. **Validate layout.** Expect top-level items such as `om.db` and, when bucket snapshots were backed up, `db.snapshots`. Confirm `om.db/CURRENT` exists. + +6. **Install into OM metadata storage.** Let `OM_DB_DIRS` be the path configured in `ozone.om.db.dirs` (or `ozone.metadata.dirs`). Back up any recoverable existing metadata, then install checkpoint items: + +```shell +OM_DB_DIRS=/var/lib/ozone/om/metadata # your ozone.om.db.dirs + +# Optional: preserve the om/ VERSION subtree if the disk is partially readable +# before replacing RocksDB directories. + +mkdir -p "$OM_DB_DIRS" +for name in om.db db.snapshots; do + if [[ -e "$STAGING/$name" ]]; then + rm -rf "$OM_DB_DIRS/$name" + mv "$STAGING/$name" "$OM_DB_DIRS/" + fi +done +# Move any remaining top-level checkpoint items (compaction backup dirs, etc.) +find "$STAGING" -mindepth 1 -maxdepth 1 -exec mv -t "$OM_DB_DIRS" {} + +``` + +Preserve the existing `om/` VERSION subdirectory under `OM_DB_DIRS` when it survives disk failure. Do not delete OM cluster identity files unless you are intentionally rebuilding the OM from scratch. + +7. **Standalone OM:** Start SCM (if required), then start the Ozone Manager. Verify with `ozone sh volume list` and spot-check buckets and keys. + +8. **HA OM (all peers lost metadata):** The tarball replaces RocksDB metadata only; Ratis state lives under `ozone.om.ratis.storage.dir` on each node. After installing the checkpoint on a designated seed OM: + + - Clear `ozone.om.ratis.storage.dir` on **every** OM node (empty directory or fresh disk). + - Start the seed OM and confirm it serves reads. + - Bootstrap remaining OMs from the seed with `ozone om --bootstrap` once the seed is healthy. If the Ratis ring cannot reform, engage the [Ozone community](https://ozone.apache.org/community/) before forcing further metadata changes. + +**Limitations** + +- v1 (`/dbCheckpoint`) backups omit bucket snapshot data; use them only for AOS-only recovery on legacy clusters. +- There is no single supported CLI for cold install. Validate the procedure on a non-production clone before relying on it for production DR. + +Store tarball backups off-cluster (object store, NFS, or backup appliance) according to your retention policy. + +## Operational notes + +- Run backups against the **leader** OM. Followers redirect or reject checkpoint creation. +- Checkpoint creation takes a short-lived snapshot cache lock when snapshot data is included. Schedule backups during low snapshot activity when possible. +- Recon's OM sync uses `includeSnapshotData=false` by design; do **not** use Recon as a full-metadata backup source. +- Bucket **Ozone Snapshots** (user-visible point-in-time bucket images) are included in this backup. **Ratis snapshots** (OM HA replication checkpoints) are a related but separate concept; see [OM high availability](../../../system-internals/components/ozone-manager/high-availability). + +## Related configuration + +| Property | Default | Relevance | +| ---------- | --------- | --------- | +| `ozone.om.db.checkpoint.use.inode.based.transfer` | `true` | Use `/v2/dbCheckpoint` | +| `ozone.om.ratis.snapshot.max.total.sst.size` | `10GB` | Batch size threshold for snapshot SST transfer | +| `ozone.om.bootstrap.min.space` | `5GB` | Follower disk pre-check fallback (bootstrap) | +| `ozone.om.bootstrap.checkpoint.estimated.space.headroom.ratio` | `2.0` | Follower disk pre-check vs leader SST estimate | +| `ozone.administrators` | (none) | HTTP checkpoint access in secure mode | + +## See also + +- [OM bootstrapping with snapshots (design)](../../../system-internals/features/om-bootstrapping-with-snapshots) +- [OM HA configuration](../../configuration/high-availability/om-ha) +- [Replacing Ozone Manager disks](../disk-replacement/ozone-manager) +- [Troubleshooting OM HA snapshot installation](../../../troubleshooting/om-ha-snapshot-installation-issues) +- [Snapshots overview](../snapshots/overview) diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx new file mode 100644 index 0000000000..c518ac0767 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/README.mdx @@ -0,0 +1,11 @@ +--- +sidebar_label: Backup and recovery +--- + +# Backup and recovery + +import DocCardList from '@theme/DocCardList'; + +Operator procedures for backing up Ozone metadata and related disaster-recovery planning. + + diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh new file mode 100755 index 0000000000..0e3354bd63 --- /dev/null +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/om-metadata-backup.sh @@ -0,0 +1,236 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Download a consistent OM metadata checkpoint tarball from the leader OM HTTP(S) +# endpoint, including bucket snapshot RocksDB state. Handles multi-batch transfers +# when snapshot SST data exceeds ozone.om.ratis.snapshot.max.total.sst.size. +# +# See om-metadata-backup.md in this directory for prerequisites and restore notes. + +set -euo pipefail + +readonly COMPLETE_SENTINEL='OZONE_RATIS_SNAPSHOT_COMPLETE' +readonly HARDLINK_FILE='hardLinkFile' +readonly DEFAULT_ENDPOINT='v2/dbCheckpoint' +readonly QUERY='includeSnapshotData=true&flushBeforeCheckpoint=true' + +usage() { + cat <<'EOF' +Usage: om-metadata-backup.sh [options] --base-url URL + +Download OM metadata (AOS + bucket snapshots) from the leader OM checkpoint API. +Extracts all batches into a staging directory until OZONE_RATIS_SNAPSHOT_COMPLETE +is present. + +Required: + --base-url URL Leader OM base URL, e.g. https://om1.example.com:9874 + (no trailing slash; do not include /v2/dbCheckpoint) + +Options: + --staging-dir DIR Directory to accumulate extracted checkpoint files + (default: temporary directory) + --batch-dir DIR Directory for per-batch .tar downloads (default: staging-dir/batches) + --archive FILE After success, create a gzip tarball of the staging directory + --kerberos Use curl SPNEGO (--negotiate -u :); run kinit first + --endpoint PATH Checkpoint path (default: v2/dbCheckpoint; v1 omits bucket snapshots) + --max-batches N Stop after N batches (default: 1000) + --keep-batches Do not delete per-batch .tar files after successful extract + -h, --help Show this help + +Examples: + kinit -k -t /path/to/admin.keytab admin@REALM + ./om-metadata-backup.sh --kerberos --base-url https://om-leader:9874 \\ + --archive om-metadata-$(date +%Y%m%d-%H%M%S).tar.gz + + ./om-metadata-backup.sh --base-url http://om-leader:9874 --staging-dir /backup/om-staging +EOF +} + +log() { + printf '[om-metadata-backup] %s\n' "$*" +} + +die() { + log "ERROR: $*" >&2 + exit 1 +} + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || die "Required command not found: $1" +} + +collect_exclude_ids() { + local root="$1" + if [[ ! -d "$root" ]]; then + return 0 + fi + find "$root" -type f -exec basename {} \; | sort -u +} + +is_complete() { + [[ -f "$1/$COMPLETE_SENTINEL" ]] +} + +download_batch() { + local url="$1" + local output="$2" + shift 2 + local -a curl_auth=("$@") + local -a curl_form=() + local inode_id + + if [[ ${#EXCLUDE_IDS[@]} -eq 0 ]]; then + curl_form+=(-F 'toExcludeSST[]=') + else + for inode_id in "${EXCLUDE_IDS[@]}"; do + curl_form+=(-F "toExcludeSST[]=${inode_id}") + done + fi + + log "POST ${url} (excluding ${#EXCLUDE_IDS[@]} file(s))" + curl -f -sS "${curl_auth[@]}" \ + -X POST \ + "${url}?${QUERY}" \ + "${curl_form[@]}" \ + -o "$output" +} + +BASE_URL='' +STAGING_DIR='' +BATCH_DIR='' +ARCHIVE_FILE='' +ENDPOINT="$DEFAULT_ENDPOINT" +MAX_BATCHES=1000 +KEEP_BATCHES=false +KERBEROS=false +EXCLUDE_IDS=() + +while [[ $# -gt 0 ]]; do + case "$1" in + --base-url) + BASE_URL="${2:-}" + shift 2 + ;; + --staging-dir) + STAGING_DIR="${2:-}" + shift 2 + ;; + --batch-dir) + BATCH_DIR="${2:-}" + shift 2 + ;; + --archive) + ARCHIVE_FILE="${2:-}" + shift 2 + ;; + --endpoint) + ENDPOINT="${2:-}" + shift 2 + ;; + --max-batches) + MAX_BATCHES="${2:-}" + shift 2 + ;; + --keep-batches) + KEEP_BATCHES=true + shift + ;; + --kerberos) + KERBEROS=true + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + die "Unknown argument: $1 (use --help)" + ;; + esac +done + +[[ -n "$BASE_URL" ]] || die "--base-url is required" +require_cmd curl +require_cmd tar + +BASE_URL="${BASE_URL%/}" +CHECKPOINT_URL="${BASE_URL}/${ENDPOINT#/}" + +if [[ -z "$STAGING_DIR" ]]; then + STAGING_DIR="$(mktemp -d "${TMPDIR:-/tmp}/om-metadata-backup.XXXXXX")" + log "Using temporary staging directory: ${STAGING_DIR}" +else + mkdir -p "$STAGING_DIR" +fi + +if [[ -z "$BATCH_DIR" ]]; then + BATCH_DIR="${STAGING_DIR}/batches" +fi +mkdir -p "$BATCH_DIR" "$STAGING_DIR" + +CURL_AUTH=() +if [[ "$KERBEROS" == true ]]; then + CURL_AUTH=(--negotiate -u :) +fi + +batch=1 +while [[ "$batch" -le "$MAX_BATCHES" ]]; do + batch_tar="${BATCH_DIR}/batch-$(printf '%04d' "$batch").tar" + EXCLUDE_IDS=() + while IFS= read -r inode_id; do + [[ -n "$inode_id" ]] && EXCLUDE_IDS+=("$inode_id") + done < <(collect_exclude_ids "$STAGING_DIR") + + download_batch "$CHECKPOINT_URL" "$batch_tar" "${CURL_AUTH[@]}" + + if [[ ! -s "$batch_tar" ]]; then + die "Batch ${batch} download is empty: ${batch_tar}" + fi + + log "Extracting batch ${batch} into ${STAGING_DIR}" + tar -xf "$batch_tar" -C "$STAGING_DIR" + + if [[ "$KEEP_BATCHES" != true ]]; then + rm -f "$batch_tar" + fi + + if is_complete "$STAGING_DIR"; then + log "Found ${COMPLETE_SENTINEL} after batch ${batch}" + break + fi + + log "Batch ${batch} incomplete; requesting next batch" + batch=$((batch + 1)) +done + +if ! is_complete "$STAGING_DIR"; then + die "Checkpoint incomplete after ${MAX_BATCHES} batch(es); ${COMPLETE_SENTINEL} not found in ${STAGING_DIR}" +fi + +if [[ ! -f "${STAGING_DIR}/${HARDLINK_FILE}" ]]; then + log "WARNING: ${HARDLINK_FILE} not found in staging directory (expected on final batch)" +fi + +if [[ -n "$ARCHIVE_FILE" ]]; then + log "Creating archive ${ARCHIVE_FILE}" + tar -czf "$ARCHIVE_FILE" -C "$STAGING_DIR" . +fi + +log "Backup complete. Staging directory: ${STAGING_DIR}" +if [[ -n "$ARCHIVE_FILE" ]]; then + log "Archive: ${ARCHIVE_FILE}" +fi From 85b286f4e4e7e5b09f7335f45afd5f0e343c7440 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Thu, 13 Aug 2026 15:52:39 -0700 Subject: [PATCH 2/3] HDDS-16147. Document ozone repair om download for OM metadata restore. Replace manual live-OM restore steps with the HDDS-16171 repair CLI and add download subcommand reference in the Ozone Repair guide. Co-authored-by: Cursor --- .../04-disk-replacement/01-ozone-manager.md | 2 +- .../03-operations/11-tools/01-ozone-repair.md | 26 ++++++- .../01-om-metadata-backup.md | 78 +++++++++++++++---- 3 files changed, 89 insertions(+), 17 deletions(-) diff --git a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md index 17a59f92cf..83b1e0edaa 100644 --- a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md +++ b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md @@ -49,7 +49,7 @@ This is a high-risk, manual disaster recovery process that will require cluster 2. **Attempt Data Recovery:** If the failed disk is still partially readable, make a best-effort attempt to copy the contents of the `ozone.om.db.dirs` directory to a safe, temporary location. -3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, restore from your most recent backup. See [OM metadata backup](../backup-and-recovery/om-metadata-backup) for backup creation and [cold restore from an offline tarball](../backup-and-recovery/om-metadata-backup#cold-restore-from-an-offline-tarball-without-a-live-leader) when no live OM leader is available. +3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, restore from your most recent backup. When another OM is reachable, use [restore metadata from a live OM](../backup-and-recovery/om-metadata-backup#restore-metadata-from-a-live-om-ozone-repair-om-download) (`ozone repair om download`). If no OM is available, follow [restore from an offline tarball](../backup-and-recovery/om-metadata-backup#restore-from-an-offline-tarball-no-live-om). 4. **Replace and Configure Disk:** Physically replace the hardware and ensure the new, empty disk is mounted at the correct path defined in `ozone.om.db.dirs`. diff --git a/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md b/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md index b687ef2103..9a03c09730 100644 --- a/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md +++ b/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md @@ -1,6 +1,6 @@ # Ozone Repair -Ozone Repair (`ozone repair`) is an advanced tool to repair Ozone. The nodes being repaired must be stopped before the tool is run. +Ozone Repair (`ozone repair`) is an advanced tool to repair Ozone. Most subcommands require the node being repaired to be stopped before the tool is run. The `ozone repair om download` subcommand is an exception: it reads from a **running** OM over HTTP(S) while you restore metadata onto a stopped node. :::note All repair commands support a `--dry-run` option which allows a user to see what repair the command will be performing without actually making any changes to the cluster. @@ -69,6 +69,7 @@ Operational tool to repair OM. - quota - compact - skip-ratis-transaction +- download ### FSO-tree @@ -229,6 +230,29 @@ from one of the good OMs to the crashing OM instead. Path of the input segment file ``` +#### download + +Download and construct `om.db` from a running OM using the same v2 inode-based checkpoint transfer that follower bootstrap uses (including bucket snapshot data in the transfer). Unlike most repair subcommands, the **source** OM cluster must be **running** and reachable over HTTP(S). Stop the **target** OM before replacing its local `om.db` with the downloaded output. + +Requires Ozone 2.2+ with HDDS-16171. See [OM metadata backup](../backup-and-recovery/om-metadata-backup#restore-metadata-from-a-live-om-ozone-repair-om-download) for a full restore walkthrough. + +```bash +Usage: ozone repair om download [-hV] [--overwrite] [--verbose] + [--node-id=] + [--om-service-id=] + --output-dir= +Downloads and constructs om.db from an OM node using the same checkpoint +transfer flow as follower bootstrap. + --node-id= + Node ID of the OM to download om.db from. Required when + OM HA is configured. + --om-service-id, --service-id= + Ozone Manager Service ID + --output-dir= + Path where the constructed om.db directory will be written + --overwrite Overwrite output directory if it already exists. +``` + ## ozone repair SCM Operational tool to repair SCM. diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md index de0e8d43c2..a67d886bf8 100644 --- a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md @@ -122,9 +122,65 @@ The v1 `/dbCheckpoint` endpoint (selected when `ozone.om.db.checkpoint.use.inode Use `ozone om --bootstrap` (downloads a fresh checkpoint from the live leader, including snapshots). See [Replacing Ozone Manager disks](../disk-replacement/ozone-manager). -### Cold restore from an offline tarball without a live leader +### Restore metadata from a live OM (`ozone repair om download`) -Use when **no OM leader is available** to serve bootstrap—for example, total metadata loss on a standalone OM, or every HA OM lost its metadata volume. This installs a previously saved **v2** checkpoint tarball manually, following the same hardlink reconstruction and directory layout that `ozone om --bootstrap` applies after download. +When at least one OM is reachable (for example, HA peers are healthy but one host lost its disk), use the repair CLI added in [HDDS-16171](https://issues.apache.org/jira/browse/HDDS-16171). It downloads and constructs `om.db` using the same v2 inode-based checkpoint transfer and hardlink reconstruction that OM followers use during bootstrap—no manual `curl` or shell hardlink steps. + +**Requirements** + +- Ozone **2.2+** build that includes `ozone repair om download` (HDDS-16171). +- A reachable OM HTTP(S) endpoint and `ozone-site.xml` on the host where you run the command. +- Caller identity in `ozone.administrators` when `ozone.security.enabled=true` (run `kinit` first in secure clusters). +- The **target** OM whose disk you are restoring must be **stopped** before you replace its local `om.db`. The **source** OM cluster must remain up to serve the checkpoint. + +**Download** + +Non-HA (single OM): + +```shell +kinit -k -t /path/to/admin.keytab admin@REALM # secure clusters only + +ozone repair om download \ + --output-dir /restore/om.db \ + --overwrite +``` + +OM HA (specify the service ID and the node to pull from—typically the current leader): + +```shell +ozone repair om download \ + --service-id \ + --node-id \ + --output-dir /restore/om.db \ + --overwrite +``` + +The command always uses `/v2/dbCheckpoint` with snapshot data included. It fails if `--output-dir` already exists unless you pass `--overwrite`. + +**Install on the failed node** + +1. Stop the Ozone Manager on the node being restored (and block client writes if this is a standalone deployment). +2. Back up any partial data under `ozone.om.db.dirs` if the disk is still readable. +3. Replace the local database directory: + +```shell +OM_DB_DIRS=/var/lib/ozone/om/metadata # your ozone.om.db.dirs + +rm -rf "$OM_DB_DIRS/om.db" +mv /restore/om.db "$OM_DB_DIRS/om.db" +``` + +Preserve the existing `om/` VERSION subtree under `OM_DB_DIRS` when it survives disk failure. Do not delete OM cluster identity files unless you are intentionally rebuilding the OM from scratch. + +4. Start the Ozone Manager and verify with `ozone sh volume list`. + +For HA nodes with an empty metadata disk, prefer `ozone om --bootstrap` when the Ratis ring is intact. Use `ozone repair om download` when you need a constructed `om.db` on disk for manual replacement or offline inspection. + +See also [Ozone Repair](../tools/ozone-repair#download). + +### Restore from an offline tarball (no live OM) + +Use only when **no OM is available** to serve a checkpoint—for example, total metadata loss on a standalone deployment, or every HA OM lost its metadata volume. Offline tarballs cannot be consumed by `ozone repair om download`; install the saved checkpoint manually. **Requirements** @@ -172,14 +228,11 @@ rm -f "$STAGING/hardLinkFile" "$STAGING/OZONE_RATIS_SNAPSHOT_COMPLETE" 5. **Validate layout.** Expect top-level items such as `om.db` and, when bucket snapshots were backed up, `db.snapshots`. Confirm `om.db/CURRENT` exists. -6. **Install into OM metadata storage.** Let `OM_DB_DIRS` be the path configured in `ozone.om.db.dirs` (or `ozone.metadata.dirs`). Back up any recoverable existing metadata, then install checkpoint items: +6. **Install into OM metadata storage.** Let `OM_DB_DIRS` be the path configured in `ozone.om.db.dirs` (or `ozone.metadata.dirs`): ```shell OM_DB_DIRS=/var/lib/ozone/om/metadata # your ozone.om.db.dirs -# Optional: preserve the om/ VERSION subtree if the disk is partially readable -# before replacing RocksDB directories. - mkdir -p "$OM_DB_DIRS" for name in om.db db.snapshots; do if [[ -e "$STAGING/$name" ]]; then @@ -191,20 +244,14 @@ done find "$STAGING" -mindepth 1 -maxdepth 1 -exec mv -t "$OM_DB_DIRS" {} + ``` -Preserve the existing `om/` VERSION subdirectory under `OM_DB_DIRS` when it survives disk failure. Do not delete OM cluster identity files unless you are intentionally rebuilding the OM from scratch. - -7. **Standalone OM:** Start SCM (if required), then start the Ozone Manager. Verify with `ozone sh volume list` and spot-check buckets and keys. - -8. **HA OM (all peers lost metadata):** The tarball replaces RocksDB metadata only; Ratis state lives under `ozone.om.ratis.storage.dir` on each node. After installing the checkpoint on a designated seed OM: +7. **Standalone OM:** Start SCM (if required), then start the Ozone Manager. Verify with `ozone sh volume list`. - - Clear `ozone.om.ratis.storage.dir` on **every** OM node (empty directory or fresh disk). - - Start the seed OM and confirm it serves reads. - - Bootstrap remaining OMs from the seed with `ozone om --bootstrap` once the seed is healthy. If the Ratis ring cannot reform, engage the [Ozone community](https://ozone.apache.org/community/) before forcing further metadata changes. +8. **HA OM (all peers lost metadata):** Clear `ozone.om.ratis.storage.dir` on **every** OM node, install the checkpoint on a seed OM, start it, then bootstrap remaining peers with `ozone om --bootstrap` once the seed is healthy. **Limitations** - v1 (`/dbCheckpoint`) backups omit bucket snapshot data; use them only for AOS-only recovery on legacy clusters. -- There is no single supported CLI for cold install. Validate the procedure on a non-production clone before relying on it for production DR. +- Validate the offline tarball procedure on a non-production clone before relying on it for production DR. Store tarball backups off-cluster (object store, NFS, or backup appliance) according to your retention policy. @@ -227,6 +274,7 @@ Store tarball backups off-cluster (object store, NFS, or backup appliance) accor ## See also +- [Ozone Repair](../tools/ozone-repair#download) - [OM bootstrapping with snapshots (design)](../../../system-internals/features/om-bootstrapping-with-snapshots) - [OM HA configuration](../../configuration/high-availability/om-ha) - [Replacing Ozone Manager disks](../disk-replacement/ozone-manager) From 1a9168ea3f9cd0d56f2cc96ec612981f98492540 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Mon, 24 Aug 2026 16:02:10 -0700 Subject: [PATCH 3/3] HDDS-16147. Simplify OM metadata backup docs for ozone repair om download. Lead with the repair CLI for live-OM backup and restore, keep HTTP/curl as an alternative, and align command examples with the HDDS-16171 tool behavior. Co-authored-by: Cursor --- .../04-disk-replacement/01-ozone-manager.md | 2 +- .../03-operations/11-tools/01-ozone-repair.md | 20 +- .../01-om-metadata-backup.md | 285 +++++------------- 3 files changed, 79 insertions(+), 228 deletions(-) diff --git a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md index 83b1e0edaa..b02233b9c0 100644 --- a/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md +++ b/docs/05-administrator-guide/03-operations/04-disk-replacement/01-ozone-manager.md @@ -49,7 +49,7 @@ This is a high-risk, manual disaster recovery process that will require cluster 2. **Attempt Data Recovery:** If the failed disk is still partially readable, make a best-effort attempt to copy the contents of the `ozone.om.db.dirs` directory to a safe, temporary location. -3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, restore from your most recent backup. When another OM is reachable, use [restore metadata from a live OM](../backup-and-recovery/om-metadata-backup#restore-metadata-from-a-live-om-ozone-repair-om-download) (`ozone repair om download`). If no OM is available, follow [restore from an offline tarball](../backup-and-recovery/om-metadata-backup#restore-from-an-offline-tarball-no-live-om). +3. **If Recovery Fails, Restore from Backup:** If the OM database files are unrecoverable, restore from your most recent backup. When another OM is reachable, use [backup and restore from a live OM](../backup-and-recovery/om-metadata-backup#backup-and-restore-from-a-live-om) (`ozone repair om download`). Otherwise extract and install a saved tarball (same page). 4. **Replace and Configure Disk:** Physically replace the hardware and ensure the new, empty disk is mounted at the correct path defined in `ozone.om.db.dirs`. diff --git a/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md b/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md index 9a03c09730..c4b9cc65a8 100644 --- a/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md +++ b/docs/05-administrator-guide/03-operations/11-tools/01-ozone-repair.md @@ -232,25 +232,17 @@ from one of the good OMs to the crashing OM instead. #### download -Download and construct `om.db` from a running OM using the same v2 inode-based checkpoint transfer that follower bootstrap uses (including bucket snapshot data in the transfer). Unlike most repair subcommands, the **source** OM cluster must be **running** and reachable over HTTP(S). Stop the **target** OM before replacing its local `om.db` with the downloaded output. - -Requires Ozone 2.2+ with HDDS-16171. See [OM metadata backup](../backup-and-recovery/om-metadata-backup#restore-metadata-from-a-live-om-ozone-repair-om-download) for a full restore walkthrough. +Download OM metadata from a running OM (same v2 checkpoint flow as follower bootstrap). The source cluster must be **running**; stop the target OM before installing the output. Requires Ozone 2.2+ (HDDS-16171). See [OM metadata backup](../backup-and-recovery/om-metadata-backup#backup-and-restore-from-a-live-om). ```bash -Usage: ozone repair om download [-hV] [--overwrite] [--verbose] +Usage: ozone repair om download [-hV] [--dry-run] [--overwrite] [--verbose] [--node-id=] [--om-service-id=] --output-dir= -Downloads and constructs om.db from an OM node using the same checkpoint -transfer flow as follower bootstrap. - --node-id= - Node ID of the OM to download om.db from. Required when - OM HA is configured. - --om-service-id, --service-id= - Ozone Manager Service ID - --output-dir= - Path where the constructed om.db directory will be written - --overwrite Overwrite output directory if it already exists. + --node-id= OM node to download from (recommended: leader) + --om-service-id, --service-id Ozone Manager Service ID + --output-dir= Output directory (om.db and db.snapshots) + --overwrite Overwrite output directory if it exists ``` ## ozone repair SCM diff --git a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md index a67d886bf8..b289b7535d 100644 --- a/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md +++ b/docs/05-administrator-guide/03-operations/16-backup-and-recovery/01-om-metadata-backup.md @@ -4,277 +4,136 @@ sidebar_label: OM metadata backup # OM metadata backup (including bucket snapshots) -This procedure describes how to take a **consistent tarball backup of Ozone Manager (OM) metadata**, including **bucket snapshot** RocksDB state. It uses the same checkpoint mechanism that OM HA bootstrap and Ratis snapshot installation rely on internally ([OM bootstrapping with snapshots](../../../system-internals/features/om-bootstrapping-with-snapshots)). +Back up **Ozone Manager (OM) namespace metadata**, including **bucket snapshot** RocksDB state. This is not a substitute for Datanode block or SCM metadata backups. -## What is backed up +The transfer uses the same v2 checkpoint mechanism as OM HA follower bootstrap ([design doc](../../../system-internals/features/om-bootstrapping-with-snapshots)). -When `includeSnapshotData=true`, the leader OM streams a tar archive containing: +## What is included -- The Active Object Store (AOS) RocksDB checkpoint (`om.db`). -- All bucket snapshot RocksDB directories registered in the snapshot metadata table. -- Snapshot compaction backup and compaction log directories required to interpret snapshot SST files. -- Snapshot-local property files used by snapshot defragmentation. -- A `hardLinkFile` mapping so hard-linked SST files can be reconstructed on restore. -- An `OZONE_RATIS_SNAPSHOT_COMPLETE` sentinel on the **final** batch when the transfer finishes. +- `om.db` — volumes, buckets, keys, and related AOS metadata +- `db.snapshots` — bucket snapshot RocksDB state (when snapshots exist on the leader) +- Compaction backup and log directories needed to read snapshot SST files -This is **OM namespace metadata only** (volumes, buckets, keys, snapshot definitions, and related RocksDB state). It does **not** replace Datanode block backups or SCM metadata backups. +Requires Ozone **2.2+** with inode-based checkpoint transfer (default: `ozone.om.db.checkpoint.use.inode.based.transfer=true`). -## Prerequisites +## Backup and restore (recommended) -1. An OM HA cluster with a reachable **leader** OM HTTP(S) endpoint (default port `9874`). -2. A caller identity in `ozone.administrators` when `ozone.security.enabled=true` (Kerberos SPNEGO or equivalent HTTP auth). -3. Sufficient disk space on the backup host for the tarball(s). The leader sets response header `X-Ozone-Om-Checkpoint-Estimated-Sst-Bytes` with an uncompressed SST size estimate; plan for at least **2×** that value to allow for tar overhead and unpack headroom (same guidance as [OM HA](../../../system-internals/components/ozone-manager/high-availability)). -4. Ozone **2.2+** with inode-based checkpoint transfer enabled (default): +Use `ozone repair om download` ([HDDS-16171](https://issues.apache.org/jira/browse/HDDS-16171)) when a live OM is reachable. It handles multi-batch transfer, hard link reconstruction, and writes a **ready-to-install** copy of everything under `--output-dir` (including `db.snapshots` when snapshots exist on the leader). No separate snapshot steps. -```xml - - ozone.om.db.checkpoint.use.inode.based.transfer - true - -``` - -## Important: include snapshot data explicitly - -A checkpoint request **without** `includeSnapshotData=true` returns **AOS metadata only** and omits bucket snapshots. This was the limitation of older backup procedures. - -Always pass: - -| Query parameter | Required value | Purpose | -| ----------------------- | -------------- | ------------------------------------------------------------- | -| `includeSnapshotData` | `true` | Include bucket snapshot RocksDB directories and related files | -| `flushBeforeCheckpoint` | `true` | Flush memtables before taking the AOS checkpoint | - -## Single-batch backup (small to medium metadata) - -Use when total snapshot SST size fits in one batch (typically when snapshot SST footprint is below `ozone.om.ratis.snapshot.max.total.sst.size`, default **10 GB**). - -Replace ``, ``, and auth flags for your environment. - -**Secure cluster (Kerberos):** +**Requirements:** `ozone-site.xml` on the host; `kinit` in secure clusters (`ozone.administrators`); stop the **target** OM before installing downloaded metadata. ```shell -kinit -k -t /path/to/admin.keytab admin@REALM -curl -f --negotiate -u : \ - -X POST \ - "https://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ - -F "toExcludeSST[]=" \ - -o "om-metadata-$(date +%Y%m%d-%H%M%S).tar" -``` +# Download +ozone repair om download --output-dir /backup/om-metadata --overwrite -**Non-secure cluster:** +# OM HA — add --service-id; --node-id targets a specific OM (leader recommended) +ozone repair om download \ + --service-id \ + --node-id \ + --output-dir /backup/om-metadata \ + --overwrite -```shell -curl -f \ - -X POST \ - "http://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ - -F "toExcludeSST[]=" \ - -o "om-metadata-$(date +%Y%m%d-%H%M%S).tar" +# Archive off-cluster (top-level om-metadata/ directory in the tarball) +tar -czf om-metadata-$(date +%Y%m%d).tar.gz -C /backup om-metadata ``` -### Verify the archive - -The tarball uses inode-based file names during transfer. On the **final** batch, expect: - -- `hardLinkFile` — tab-separated mapping of relative paths to inode identifiers -- `OZONE_RATIS_SNAPSHOT_COMPLETE` — completion sentinel - -If your backup stopped after one batch but lacks the completion sentinel, the snapshot set is **incomplete** (see multi-batch section below). - -## Multi-batch backup (large snapshot footprint) - -When snapshot SST data exceeds `ozone.om.ratis.snapshot.max.total.sst.size`, the leader splits the transfer across multiple POST responses. Each batch excludes inode IDs already received via the `toExcludeSST[]` multipart form field. - -Use the helper script in this directory (same logic as OM bootstrap's `OmRatisSnapshotProvider` / `HAUtils.getExistingFiles`): +**Restore** — stop the target OM first. If a live OM is still up, re-run `ozone repair om download`. Otherwise extract a CLI backup archive and install: ```shell -chmod +x om-metadata-backup.sh +mkdir -p /restore +tar -xzf om-metadata-YYYYMMDD.tar.gz -C /restore # yields /restore/om-metadata/ -# Secure cluster (run kinit first) -./om-metadata-backup.sh --kerberos \ - --base-url "https://:" \ - --archive "om-metadata-$(date +%Y%m%d-%H%M%S).tar.gz" +OM_DB_DIRS=/var/lib/ozone/om/metadata # ozone.om.db.dirs +rm -rf "$OM_DB_DIRS/om.db" "$OM_DB_DIRS/db.snapshots" +rsync -a /restore/om-metadata/ "$OM_DB_DIRS/" -# Non-secure cluster -./om-metadata-backup.sh \ - --base-url "http://:" \ - --staging-dir "/backup/om-metadata-staging" +# Start OM; verify with: ozone sh volume list ``` -The script: - -1. POSTs to `/v2/dbCheckpoint` with `includeSnapshotData=true` and `flushBeforeCheckpoint=true`. -2. Saves each response tarball, extracts it into a staging directory, and collects file basenames (inode IDs) already on disk. -3. Repeats with `toExcludeSST[]=` for each collected file until `OZONE_RATIS_SNAPSHOT_COMPLETE` appears in the staging directory. - -Optional flags: `--staging-dir`, `--archive` (gzip tarball of the final staging tree), and `--max-batches` as a safety limit. Run `./om-metadata-backup.sh --help` for the full list. - -Verify the staging directory contains `hardLinkFile` and `OZONE_RATIS_SNAPSHOT_COMPLETE` before treating the backup as complete. - -## Legacy v1 endpoint +For HA with an intact Ratis ring, prefer `ozone om --bootstrap` over manual install. See [Replacing Ozone Manager disks](../disk-replacement/ozone-manager) and [Ozone Repair](../tools/ozone-repair#download). -The v1 `/dbCheckpoint` endpoint (selected when `ozone.om.db.checkpoint.use.inode.based.transfer=false`) transfers **AOS (`om.db`) metadata only**. It does **not** include bucket snapshot RocksDB state in the checkpoint, even when `includeSnapshotData=true` is passed. +## Alternative: HTTP checkpoint download -**New deployments should run Ozone 2.2+** with inode-based transfer enabled (the default) and use `/v2/dbCheckpoint` for backups that include bucket snapshots. Reserve v1 only for legacy clusters that have not yet migrated to inode-based transfer. +Use `curl` or `om-metadata-backup.sh` only when the backup host has no Ozone CLI. This runs the same `/v2/dbCheckpoint` transfer, but the saved tarball stays in inode-based form and needs hard link reconstruction before install. -## Restore scenarios +Target the **leader** OM. Plan for at least **2×** the `X-Ozone-Om-Checkpoint-Estimated-Sst-Bytes` response header value (see [OM HA](../../../system-internals/components/ozone-manager/high-availability)). -### Add or replace an OM node in an existing HA cluster +Always pass `includeSnapshotData=true&flushBeforeCheckpoint=true`. -Use `ozone om --bootstrap` (downloads a fresh checkpoint from the live leader, including snapshots). See [Replacing Ozone Manager disks](../disk-replacement/ozone-manager). - -### Restore metadata from a live OM (`ozone repair om download`) - -When at least one OM is reachable (for example, HA peers are healthy but one host lost its disk), use the repair CLI added in [HDDS-16171](https://issues.apache.org/jira/browse/HDDS-16171). It downloads and constructs `om.db` using the same v2 inode-based checkpoint transfer and hardlink reconstruction that OM followers use during bootstrap—no manual `curl` or shell hardlink steps. - -**Requirements** - -- Ozone **2.2+** build that includes `ozone repair om download` (HDDS-16171). -- A reachable OM HTTP(S) endpoint and `ozone-site.xml` on the host where you run the command. -- Caller identity in `ozone.administrators` when `ozone.security.enabled=true` (run `kinit` first in secure clusters). -- The **target** OM whose disk you are restoring must be **stopped** before you replace its local `om.db`. The **source** OM cluster must remain up to serve the checkpoint. - -**Download** - -Non-HA (single OM): +**Single batch** (snapshot SST below `ozone.om.ratis.snapshot.max.total.sst.size`, default 10 GB): ```shell -kinit -k -t /path/to/admin.keytab admin@REALM # secure clusters only - -ozone repair om download \ - --output-dir /restore/om.db \ - --overwrite -``` - -OM HA (specify the service ID and the node to pull from—typically the current leader): +# Non-secure +curl -f -X POST \ + "http://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ + -F "toExcludeSST[]=" \ + -o "om-metadata-$(date +%Y%m%d).tar" -```shell -ozone repair om download \ - --service-id \ - --node-id \ - --output-dir /restore/om.db \ - --overwrite +# Secure (run kinit first) +curl -f --negotiate -u : -X POST \ + "https://:/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true" \ + -F "toExcludeSST[]=" \ + -o "om-metadata-$(date +%Y%m%d).tar" ``` -The command always uses `/v2/dbCheckpoint` with snapshot data included. It fails if `--output-dir` already exists unless you pass `--overwrite`. - -**Install on the failed node** - -1. Stop the Ozone Manager on the node being restored (and block client writes if this is a standalone deployment). -2. Back up any partial data under `ozone.om.db.dirs` if the disk is still readable. -3. Replace the local database directory: +**Multiple batches** — `om-metadata-backup.sh` loops until `OZONE_RATIS_SNAPSHOT_COMPLETE`: ```shell -OM_DB_DIRS=/var/lib/ozone/om/metadata # your ozone.om.db.dirs - -rm -rf "$OM_DB_DIRS/om.db" -mv /restore/om.db "$OM_DB_DIRS/om.db" +chmod +x om-metadata-backup.sh +./om-metadata-backup.sh --base-url "http://:" \ + --archive "om-metadata-$(date +%Y%m%d).tar.gz" +# Add --kerberos for secure clusters (run kinit first) ``` -Preserve the existing `om/` VERSION subtree under `OM_DB_DIRS` when it survives disk failure. Do not delete OM cluster identity files unless you are intentionally rebuilding the OM from scratch. - -4. Start the Ozone Manager and verify with `ozone sh volume list`. - -For HA nodes with an empty metadata disk, prefer `ozone om --bootstrap` when the Ratis ring is intact. Use `ozone repair om download` when you need a constructed `om.db` on disk for manual replacement or offline inspection. - -See also [Ozone Repair](../tools/ozone-repair#download). - -### Restore from an offline tarball (no live OM) - -Use only when **no OM is available** to serve a checkpoint—for example, total metadata loss on a standalone deployment, or every HA OM lost its metadata volume. Offline tarballs cannot be consumed by `ozone repair om download`; install the saved checkpoint manually. - -**Requirements** - -- Backup taken from `/v2/dbCheckpoint` with `includeSnapshotData=true` on Ozone **2.2+**. -- Extracted staging tree contains `hardLinkFile` and `OZONE_RATIS_SNAPSHOT_COMPLETE`. -- Plan for **cluster downtime** while OM metadata is offline. - -**Procedure** - -1. **Stop writers.** Stop all Ozone Manager processes, clients, and services that write to the cluster. - -2. **Verify the backup.** For multi-batch backups, confirm the staging directory contains `hardLinkFile` and `OZONE_RATIS_SNAPSHOT_COMPLETE`. Do not proceed with an incomplete transfer. - -3. **Extract the archive** into a clean staging directory: +HTTP archives contain a **flat** inode-based tree (not the `om-metadata/` wrapper used by the CLI). Stop the target OM, then reconstruct hard links and install: ```shell -mkdir -p /restore/om-checkpoint-staging -tar -xf om-metadata-YYYYMMDD.tar -C /restore/om-checkpoint-staging -# If you used om-metadata-backup.sh --archive: -tar -xzf om-metadata-YYYYMMDD.tar.gz -C /restore/om-checkpoint-staging -``` - -4. **Reconstruct hard links.** v2 checkpoints store many files under inode IDs; `hardLinkFile` maps logical paths (for example `om.db/000012.sst`) to those IDs: +mkdir -p /restore/om-staging +OM_DB_DIRS=/var/lib/ozone/om/metadata -```shell -STAGING=/restore/om-checkpoint-staging +tar -xf om-metadata-YYYYMMDD.tar -C /restore/om-staging # .tar from curl +# tar -xzf om-metadata-YYYYMMDD.tar.gz -C /restore/om-staging # .tar.gz from om-metadata-backup.sh --archive while IFS=$'\t' read -r dest src; do - [[ -z "$dest" ]] && continue - install -d "$(dirname "$STAGING/$dest")" - ln "$STAGING/$src" "$STAGING/$dest" -done < "$STAGING/hardLinkFile" + [[ -z "$dest" || -z "$src" ]] && continue + install -d "$(dirname "/restore/om-staging/$dest")" + ln "/restore/om-staging/$src" "/restore/om-staging/$dest" +done < /restore/om-staging/hardLinkFile while IFS=$'\t' read -r dest src; do [[ -z "$src" ]] && continue - if [[ -d "$STAGING/$src" ]]; then - rm -rf "$STAGING/$src" + if [[ -d "/restore/om-staging/$src" ]]; then + rm -rf "/restore/om-staging/$src" else - rm -f "$STAGING/$src" + rm -f "/restore/om-staging/$src" fi -done < "$STAGING/hardLinkFile" - -rm -f "$STAGING/hardLinkFile" "$STAGING/OZONE_RATIS_SNAPSHOT_COMPLETE" -``` - -5. **Validate layout.** Expect top-level items such as `om.db` and, when bucket snapshots were backed up, `db.snapshots`. Confirm `om.db/CURRENT` exists. +done < /restore/om-staging/hardLinkFile -6. **Install into OM metadata storage.** Let `OM_DB_DIRS` be the path configured in `ozone.om.db.dirs` (or `ozone.metadata.dirs`): +rm -f /restore/om-staging/hardLinkFile /restore/om-staging/OZONE_RATIS_SNAPSHOT_COMPLETE -```shell -OM_DB_DIRS=/var/lib/ozone/om/metadata # your ozone.om.db.dirs - -mkdir -p "$OM_DB_DIRS" -for name in om.db db.snapshots; do - if [[ -e "$STAGING/$name" ]]; then - rm -rf "$OM_DB_DIRS/$name" - mv "$STAGING/$name" "$OM_DB_DIRS/" - fi -done -# Move any remaining top-level checkpoint items (compaction backup dirs, etc.) -find "$STAGING" -mindepth 1 -maxdepth 1 -exec mv -t "$OM_DB_DIRS" {} + +rm -rf "$OM_DB_DIRS/om.db" "$OM_DB_DIRS/db.snapshots" +rsync -a /restore/om-staging/ "$OM_DB_DIRS/" ``` -7. **Standalone OM:** Start SCM (if required), then start the Ozone Manager. Verify with `ozone sh volume list`. - -8. **HA OM (all peers lost metadata):** Clear `ozone.om.ratis.storage.dir` on **every** OM node, install the checkpoint on a seed OM, start it, then bootstrap remaining peers with `ozone om --bootstrap` once the seed is healthy. - -**Limitations** - -- v1 (`/dbCheckpoint`) backups omit bucket snapshot data; use them only for AOS-only recovery on legacy clusters. -- Validate the offline tarball procedure on a non-production clone before relying on it for production DR. - -Store tarball backups off-cluster (object store, NFS, or backup appliance) according to your retention policy. +The legacy v1 `/dbCheckpoint` endpoint omits bucket snapshot data even with `includeSnapshotData=true`. Use v2 only. ## Operational notes -- Run backups against the **leader** OM. Followers redirect or reject checkpoint creation. -- Checkpoint creation takes a short-lived snapshot cache lock when snapshot data is included. Schedule backups during low snapshot activity when possible. -- Recon's OM sync uses `includeSnapshotData=false` by design; do **not** use Recon as a full-metadata backup source. -- Bucket **Ozone Snapshots** (user-visible point-in-time bucket images) are included in this backup. **Ratis snapshots** (OM HA replication checkpoints) are a related but separate concept; see [OM high availability](../../../system-internals/components/ozone-manager/high-availability). +- Run checkpoints against the **leader** OM. +- Schedule during low snapshot activity; including snapshot data takes a short-lived cache lock. +- Do **not** use Recon as a backup source (`includeSnapshotData=false` by design). +- User-visible **Ozone Snapshots** are included. **Ratis snapshots** (OM HA replication) are a separate concept — see [OM high availability](../../../system-internals/components/ozone-manager/high-availability). ## Related configuration | Property | Default | Relevance | | ---------- | --------- | --------- | -| `ozone.om.db.checkpoint.use.inode.based.transfer` | `true` | Use `/v2/dbCheckpoint` | -| `ozone.om.ratis.snapshot.max.total.sst.size` | `10GB` | Batch size threshold for snapshot SST transfer | -| `ozone.om.bootstrap.min.space` | `5GB` | Follower disk pre-check fallback (bootstrap) | -| `ozone.om.bootstrap.checkpoint.estimated.space.headroom.ratio` | `2.0` | Follower disk pre-check vs leader SST estimate | -| `ozone.administrators` | (none) | HTTP checkpoint access in secure mode | +| `ozone.om.db.checkpoint.use.inode.based.transfer` | `true` | Required for snapshot-inclusive backup | +| `ozone.om.ratis.snapshot.max.total.sst.size` | `10GB` | HTTP multi-batch threshold | +| `ozone.administrators` | (none) | HTTP/CLI access in secure mode | ## See also -- [Ozone Repair](../tools/ozone-repair#download) - [OM bootstrapping with snapshots (design)](../../../system-internals/features/om-bootstrapping-with-snapshots) - [OM HA configuration](../../configuration/high-availability/om-ha) - [Replacing Ozone Manager disks](../disk-replacement/ozone-manager)