diff --git a/docs/accessanalyzer/26.1/install/index.md b/docs/accessanalyzer/26.1/install/index.md index b45f667958..88f63c1f0e 100644 --- a/docs/accessanalyzer/26.1/install/index.md +++ b/docs/accessanalyzer/26.1/install/index.md @@ -13,7 +13,7 @@ An installation takes three steps, each on its own page. After the first sign-in, the [Guides](../guides/index.md) show you how to scan your first source. -After you're running, see [Upgrade to a new version](upgrade-to-a-new-version.md) for how new releases roll out and when you need to act. +After you're running, see [Upgrade to a new version](upgrade-to-a-new-version.md) for how new releases roll out and when you need to act. To replace the TLS certificate without reinstalling, see [Rotate the TLS certificate](rotate-the-tls-certificate.md). ## People You Need diff --git a/docs/accessanalyzer/26.1/install/installer-reference.md b/docs/accessanalyzer/26.1/install/installer-reference.md index 38437cecd8..b023e7acf7 100644 --- a/docs/accessanalyzer/26.1/install/installer-reference.md +++ b/docs/accessanalyzer/26.1/install/installer-reference.md @@ -1,14 +1,16 @@ --- title: Installer Reference description: The dspm-installer flags, environment variables, configuration file keys, exit codes, preflight checks, and log locations. -sidebar_position: 4 +sidebar_position: 6 --- -`dspm-installer` takes its settings from four places. A flag wins over an environment variable, an environment variable wins over the configuration file, and the configuration file wins over the built-in default. When the installer runs in a terminal, it prompts for any required value still missing; without a terminal, a missing required value is an error. +`dspm-installer` takes its settings from four places. A flag overrides an environment variable, an environment variable overrides the configuration file, and the configuration file overrides the built-in default. When the installer runs in a terminal, it prompts for any required value still missing; without a terminal, a missing required value is an error. ```bash dspm-installer [flags] dspm-installer wait-for-apps [flags] +dspm-installer update-cert [flags] +dspm-installer rollback-cert [flags] dspm-installer --help dspm-installer --version ``` @@ -34,10 +36,11 @@ Two environment variable names need care: `--hostname` reads `DSPM_HOSTNAME`, no | `--assume-yes` | `DSPM_ASSUME_YES` | `false` | Skip the review screen that appears when the configuration file already supplies every required value. | | `--dry-run` | `DRY_RUN` | `false` | Print the planned actions and exit without installing. Needs no TLS files and writes no configuration file. | | `--log-level` | `LOG_LEVEL` | `info` | Detail written to the log file: `debug`, `info`, `warn`, or `error`. | +| `--log-path` | `LOG_PATH` | `/var/log/dspm-installer.log` | Path to the installer's log file. If you set this explicitly (flag, environment variable, or configuration file) and the path isn't writable or is a symlink, the installer stops with an error instead of falling back to the terminal. | | `--postgres-data-dir` | `POSTGRES_DATA_DIR` | none | Custom directory for the application database's data. | | `--clickhouse-data-dir` | `CLICKHOUSE_DATA_DIR` | none | Custom directory for the analytics store's data. | | `--log-exports-storage` | `LOG_EXPORTS_STORAGE` | none | Persistent volume claim (PVC) size for log exports, such as `10Gi`. | -| `--skip-preflight` | `SKIP_PREFLIGHT` | `false` | Skip the preflight checks. Intended for testing only. | +| `--skip-preflight` | `SKIP_PREFLIGHT` | `false` | Skip the preflight checks. For testing only. | | `--version` | — | — | Print the installer version and exit. | | `--help` | — | — | Print flag help and exit. | @@ -107,8 +110,8 @@ When the file supplies every required value and the installer runs in a terminal | Code | Meaning | |---|---| | 0 | Success. | -| 1 | General failure: an invalid flag value, a hostname or TLS validation error, a required value missing in a non-interactive run, or prompts canceled with Esc or Ctrl-C (`installation cancelled`). | -| 10 | License key error. The key is expired, suspended, not found, or invalid. | +| 1 | General failure: an invalid flag value, a hostname or TLS validation error, a required value missing in a non-interactive run, or you canceled the prompts with Esc or Ctrl-C (`installation cancelled`). | +| 10 | License key error. The key is expired, suspended, unknown, or invalid. | | 20 | The release version you requested with `--target-revision` isn't available for this license key. | | 50 | The installer couldn't install the platform, or the platform didn't become ready within 5 minutes. | | 60 | The installer couldn't install a platform component. | @@ -116,6 +119,8 @@ When the file supplies every required value and the installer runs in a terminal | 71 | A service stayed in a failed state for 5 minutes. Only `wait-for-apps` returns this code; during an install the same condition exits 70. | | 80 | Preflight checks failed (`preflight checks failed`), or you didn't accept warnings (`preflight warnings detected; use --accept-warnings to continue` or `installation stopped at preflight warnings`). | +The `update-cert` and `rollback-cert` commands return their own codes. See [The `update-cert` command](#the-update-cert-command) and [The `rollback-cert` command](#the-rollback-cert-command). + ## Preflight Checks Eleven checks run before the installer changes anything on the server, in the order the following table lists them. Each ends as PASS, WARN, or FAIL. The installer prints only WARN and FAIL results, as ` [FAIL] ` or ` [WARN] `. Any FAIL stops the install; `--accept-warnings` doesn't override it. Any WARN stops it too unless you answer **Yes** to **Continue despite these warnings?** or pass `--accept-warnings`. @@ -132,7 +137,7 @@ The installer compares RAM and disk against their thresholds with a 5% tolerance | `kernel-modules` | The kernel has the `br_netfilter` and `overlay` modules loaded or built in. The install loads missing modules itself, so this check warns only when it can't inspect a module, or during a dry run when a module isn't loaded. | WARN | `kernel module issues: : could not check module: ` or `kernel module issues: : not loaded (dry run; will not be modprobed)` | | `os` | The Linux distribution belongs to a recognized family. | WARN | `unrecognised Linux distribution; installation may not be supported` | | `selinux` | SELinux isn't in enforcing mode. | WARN | The message says SELinux is enforcing and asks you to allow the platform's container policy or set SELinux to permissive. | -| `antivirus` | No known antivirus product is installed or running: `mdatp`, CrowdStrike, ClamAV, Sophos, Carbon Black, or Trend Micro. | WARN | `antivirus software detected: (exclusion hint: )` | +| `antivirus` | The server has no known antivirus product installed or running: `mdatp`, CrowdStrike, ClamAV, Sophos, Carbon Black, or Trend Micro. | WARN | `antivirus software detected: (exclusion hint: )` | | `network` | Each of the 18 required hosts resolves in DNS and accepts a connection on port 443 within 5 seconds. | FAIL when a name doesn't resolve; WARN when a connection times out or the host refuses it | `DNS resolution failed for: ` or `connection failed (timeout/refused) for: ` | | `domain-join` | Whether the server belongs to an Active Directory domain. Informational only. | — | `no AD domain detected`, or a message naming the detected domain | | `clock-sync` | A time-sync service (`chronyd`, `ntpd`, or `systemd-timesyncd`) is running. | WARN | `no clock sync daemon detected; Kerberos authentication requires clocks within 5 minutes of the AD domain controller — install chronyd, ntpd, or systemd-timesyncd to eliminate clock-skew risk` | @@ -157,9 +162,74 @@ It prints `Waiting for applications to become Synced and Healthy…` and exits w Exit codes: 0 when everything is healthy, 70 when the timeout passes, 71 when a service stays in a failed state for 5 minutes, and 1 for any other error. Ctrl-C exits 1. +## The `update-cert` Command + +`update-cert` installs a new TLS certificate on a running Access Analyzer installation, without re-running the full installer. Use it to replace a certificate that's expiring or expired, to replace one that pods don't trust, or to swap a self-signed certificate for a CA-issued one. + +```bash +sudo dspm-installer update-cert \ + --tls-cert /etc/dspm/tls.crt \ + --tls-key /etc/dspm/tls.key \ + --ca-bundle /etc/dspm/internal-root-ca.pem +``` + +`update-cert` validates the certificate and key pair, confirms the certificate's Subject Alternative Names cover the installed hostname, snapshots the certificate the cluster serves, applies the new certificate and CA bundle, restarts every workload that mounts the CA bundle, and verifies the result before it exits. See [Rotate the TLS certificate](rotate-the-tls-certificate.md) for the full procedure, including how to roll back with `rollback-cert`. + +| Flag | Default | Description | +|---|---|---| +| `--tls-cert` | (required) | PEM certificate file, full chain with the leaf certificate first. | +| `--tls-key` | (required) | PEM private key file matching `--tls-cert`. | +| `--ca-bundle` | none | PEM CA bundle the certificate chains to. Required unless the certificate is self-signed. | +| `--hostname` | from `/etc/dspm/installer.yaml` | Hostname the certificate must cover. | +| `--port` | `443` | External HTTPS port for probing the certificate the cluster serves. | +| `--timeout` | `30m0s` | Time budget for the whole rotation. A rollback, if needed, gets its own budget of the same size. | +| `--dry-run` | `false` | Validate the certificate and print the plan without changing the cluster. Doesn't need cluster access. | +| `--no-rollback` | `false` | Leave the new certificate in place if verification fails, instead of restoring the previous one automatically. | +| `--kubeconfig` | `/etc/rancher/k3s/k3s.yaml` | Path to the kubeconfig file. | +| `--argocd-namespace` | `argocd` | Kubernetes namespace for ArgoCD. | + +If verification fails, `update-cert` restores the previous certificate from its snapshot—unless you passed `--no-rollback`, or it couldn't reach the ingress at all. It exits with a code that tells you what state the cluster is in: + +| Code | Meaning | +|---|---| +| 0 | `update-cert` applied and verified the new certificate. | +| 1 | A check failed before `update-cert` wrote anything. The cluster is unchanged. | +| 72 | Verification failed; `update-cert` restored and verified the previous certificate. `update-cert` also returns 72 when you pass `--no-rollback` (nothing restored) or when it can't load the saved snapshot. | +| 73 | Verification failed; `update-cert` restored the previous certificate but couldn't verify it. | +| 74 | Verification failed and `update-cert` couldn't apply the rollback. | +| 75 | `update-cert` couldn't reach the ingress, so it verified nothing and rolled nothing back. The new certificate is still in place. | + +## The `rollback-cert` Command + +`rollback-cert` restores a certificate from a snapshot `update-cert` saved during an earlier rotation. Snapshots live under `/etc/dspm/cert-snapshots/`, and Access Analyzer never prunes them automatically. Snapshots contain private key material. See [Roll back a certificate](rotate-the-tls-certificate.md#roll-back-a-certificate) for how to remove ones you no longer need. + +```bash +sudo dspm-installer rollback-cert --latest +``` + +| Flag | Default | Description | +|---|---|---| +| `--list` | `false` | List available snapshots: timestamp, hostname, leaf certificate fingerprint, and expiry. Doesn't need cluster access. | +| `--latest` | `false` | Restore the most recent snapshot. | +| `--snapshot` | none | Restore the snapshot at the given path, such as `/etc/dspm/cert-snapshots/`. | +| `--hostname` | from `/etc/dspm/installer.yaml` | Hostname the restored certificate must cover. | +| `--port` | `443` | External HTTPS port for verifying the restore. | +| `--timeout` | `30m0s` | Time budget for the restore. | +| `--kubeconfig` | `/etc/rancher/k3s/k3s.yaml` | Path to the kubeconfig file. | +| `--argocd-namespace` | `argocd` | Kubernetes namespace for ArgoCD. | + +`--list`, `--latest`, and `--snapshot` are mutually exclusive. + +| Code | Meaning | +|---|---| +| 0 | `rollback-cert` applied and verified the restore. | +| 1 | A check failed before `rollback-cert` wrote anything, such as combining mutually exclusive flags or naming a snapshot that doesn't exist. | +| 73 | `rollback-cert` applied the restore, but verification failed. | +| 74 | `rollback-cert` couldn't apply the restore. | + ## Logs | File | Contents | |---|---| -| `/var/log/dspm-installer.log` | Everything the installer does, as one JavaScript Object Notation (JSON) object per line, at the detail `--log-level` sets. The installer appends to the file on every run, with mode `0640`. If the installer can't write the file, it sends the same output to the terminal's standard error as text. | +| The `--log-path` file (default `/var/log/dspm-installer.log`) | Everything the installer does, as one JavaScript Object Notation (JSON) object per line, at the detail `--log-level` sets. The installer appends to the file on every run, with mode `0640`, and rejects a symlink at that path. At the default path, a write failure is non-fatal and the installer sends the same output to the terminal's standard error as text instead; with `--log-path` set explicitly, the same failure stops the installer with an error. | | `/var/log/dspm-preflight.json` | The full result of the most recent preflight run: `timestamp`, `overallStatus`, and a `checks` list with `name`, `status`, and `message` for every check, including the ones that passed. `--dry-run` doesn't write it. | diff --git a/docs/accessanalyzer/26.1/install/rotate-the-tls-certificate.md b/docs/accessanalyzer/26.1/install/rotate-the-tls-certificate.md new file mode 100644 index 0000000000..425b5f21d9 --- /dev/null +++ b/docs/accessanalyzer/26.1/install/rotate-the-tls-certificate.md @@ -0,0 +1,199 @@ +--- +title: Rotate the TLS Certificate +description: Replace the TLS certificate on a running Access Analyzer installation with dspm-installer update-cert, and roll back to a previous certificate if needed. +sidebar_position: 5 +--- + +Rotate the TLS certificate with `update-cert`, a subcommand of the same `dspm-installer` binary you used to install Access Analyzer. `update-cert` and its counterpart, `rollback-cert`, talk to the cluster directly with `kubectl` instead of through the product API, so they work even when the `platform-service` workload is failing because it doesn't trust the current certificate. + +Run both commands with `sudo`. The default kubeconfig at `/etc/rancher/k3s/k3s.yaml` is readable only by root, so without `sudo`, `kubectl` falls back to `localhost:8080` and fails with "connection refused." + +## When to Rotate the Certificate + +Use `update-cert` when: + +- The current certificate is expiring or has expired. +- An internal certificate authority (CA) issued the current certificate and pods are failing because they don't trust it, even though the install itself completed. +- You want to replace a self-signed demo certificate with a CA-issued one. + +:::warning +Don't re-run the installer to change the certificate. The installer writes only the Kubernetes Secret that holds the certificate and key (`dspm-tls`)—it doesn't update the CA bundle every pod trusts, and ArgoCD reverts a hand-edited Secret on its next sync. `update-cert` updates both and waits for the cluster to load them. +::: + +## Before You Start + +1. Stage the new certificate and key on the install host. The conventional paths are: + + ```bash + /etc/dspm/tls.crt # PEM, full chain, leaf certificate first + /etc/dspm/tls.key # PEM private key matching the certificate + ``` + + Both files must be PEM. If you received a PFX or P12 file, convert it first: + + ```bash + openssl pkcs12 -in cert.pfx -clcerts -nokeys -out /etc/dspm/tls.crt + openssl pkcs12 -in cert.pfx -nocerts -nodes -out /etc/dspm/tls.key + ``` + +2. If a private CA issued the certificate, get the issuing root CA in PEM form too, for example `/etc/dspm/internal-root-ca.pem`. A certificate from a private CA requires this: a full-chain PEM omits the root by convention, so the certificate file alone doesn't tell `update-cert` which CA to trust. Only a self-signed certificate can skip this. + +3. Confirm the certificate covers the installed hostname. `update-cert` reads the hostname from `/etc/dspm/installer.yaml` and stops if the certificate's Subject Alternative Names don't cover it. Pass `--hostname` to override the value in that file, or to supply it when the file is missing. + +If a load balancer, reverse proxy, or split-horizon DNS sits in front of the cluster's ingress, read [If the probe fails behind a reverse proxy](#if-the-probe-fails-behind-a-reverse-proxy) before you start. + +## Rotate the Certificate + +1. Run a dry run first. It validates the files and prints what would change without touching the cluster, so you can run it without cluster access. + + ```bash + sudo dspm-installer update-cert \ + --tls-cert /etc/dspm/tls.crt \ + --tls-key /etc/dspm/tls.key \ + --ca-bundle /etc/dspm/internal-root-ca.pem \ + --dry-run + ``` + + Leave out `--ca-bundle` for a self-signed certificate. The output shows the hostname, the fingerprint of the replacement certificate, where the CA bundle comes from, and, if the cluster is reachable, the fingerprint the cluster serves. Fix the certificate or key files if validation fails here—`update-cert` hasn't written anything yet. + +2. Run the rotation. + + :::note + Restarting the workloads that mount the CA bundle briefly interrupts the web application. A rotation typically finishes in a few minutes; the default `--timeout` gives it up to 30 minutes. Run it during a maintenance window. + ::: + + ```bash + sudo dspm-installer update-cert \ + --tls-cert /etc/dspm/tls.crt \ + --tls-key /etc/dspm/tls.key \ + --ca-bundle /etc/dspm/internal-root-ca.pem + ``` + + `update-cert` validates the certificate and key pair, confirms the certificate covers the hostname, and (with `--ca-bundle`) confirms the certificate chains to the bundle. It then snapshots the certificate the cluster serves to `/etc/dspm/cert-snapshots//`, applies the new certificate and CA bundle, restarts every workload that mounts the CA bundle, and verifies the ingress serves the new certificate before it exits. On success, it prints `New certificate applied and verified (leaf )`. + +3. Confirm the certificate from a client machine. Substitute your installed hostname for ``. + + ```bash + openssl s_client -connect :443 -servername /dev/null \ + | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 + ``` + + The fingerprint should match the `leaf` value `update-cert` printed. + +If verification fails, `update-cert` restores the previous certificate from its snapshot and exits with a non-zero code—unless you passed `--no-rollback`, or it couldn't reach the ingress at all, in which case the new certificate stays in place. See [The `update-cert` command](installer-reference.md#the-update-cert-command) in the installer reference for what each code means. If it exits `74`, see the "Recovering when rollback-cert can't restore a snapshot" troubleshooting section under [Check the Result](#check-the-result). + +## If the Probe Fails Behind a Reverse Proxy + +`update-cert` verifies the certificate by connecting to `:` from the install host. If a load balancer, reverse proxy, or split-horizon DNS sits in front of the cluster's ingress, that verification connects to the intermediary's certificate instead of the one you just installed, and `update-cert` rolls back a certificate that actually installed correctly. The error message names the expected fingerprint and points you to `--no-rollback`. + +In that topology, run with `--no-rollback`, then verify the certificate yourself from a client that reaches the ingress directly, or read it from the Secret: + +```bash +sudo dspm-installer update-cert \ + --tls-cert /etc/dspm/tls.crt \ + --tls-key /etc/dspm/tls.key \ + --ca-bundle /etc/dspm/internal-root-ca.pem \ + --no-rollback + +sudo kubectl get secret dspm-tls -n kube-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -fingerprint -sha256 +``` + +If the new certificate turns out to be wrong, roll it back with `rollback-cert`. + +## Roll Back a Certificate + +Every rotation leaves a snapshot under `/etc/dspm/cert-snapshots/`. To restore a previous certificate: + +1. List the available snapshots. This doesn't need cluster access. + + ```bash + sudo dspm-installer rollback-cert --list + ``` + + The output shows the timestamp, hostname, leaf certificate fingerprint, and expiry of each snapshot. + +2. Restore one. + + ```bash + # the most recent snapshot + sudo dspm-installer rollback-cert --latest + + # or a specific one + sudo dspm-installer rollback-cert --snapshot /etc/dspm/cert-snapshots/ + ``` + + `rollback-cert` restores the CA bundle along with the certificate and key, restarts the workloads that consume them, and verifies the result. On success, it prints `Restored and verified certificate from `. See [The `rollback-cert` command](installer-reference.md#the-rollback-cert-command) in the installer reference for its flags and exit codes. + +:::note +Snapshots contain private key material. Access Analyzer writes them with restricted file permissions and never prunes them automatically. Remove ones you no longer need: + +```bash +sudo ls -l /etc/dspm/cert-snapshots/ +sudo rm -rf /etc/dspm/cert-snapshots/ +``` +::: + +## Check the Result + +Confirm the cluster's state directly if an exit code left you unsure what happened: + +```bash +# the certificate and key the ingress serves +sudo kubectl get secret dspm-tls -n kube-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -subject -dates -fingerprint -sha256 + +# the CA bundle pods trust +sudo kubectl get configmap ca-bundle -n access-analyzer -o jsonpath='{.data}' | head -c 400 + +# ArgoCD application health +sudo kubectl get applications -n argocd + +# platform-service came up after the restart +sudo kubectl rollout status deploy/platform-service -n access-analyzer +sudo kubectl logs deploy/platform-service -n access-analyzer --tail=50 +``` + +The CA bundle should start with `-----BEGIN CERTIFICATE-----`, every application should show `Synced` and `Healthy`, and the `platform-service` log should show OpenID Connect (OIDC) discovery completing rather than exiting on a certificate error. + +
+Troubleshooting: recovering when rollback-cert can't restore a snapshot + +If `rollback-cert` itself can't apply a snapshot, the snapshot directory still holds everything you need to recover manually. Each snapshot contains: + +| File | Contents | +|---|---| +| `tls.crt` | The previous certificate chain. | +| `tls.key` | The previous private key. | +| `params.json` | The previous `customCaBundle` and `caBundle` Helm parameter values (`caBundle` is already base64-encoded). | +| `meta.yaml` | Timestamp, hostname, leaf certificate fingerprint, and expiry. | + +Apply them directly: + +```bash +SNAP=/etc/dspm/cert-snapshots/ +sudo ls "$SNAP" + +# 1. Restore the certificate and key Secret. +sudo kubectl create secret tls dspm-tls -n kube-system \ + --cert="$SNAP/tls.crt" --key="$SNAP/tls.key" \ + --dry-run=client -o yaml | sudo kubectl apply -f - + +# 2. Restore the CA bundle parameters on the netwrix application. +# List the parameters, find the 0-based positions of ingress.customCaBundle +# and ingress.caBundle, and use them as N and M. +sudo kubectl get application netwrix -n argocd \ + -o jsonpath='{range .spec.source.helm.parameters[*]}{.name}{"\n"}{end}' +CUSTOM=$(sudo jq -r .customCaBundle "$SNAP/params.json") +BUNDLE=$(sudo jq -r .caBundle "$SNAP/params.json") +sudo kubectl patch application netwrix -n argocd --type json \ + -p "[{\"op\":\"replace\",\"path\":\"/spec/source/helm/parameters/N/value\",\"value\":\"$CUSTOM\"}, + {\"op\":\"replace\",\"path\":\"/spec/source/helm/parameters/M/value\",\"value\":\"$BUNDLE\"}]" + +# 3. Force a refresh and restart the consumers. +sudo kubectl annotate application netwrix -n argocd argocd.argoproj.io/refresh=hard --overwrite +sudo kubectl rollout restart deploy/platform-service -n access-analyzer +sudo kubectl get applications -n argocd -w +``` + +