[RFD]: BMC Credential Secret Management in OpenCHAMI via Vault

[bmcdonald3]

Decision Goal

Determine the architectural approach for securely storing, generating, and accessing BMC credentials managed by the BMC Manager service in OpenCHAMI, specifically evaluating HashiCorp Vault against alternatives in an air-gapped, Podman/Docker quadlet deployment.

Category

Architecture

Stakeholders / Affected Areas

Determine the architectural approach for securely storing, generating, and accessing BMC credentials managed by the BMC Manager service in OpenCHAMI, specifically evaluating HashiCorp Vault against alternatives in an air-gapped, Podman/Docker quadlet deployment.

Decision Needed By

No response

Problem Statement

The BMC Manager service operates using a reconciliation loop that accepts BmcCredential requests to apply password rotations to target hardware via Redfish HTTP PATCH requests. Currently, the server executes with a SQLite database driver, writing state to a local file (data.db).

Because the BmcCredentialSpec includes both currentPassword and newPassword fields to facilitate authentication and rotation, these sensitive strings are logged and persisted in plaintext within the SQLite flat file. If a host system or container is compromised, the attacker gains immediate, unencrypted access to the credentials for the entire fleet of managed BMCs. Furthermore, flat-file storage provides 0 auditing capabilities to track which internal components or administrators access the credentials.

Proposed Solution

Integrate HashiCorp Vault Community Edition as the centralized secret manager.

Service Adjustments for Vault Integration

The BMC Manager will transition from storing raw credentials in SQLite to storing them in Vault. The local SQLite database will retain only the operational metadata (targetAddress, targetAccount, rotationSucceeded, lastRotationAttempt) and a reference URI mapping the resource to its corresponding Vault path (e.g., secret/data/openchami/bmc/172.24.0.3/account/3). Upon a successful HTTP 200 or 204 response from the Redfish endpoint, the service will write the updated password to Vault and clear the previous password from memory.

Password Generation Operational Models

To establish the password lifecycle, one of the following models must be adopted:

Model A: Service-Generated. The BMC Manager generates a random, cryptographically secure password (e.g., 256-bit entropy, 32 characters) internally. It applies this password to the BMC via Redfish and writes the resulting credential to Vault. The BmcCredentialSpec is modified to remove the newPassword requirement, acting purely as a rotation trigger.

Model B: User-Provided. An administrator supplies the specific newPassword within the API payload. The service attempts the rotation using the provided string and writes it to Vault upon success. This requires transmitting the plaintext password over the API to the BMC Manager.

Read Access and Least-Privilege Design

Vault policies will enforce strict boundaries on credential retrieval. Human administrators will be explicitly denied read access to the raw password paths within Vault. Read access will be exclusively granted to specific OpenCHAMI microservices that possess a functional requirement to communicate with the BMCs. Access logs generated by Vault will provide an audit trail of which services requested which credentials and at what time.

Service Authentication Mechanism

Because Kubernetes Service Accounts are incompatible with the Podman/Docker quadlet architecture, the BMC Manager will authenticate to Vault using the Vault AppRole auth method. AppRole is designed for machine-to-machine authentication. The quadlet will be provisioned with a RoleID and a SecretID. The SecretID can be configured with a strict time-to-live (e.g., 30 to 90 days) and rotation schedule. These identifiers will be passed to the BMC Manager container as environment variables at deployment, allowing it to request short-lived Vault tokens for writing credentials.

Alternatives Considered

• Docker / Podman Secrets: Native container secret mechanisms were considered. While they avoid adding a new infrastructure dependency, they are strictly designed for read-only consumption by containers at runtime. They do not provide an API for the credential-manager to dynamically write newly generated passwords back to the secret store, rendering them incompatible with the Service-Generated operational model.
• SOPS (Secrets OPerationS): SOPS was evaluated as a file-based encryption alternative. It was rejected because it is engineered primarily for GitOps workflows to encrypt configuration files at rest. It lacks the dynamic HTTP API required for the service to write credentials at runtime and does not offer the granular, centralized Access Control Lists required to enforce the least-privilege read access model across multiple OpenCHAMI microservices.

Other Considerations

Storage Backend: The integrated Raft storage backend is recommended for Vault to avoid external database dependencies, but it requires allocating dedicated, persistent host volumes within the quadlet configuration to prevent data loss across container restarts.

Related Docs / PRs

https://github.com/bmcdonald3/credential-manager

[bmcdonald3]

I have spent some time investigating the Vault integration and I am starting to see some of the complexities and why we haven't just adopted that already. I think an alternative that might be simpler, more in line with OpenCHAMI could be Infisical.

Vault is difficult to get up and running and requires a lot of resources. Running Vault securely requires a distributed storage backend for Raft consensus, and it has a sealed-by-default architecture meaning a cold boot of the data center requires manual unsealing or external cloud KMS dependencies. When working with something small, this feels like overkill to me.

Infisical seems a better middle ground than Vault and what Magellan is doing today with the flat files. It provides the necessary Role-Based Access Control and audit logging required for secure environments, but it runs on standard PostgreSQL and Redis. It boots directly into a ready state, which solves the cold-start bootstrapping problem. We can authenticate the container quadlets using Infisical's Universal Auth via injected environment variables, which map to the AppRole concept outlined in the original RFD.

I've got a draft of the credential manager using this approach here: https://github.com/bmcdonald3/credential-manager/tree/infiscal

And it's based on this approach:

1. The Adapter Pattern
Instead of hardcoding a specific backend, there is a generic SecretStore interface in the codebase. This allows us to inject the Infisical Go SDK adapter at runtime. It also lets us retain a local, file-based secret store as a fallback implementation for local development without modifying the core reconciliation logic.

2. Zero-Payload Passwords (Adopting Model A)
Passwords will no longer be transmitted over the network in the API payload. The BmcCredentialSpec will be stripped down to just the target address, the target account ID, and a unique node identifier.

3. The Reconciliation Loop
When a credential rotation is triggered, the service will execute the following steps:

• Use the node identifier to fetch the current BMC password directly from Infisical.
• Generate a new cryptographically secure 24-byte string internally.
• Execute the Redfish HTTP PATCH using the fetched current credentials.
• Upon a successful HTTP 200 or 204 response from the BMC, the service will write the newly generated password back to Infisical.

So, if we choose to go with this Infiscal approach, we could use this in things like Magellan as well.

https://github.com/infisical/infisical

[alexlovelltroy]

Cray-HPE/hms-securestorage#20

[bmcdonald3]

Summary of Vault/Credential Store Discussion From Community Meeting

The thought is to avoid introducing external dependencies like Vault or Infisical for the credential rotation service if a file-based approach is good enough. While Vault remains an acceptable fallback if strictly necessary, the immediate direction is to use a local secure file store to minimize overhead.

The referenced PR supports this approach by implementing a file-based secret store (LocalStore). The core technical additions include:

• Local JSON Storage: Secrets are stored in a local JSON file, which is designed to automatically reload if the file is modified externally.
• AES-GCM Encryption: All stored data is encrypted using AES-GCM before being written to the file.
• Key Derivation: Unique encryption keys are derived for each secret using HKDF. This derivation is based on the specific secret's ID and a central master key (supplied via the MASTER_KEY environment variable).
• CLI Implementation: A Cobra-based CLI example is provided to demonstrate storing, retrieving, and listing secrets, as well as generating new 32-byte master keys.

I'll switch the credential manager service to using this approach, similar to SMD and Magellan.

[Masber]

I am ok with Hashicorp Vault because we use it quite extensively at CSCS and it is integrated with other tools like gitlab and keycloak. We also have another team to manage tools like Hashicorp Vaule