diff --git a/oak-doc/src/site/markdown/security/audit-commit-flow.png b/oak-doc/src/site/markdown/security/audit-commit-flow.png new file mode 100644 index 00000000000..3c78c3692b8 Binary files /dev/null and b/oak-doc/src/site/markdown/security/audit-commit-flow.png differ diff --git a/oak-doc/src/site/markdown/security/audit-design.md b/oak-doc/src/site/markdown/security/audit-design.md new file mode 100644 index 00000000000..90ea2f1cd25 --- /dev/null +++ b/oak-doc/src/site/markdown/security/audit-design.md @@ -0,0 +1,510 @@ + + +Audit Pipeline Design +-------------------------------------------------------------------------------- + +This document describes the design of Oak's audit pipeline: the SPI surface +in `oak-core-spi`, the security-domain constants in `oak-security-spi`, the +pipeline implementation in `oak-core`, OSGi and embedded wiring, and the +threading and ordering rules the implementation relies on. + +For the consumer-facing guide (event model, listener contract, trust model), +see [Audit SPI](audit.html). + + +### Overview + +The audit pipeline transports structured `AuditEvent`s from producers to +bundle-registered `AuditEventListener` consumers, gated by a feature toggle +and a per-domain listener registry. + +A *capture site* is a place in Oak's own code that records an audit event. +Today the only ones are in `UserManagerImpl`, on group membership add and +remove. The term is used throughout this document for that kind of call +site, as distinct from a bundle emitting its own events. + +There are two delivery paths: + +- **Commit-attached.** Oak-internal capture sites (e.g. `UserManagerImpl`) + call `AuditDispatch.record(root, event)`. Events land in a per-session + `ThreadLocal` buffer (`AuditBuffer`), and a `NodeStore` `Observer` + (`AuditDrainObserver`) drains and dispatches them once the following + commit has durably persisted, whether the caller issued that commit + explicitly or an operation issued it on their behalf. At drain time each + event is decorated with `oak.commit.sessionId`, `oak.commit.userId`, and + `oak.commit.timestamp` payload entries; a failed commit drops the buffer. +- **Fire-and-forget.** Any OSGi bundle resolves `AuditEventEmitter` via + `@Reference` and calls `emit(event)`. The event is dispatched synchronously + on the calling thread: no buffering, no commit boundary, no payload + decoration. Caller-supplied values for the three reserved `oak.commit.*` + attestation keys are stripped before delivery (the trust contract on + `AuditEvent#getPayload()` is the normative statement). + +Both paths converge on `AuditEventListener.onEvents(List)` and +share one listener registry. Failure isolation is layered: an outer +`Throwable` barrier in `AuditDrainObserver.contentChanged` keeps audit from +masquerading as a commit failure, and an inner per-listener `Throwable` +barrier on both paths keeps one misbehaving listener from stopping the +others. + +Pipeline state is owned by `AuditPipeline` in `oak-core`, which +holds the feature toggle, the buffer, the listener registry, the sink +installed into the `AuditDispatch` facade, and the singleton drain observer. +It is registered as an OSGi service of type `AuditConfiguration`. Audit is a +top-level Oak concern, not a `SecurityConfiguration`. + + +### Pipeline diagram + +![Audit pipeline](audit-pipeline.png) + +The upper row is the commit-attached path, the lower row fire-and-forget. +They converge on the listener registry, which filters by domain and orders by +rank before invoking each listener. + +On the commit-attached path, `BufferSink.record` gates on the feature toggle +and on whether any listener is registered for the event's domain, so a +capture site allocates nothing when audit is off. The observer runs on the +commit thread once the merge has persisted, drains the buffer for that +session, and stamps the three `oak.commit.*` keys. On the fire-and-forget path, +`BufferSink.dispatch` applies the same toggle and listener gates, strips +caller-supplied `oak.commit.*` values, and dispatches inline. + +The short-circuit order in the observer, and the exception barriers on both +paths, are described under Implementation below. + + +### Commit flow + +![Commit flow](audit-commit-flow.png) + +The observer fires synchronously on the commit thread, after durable +persistence and before `store.merge` returns. That is the property the +per-session buffer depends on: the drain has to happen on the thread that +filled it. Every production `NodeStore` notifies observers that way, though +not all by the same route. The document and composite stores dispatch +through `ChangeDispatcher`; `MemoryNodeStore` iterates its registered +observers directly from `setRoot`. Either way the notification completes +before `merge` returns. + +The drain hangs off the commit, not off the API call that produced the event, +so operations that persist without an explicit `Session.save()` are covered +too. The buffer is keyed by `CommitInfo.getSessionId()`, which +`MutableRoot.commit` fills from the `ContentSession`, and anything reaching +`MutableRoot.commit` drains it. That covers the commits issued internally by +`Workspace.move` and by `VersionManager.checkin` / `checkout`, none of which +require a `Session.save()` from the caller. `Workspace.move` builds a fresh +`MutableRoot` from `ContentSession.getLatestRoot()`, but it is the same +`ContentSession`, so the buffer key still matches. The one case with no drain +is a write that never goes through `MutableRoot` at all, covered under +Migration commits below. + +Two segment-store configurations are worth knowing about, because in both the +commit-attached path silently produces nothing in the segment store. +`SegmentNodeStore.addObserver` returns a no-op handle unless change dispatch +is enabled, and an observer attached through that handle is never notified. +That applies to a cold-standby instance, where the primary store turns +dispatch off, and to a store configured through `SegmentNodeStoreFactory`, +where dispatch is off unless `dispatchChanges` is set explicitly. Neither +caveat extends to the document or composite stores: +`DocumentNodeStore.addObserver` always returns a live handle, and +`DocumentNodeStore` dispatches through `ChangeDispatcher` in the `merge` +path. + +Draining from an `Observer` rather than from a commit hook has two +consequences worth spelling out. Events never transit the `CommitContext`, +which is a shared string-keyed channel readable by any `CommitHook` in the +same commit; keeping audit events out of it avoids a class of cross-bundle +information disclosure. And dispatch happens only after the commit is +durable, so there is no window in which a listener sees an event for a +write that subsequently fails. + + +### SPI layout + +#### oak-core-spi + +Package `org.apache.jackrabbit.oak.spi.audit` holds the domain-neutral SPI: + +| Type | Role | +|---|---| +| `AuditEvent` | Event interface: domain, type, timestamp, payload. Static factory `AuditEvent.of(...)`. Publishes the three reserved key names as `COMMIT_SESSION_ID`, `COMMIT_USER_ID`, `COMMIT_TIMESTAMP`, and the `isCommitAttested(event)` predicate over them. | +| `AuditDomain` / `AuditType` | Value types wrapping the domain and type strings, created via `of(name)` and validated there. | +| `AuditEventListener` | Consumer SPI: `onEvents(List)`, scoped to one domain via `getDomain()`, ordered by `getRank()`. | +| `AuditEventEmitter` | OSGi service surface for fire-and-forget emission from any bundle. | +| `AuditDispatch` | Static facade: `record(root, event)` and `dispatch(event)`, routing to the installed `Sink`; `isEnabled()` / `isEnabledFor(domain)` gates. | +| `AuditDispatch.Sink` | SPI implemented by the pipeline. `AuditPipeline` installs a `BufferSink`. | +| `AuditBufferLifecycle` | Session lifecycle callouts: drain on refresh and on commit failure. | +| `AuditConfiguration` | Typed handle on pipeline state (`isActive()`, `NOOP`). | + +One class in the package is not part of the SPI: `AuditEventImpl`, the +immutable holder behind `AuditEvent.of(...)`. It lives in +`org.apache.jackrabbit.oak.spi.audit.impl`, which is absent from the +bundle's `Export-Package`, so it is unreachable outside `oak-core-spi` +despite being `public`. It has to be public because the factory that builds +it sits on the interface in the parent package. Keeping it out of the +exported package also keeps edits to it from moving that package's baseline +version, which BND computes per package rather than per class. + +Domain and type are value types rather than bare strings so the constraint +on them has somewhere to live. A listener that persists events into the +repository wants to build a path from the domain, so a domain has to be +usable as a JCR node name. `AuditDomain.of(...)` and `AuditType.of(...)` +enforce that: the name must be non-blank, must pass `JcrNameParser` (which +rules out `/`, `[`, `]`, `|` and `*`), and must contain no colon and no +whitespace. The colon is rejected rather than escaped because in JCR it +denotes a namespace prefix, which means nothing for a flat audit +identifier; whitespace is rejected because it has no place in one either. + +Validating in the factory puts the failure at the producer that supplied +the bad name, rather than at whichever listener later tried to build a path +out of it. Both types are final, with `equals`/`hashCode` over the wrapped +string, so the registry can route on them and `name()` gives listeners the +raw value back. Neither is an enum: the set of domains is open, and +consumer bundles define their own. + +`AuditConfiguration.isActive()` returns `true` when the feature toggle is +enabled and at least one listener is registered. The two predicates AND +together so a deployed-but-unused pipeline reports `false`, matching the +no-allocation semantics of `AuditDispatch.isEnabled()`. Both read the same +volatile sink state, so they cannot drift apart. The interface ships a +`NOOP` constant for callers that want a guaranteed-non-null handle. + +Cardinality is unary optional: multiple `AuditConfiguration` implementations +are not supported. The buffer lifecycle is a singleton install, and two +observers on the same root `NodeStore` would each produce a dispatch. +Multiplexing belongs at the listener layer. + +#### oak-security-spi + +Security-domain constants live next to the SPI they describe: + +- `spi/security/audit/SecurityAuditDomain` holds `DOMAIN`, the single + `AuditDomain` wrapping `"oak.security"`, shared by all events Oak's security + stack emits. The `oak.` prefix namespaces the domain so listeners in mixed + deployments (Sling, application bundles) can tell Oak's security events + apart from same-named domains defined by other layers. +- `spi/security/user/UserAuditTypes` holds the user-membership vocabulary: + `AuditType` constants (`MEMBER_ADDED`, `MEMBER_REMOVED`) and payload keys + (`PAYLOAD_GROUP_PATH`, `PAYLOAD_MEMBER_IDS`, `PAYLOAD_MEMBER_PATHS`, + `PAYLOAD_MEMBERSHIP_SOURCE`, `PAYLOAD_IS_CONTENT_ID`, + `PAYLOAD_FAILED_IDS`). A single and a bulk membership change share the + same type; a bulk change is one whose `PAYLOAD_MEMBER_IDS` list holds more + than one entry. + +Future ACL, principal, or token events declare their own `*AuditTypes` +classes in the respective SPI sub-packages. + +Producer-side factories are deliberately not part of the SPI. They live as +package-private classes next to their only callers, e.g. +`UserAuditEvents` next to `UserManagerImpl` in `oak-core`. The asymmetry +(read-side vocabulary public, write-side factories impl-private) raises the +bar for casually forging Oak-attested events, but it is not a hard boundary: +any bundle can call `AuditEvent.of(domain, type, payload)` directly. +Listeners that need to distinguish Oak-attested commit-attached events from +fire-and-forget emissions call `AuditEvent.isCommitAttested(event)`. + +The helper exists so listeners do not hardcode the key names or re-derive +the rule. It is named for what it checks: all three reserved keys are +present and non-null. Oak does not sign events, so a positive result means +the event came through Oak's commit-attached dispatch, which is exactly the +guarantee the trust contract on `AuditEvent#getPayload()` states, and no +more. The key names themselves are public as `AuditEvent.COMMIT_SESSION_ID`, +`COMMIT_USER_ID`, and `COMMIT_TIMESTAMP`, for listeners that read individual +values rather than testing for attestation. Helper and constants both sit in +`oak-core-spi`, so a listener bundle still depends on that module alone. + + +### Implementation (oak-core) + + +#### Design rules + +The rules the components below are built on: + +1. **Observers fire synchronously on the commit thread for local commits.** + Every production store notifies observers before `merge` returns, whether + through `ChangeDispatcher` or, as in `MemoryNodeStore`, by iterating its + observers directly. The per-thread buffer depends on this. +2. **Observers fire after durable persistence, or not at all.** A failed + merge never reaches the observer, so a dispatched event always + corresponds to a persisted write. +3. **External commits are ignored at observer entry.** One predicate covers + cluster sync, the `addObserver` replay, and external head movement. +4. **The buffer key equals `CommitInfo.getSessionId()`.** `MutableRoot` + sets the commit info's session id from the `ContentSession`, which is the + same key the sink used at capture time. +5. **`CompositeObserver` provides no per-observer isolation.** Hence the + outer `Throwable` barrier in `contentChanged`. +6. **No ordering guarantee among observers.** Audit does not depend on + observer order; listener order within the audit dispatch is defined by + `getRank()`. +7. **Never wrap the drain observer in `BackgroundObserver`.** Queue overflow + replaces the commit info with `CommitInfo.EMPTY_EXTERNAL`, losing the + session id and with it the buffered events. +8. **Audit never masquerades as a commit failure.** The outer barrier + guarantees `contentChanged` returns normally no matter what the drain, + the decorator, or a listener does. +9. **Lifecycle callouts are unconditional.** Gating them on the toggle or on + listener presence would let events captured while the toggle was on stay + in the buffer across a lifecycle transition, to be dispatched later + against an unrelated commit and stamped with that commit's metadata. + +Two consequences of the destructive `ThreadLocal` drain are worth noting. +When a composite store causes the observer to be invoked twice for one +merge, the first invocation drains the buffer and the second finds it empty +and returns, so double-dispatch dedupes itself. And because `clearAll()` +removes only the calling thread's `ThreadLocal` entry, disposing the +pipeline while other threads hold sessions mid-flight leaves their staged +events behind; that residue is bounded by the per-session cap and released +when the thread is reused or discarded. + +#### Components + +| Component | Role | +|---|---| +| `AuditBuffer` | `ThreadLocal` per-session staging area, keyed by `ContentSession` id. Caps a session at 10,000 staged events: past that, further events are dropped and one WARN is logged for the session rather than one per event. The cap re-arms on the next drain, refresh, or commit failure, so it bounds the memory a single large or non-committing session can pin. | +| `BufferSink` (inner class of `AuditPipeline`) | The installed `AuditDispatch.Sink`. Gates on the feature toggle and listener presence, buffers on `record`, dispatches inline on `dispatch`. | +| `AuditDrainObserver` | `Observer` that drains the buffer on commit success. Carries the outer and inner `Throwable` barriers. | +| `CommitMetadataDecorator` | Stamps the three reserved `oak.commit.*` entries at drain time (commit-attached) and strips caller-supplied values for the same keys at dispatch (fire-and-forget). | +| `AuditEventEmitterImpl` | OSGi `@Component` implementing `AuditEventEmitter`; delegates to `AuditDispatch.dispatch`. | +| `WhiteboardAuditEventListenerRegistry` | Tracks `AuditEventListener` services on the Whiteboard. `getListeners()` returns them sorted by rank descending; `hasListenerFor(domain)` backs the pre-allocation gate. | +| `AuditMonitor` | Wraps the `StatisticsProvider`: per-domain event meter, per-listener timer and failure meter, dropped-event meter. Falls back to a no-op when no provider is bound. | +| `AuditPipeline` | Pipeline owner: feature toggle, buffer, registry, sink, drain observer, monitor. Published as `AuditConfiguration`. | + +`initialize` looks the `StatisticsProvider` up on the whiteboard rather than +taking a DS `@Reference`, so the OSGi and embedded paths share one lookup. +Deployments that publish no provider, which is the normal case for tests and +embedded callers, get `AuditMonitor.NOOP` and record nothing. The monitor is +passed to the components that record: the buffer for dropped events, and both +dispatch paths for the event meter, the per-listener timer, and the failure +meter. Recording sits inside the existing per-listener `Throwable` barrier, so +a metrics failure cannot break a dispatch. The metric names and their +operational meaning are listed under [Monitoring](audit.html#Monitoring). + +The event meter counts an event once per domain, when at least one listener +consumed it. Both are deliberate: counting per listener would multiply the +rate by the number of subscribers, and counting before the dispatch loop would +include events whose listener unregistered between capture and drain. + +#### AuditDrainObserver + +The observer's `contentChanged(root, info)` short-circuits on +`CommitInfo.isExternal()`, then drains the buffer for `info.getSessionId()`. +If events came out and the toggle is still enabled, it decorates them via +`CommitMetadataDecorator`, groups them by domain, and dispatches each group +to the listeners registered for that domain, in rank order. + +The drain runs before the toggle check, not after. Draining unconditionally +means a toggle flip between capture and commit discards the staged events +cleanly; checking the toggle first would leave them in the buffer, where a +later commit on the same session would pick them up and stamp them with the +wrong commit metadata. + +Three rules govern this class: + +- **Outer `Throwable` barrier.** `CompositeObserver` iterates its observers + with no per-observer isolation, and the `NodeStore` implementations invoke + the observer chain after the commit is already durable. An exception + escaping `contentChanged` would therefore surface as a commit failure to + the merge caller even though the commit succeeded, and could mask other + observers' work. The entire method body runs inside a + `try { ... } catch (Throwable t) { log.warn(...); }`. +- **Inner per-listener barrier.** Each listener dispatch is individually + wrapped, also at `Throwable` width, so a listener throwing a + `LinkageError` or similar does not stop the remaining listeners. The same + isolation covers the `getDomain()` / `getRank()` accessors consulted + during routing. +- **Never wrap in `BackgroundObserver`.** The async wrapper replaces the + latest queued entry with `CommitInfo.EMPTY_EXTERNAL` on queue overflow, + which discards the session id. The drain keys exclusively on + `CommitInfo.getSessionId()` to find the per-thread buffer, so losing the + session id silently loses audit events for high-rate writers. Synchronous + dispatch is mandatory, and the buffer's `ThreadLocal` semantics require + draining on the capturing thread anyway. + +The external-commit short-circuit covers cluster sync from peer nodes, the +synthetic replay invocation that `Observable.addObserver` makes at +registration time, and external head movement in the segment store. By +construction the buffer is empty for external commits (no local capture site +fired), so the explicit gate is defense in depth rather than a correctness +requirement. + +#### Session lifecycle and buffer draining + +The observer only sees successful commits. `MutableRoot` covers the other +paths through `AuditBufferLifecycle` callouts: + +| Case | Who drains the buffer | +|---|---| +| `Root.commit()` succeeds | `AuditDrainObserver.contentChanged` | +| `Root.commit()` fails (merge throws) | `MutableRoot.commit` finally block, via `AuditBufferLifecycle.onCommitFailed` | +| `Root.refresh()` | `MutableRoot.refresh`, via `AuditBufferLifecycle.onRefresh` | +| Pipeline shutdown while sessions are mid-flight | `AuditPipeline.dispose`, via `buffer.clearAll()` | + +`Root.rebase()` intentionally does not drain: rebase preserves transient +changes, so the audit events staged alongside them survive and are +dispatched when the session eventually commits. + +The lifecycle callouts always fire, for the reason given in design rule 9. +The cost of that is negligible: when no pipeline is installed the callout is +one volatile read plus a virtual call into a no-op listener. + +`AuditDispatch.record(root, event)` requires that `root` is the `MutableRoot` +of an active JCR session, since the lifecycle callouts above are what keep +the buffer consistent. Non-JCR commits must not call it. + + +### OSGi wiring + +`AuditPipeline` is declared as +`@Component(service = AuditConfiguration.class)`. It takes no reference to +the `NodeStore` or to `Observable`. Instead it follows Oak's established +observer-registration idiom (the same one the Lucene index observer uses): +`@Activate` registers the drain observer as an `Observer` service on the +`BundleContext`, and the `ObserverTracker` that each NodeStore service runs +picks the service up and subscribes it to the root `NodeStore`. This keeps +the audit bundle decoupled from store selection; composite deployments get +the same root store the rest of the stack uses. + +Activation wires the pipeline internals first and publishes the `Observer` +service last; deactivation unregisters that service first and only then tears +the internals down: + +![Activation and deactivation order](audit-lifecycle.png) + +Registering the observer last means a commit thread racing with activation +either misses the observer entirely (events stay buffered for the next +commit) or sees a fully wired pipeline. Unregistering it first means +`ObserverTracker` closes the subscription before anything is dismantled, so +no further `contentChanged` call reaches a half-torn-down buffer or registry. + +Detach first, tear down internals second: the same shape as +`ChangeProcessor` in `oak-jcr`. `dispose()` checks that the observer service +has been unregistered and fails loudly otherwise, turning an out-of-order +teardown into an error instead of a dangling observer subscription over +torn-down state. The drain observer is constructed once in `initialize` and +zeroed in `dispose`; `getDrainObserver()` throws `IllegalStateException` +outside that window. It is a singleton by design: two observer instances +sharing one buffer would double-dispatch. + + +### Embedded (non-OSGi) wiring + +Embedded callers (tests, `oak-run` tooling, custom embeds) wire the pipeline +explicitly. Several of the types below, including `AuditPipeline` +and `SecurityProviderBuilder`, live in packages oak-core does not export, so +this path is available to code on a flat classpath rather than to a bundle +running inside an OSGi framework; there, use the DS service instead. + +```java +MemoryNodeStore store = new MemoryNodeStore(); +DefaultWhiteboard whiteboard = new DefaultWhiteboard(); + +AuditPipeline audit = new AuditPipeline(); +audit.initialize(whiteboard); // toggle, registry, buffer, sink + +// The toggle is created disabled. Flip it on through the FeatureToggle +// that initialize() registered, or the pipeline stays silent. +Tracker toggles = whiteboard.track(FeatureToggle.class); +try { + for (FeatureToggle ft : toggles.getServices()) { + if (AuditPipeline.FEATURE_TOGGLE_NAME.equals(ft.getName())) { + ft.setEnabled(true); + } + } +} finally { + toggles.stop(); +} + +// Register listeners before driving any commit. +whiteboard.register(AuditEventListener.class, new MyListener(), Map.of()); + +Closeable observerHandle = store.addObserver(audit.getDrainObserver()); + +SecurityProvider securityProvider = SecurityProviderBuilder.newBuilder() + .withWhiteboard(whiteboard) + .build(); + +ContentRepository repo = new Oak(store) + .with(new InitialContent()) // required: security setup needs jcr:system + .with(securityProvider) + .with(whiteboard) + .createContentRepository(); + +// ... drive commits ... + +observerHandle.close(); // detach observer first, as in OSGi teardown +if (repo instanceof Closeable) { + ((Closeable) repo).close(); // ContentRepository itself declares no close() +} +audit.dispose(); +``` + +Four things in that sequence are easy to get wrong. The toggle starts +disabled, so a pipeline that is otherwise wired correctly dispatches nothing +until something enables it. `InitialContent` is needed because the security +setup expects `jcr:system` to exist. Listeners have to be registered before +the commits you want to capture, since the capture gate checks for a listener +on the event's domain. And `ContentRepository` declares no `close()`, so the +teardown has to test for `Closeable`. + +The explicit `addObserver` call is required. `Oak.with(Observer)` relies on +an auto-attach side effect of Oak's default whiteboard, and that side effect +is lost as soon as the embedder replaces the whiteboard via +`Oak.with(Whiteboard)`. Sharing one whiteboard between audit and Oak is the +common case for embedded setups (listener registrations and the audit +tracker should see the same whiteboard), so the safe path is always the +direct `Observable.addObserver(...)`. OSGi deployments are unaffected: their +subscription runs through `ObserverTracker`, not through the default +whiteboard. + + +### Migration commits + +Migration tooling (`oak-upgrade`) calls `NodeStore.merge(...)` directly, +bypassing `MutableRoot` and the capture sites. Migration tools do not set the +pipeline up, so when one runs standalone there is no observer attached and no +audit machinery on the path at all. Run inside a container where audit is +deployed, the observer does fire on migration commits, finds an empty buffer +for the session id, and returns. Either way migration mutations are not +audited; if that is ever wanted, it is a capture-site addition, not a +pipeline change. + + +### Performance characteristics + +With audit off (toggle disabled, or no listener registered for the domain), +capture sites short-circuit at `AuditDispatch.isEnabledFor(domain)` before +constructing an event: no allocation, no buffer touch. The check is a +volatile read of the installed sink and the toggle, and, when the toggle is +on, a linear scan of the registered listeners comparing each `getDomain()` +against the requested domain. Deployments carry a handful of listeners, so +the scan stays cheaper than maintaining a domain index. The per-commit cost +of a deployed-but-idle pipeline is the external-commit check plus an empty +buffer lookup in the observer. + +With audit on, the per-event cost is the event allocation, a buffer append, +the drain, three decorator entries, the metric updates, and the listener +dispatch itself. Benchmarks in `oak-benchmarks` cover both shapes: the +pipeline running with no capture site firing, and the full captured-event +path. In both, the overhead sits below the resolution of the surrounding +commit machinery, so turning audit on does not measurably change commit +throughput. Listener work is on top of that and belongs to the listener; +implementations that do I/O are expected to hand off to their own async +executor. diff --git a/oak-doc/src/site/markdown/security/audit-lifecycle.png b/oak-doc/src/site/markdown/security/audit-lifecycle.png new file mode 100644 index 00000000000..5a0d8e128a3 Binary files /dev/null and b/oak-doc/src/site/markdown/security/audit-lifecycle.png differ diff --git a/oak-doc/src/site/markdown/security/audit-pipeline.png b/oak-doc/src/site/markdown/security/audit-pipeline.png new file mode 100644 index 00000000000..e7eb869fd4b Binary files /dev/null and b/oak-doc/src/site/markdown/security/audit-pipeline.png differ diff --git a/oak-doc/src/site/markdown/security/audit.md b/oak-doc/src/site/markdown/security/audit.md new file mode 100644 index 00000000000..aff178545e5 --- /dev/null +++ b/oak-doc/src/site/markdown/security/audit.md @@ -0,0 +1,525 @@ + + +Audit SPI +-------------------------------------------------------------------------------- + +### General + +The Oak audit SPI records structured events about repository activity and +dispatches them to in-process consumers. Listeners are registered on the OSGi +Whiteboard and invoked synchronously when events are produced. Typical +consumers forward events to a SIEM, write to a compliance archive, or apply +runtime policy. + +The SPI is small: an event type, a listener interface, and an emitter service. +It does not prescribe transport, persistence, or out-of-process delivery. +Those are listener concerns. + +A *capture site* is a place in Oak's own code that records an audit event. +Today the only ones are in the user-management implementation, on group +membership add and remove. + +Two producer paths feed a single listener registry: + +- A **commit-attached** path used by Oak-internal capture sites, currently + group membership changes in the user-management implementation. Events are + buffered for the duration of a session write, drained on the commit that + follows, and decorated with commit metadata before dispatch. Events are + dropped if the commit fails. +- A **fire-and-forget** path exposed to any OSGi bundle through the + [AuditEventEmitter] service. Events are dispatched immediately on the + calling thread. They are not tied to a commit and are not buffered. + +Both paths converge on the same `AuditEventListener.onEvents(List)` +method, so a single listener can consume Oak-internal security events and +bundle-emitted custom events through one entry point. + + +### Module layout + +| Module | Role | +|---|---| +| `oak-core-spi` | Domain-neutral SPI: [AuditEvent], [AuditEventListener], [AuditEventEmitter], the [AuditDispatch] static facade, and [AuditConfiguration] (typed handle on the pipeline's runtime state). | +| `oak-security-spi` | Security-domain constants: `SecurityAuditDomain.DOMAIN` (the `"oak.security"` domain) and per-sub-domain vocabulary classes such as `UserAuditTypes` in the `spi.security.user` package. | +| `oak-core` | Pipeline implementation: listener registry, commit-attached buffer, the observer that drains it on commit success, the emitter, and the configuration component. | + +Consumer bundles depend on `oak-core-spi` only. Implementing a listener or +emitting events requires no dependency on `oak-core`, `oak-jcr`, or +`oak-security-spi`. + + +### Event model + +#### AuditEvent + +```java +public interface AuditEvent { + @NotNull AuditDomain getDomain(); + @NotNull AuditType getType(); + long getTimestamp(); + @NotNull Map getPayload(); +} +``` + +- **Domain**: namespace identifying the event source category, as an + `AuditDomain`. Oak's security stack uses `SecurityAuditDomain.DOMAIN`, which + wraps `"oak.security"`. Bundles defining new event types build their own with + `AuditDomain.of("...")`; the SPI imposes no schema. +- **Type**: stable identifier within the domain, as an `AuditType`, e.g. + `AuditType.of("membership.added")`. Consumers dispatch on it. +- **Timestamp**: milliseconds since epoch at event construction time. +- **Payload**: open map of supplementary data. Consumers MUST tolerate missing + keys; producers MAY add keys without versioning. + +`AuditDomain` and `AuditType` wrap their names rather than passing plain +strings around, and both validate in `of(...)`: a name must be non-blank and +usable as a JCR node name, with no colon and no whitespace. That keeps a +domain safe to use as a path element for listeners that persist events into +the repository, and it means a bad name fails at the producer instead of +reaching a listener. Call `name()` for the underlying string. The +[design document](audit-design.html#SPI_layout) has the full rules. + +The public SPI keeps only the `AuditEvent` interface. Concrete events are +built with the static factory `AuditEvent.of(domain, type, payload)`, and +consumers discriminate events by `getDomain()` plus `getType()` rather than by +`instanceof` checks. + +The `oak.security` domain pins its type strings and payload keys in +per-sub-domain classes next to the security area they describe. User-membership +constants live in `UserAuditTypes` in the `spi.security.user` package +(`MEMBER_ADDED`, `PAYLOAD_GROUP_PATH`, and so on). Bundles emitting custom +events implement `AuditEvent` directly or call `AuditEvent.of(...)` with their +own domain string. + + +#### Commit metadata payload keys + +Events produced by the commit-attached pipeline are decorated at drain time +with three additional payload entries: + +| Key | Value | Source | +|---|---|---| +| `oak.commit.sessionId` | session identifier of the writing session | `CommitInfo.getSessionId()` | +| `oak.commit.userId` | acting user id (`CommitInfo.OAK_UNKNOWN`, i.e. `"oak:unknown"`, for system commits) | `CommitInfo.getUserId()` | +| `oak.commit.timestamp` | commit timestamp in milliseconds since epoch | `CommitInfo.getDate()` | + +The three key names are published as `AuditEvent.COMMIT_SESSION_ID`, +`AuditEvent.COMMIT_USER_ID`, and `AuditEvent.COMMIT_TIMESTAMP`; use those +rather than string literals. + +Events arriving through the fire-and-forget pipeline cannot carry these keys: +Oak strips caller-supplied values for exactly these three at dispatch. For +events delivered through Oak dispatch, their presence is therefore a reliable +commit-attached signal. The Javadoc on `AuditEvent#getPayload()` is the +normative statement of this contract. Consumers that need to tell the two +sources apart call `AuditEvent.isCommitAttested(event)`, which returns +`true` when all three keys are present and non-null. The `oak.commit.userId` +value `"oak:unknown"` is a deliberate anonymity marker for system commits; +listeners MUST NOT attempt to resolve it to a real user. + + +### Pipelines + +#### Commit-attached pipeline + +Used by Oak-internal capture sites in the user-management implementation. Events +are buffered against the writing session and only dispatched when +`Root.commit()` succeeds, strictly **after** durable persistence rather than +inside the commit hook chain. If validators reject the commit or the merge +fails, the buffered events are discarded. + +The dispatch sequence: + +1. A capture site appends an event to the per-session buffer. +2. The session reaches `Root.commit()`; commit hooks and validators run; the + merge persists durably. +3. An `Observer` registered by the audit configuration fires on the commit + thread, drains the buffer for the originating session, and decorates each + event with `oak.commit.sessionId`, `oak.commit.userId`, and + `oak.commit.timestamp`. +4. The registry sorts listeners by rank, filters by domain, and invokes each + matching listener's `onEvents(List)`. + +Step 2 does not require an explicit `Session.save()`. Operations that commit +on their own, such as `Workspace.move` or `VersionManager.checkin`, reach +`Root.commit()` too and drain the buffer the same way. + +Because dispatch happens after durable persistence, a delivered event implies +the corresponding write actually landed. A failed commit never produces an +audit event. + +The converse does not hold, and consumers building a compliance trail need to +know it. The per-session buffer is capped, so a session that records more +than 10,000 events before committing has its later events dropped, with a +single WARN logged for that session rather than one per dropped event. A +persisted write can therefore leave no audit event behind. The cap exists to +bound the memory one runaway session can pin; it resets on the next commit, +refresh, or commit failure. Treat that WARN as a gap in the trail. + +This path is internal to Oak; bundles that want to record their own events use +the fire-and-forget pipeline below. + +#### Fire-and-forget pipeline + +Available to any OSGi bundle that wants to record an event for its own domain. +Events fire immediately on the calling thread; there is no buffering and no +rollback: + +1. The caller resolves `AuditEventEmitter` via `@Reference`. +2. The caller gates allocation with `isEnabledFor(domain)`. +3. The caller invokes `emit(event)`. +4. The registry sorts listeners by rank, filters by domain, and invokes each + matching listener's `onEvents(List)`. + +Properties: + +- **No commit boundary.** The event is dispatched as soon as `emit` is called; + subsequent JCR operations do not affect it. +- **Synchronous on the calling thread.** Listeners performing I/O are + responsible for wrapping themselves in an async dispatcher. +- **Per-listener isolation.** Exceptions thrown by one listener, whether from + `onEvents` or from the `getDomain()` / `getRank()` accessors consulted + during routing, are logged and swallowed; remaining listeners still run. + `emit` never propagates a listener exception back to the caller. +- **No payload decoration, but reserved keys are stripped.** No `oak.commit.*` + keys are added; caller-supplied values for the three reserved attestation + keys (`oak.commit.sessionId`, `oak.commit.userId`, + `oak.commit.timestamp`) are removed before delivery. Every other entry + reaches listeners exactly as the caller provided it. + + +### Configuration + +The pipeline is gated by a feature toggle and is **off by default**. Nothing +is captured or dispatched until the toggle is enabled, which keeps the cost +of a deployed-but-unused pipeline at zero. + +Registering a listener is not enough on its own. Both conditions have to +hold: the toggle is enabled, and at least one listener is registered for the +event's domain. Enable the toggle the same way as any other Oak feature +toggle, through the `FeatureToggle` service published on the Whiteboard; the +[OSGi configuration](../osgi_config.html) page describes the mechanism under +Feature Toggles. For a worked example of locating this toggle and flipping +it, see the embedded wiring snippet in +[Audit Pipeline Design](audit-design.html). + +`AuditPipeline` in `oak-core` owns the pipeline and is published as +an OSGi service of type `AuditConfiguration`. It carries an OSGi +object-class definition, so it appears in the Felix console alongside Oak's +other components. + + +### Probing pipeline state + +Components can ask whether the audit pipeline is currently active via +[AuditConfiguration]`.isActive()`, without depending on the implementation +class. `AuditConfiguration` is an OSGi service; resolve it via a DS +`@Reference`: + +```java +@Component(service = MyComponent.class) +public class MyComponent { + + @Reference + private AuditConfiguration audit; + + public void doWork() { + if (audit.isActive()) { + // Feature toggle is ON and at least one listener is registered. + // Safe to do work that only matters when audit will actually + // dispatch (e.g. allocate richer payload context). + } + } +} +``` + +`AuditConfiguration` is published as an OSGi service only. Embedded callers +(tests, `oak-run` tools) use `AuditDispatch.isEnabled()` on the static facade +instead, which evaluates the same two conditions. + +`isActive()` returns `true` when the audit feature toggle is enabled AND at +least one `AuditEventListener` is registered on the Whiteboard. A +deployed-but-unused pipeline (toggle ON, no listener registered) reports +`false`, matching the no-allocation semantics of `AuditDispatch.isEnabled()`. +The NOOP `AuditConfiguration`, returned when no implementation is bound at +all, reports `false`. + +Note that audit is a top-level Oak concern, not a `SecurityConfiguration`: +`AuditConfiguration` is not reachable via +`SecurityProvider.getConfiguration(...)`. Use a `@Reference` to +`AuditConfiguration`. + + +### Monitoring + +The pipeline registers metrics through the `StatisticsProvider` it resolves at +activation, so they surface via JMX or Sling Metrics like Oak's other metrics. +Nothing is registered when no `StatisticsProvider` is bound. + +| Name | Type | Description | +|---|---|---| +| `security.audit.events;domain=` | Meter | Events dispatched, per domain. Counts events that reached at least one listener, so it excludes anything dropped at the toggle or the listener gate. | +| `security.audit.listener.duration;listener=` | Timer | Wall-clock duration of one `onEvents` call, per listener class. | +| `security.audit.listener.failures;listener=` | Meter | Dispatches that ended in a `Throwable` from the listener. | +| `security.audit.events.dropped;domain=` | Meter | Events discarded because the originating session hit the per-session buffer cap. | + +The `domain=` and `listener=` suffixes follow Oak's `StatsProviderUtil` +label convention, which Prometheus and similar systems split back into a +metric name plus labels. + +The listener timer is worth an alert. Listeners run synchronously on the +commit thread, so time spent in `onEvents` is added directly to commit +latency for the writing session. A listener that starts doing I/O inline +shows up here before it shows up as a user complaint. + +The dropped-events meter is the one that matters for a compliance trail: a +non-zero value means a persisted write left no audit event behind. The same +condition logs a WARN, but the meter is what you can alert on. + + +### User-API-level audit, not a transaction log + +Oak's audit SPI captures activity at the level of user API calls, not at the +level of the transaction log. The distinction matters when choosing whether +the audit SPI fits a given use case. + +- **What fires audit events:** capture sites in Oak's user-management + implementation. Group membership changes record member add/remove events. + The usual route is a `Group.addMember(...)` / `.removeMember(...)` call from + the Jackrabbit user-management API, but the same capture site also covers + membership applied by the protected-item importer during XML import, which + reaches it without any user-facing API call. Equivalent capture sites can + cover other security-relevant areas. +- **What does NOT fire audit events:** changes made by commit hooks, editors, + or validators during commit processing. If a hook transforms the tree in + flight (autocreated properties, denormalised indexes, side-effect writes + from a `Validator` or `Editor`), those tree changes are not recorded even + though they end up in the merged `NodeState`. + +This is intentional. The audit SPI answers "who called the API", which is the +right level for security audit, compliance trails, and "who removed user X +from group Y" investigations. It does not enumerate every node mutation that +landed in the merged commit. + +Consumers needing every node mutation (event sourcing, change-data capture, +derived index rebuilding) should use Oak's `NodeStore.addObserver(...)` / +`BackgroundObserver` mechanism instead. Those observers see the post-merge +`NodeState` diff and capture mutations regardless of which API surface or +commit hook produced them. The audit SPI and a `NodeStore` observer answer +different questions; deploy the one that matches your use case. + + +### Clustering + +Audit events are node-local. A write on one cluster node produces events on +that node only, dispatched to the listeners registered there. The drain +observer ignores commits for which `CommitInfo.isExternal()` is `true`, and +cluster sync from a peer node is exactly that, so the same write does not +produce a second event when it reaches the other nodes. + +For a listener deployed on every node this gives the property you want: each +audited write is delivered once, on the node that performed it. Aggregating +into a single SIEM or compliance archive therefore needs the listener to tag +events with the node they came from, since the SPI adds no node-identity +payload key. `DocumentNodeStore.getClusterId()` is the per-node identifier; +note that `ClusterRepositoryInfo.getId(...)` is not, since it returns one id +shared by the whole cluster. + +Two consequences to plan for. A listener deployed on only some nodes sees +only the writes performed on those nodes, which for a compliance trail is a +silent gap rather than an error. And a node going down loses whatever its +listeners had buffered in their own async queues, if they use one; the audit +SPI dispatches synchronously and holds no cross-node state, so durability +past the dispatch call belongs to the listener. + +Everything above applies to the document store, where clustering is +supported. A segment-store cold-standby instance produces no commit-attached +events at all, for the reason given under +[Commit flow](audit-design.html#Commit_flow). + + +### Emitting events from a bundle + +Bundles emit events through the [AuditEventEmitter] OSGi service. A single +implementation is registered by `oak-core`. + +```java +@Component +public class ContentPublishAuditor { + + private static final AuditDomain DOMAIN = AuditDomain.of("example.content"); + + @Reference + private AuditEventEmitter audit; + + public void onPublished(String path, String variant) { + if (audit.isEnabledFor(DOMAIN)) { + audit.emit(new ContentPublishedEvent(path, variant)); + } + } +} +``` + +The `isEnabledFor` gate short-circuits when no listener is registered for the +domain, so callers can skip event construction on hot paths. The check is +cheap; producers SHOULD use it. + +A minimal event implementation: + +```java +class ContentPublishedEvent implements AuditEvent { + + private static final AuditDomain DOMAIN = AuditDomain.of("example.content"); + private static final AuditType TYPE = AuditType.of("content.published"); + + private final String path; + private final String variant; + private final long timestamp = System.currentTimeMillis(); + + ContentPublishedEvent(String path, String variant) { + this.path = path; + this.variant = variant; + } + + @Override public AuditDomain getDomain() { return DOMAIN; } + @Override public AuditType getType() { return TYPE; } + @Override public long getTimestamp() { return timestamp; } + @Override public Map getPayload() { + return Map.of("path", path, "variant", variant); + } +} +``` + +Events emitted this way are not tied to a JCR session or commit. The caller +need not hold a `Session` or `Root`, so lifecycle events such as workflow +transitions, replication outcomes, or background-job completion are valid +producers. + + +### Implementing a listener + +A listener is an OSGi component registered as a service of type +[AuditEventListener]. The Whiteboard registry discovers it automatically. + +```java +@Component(service = AuditEventListener.class) +public class SiemForwarder implements AuditEventListener { + + @Override + public AuditDomain getDomain() { + return SecurityAuditDomain.DOMAIN; + } + + @Override + public int getRank() { + return 0; + } + + @Override + public void onEvents(List events) { + for (AuditEvent e : events) { + if (!AuditEvent.isCommitAttested(e)) { + continue; // caller-asserted, not an Oak-attested write + } + Map p = e.getPayload(); + String sessionId = (String) p.get(AuditEvent.COMMIT_SESSION_ID); + String userId = (String) p.get(AuditEvent.COMMIT_USER_ID); + siem.forward(e, sessionId, userId); + } + } +} +``` + +Contract notes: + +- **`getDomain()`** is queried on every dispatch and MUST return a stable, + non-null value across the listener's lifetime. A listener subscribes to + exactly one domain. To consume multiple domains, register multiple listener + components. +- **`getRank()`** orders listeners within a domain, higher rank first, default + 0. Useful when one listener must observe state set by another (for example, + a redaction listener running before a SIEM forwarder). +- Both accessors are treated as listener code. A listener whose `getDomain()` + or `getRank()` throws is skipped for that dispatch and picked up again once + the accessor stops throwing; the failure is logged at WARN the first time + for that listener instance and at DEBUG afterwards, so a broken listener + cannot flood the log. Other listeners are unaffected. +- **`onEvents(List)`** is invoked with a non-empty, non-null list + of events in capture order. The same method serves both pipelines: + commit-attached events arrive in a batch sized by the originating session's + buffer; fire-and-forget events arrive in singleton lists. +- Implementations MUST be non-blocking. Expensive I/O belongs in an async + wrapper owned by the listener. +- Implementations MUST tolerate unknown payload keys and missing optional + keys. The payload schema is open. + + +### Trust model + +The fire-and-forget producer surface is open by design. + +- Any bundle that resolves `AuditEventEmitter` can emit any event for any + domain, including `"oak.security"`. There is no compile-time check, no + reserved domain registry, and no runtime gate on the emitting bundle. +- Listeners therefore receive caller-asserted data. An event arriving through + `onEvents` reflects the emitting bundle's claim, not Oak-verified truth. +- Oak does not verify, sign, or annotate events with their originating + bundle. Consumers that require Oak attestation MUST distinguish events at + the consumer side. + +The distinguishing signal is payload-based and enforced at dispatch: events +produced by the commit-attached pipeline carry the `oak.commit.sessionId`, +`oak.commit.userId`, and `oak.commit.timestamp` keys, unconditionally +overwritten from the commit's `CommitInfo`. Fire-and-forget events cannot +carry them, because Oak strips caller-supplied values for exactly these +three keys before delivery. `AuditEvent.isCommitAttested(event)` does the +check, so listeners need neither the key names nor the rule. A SIEM +forwarder that treats only attested events as Oak-verified mutations is +operating within the contract. The Javadoc on `AuditEvent#getPayload()` is +the normative statement, including the boundaries of the attestation: it +applies to Oak dispatch only and does not survive re-emission. + +The open surface is a deliberate trade-off. A reserved-domain registry or +typed event subclasses would put Oak in the middle of every producer bundle's +policy decision; the open surface lets any higher-stack bundle emit on its own +schedule and shifts allowlisting to the consumer side, where the deployment +owner already controls listener registration. + +Recommended consumer-side discipline: + +| Need | Approach | +|---|---| +| Distinguish Oak-attested mutations from caller-asserted events. | Call `AuditEvent.isCommitAttested(event)`. It anchors on the three reserved keys, not on the `oak.commit.` prefix in general. | +| Restrict trusted producers. | Maintain a consumer-side allowlist of trusted domain prefixes and reject unknown domains. | +| Compliance audit (Oak-verified writes only). | Subscribe to `"oak.security"` and keep only events for which `AuditEvent.isCommitAttested(event)` is `true`. | + + +### Further Reading + +- [Audit Pipeline Design](audit-design.html): the design document covering + the SPI shape, pipeline internals, OSGi and embedded wiring, threading + invariants, and performance characteristics. +- [OAK-12331](https://issues.apache.org/jira/browse/OAK-12331): the issue + that introduced the audit SPI. + + +[AuditEvent]: /oak/docs/apidocs/org/apache/jackrabbit/oak/spi/audit/AuditEvent.html +[AuditEventListener]: /oak/docs/apidocs/org/apache/jackrabbit/oak/spi/audit/AuditEventListener.html +[AuditEventEmitter]: /oak/docs/apidocs/org/apache/jackrabbit/oak/spi/audit/AuditEventEmitter.html +[AuditDispatch]: /oak/docs/apidocs/org/apache/jackrabbit/oak/spi/audit/AuditDispatch.html +[AuditConfiguration]: /oak/docs/apidocs/org/apache/jackrabbit/oak/spi/audit/AuditConfiguration.html diff --git a/oak-doc/src/site/markdown/security/overview.md b/oak-doc/src/site/markdown/security/overview.md index e1fd787a8dd..9eb2de246a2 100644 --- a/oak-doc/src/site/markdown/security/overview.md +++ b/oak-doc/src/site/markdown/security/overview.md @@ -22,6 +22,7 @@ The Oak Security Layer * [Introduction to Oak Security](introduction.html) * [Security Reports](reports.html) + * [Audit SPI](audit.html) ### Authentication diff --git a/oak-doc/src/site/site.xml b/oak-doc/src/site/site.xml index 00d2845c724..a37e8ee7ec6 100644 --- a/oak-doc/src/site/site.xml +++ b/oak-doc/src/site/site.xml @@ -78,6 +78,9 @@ under the License. + + +