Skip to content

fix(macos): align native log fields with security rule consumers #2622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rvald26 wants to merge 2 commits into from
Closed

fix(macos): align native log fields with security rule consumers #2622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
01589d2
fix: align macOS filter and rule contracts
rvald26 Sep 17, 2026
ff651dd
fix(macos): align native log fields and correlation consumers
rvald26 Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 139 additions & 0 deletions filters/audits/macos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# macOS normalization and correlation review

This draft targets UTMStack `v11` and changes both macOS filter configurations
and all 36 macOS rules. The pinned `go-sdk v1.1.31` Event/Side protobuf is the
schema authority. The local dictionary is supporting material.

## Producer and consumer corrections

The 30 sampled native raw records contain JSON with `message`, `process`,
`class_name`, `level`, `timestamp`, and identifiers whose names contain underscores.
The Go agent source supplies the dataType/dataSource envelope and forwards lines
from an external collector; the collector implementation was not available in
the inspected source tree. Its JSON shape is established by observed records,
while Apple documentation establishes class and level semantics. SDK JSON field
sanitization removes underscores and preserves letter case. The existing filter
already produces `log.message`; 35 rules instead read `log.eventMessage`. Those
rules now read the emitted message. The filter copies the process name to
`origin.process`, the raw timestamp to `deviceTime`, and meaningful agent
`dataSource` to `origin.host`. Every native record gets `dataSource` from the
agent hostname; the native producer uses `unknown` if hostname lookup fails.

Legacy `eventMessage` and `messageType` remain available while supplying
`log.message` and `log.level` when the native fields are absent. Native values
win if both forms exist. `class_name` becomes the compatible `log.className`
alias; the original sanitized `log.classname` remains available. The four
existing activity/process/store/thread identifier aliases remain unchanged.
There is no PID field in the SDK Side message, so the numeric process identifier
stays in `log.processIdentifier`. No non-schema `origin.pid` is introduced.

Apple documents `OSLogEntryLog` as an entry generated by the logging API. Three
class-gated rules accept this native class alongside their legacy `eventType`
checks; the filter does not invent `eventType`. Keychain and Endpoint Security
consume `log.level`. Apple debug, info/notice, error, and fault map respectively
to standard severity debug, info, error, and critical; undefined remains unmapped.
Vendor levels remain intact. A log level does not establish operation success,
so the filter does not synthesize `actionResult`.

A native process string identifies the process name, not an executable path.
Only an explicitly supplied legacy `processImagePath` is copied to `origin.path`,
with its vendor alias preserved. Neither process nor sender is fabricated into
a path. All 36 rules retain `adversary: origin`; the host/process promotions feed
those existing actor roles. Host grouping now has an actual source identity;
process grouping uses `adversary.process`. Optional user grouping remains empty
when the producer has no user. Existing message grouping uses `lastEvent.log.message`.

## Wrappers and history

The optional forwarding prefix is anchored at the start of raw input. It first
captures `log.datasourceoverride`, then promotes only a nonempty, whitespace-free
value other than `unknown` to `dataSource`. Invalid overrides stay in their
vendor field without replacing ingress identity. JSON `datasource_override`
uses the same sanitized spelling and validation. The native filter handles the
wrapper itself and both filter execution orders are tested. Wrapper and legacy
message copies preserve newlines. The compatibility filter skips extraction
when a message is already available, avoiding duplicate capture. The original
noise-drop policy runs after JSON extraction and legacy level normalization,
before the remaining normalization work.

Endpoint Security and ransomware history searches use the meaningful per-agent
`dataSource` and an exact candidate marker computed by the filter. Without that
marker, any earlier macOS event from the same source could satisfy a security
threshold. Marker conditions repeat the complete rule trigger; tests check their
parity. Untrusted incoming markers are removed and recomputed. The rules retain
their existing counts/windows and reject empty/unknown identities. Filter and
rules must deploy together; old indexed events do not have candidate markers,
so each history window must warm up after deployment.

The kernel-extension rule no longer treats a missing or non-system executable
path as evidence that the loaded extension is untrusted. Its first branch
requires explicit unsigned or invalid-signature message evidence. Process
exclusions in kernel-extension, XProtect and TCC checks require a present process
name. The other negative predicates were checked for positive same-field gates.

## Validation and bounded evidence

- Standalone `macos_contract_test.go` tests 99 synthetic raw fixtures in three
configurations: native filter alone, compatibility filter first, and
compatibility filter last. This includes a native positive and benign negative
for each of the 36 rules, wrapper edge cases, multiline/legacy messages,
severity/identity mappings, noise drops and absent-field regressions.
- All 36 real SDK CEL predicates are compiled/evaluated. Forty-two positive
fixtures assert intended matches, source grouping identities, and available
history placeholders. The two candidate-marker predicates must agree with
their consumers. Negative fixtures assert that none of the 36 rules matches.
- Real SDK historical requests execute against a local mock OpenSearch service.
Both history rules are checked below/at count thresholds, against another
source, benign non-candidate history, and expired events. The mock test runs
in a subprocess because the SDK OpenSearch client is a process-wide singleton.
- The shared contract manifest covers 90 raw JSON cases with opt-in decoding and recursive
history-placeholder preflight supplied by draft #2590. The standalone macOS
tests also cover wrappers, which the shared harness deliberately does not model.
- Thirty distinct raw/normalized records from three instances were inspected
read-only. All use native JSON; all lack `log.eventMessage`, `origin.host` and
`origin.process`; all carry a raw timestamp differing from stored `deviceTime`.
The relevant deployed repository filters match the baseline mappings. Private
evidence retains instance/document anchors, raw records and configuration
hashes without publishing customer payloads.
- Replaying those 30 raw records through the offline parser model and actual SDK
CEL yields zero candidates. No wrapper override was observed in bounded
30-day searches on those instances. Wrapper defects are supported by source,
deployed filter configuration and synthetic tests, not observed wrapper events.

Filter extraction and transformations are an offline model using SDK field
sanitization and documented steps. Configuration decoding, CEL, Event conversion,
placeholder expansion, query construction and count decisions use the actual
SDK. This does not execute the closed EventProcessor or validate live alert
creation/delivery. No customer writes or production rollout were performed.

## Activation and remaining validation

Correcting 35 message consumers can materially increase alert candidates.
Populating host grouping can also consolidate events that previously had no
usable grouping identity. The bounded sample had no candidates; it does not
measure false-positive rates or establish detection coverage. Before rollout,
staging must compare actual raw/output/alert records and per-rule volumes,
identity grouping, time/count behavior, and expected benign traffic.

Native samples do not include a process image path. Legacy branches comparing
that executable path to a TCC database or Gatekeeper resource path remain
semantically uncertain and unexercised by the native positive fixtures. No
resource path was invented as a process image to make those branches pass;
message-based branches supply the tested matches. Further producer evidence is
required before changing those resource-specific heuristics.

The shared alert-grouping correction in draft #2590 is required for runtime
resolution of the remaining `lastEvent.*` grouping keys. Review and deploy the
filter, its consumers and that grouping support together. This draft does not
claim that existing heuristic matches prove malware or compromise.

## References

- [SDK schema v1.1.31](https://github.com/threatwinds/go-sdk/blob/v1.1.31/plugins/plugins.proto)
- [SDK sanitization](https://github.com/threatwinds/go-sdk/blob/v1.1.31/utils/fields.go)
- [Native macOS producer](https://github.com/utmstack/UTMStack/blob/v11/agent/collector/platform/darwin.go)
- [Filter steps](https://github.com/threatwinds/go-sdk/wiki/Filter-Steps-Reference)
- [Standard event schema](https://github.com/threatwinds/go-sdk/wiki/Standard-Event-Schema)
- [Correlation rules](https://github.com/threatwinds/go-sdk/wiki/Implementing-Rules)
- [Apple OSLogEntryLog](https://developer.apple.com/documentation/oslog/oslogentrylog)
- [Apple log entry levels](https://developer.apple.com/documentation/oslog/oslogentrylog/level-swift.enum)
35 changes: 21 additions & 14 deletions filters/macos/macos-syslog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,29 @@
# Compatibility wrapper extraction; macos.yml also handles this format.
pipeline:
- dataTypes:
- macos
- dataTypes: [macos]
steps:
# 1. Handle Agent Wrapper (Optional)
# Optional forwarding wrapper. Anchoring prevents payload text from
# overriding the source, and the message is captured exactly once.
- grok:
source: raw
patterns:
- fieldName: log.wrapper_open
pattern: '\[utm_stack_agent_ds='
- fieldName: log.datasource_override
pattern: '{{.data}}'
- fieldName: log.wrapper_close
- fieldName: ''
pattern: '^\[utm_stack_agent_ds='
- fieldName: log.datasourceoverride
pattern: '[^\]\r\n]*'
- fieldName: ''
pattern: '\]-'
- fieldName: log.wrapperMessage
pattern: '(?s:.*)'
where: startsWith("raw", "[utm_stack_agent_ds=") && !exists("log.message")
- json:
source: log.wrapperMessage
where: regexMatch("log.wrapperMessage", "^\\s*\\{") && !exists("log.message")
- grok:
source: log.wrapperMessage
patterns:
- fieldName: log.message
pattern: '{{.greedy}}'
- fieldName: log.message
pattern: '{{.greedy}}'

# 2. Cleanup
pattern: '(?s:.*)'
where: exists("log.wrapperMessage") && !regexMatch("log.wrapperMessage", "^\\s*\\{") && !exists("log.message")
- delete:
fields: [log.wrapper_open, log.wrapper_close]
fields: [log.wrapperMessage]
163 changes: 145 additions & 18 deletions filters/macos/macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,159 @@
# Macos filter, version 3.0.1
# macOS native JSON and forwarding-wrapper filter, version 3.1.0
pipeline:
- dataTypes:
- macos
- dataTypes: [macos]
steps:
# Optional forwarding wrapper. Anchoring prevents payload text from
# overriding the source, and the message is captured exactly once.
- grok:
source: raw
patterns:
- fieldName: ''
pattern: '^\[utm_stack_agent_ds='
- fieldName: log.datasourceoverride
pattern: '[^\]\r\n]*'
- fieldName: ''
pattern: '\]-'
- fieldName: log.wrapperMessage
pattern: '(?s:.*)'
where: startsWith("raw", "[utm_stack_agent_ds=") && !exists("log.message")
- json:
source: raw
where: regexMatch("raw", "^\\s*\\{")
- json:
source: log.wrapperMessage
where: regexMatch("log.wrapperMessage", "^\\s*\\{")
- grok:
source: log.wrapperMessage
patterns:
- fieldName: log.message
pattern: '(?s:.*)'
where: exists("log.wrapperMessage") && !regexMatch("log.wrapperMessage", "^\\s*\\{")
# Normalize the legacy level before applying the existing noise policy.
- grok:
source: log.messageType
patterns:
- fieldName: log.level
pattern: '(?s:.*)'
where: '!exists("log.level")'
# All fields used by the drop predicate are now available.
- drop:
where: equals("log.level", "notice") && contains("log.subsystem", "com.apple.cloudkit") || contains("log.subsystem", "com.apple.CoreDuet") || oneOf("log.subsystem", ["com.apple.apsd", "com.apple.bluetooth", "com.apple.SkyLight", "com.apple.mDNSResponder", "com.apple.homed", "com.apple.identityservicesd", "com.apple.powerlogd", "com.apple.analyticsd", "com.apple.UIKit", "com.apple.runningboard", "com.apple.WiFiManager", "com.apple.xpc", "com.apple.cache_delete", "com.apple.spotlightindex"])

# Renaming useful fields
- rename:
from:
- log.activityidentifier
from: [log.activityidentifier]
to: log.activityIdentifier

- rename:
from:
- log.processidentifier
from: [log.processidentifier]
to: log.processIdentifier

- rename:
from:
- log.storecategory
from: [log.storecategory]
to: log.storeCategory

- rename:
from:
- log.threadidentifier
from: [log.threadidentifier]
to: log.threadIdentifier
- grok:
source: log.classname
patterns:
- fieldName: log.className
pattern: '(?s:.*)'
where: '!exists("log.className")'
- grok:
source: log.eventMessage
patterns:
- fieldName: log.message
pattern: '(?s:.*)'
where: '!exists("log.message")'

# Drop unnecessary events
- drop:
where: equals("log.level", "notice") && contains("log.subsystem", "com.apple.cloudkit") || contains("log.subsystem", "com.apple.CoreDuet") || oneOf("log.subsystem", ["com.apple.apsd", "com.apple.bluetooth", "com.apple.SkyLight", "com.apple.mDNSResponder", "com.apple.homed", "com.apple.identityservicesd", "com.apple.powerlogd", "com.apple.analyticsd", "com.apple.UIKit", "com.apple.runningboard", "com.apple.WiFiManager", "com.apple.xpc", "com.apple.cache_delete", "com.apple.spotlightindex"])
# SDK JSON key sanitization removes the underscore in datasource_override.
- rename:
from: [log.datasourceoverride]
to: dataSource
where: regexMatch("log.datasourceoverride", "^[^\\s]+$") && !equals("log.datasourceoverride", "unknown")
- grok:
source: dataSource
patterns:
- fieldName: origin.host
pattern: '(?s:.*)'
where: '!oneOf("dataSource", ["", "unknown"])'
- grok:
source: log.process
patterns:
- fieldName: origin.process
pattern: '(?s:.*)'
where: exists("log.process")
- grok:
source: log.processImagePath
patterns:
- fieldName: origin.path
pattern: '(?s:.*)'
where: exists("log.processImagePath")
- grok:
source: log.timestamp
patterns:
- fieldName: deviceTime
pattern: '(?s:.*)'
where: regexMatch("log.timestamp", "^\\d{4}-\\d{2}-\\d{2}T")
- add:
function: string
params:
key: severity
value: debug
where: equals("log.level", "debug")
- add:
function: string
params:
key: severity
value: info
where: equals("log.level", "info")
- add:
function: string
params:
key: severity
value: info
where: equals("log.level", "notice")
- add:
function: string
params:
key: severity
value: error
where: equals("log.level", "error")
- add:
function: string
params:
key: severity
value: critical
where: equals("log.level", "fault")

# Only matching security candidates may satisfy historical thresholds.
# The raw contract tests check parity against the two rule predicates.
- delete:
fields: [log.correlationCandidate, log.wrapperMessage]
- add:
function: string
params:
key: log.correlationCandidate.endpointSecurity
value: match
where: |
!oneOf("dataSource", ["", "unknown"]) &&
(equals("log.subsystem", "com.apple.endpointsecurity") ||
contains("origin.process", "EndpointSecurity") ||
contains("log.message", "ES_EVENT_TYPE")) &&
(contains("log.message", "bypass") ||
contains("log.message", "failed authorization") ||
contains("log.message", "client disconnected") ||
contains("log.message", "authorization denied") ||
contains("log.message", "invalid client") ||
contains("log.message", "ES_RETURN_ERROR") ||
equals("log.level", "error") ||
contains("log.message", "memory leak detected"))
- add:
function: string
params:
key: log.correlationCandidate.ransomware
value: match
where: |
!oneOf("dataSource", ["", "unknown"]) &&
( (contains("log.message", ["encrypted", "encrypt", ".locked", ".crypto", ".crypt"]) ||
regexMatch("log.message", "(?i)\\.(locky|cerber|wannacry|ryuk|maze|conti|lockbit)$")) &&
(contains("log.message", "rename") || contains("log.message", "modified") ||
contains("log.message", "write")))
Loading
Loading