diff --git a/filters/audits/fortigate.md b/filters/audits/fortigate.md new file mode 100644 index 000000000..74880ad49 --- /dev/null +++ b/filters/audits/fortigate.md @@ -0,0 +1,173 @@ +# FortiGate filter and correlation review + +This draft updates one FortiGate filter and six of its seven rules on UTMStack +`v11`. The sandbox rule is reviewed and covered by regression fixtures without a +rule edit. No customer configuration was changed, no live alert was generated, +and no reduction in customer false positives has been measured. + +## Source and evidence + +The data model authority is the [go-sdk protobuf at v1.1.31](https://github.com/threatwinds/go-sdk/blob/v1.1.31/plugins/plugins.proto), +the version pinned by `plugins/alerts/go.mod` in the reviewed v11 source. SDK main +at `92f4588df51441ae7f273e0e1eac457c9b20f0b7` has the same protobuf for this +review. The filter and rule wiki was read at commit +`c18b54bd5ea5a34abb0e690458d73f89835edd29`. The v11 comparison point is +`660c796f168670fd6ccb079995c09c46d4f0068d`; the prior source draft is +`31b82b551f816c229f983c72ba5164bea0991f14`. + +Read-only sampling collected 63 distinct native key/value records from three +instances, using recent records and bounded representatives of action and event +classes in the preceding seven days. All three deployed filter copies have +SHA-256 `820b44dc36135080131c537fb44736dec2c22915f69cc4d551cadf78ab651328`. +Raw records, document IDs, instance provenance and deployed copies stay in the +private evidence pack; only aggregate observations and fabricated fixtures are +published here. The sample is stratified, not a prevalence estimate. + +All 63 records have ingress `dataSource`, appliance `log.devid` and VDOM `log.vd`. +None has an indexed `actionResult`. Existing populated IPs, ports, source MAC, +byte counters and packet counters have the correct physical source/destination +orientation in this sample. Their absence in other event classes is not, by +itself, a defect. No malformed IP, alias-form input, CEF record, DLP event or +malicious sandbox result was observed in this bounded sample. + +## Filter corrections + +- Recover complete consumed key/value strings, including a quoted final field. + The deployed output loses or truncates relevant `msg`, `logdesc`, user and + multiword OS values. The updated recovery preserves all 40 sampled messages + and 13 descriptions exactly, apart from surrounding quotes. +- Restrict syslog priority and CEF header recognition to their actual envelopes. + Quoted message text containing HTML or a CEF-looking string cannot become a + header. Consumed keys are reconstructed only at quote-aware field boundaries, + so text such as a fake `srcip=` inside a message cannot supply an identity. + Unterminated quoted values cannot fall back to an unquoted identity. +- Validate an IP at its original vendor-field rename before geolocation. Native + aliases keep precedence; an invalid preferred value remains under its vendor + key instead of silently falling back to a second alias. IPv4, IPv6 and quoted + addresses are covered, including alternate unspecified representations. + Invalid values do not enter `origin.ip` or `target.ip`. +- Preserve physical traffic roles. Promote `remip` only for explicit SSL-VPN + authentication events, including the exact failed-login ID. All six sampled + SSL login failures had `remip` but no `origin.ip`; ordinary tunnel shutdown and + IPsec negotiation errors do not acquire authentication or adversary semantics. +- Copy supported endpoint fields into `origin.host`, `target.host`, + `origin.operatingSystem`, `target.operatingSystem`, `target.mac`, + `origin.user`, `origin.group` and `target.user` when present and meaningful. + Vendor originals remain available. In the modelled sample this adds 13 source + hosts, 16 source OS values, three destination OS values and eight destination + MACs. Appliance identity and ingress `dataSource` remain unchanged. +- Copy six absolute HTTP/FTP URLs to `target.url` and seven valid UTM destination + domain names to `target.domain`. Relative URLs remain in `log.url`; an embedded + `://` does not make a relative path absolute. SDK `path` denotes a filesystem + directory, so it is not used for a URL path. +- Derive outcome after quote cleanup. Policy `accept` maps to `success`; explicit + policy or UTM denial maps to `denied` and wins over an accept. Exact failed + connection/admin/SSL authentication classes map to `failure`. Bare `dns` or + `ip-conn` action text is insufficient evidence of failure. The observed + traffic `dns` records carry failed-connection log ID 11, which independently + justifies their failure result. Close/reset/timeout describe a closed + connection without manufacturing a successful action. +- Preserve existing numeric protocol conversions: SDK v1.1.31 supports numeric + comparisons against the parsed numeric strings. Map vendor logging levels to + standard severity while retaining separate IPS attack severity under `log`. + +The protobuf defines `actionResult` as a string, not an enforced enum. This patch +uses the wiki's `success`/`failure`/`denied` convention consistently with its +consumers; it does not claim that alternative strings are rejected by the SDK. +A permitted policy action is not proof of a completed TCP handshake or successful +application operation. + +## Consumers and correlation + +| Rule | Corrected contract | +|---|---| +| Admin account compromise | Recover full success description and `origin.user`; failed-login history requires the same source IP, user, ingress, appliance and VDOM, with the exact failed-admin event class. Group user identity on the adversary side. | +| Admin session anomaly | Preserve the existing heuristic but exclude IPv6 loopback, link-local and unique-local addresses from its external-source branch. Public IPv6 remains eligible. This edge is a synthetic predicate finding, not observed live incidence. | +| SSL-VPN brute force | Count explicit SSL authentication failures from one remote source and device/VDOM/ingress. Routine tunnel-down and generic IPsec negotiation messages are not authentication failures. | +| Critical IPS activity | Consume actual sibling type/subtype/severity fields and a denied outcome; history counts matching high/critical denied IPS candidates rather than unrelated traffic from that source. | +| DLP exfiltration | Require UTM/DLP context, consume the produced action/profile fields, and restrict history to matching DLP candidates and source/device/VDOM/ingress. Generic administrative messages mentioning DLP do not qualify. | +| Antivirus outbreak | Require the infected-file event class and denied outcome; count repeated blocked infected files from one physical source and appliance/VDOM. Analytics submissions, file-policy blocks and scan errors are not malware proof. The rule does not establish endpoint infection or successful delivery. | +| Sandbox malicious verdict | Retain the rule; trim consumed verdict/risk fields and verify positive and clean synthetic cases. No live malicious sandbox example was available. | + +The five existing history counts and windows are preserved. Four candidate +markers under `log.correlationCandidate` have conditions exactly matching their +consuming trigger predicates. Input-supplied markers are cleared first. Every +history requires the identity needed to resolve its placeholders; missing +appliance/VDOM/actor data skips that correlation instead of pooling unknown +identities. Grouping includes appliance and VDOM so overlapping private IPs on +different devices do not collapse into one alert group. The antivirus history +uses physical appliance/VDOM/source identity without an additional ingress term. + +SDK history counts are at-least thresholds over a processing-time `@timestamp` +lower bound. They do not establish strict event-time ordering, unique sources or +distinct destinations. Existing records without the new markers or promoted +identities will not satisfy the updated histories; allow the retained 15-minute +and one-hour windows to warm up after staged rollout. + +## Validation and boundaries + +- `go test ./... -count=1 -v` passes in `plugins/alerts` with cached dependencies + and the optional private-evidence input enabled. The history test uses a local + loopback HTTP mock, not a customer OpenSearch cluster. +- 72 fabricated raw fixtures exercise extraction, alias precedence, invalid and + quoted IPs, quoted-key injection, malformed quotes, header ambiguity, new + standard mappings, outcomes and all seven rule predicates. Every fixture + asserts both matching and nonmatching rules, and candidate-marker parity. +- Actual SDK v1.1.31 CEL, protobuf conversion, placeholder expansion, generated + search requests, mapping resolution and threshold decisions are exercised. + All five history rules have count-minus-one/count, inside/expired-window, + wrong-identity, wrong-event-class and missing-placeholder checks. Benign raw + histories are reparsed to verify that they do not gain candidate markers. +- All 63 private raw records pass the same offline extraction model and actual + SDK predicates, preserving their existing populated standard network fields. + The model produces 35 outcomes, six additional SSL peer IPs, seven users and + four groups. Predicate matches are one admin-success candidate, six SSL login + failures and one critical denied IPS event; these are not generated alerts or + proof that live history thresholds were met. The prior predicates matched no + sampled records, which alone would not prove all seven rules defective. +- The shared contract manifest lists all seven consumers and contains nine + isolated normalization checks for outcomes and numeric protocol strings. It + requires the separate shared contract runner and does not test raw extraction. + The standalone raw and SDK-history tests in this draft run without that runner. + An overlay of the current shared runner passed the schema/CEL checks and all + nine normalization cases alongside this draft's standalone tests. + +The filter executor is closed. Raw parsing here is an explicitly limited offline +model of the YAML grok/rename/trim/cast/add/delete steps and observed KV splitting; +it is not an execution of the closed EventProcessor. Geolocation lookups, live +OpenSearch behaviour, alert creation and end-to-end delivery were not executed. +CEF compatibility is documented and synthetic, including a bounded vendor-shaped +example; arbitrary CEF escaping and unobserved vendor variants remain unverified. +The recovery adds gated scans for 129 consumed/mapped keys and aliases, not every +vendor field. Its CPU cost and actual parser behaviour require staging validation +at realistic message sizes and event rates. + +`lastEvent.*` grouping requires the separate alert-foundation correction that +resolves indexed event aliases against the event carried by the Alert. This +source draft does not duplicate that runtime fix; deploy the foundation before +depending on these grouping scopes. Verify resulting alerts and history windows +in staging before production approval. + +Outcome normalization changes queries that depend on prior literal values or +missing outcomes. Maintained source consumers were reviewed, but private saved +queries and customer dashboards were not enumerated. Review that compatibility +and the history warm-up during rollout; retained vendor fields support migration. + +## Deliberately unresolved mappings + +All sampled `eventtime` values are 19-digit Unix nanoseconds and agree with vendor +date/time/timezone; indexed `deviceTime` equals ingestion time. The available +filter wiki documents Go-layout time parsing, not a verified Unix-nanosecond +conversion. Preserve the original time under `log` until a supported conversion +can be proven; do not put an epoch-nanosecond string directly into `deviceTime`. +NAT pre/post-translation fields remain vendor fields because this SDK has no +dedicated equivalent. Vendor file-type labels are not MIME types, and the sample +does not establish a safe file-side role for incoming content, so those fields +are not promoted speculatively. + +## References + +- [Filter steps](https://github.com/threatwinds/go-sdk/wiki/Filter-Steps-Reference), [standard event schema](https://github.com/threatwinds/go-sdk/wiki/Standard-Event-Schema), [rule implementation](https://github.com/threatwinds/go-sdk/wiki/Implementing-Rules). +- [FortiOS log fields](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/357866/log-message-fields), [failed connection ID 11](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/11/11-log-id-traffic-fail-conn), [DNS query event](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/54000/54000-log-id-dns-query). +- [Admin login success](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/32001/32001-log-id-admin-login-succ), [admin login failure](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/32002/32002-log-id-admin-login-fail), [SSL login failure](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/39426/39426-log-id-event-ssl-vpn-user-ssl-login-fail). +- [Traffic CEF examples](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/949981/traffic-log-support-for-cef), [IPS CEF examples](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/311596/ips-log-support-for-cef), [DLP CEF examples](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/223332/dlp-log-support-for-cef), [antivirus CEF examples](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/807724/antivirus-log-support-for-cef), [infected-file event](https://docs.fortinet.com/document/fortigate/7.0.4/fortios-log-message-reference/8192/8192-mesgid-infect-warning). diff --git a/filters/fortinet/fortinet.yml b/filters/fortinet/fortinet.yml index 54ecdb9bf..8d6042fe3 100644 --- a/filters/fortinet/fortinet.yml +++ b/filters/fortinet/fortinet.yml @@ -1,4 +1,4 @@ -# Fortinet firewall module filter, version 3.1.0 +# Fortinet firewall module filter, version 3.2.0 # Based in docs and samples provided # # Documentations @@ -15,68 +15,1081 @@ pipeline: #Using grok to parse priority if present - grok: patterns: - - fieldName: log.priority - pattern: '\<{{.data}}\>' - - fieldName: log.kvMessage - pattern: '{{.greedy}}' + - {fieldName: log.priority, pattern: '^<[0-9]{1,3}>'} + - {fieldName: log.kvMessage, pattern: '{{.greedy}}'} + + # Vendor disk-format samples do not include a syslog priority prefix. + - grok: + source: raw + patterns: + - {fieldName: log.kvMessage, pattern: '(?s:.*)'} + where: '!exists("log.kvMessage")' + + # Extract the CEF extension before KV parsing; header text is not a KV key. + # This also accepts the space after "CEF:" used by Fortinet's examples. + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:[A-Za-z]{3}\s+[0-9]{1,2}\s+[0-9]{2}:[0-9]{2}:[0-9]{2}\s+\S+\s+)?CEF:\s*'} + - {fieldName: log.cefVersion, pattern: '[0-9]+'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.cefVendor, pattern: '[^|]*'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.cefProduct, pattern: '[^|]*'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.cefProductVersion, pattern: '[^|]*'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.cefSignatureId, pattern: '[^|]*'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.cefName, pattern: '[^|]*'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.cefSeverity, pattern: '[^|]*'} + - {fieldName: '', pattern: '\|'} + - {fieldName: log.kvMessage, pattern: '(?s:.*)'} - #Using the kv filter with default config, usefull in key-value logs - kv: source: log.kvMessage - fieldSplit: " " - valueSplit: "=" - - # Remove fields that have issues with kv filter - - delete: - fields: - - log.msg - - # CEF support: re-parse log.kvMessage to extract CEF header. - # Fails silently (all-or-nothing) on non-CEF events. - - grok: - source: log.kvMessage - patterns: - - fieldName: log.syslogTimestamp - pattern: '{{.syslogDate}}' - - fieldName: log.syslogHost - pattern: '{{.notSpace}}' - - fieldName: "" - pattern: 'CEF:' - - fieldName: log.cefVersion - pattern: '{{.integer}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.cefVendor - pattern: '{{.data}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.cefProduct - pattern: '{{.data}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.cefProductVersion - pattern: '{{.data}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.cefSignatureId - pattern: '{{.data}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.cefName - pattern: '{{.data}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.cefSeverity - pattern: '{{.integer}}' - - fieldName: "" - pattern: '\|' - - fieldName: log.kvMessage - pattern: '{{.greedy}}' - - # Recover deviceExternalId mangled by kv (CEF header token). - - rename: - from: [log.serverrst3deviceExternalId, log.clientrst3deviceExternalId] - to: log.deviceExternalId + fieldSplit: ' ' + valueSplit: '=' + + # Rebuild consumed KV fields so tokens inside quoted messages cannot + # manufacture an address, identity, action, or rule predicate field. + - delete: + fields: + - log.FTNTFGTapp + - log.FTNTFGTappcat + - log.FTNTFGTdevtype + - log.FTNTFGTdstcity + - log.FTNTFGTdstcountry + - log.FTNTFGTdstinetsvc + - log.FTNTFGTdstintfrole + - log.FTNTFGTdstmac + - log.FTNTFGTdstname + - log.FTNTFGTdstosname + - log.FTNTFGTdstregion + - log.FTNTFGTdstreputation + - log.FTNTFGTduration + - log.FTNTFGTeventtime + - log.FTNTFGTeventtype + - log.FTNTFGTfilehash + - log.FTNTFGTfilename + - log.FTNTFGTfilesize + - log.FTNTFGTgroup + - log.FTNTFGTlevel + - log.FTNTFGTlocip + - log.FTNTFGTlocport + - log.FTNTFGTlogdesc + - log.FTNTFGTlogid + - log.FTNTFGTmastersrcmac + - log.FTNTFGTosname + - log.FTNTFGTpolicyid + - log.FTNTFGTpolicyname + - log.FTNTFGTpolicytype + - log.FTNTFGTpoluuid + - log.FTNTFGTprofile + - log.FTNTFGTrcvdpkt + - log.FTNTFGTremip + - log.FTNTFGTremport + - log.FTNTFGTrisk + - log.FTNTFGTsentpkt + - log.FTNTFGTseverity + - log.FTNTFGTsrccountry + - log.FTNTFGTsrchwvendor + - log.FTNTFGTsrcintfrole + - log.FTNTFGTsrcmac + - log.FTNTFGTsrcname + - log.FTNTFGTsrcserver + - log.FTNTFGTstatus + - log.FTNTFGTsubtype + - log.FTNTFGTtrandisp + - log.FTNTFGTtz + - log.FTNTFGTuser + - log.FTNTFGTutmaction + - log.FTNTFGTvd + - log.FTNTFGTverdict + - log.FTNTFGTvwlid + - log.act + - log.action + - log.agent + - log.app + - log.cat + - log.catdesc + - log.dest_ip + - log.dest_port + - log.deviceExternalId + - log.deviceInboundInterface + - log.deviceOutboundInterface + - log.devid + - log.devtype + - log.dlpaction + - log.dlpprofile + - log.dpt + - log.dst + - log.dstcountry + - log.dstip + - log.dstmac + - log.dstname + - log.dstosname + - log.dstport + - log.dstuser + - log.duser + - log.eventtype + - log.externalId + - log.filehash + - log.filename + - log.group + - log.hostname + - log.in + - log.level + - log.locip + - log.logdesc + - log.logid + - log.mastersrcmac + - log.msg + - log.osname + - log.out + - log.policyname + - log.profile + - log.proto + - log.rcvdbyte + - log.rcvdpkt + - log.reason + - log.remip + - log.risk + - log.sentbyte + - log.sentpkt + - log.severity + - log.sourceTranslatedAddress + - log.sourceTranslatedPort + - log.spt + - log.src + - log.src_ip + - log.src_port + - log.srccountry + - log.srchwvendor + - log.srchwversion + - log.srcip + - log.srcmac + - log.srcname + - log.srcport + - log.srcswversion + - log.status + - log.subtype + - log.type + - log.unauthusersource + - log.url + - log.user + - log.utmaction + - log.vd + - log.verdict + - log.virus + - log.virusid + - log.vwlquality + # The anchored prefix consumes quoted strings atomically. CEF unquoted + # values end at the next key or end of input. Skip absent keys cheaply. + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTapp='} + - {fieldName: log.FTNTFGTapp, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTapp=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTappcat='} + - {fieldName: log.FTNTFGTappcat, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTappcat=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdevtype='} + - {fieldName: log.FTNTFGTdevtype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdevtype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstcity='} + - {fieldName: log.FTNTFGTdstcity, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstcity=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstcountry='} + - {fieldName: log.FTNTFGTdstcountry, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstcountry=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstinetsvc='} + - {fieldName: log.FTNTFGTdstinetsvc, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstinetsvc=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstintfrole='} + - {fieldName: log.FTNTFGTdstintfrole, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstintfrole=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstmac='} + - {fieldName: log.FTNTFGTdstmac, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstmac=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstname='} + - {fieldName: log.FTNTFGTdstname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstosname='} + - {fieldName: log.FTNTFGTdstosname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstosname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstregion='} + - {fieldName: log.FTNTFGTdstregion, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstregion=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTdstreputation='} + - {fieldName: log.FTNTFGTdstreputation, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTdstreputation=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTduration='} + - {fieldName: log.FTNTFGTduration, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTduration=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTeventtime='} + - {fieldName: log.FTNTFGTeventtime, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTeventtime=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTeventtype='} + - {fieldName: log.FTNTFGTeventtype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTeventtype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTfilehash='} + - {fieldName: log.FTNTFGTfilehash, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTfilehash=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTfilename='} + - {fieldName: log.FTNTFGTfilename, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTfilename=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTfilesize='} + - {fieldName: log.FTNTFGTfilesize, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTfilesize=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTgroup='} + - {fieldName: log.FTNTFGTgroup, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTgroup=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTlevel='} + - {fieldName: log.FTNTFGTlevel, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTlevel=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTlocip='} + - {fieldName: log.FTNTFGTlocip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTlocip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTlocport='} + - {fieldName: log.FTNTFGTlocport, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTlocport=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTlogdesc='} + - {fieldName: log.FTNTFGTlogdesc, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTlogdesc=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTlogid='} + - {fieldName: log.FTNTFGTlogid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTlogid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTmastersrcmac='} + - {fieldName: log.FTNTFGTmastersrcmac, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTmastersrcmac=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTosname='} + - {fieldName: log.FTNTFGTosname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTosname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTpolicyid='} + - {fieldName: log.FTNTFGTpolicyid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTpolicyid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTpolicyname='} + - {fieldName: log.FTNTFGTpolicyname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTpolicyname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTpolicytype='} + - {fieldName: log.FTNTFGTpolicytype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTpolicytype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTpoluuid='} + - {fieldName: log.FTNTFGTpoluuid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTpoluuid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTprofile='} + - {fieldName: log.FTNTFGTprofile, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTprofile=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTrcvdpkt='} + - {fieldName: log.FTNTFGTrcvdpkt, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTrcvdpkt=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTremip='} + - {fieldName: log.FTNTFGTremip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTremip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTremport='} + - {fieldName: log.FTNTFGTremport, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTremport=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTrisk='} + - {fieldName: log.FTNTFGTrisk, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTrisk=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsentpkt='} + - {fieldName: log.FTNTFGTsentpkt, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsentpkt=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTseverity='} + - {fieldName: log.FTNTFGTseverity, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTseverity=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsrccountry='} + - {fieldName: log.FTNTFGTsrccountry, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsrccountry=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsrchwvendor='} + - {fieldName: log.FTNTFGTsrchwvendor, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsrchwvendor=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsrcintfrole='} + - {fieldName: log.FTNTFGTsrcintfrole, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsrcintfrole=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsrcmac='} + - {fieldName: log.FTNTFGTsrcmac, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsrcmac=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsrcname='} + - {fieldName: log.FTNTFGTsrcname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsrcname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsrcserver='} + - {fieldName: log.FTNTFGTsrcserver, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsrcserver=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTstatus='} + - {fieldName: log.FTNTFGTstatus, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTstatus=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTsubtype='} + - {fieldName: log.FTNTFGTsubtype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTsubtype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTtrandisp='} + - {fieldName: log.FTNTFGTtrandisp, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTtrandisp=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTtz='} + - {fieldName: log.FTNTFGTtz, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTtz=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTuser='} + - {fieldName: log.FTNTFGTuser, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTuser=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTutmaction='} + - {fieldName: log.FTNTFGTutmaction, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTutmaction=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTvd='} + - {fieldName: log.FTNTFGTvd, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTvd=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTverdict='} + - {fieldName: log.FTNTFGTverdict, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTverdict=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)FTNTFGTvwlid='} + - {fieldName: log.FTNTFGTvwlid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "FTNTFGTvwlid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)act='} + - {fieldName: log.act, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "act=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)action='} + - {fieldName: log.action, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "action=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)agent='} + - {fieldName: log.agent, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "agent=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)app='} + - {fieldName: log.app, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "app=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)cat='} + - {fieldName: log.cat, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "cat=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)catdesc='} + - {fieldName: log.catdesc, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "catdesc=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dest_ip='} + - {fieldName: log.dest_ip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dest_ip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dest_port='} + - {fieldName: log.dest_port, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dest_port=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)deviceExternalId='} + - {fieldName: log.deviceExternalId, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "deviceExternalId=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)deviceInboundInterface='} + - {fieldName: log.deviceInboundInterface, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "deviceInboundInterface=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)deviceOutboundInterface='} + - {fieldName: log.deviceOutboundInterface, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "deviceOutboundInterface=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)devid='} + - {fieldName: log.devid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "devid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)devtype='} + - {fieldName: log.devtype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "devtype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dlp_?action='} + - {fieldName: log.dlpaction, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dlpaction=") || contains("log.kvMessage", "dlp_action=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dlp_?profile='} + - {fieldName: log.dlpprofile, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dlpprofile=") || contains("log.kvMessage", "dlp_profile=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dpt='} + - {fieldName: log.dpt, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dpt=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dst='} + - {fieldName: log.dst, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dst=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstcountry='} + - {fieldName: log.dstcountry, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstcountry=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstip='} + - {fieldName: log.dstip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstmac='} + - {fieldName: log.dstmac, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstmac=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstname='} + - {fieldName: log.dstname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstosname='} + - {fieldName: log.dstosname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstosname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstport='} + - {fieldName: log.dstport, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstport=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)dstuser='} + - {fieldName: log.dstuser, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "dstuser=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)duser='} + - {fieldName: log.duser, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "duser=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)eventtype='} + - {fieldName: log.eventtype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "eventtype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)externalId='} + - {fieldName: log.externalId, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "externalId=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)filehash='} + - {fieldName: log.filehash, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "filehash=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)filename='} + - {fieldName: log.filename, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "filename=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)group='} + - {fieldName: log.group, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "group=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)hostname='} + - {fieldName: log.hostname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "hostname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)in='} + - {fieldName: log.in, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "in=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)level='} + - {fieldName: log.level, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "level=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)locip='} + - {fieldName: log.locip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "locip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)logdesc='} + - {fieldName: log.logdesc, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "logdesc=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)logid='} + - {fieldName: log.logid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "logid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)mastersrcmac='} + - {fieldName: log.mastersrcmac, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "mastersrcmac=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)msg='} + - {fieldName: log.msg, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "msg=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)osname='} + - {fieldName: log.osname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "osname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)out='} + - {fieldName: log.out, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "out=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)policyname='} + - {fieldName: log.policyname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "policyname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)profile='} + - {fieldName: log.profile, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "profile=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)proto='} + - {fieldName: log.proto, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "proto=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)rcvdbyte='} + - {fieldName: log.rcvdbyte, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "rcvdbyte=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)rcvdpkt='} + - {fieldName: log.rcvdpkt, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "rcvdpkt=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)reason='} + - {fieldName: log.reason, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "reason=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)remip='} + - {fieldName: log.remip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "remip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)risk='} + - {fieldName: log.risk, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "risk=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)sentbyte='} + - {fieldName: log.sentbyte, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "sentbyte=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)sentpkt='} + - {fieldName: log.sentpkt, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "sentpkt=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)severity='} + - {fieldName: log.severity, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "severity=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)sourceTranslatedAddress='} + - {fieldName: log.sourceTranslatedAddress, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "sourceTranslatedAddress=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)sourceTranslatedPort='} + - {fieldName: log.sourceTranslatedPort, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "sourceTranslatedPort=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)spt='} + - {fieldName: log.spt, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "spt=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)src='} + - {fieldName: log.src, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "src=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)src_ip='} + - {fieldName: log.src_ip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "src_ip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)src_port='} + - {fieldName: log.src_port, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "src_port=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srccountry='} + - {fieldName: log.srccountry, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srccountry=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srchwvendor='} + - {fieldName: log.srchwvendor, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srchwvendor=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srchwversion='} + - {fieldName: log.srchwversion, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srchwversion=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srcip='} + - {fieldName: log.srcip, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srcip=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srcmac='} + - {fieldName: log.srcmac, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srcmac=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srcname='} + - {fieldName: log.srcname, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srcname=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srcport='} + - {fieldName: log.srcport, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srcport=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)srcswversion='} + - {fieldName: log.srcswversion, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "srcswversion=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)status='} + - {fieldName: log.status, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "status=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)subtype='} + - {fieldName: log.subtype, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "subtype=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)type='} + - {fieldName: log.type, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "type=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)unauthusersource='} + - {fieldName: log.unauthusersource, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "unauthusersource=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)url='} + - {fieldName: log.url, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "url=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)user='} + - {fieldName: log.user, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "user=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)utmaction='} + - {fieldName: log.utmaction, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "utmaction=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)vd='} + - {fieldName: log.vd, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "vd=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)verdict='} + - {fieldName: log.verdict, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "verdict=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)virus='} + - {fieldName: log.virus, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "virus=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)virusid='} + - {fieldName: log.virusid, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "virusid=") + - grok: + source: log.kvMessage + patterns: + - {fieldName: '', pattern: '^(?:"(?:\\.|[^"\\])*"|[^"])*?(?:^|\s)vwlquality='} + - {fieldName: log.vwlquality, pattern: '(?:"(?:\\.|[^"\\])*"|[^"\s][^"]*?|)'} + - {fieldName: '', pattern: '(?:\s+[A-Za-z][A-Za-z0-9_]*=|$)'} + where: contains("log.kvMessage", "vwlquality=") # Normalize FTNTFGT* fields to canonical FortiGate names. - rename: @@ -170,18 +1183,92 @@ pipeline: from: [log.FTNTFGTsrcserver] to: log.srcserver + - rename: + from: [log.FTNTFGTeventtype] + to: log.eventtype + - rename: + from: [log.FTNTFGTseverity] + to: log.severity + - rename: + from: [log.FTNTFGTutmaction] + to: log.utmaction + - rename: + from: [log.FTNTFGTlogdesc] + to: log.logdesc + - rename: + from: [log.FTNTFGTstatus] + to: log.status + - rename: + from: [log.FTNTFGTremip] + to: log.remip + - rename: + from: [log.FTNTFGTlocip] + to: log.locip + - rename: + from: [log.FTNTFGTremport] + to: log.remport + - rename: + from: [log.FTNTFGTlocport] + to: log.locport + - rename: + from: [log.FTNTFGTuser] + to: log.user + - rename: + from: [log.FTNTFGTgroup] + to: log.group + - rename: + from: [log.FTNTFGTsrcname] + to: log.srcname + - rename: + from: [log.FTNTFGTdstname] + to: log.dstname + - rename: + from: [log.FTNTFGTosname] + to: log.osname + - rename: + from: [log.FTNTFGTdstosname] + to: log.dstosname + - rename: + from: [log.FTNTFGTdstmac] + to: log.dstmac + - rename: + from: [log.FTNTFGTprofile] + to: log.profile + - rename: + from: [log.FTNTFGTverdict] + to: log.verdict + - rename: + from: [log.FTNTFGTrisk] + to: log.risk + - rename: + from: [log.FTNTFGTfilehash] + to: log.filehash + - rename: + from: [log.FTNTFGTfilename] + to: log.filename + - rename: + from: [log.FTNTFGTfilesize] + to: log.filesize + # Fortinet CEF maps its authenticated user to duser. + - rename: + from: [log.duser] + to: log.user + where: '!exists("log.user")' + + - rename: + from: [log.FTNTFGTvirus] + to: log.virus + # Split CEF "cat" (e.g. "traffic:local") → log.type + log.catSubtype. - grok: source: log.cat patterns: - - fieldName: log.type - pattern: '{{.word}}' - - fieldName: "" - pattern: ':' - - fieldName: log.catSubtype - pattern: '{{.word}}' + - {fieldName: log.type, pattern: '{{.word}}'} + - {fieldName: "", pattern: ':'} + - {fieldName: log.catSubtype, pattern: '{{.word}}'} - delete: fields: [log.cat] + where: exists("log.cefVersion") # CEF long-form aliases → canonical names. - rename: @@ -203,25 +1290,57 @@ pipeline: from: [log.sourceTranslatedPort] to: log.transport - # Using grok to parse kv issued fields - - grok: - patterns: - - fieldName: log.irrelevant - pattern: '{{.data}}(msg=)' - - fieldName: log.msg - pattern: '{{.data}}({{.word}}=)' - - fieldName: log.irrelevant - pattern: '{{.greedy}}' - source: log.kvMessage - - # Using grok to remove irrelevant data - - grok: - patterns: - - fieldName: log.msg - pattern: '{{.greedy}}{{.space}}' - - fieldName: log.irrelevant - pattern: '{{.word}}(=)' - source: log.msg + # Remove outer KV quotes before IP validation and numeric conversion. + - trim: + function: prefix + substring: '"' + fields: + - log.srcip + - log.dstip + - log.src + - log.dst + - log.dest_ip + - log.src_ip + - log.srcport + - log.dstport + - log.spt + - log.dpt + - log.src_port + - log.dest_port + - log.rcvdbyte + - log.in + - log.rcvdpkt + - log.sentbyte + - log.out + - log.sentpkt + - log.proto + - origin.packagesSent + - origin.packagesReceived + - trim: + function: suffix + substring: '"' + fields: + - log.srcip + - log.dstip + - log.src + - log.dst + - log.dest_ip + - log.src_ip + - log.srcport + - log.dstport + - log.spt + - log.dpt + - log.src_port + - log.dest_port + - log.rcvdbyte + - log.in + - log.rcvdpkt + - log.sentbyte + - log.out + - log.sentpkt + - log.proto + - origin.packagesSent + - origin.packagesReceived # Rename standard fields and fields out of kv result - rename: @@ -229,21 +1348,31 @@ pipeline: - log.action - log.act to: action + # Native address wins over its CEF alias; an invalid selected value + # stays under its original vendor key and never reaches geolocation. - rename: - from: - - log.dstip - - log.dst + from: [log.dst] + to: target.ip + where: '!exists("log.dstip") && (inCIDR("log.dst", "0.0.0.0/0") || inCIDR("log.dst", "::/0")) && !inCIDR("log.dst", "0.0.0.0/32") && !inCIDR("log.dst", "::/128")' + - rename: + from: [log.dstip] to: target.ip + where: '(inCIDR("log.dstip", "0.0.0.0/0") || inCIDR("log.dstip", "::/0")) && !inCIDR("log.dstip", "0.0.0.0/32") && !inCIDR("log.dstip", "::/128")' - rename: from: - log.dstport - log.dpt to: target.port + # Native address wins over its CEF alias; an invalid selected value + # stays under its original vendor key and never reaches geolocation. - rename: - from: - - log.srcip - - log.src + from: [log.src] + to: origin.ip + where: '!exists("log.srcip") && (inCIDR("log.src", "0.0.0.0/0") || inCIDR("log.src", "::/0")) && !inCIDR("log.src", "0.0.0.0/32") && !inCIDR("log.src", "::/128")' + - rename: + from: [log.srcip] to: origin.ip + where: '(inCIDR("log.srcip", "0.0.0.0/0") || inCIDR("log.srcip", "::/0")) && !inCIDR("log.srcip", "0.0.0.0/32") && !inCIDR("log.srcip", "::/128")' - rename: from: - log.srcport @@ -273,6 +1402,7 @@ pipeline: from: - log.dest_ip to: target.ip + where: '(inCIDR("log.dest_ip", "0.0.0.0/0") || inCIDR("log.dest_ip", "::/0")) && !inCIDR("log.dest_ip", "0.0.0.0/32") && !inCIDR("log.dest_ip", "::/128")' - rename: from: - log.dest_port @@ -281,6 +1411,7 @@ pipeline: from: - log.src_ip to: origin.ip + where: '(inCIDR("log.src_ip", "0.0.0.0/0") || inCIDR("log.src_ip", "::/0")) && !inCIDR("log.src_ip", "0.0.0.0/32") && !inCIDR("log.src_ip", "::/128")' - rename: from: - log.src_port @@ -305,24 +1436,6 @@ pipeline: to: origin.packagesSent # Adding action result - - add: - function: string - params: - key: actionResult - value: "accept" - where: 'equals("action", "accept")' - - add: - function: string - params: - key: actionResult - value: "denied" - where: 'equals("action", "deny")' - - add: - function: string - params: - key: actionResult - value: "blocked" - where: 'oneOf("action", ["client-rst", "server-rst", "ip-conn"])' # Type casting for numeric fields. - cast: @@ -435,6 +1548,156 @@ pipeline: - log.srcname - log.sourceVendor + # Preserve complete vendor values and copy semantic equivalents into SDK fields. + - trim: + function: prefix + substring: '"' + fields: + - log.msg + - log.logdesc + - log.user + - log.group + - log.dstuser + - log.dstname + - log.hostname + - log.url + - log.osName + - log.dstosname + - log.dstmac + - log.policyname + - log.catdesc + - log.devtype + - log.srchwversion + - log.vwlquality + - log.agent + - log.profile + - log.reason + - log.filename + - log.severity + - log.status + - log.utmaction + - log.verdict + - log.risk + - log.filehash + - log.remip + - log.locip + - trim: + function: suffix + substring: '"' + fields: + - log.msg + - log.logdesc + - log.user + - log.group + - log.dstuser + - log.dstname + - log.hostname + - log.url + - log.osName + - log.dstosname + - log.dstmac + - log.policyname + - log.catdesc + - log.devtype + - log.srchwversion + - log.vwlquality + - log.agent + - log.profile + - log.reason + - log.filename + - log.severity + - log.status + - log.utmaction + - log.verdict + - log.risk + - log.filehash + - log.remip + - log.locip + - grok: + source: log.srcname + patterns: + - {fieldName: origin.host, pattern: '(?s:.+)'} + where: exists("log.srcname") && !oneOf("log.srcname", ["", "-", "N/A", "unknown"]) + - grok: + source: log.dstname + patterns: + - {fieldName: target.host, pattern: '(?s:.+)'} + where: exists("log.dstname") && !oneOf("log.dstname", ["", "-", "N/A", "unknown"]) + - grok: + source: log.osName + patterns: + - {fieldName: origin.operatingSystem, pattern: '(?s:.+)'} + where: exists("log.osName") && !oneOf("log.osName", ["", "-", "N/A", "unknown"]) + - grok: + source: log.dstosname + patterns: + - {fieldName: target.operatingSystem, pattern: '(?s:.+)'} + where: exists("log.dstosname") && !oneOf("log.dstosname", ["", "-", "N/A", "unknown"]) + - grok: + source: log.dstmac + patterns: + - {fieldName: target.mac, pattern: '(?s:.+)'} + where: exists("log.dstmac") && !oneOf("log.dstmac", ["", "-", "N/A", "unknown"]) + - grok: + source: log.user + patterns: + - {fieldName: origin.user, pattern: '(?s:.+)'} + where: exists("log.user") && !oneOf("log.user", ["", "-", "N/A", "unknown"]) + - grok: + source: log.group + patterns: + - {fieldName: origin.group, pattern: '(?s:.+)'} + where: exists("log.group") && !oneOf("log.group", ["", "-", "N/A", "unknown"]) + - grok: + source: log.dstuser + patterns: + - {fieldName: target.user, pattern: '(?s:.+)'} + where: exists("log.dstuser") && !oneOf("log.dstuser", ["", "-", "N/A", "unknown"]) + - rename: + from: [log.remip] + to: origin.ip + where: '!exists("origin.ip") && equals("log.type", "event") && equals("log.subtype", "vpn") && (oneOf("action", ["ssl-login-fail", "ssl-login-succ"]) || equals("log.logid", "0101039426")) && (inCIDR("log.remip", "0.0.0.0/0") || inCIDR("log.remip", "::/0")) && !inCIDR("log.remip", "0.0.0.0/32") && !inCIDR("log.remip", "::/128")' + - add: + function: string + params: + key: severity + value: debug + where: oneOf("log.level", ["debug"]) + - add: + function: string + params: + key: severity + value: info + where: oneOf("log.level", ["information", "informational", "notice"]) + - add: + function: string + params: + key: severity + value: warning + where: oneOf("log.level", ["warning"]) + - add: + function: string + params: + key: severity + value: error + where: oneOf("log.level", ["error"]) + - add: + function: string + params: + key: severity + value: critical + where: oneOf("log.level", ["critical", "alert", "emergency"]) + + - grok: + source: log.url + patterns: + - {fieldName: target.url, pattern: '(?s:.+)'} + where: regexMatch("log.url", "(?i)^(https?|ftp)://[^\\s]+$") + - grok: + source: log.hostname + patterns: + - {fieldName: target.domain, pattern: '(?s:.+)'} + where: equals("log.type", "utm") && oneOf("log.subtype", ["webfilter", "app-ctrl", "ips", "ssl"]) && regexMatch("log.hostname", "^[A-Za-z0-9][A-Za-z0-9.-]*[A-Za-z0-9]$") && !(inCIDR("log.hostname", "0.0.0.0/0") || inCIDR("log.hostname", "::/0")) && !oneOf("log.hostname", ["unknown", "N/A"]) # Adding geolocation to origin.ip - dynamic: plugin: com.utmstack.geolocation @@ -1315,4 +2578,145 @@ pipeline: - delete: fields: - log.kvMessage - - log.proto \ No newline at end of file + - log.proto + + # Normalize explicit outcomes; an unknown outcome remains unset. + # LOG_ID_TRAFFIC_FAIL_CONN (11) denotes failure; action=dns alone does not. + - add: + function: string + params: + key: actionResult + value: success + where: equalsIgnoreCase("action", "accept") + - add: + function: string + params: + key: actionResult + value: failure + where: equals("log.type", "traffic") && equals("log.logid", "0000000011") + - add: + function: string + params: + key: actionResult + value: success + where: equals("log.type", "event") && equals("log.subtype", "system") && equals("log.logid", "0100032001") + - add: + function: string + params: + key: actionResult + value: failure + where: equals("log.type", "event") && (equals("log.subtype", "system") && equals("log.logid", "0100032002") || equals("log.subtype", "vpn") && (equals("log.logid", "0101039426") || equals("action", "ssl-login-fail"))) + - add: + function: string + params: + key: actionResult + value: denied + where: regexMatch("action", "(?i)^(deny|denied|drop|dropped|block|blocked)$") || regexMatch("log.utmaction", "(?i)^(deny|denied|drop|dropped|block|blocked)$") || regexMatch("log.FTNTFGTutmaction", "(?i)^(deny|denied|drop|dropped|block|blocked)$") + - add: + function: string + params: + key: connectionStatus + value: closed + where: oneOf("action", ["close", "client-rst", "server-rst", "timeout"]) + + - add: + function: string + params: + key: protocol + value: ICMP + where: equals("protocol",1) + - add: + function: string + params: + key: protocol + value: TCP + where: equals("protocol",6) + - add: + function: string + params: + key: protocol + value: UDP + where: equals("protocol",17) + - add: + function: string + params: + key: protocol + value: GRE + where: equals("protocol",47) + - add: + function: string + params: + key: protocol + value: ESP + where: equals("protocol",50) + - add: + function: string + params: + key: protocol + value: AH + where: equals("protocol",51) + - add: + function: string + params: + key: protocol + value: ICMPV6 + where: equals("protocol",58) + - add: + function: string + params: + key: protocol + value: SCTP + where: equals("protocol",132) + - rename: + from: + - protocol + to: log.ipProtocolNumber + where: exists("protocol") && greaterOrEqual("protocol",0) + + # Ignore input-supplied markers. History counts only equivalent candidates. + - delete: + fields: [log.correlationCandidate] + - add: + function: string + params: + key: log.correlationCandidate.vpnAuthFailure + value: match + where: | + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.type", "event") && equals("log.subtype", "vpn") && + (equals("action", "ssl-login-fail") || equals("log.logid", "0101039426")) && + exists("origin.ip")) + - add: + function: string + params: + key: log.correlationCandidate.ipsCritical + value: match + where: | + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.type", "utm") && equals("log.subtype", "ips") && + oneOf("log.severity", ["critical", "high"]) && + equals("actionResult", "denied") && exists("origin.ip")) + - add: + function: string + params: + key: log.correlationCandidate.dlp + value: match + where: | + equals("log.type", "utm") && equals("log.subtype", "dlp") && + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (exists("origin.ip") && + ((equals("log.subtype", "dlp") && oneOf("action", ["block", "blocked"])) || + contains("log.msg", "DLP sensor") || + contains("log.msg", "sensitive data detected") || + (exists("log.dlpprofile") && equals("log.dlpaction", "block")) || + contains("log.msg", "Data pattern matched"))) + - add: + function: string + params: + key: log.correlationCandidate.virusOutbreak + value: match + where: | + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.type", "utm") && equals("log.subtype", "virus") && + equals("log.eventtype", "infected") && equals("actionResult", "denied") && + exists("origin.ip")) diff --git a/plugins/alerts/fortigate_contract_test.go b/plugins/alerts/fortigate_contract_test.go new file mode 100644 index 000000000..22c24b3e0 --- /dev/null +++ b/plugins/alerts/fortigate_contract_test.go @@ -0,0 +1,502 @@ +package main + +// Offline FortiGate extraction model, not the closed EventProcessor. +// Explicit YAML grok/rename/cast/trim/add/delete and observed KV splitting are +// modeled. CEL, Event serialization, placeholder expansion, query creation and +// history thresholds use SDK v1.1.31. External geolocation is not executed. +import ( + "bytes" + "encoding/json" + "fmt" + "net" + "os" + "path/filepath" + "reflect" + "regexp" + "strings" + "testing" + "text/template" + + "github.com/threatwinds/go-sdk/plugins" + "github.com/threatwinds/go-sdk/utils" + "github.com/tidwall/gjson" + "google.golang.org/protobuf/encoding/protojson" +) + +type fortiFixture struct { + Name string `json:"name"` + DataSource string `json:"dataSource"` + Raw string `json:"raw"` + Expected map[string]any `json:"expected"` + Absent []string `json:"absent"` + Matches []string `json:"matches"` +} + +func fortiPut(m map[string]any, path string, value any, remove bool) { + p := strings.Split(path, ".") + for _, k := range p[:len(p)-1] { + n, ok := m[k].(map[string]any) + if !ok { + if remove { + return + } + n = map[string]any{} + m[k] = n + } + m = n + } + if remove { + delete(m, p[len(p)-1]) + } else { + m[p[len(p)-1]] = value + } +} +func fortiGet(m map[string]any, p string) (any, bool) { + var v any = m + for _, k := range strings.Split(p, ".") { + n, ok := v.(map[string]any) + if !ok { + return nil, false + } + v, ok = n[k] + if !ok { + return nil, false + } + } + return v, true +} +func fortiConfig(t *testing.T) *plugins.Config { + t.Helper() + b, e := utils.ReadPbYaml("../../filters/fortinet/fortinet.yml") + if e != nil { + t.Fatal(e) + } + c := new(plugins.Config) + if e = protojson.Unmarshal(b, c); e != nil { + t.Fatal(e) + } + return c +} +func fortiRegex(t *testing.T, g *plugins.Grok, cfg *plugins.Config) *regexp.Regexp { + t.Helper() + var pattern strings.Builder + for i, p := range g.Patterns { + if p.FieldName != "" { + fmt.Fprintf(&pattern, "(?P%s)", i, p.Pattern) + } else { + pattern.WriteString("(?:" + p.Pattern + ")") + } + } + pats := map[string]string{"greedy": ".*", "data": ".*?", "word": "[A-Za-z0-9_-]+", "space": "\\s+"} + for k, v := range cfg.Patterns { + pats[k] = v + } + tmpl, e := template.New("grok").Option("missingkey=error").Parse(pattern.String()) + if e != nil { + t.Fatal(e) + } + var b bytes.Buffer + if e = tmpl.Execute(&b, pats); e != nil { + t.Fatal(e) + } + r, e := regexp.Compile(b.String()) + if e != nil { + t.Fatal(e) + } + return r +} +func fortiParse(t *testing.T, cfg *plugins.Config, raw string, dataSource string, cache *plugins.CELCache) string { + t.Helper() + draft := map[string]any{"raw": raw, "dataType": "firewall-fortigate-traffic", "dataSource": dataSource, "log": map[string]any{}} + for _, stage := range cfg.Pipeline { + matched := false + for _, dataType := range stage.DataTypes { + if dataType == "firewall-fortigate-traffic" { + matched = true + } + } + if !matched { + continue + } + for _, s := range stage.Steps { + b, e := protojson.Marshal(s) + if e != nil { + t.Fatal(e) + } + var step map[string]map[string]any + if e = json.Unmarshal(b, &step); e != nil { + t.Fatal(e) + } + for kind, body := range step { + if w, ok := body["where"].(string); ok && w != "" { + snapshot, err := json.Marshal(draft) + if err != nil { + t.Fatal(err) + } + match, e := cache.Eval(w, string(snapshot)) + if e != nil { + t.Fatal(e) + } + if !match { + continue + } + } + switch kind { + case "grok": + g := s.Grok + src := g.Source + if src == "" { + src = "raw" + } + v, ok := fortiGet(draft, src) + if !ok { + continue + } + str, ok := v.(string) + if !ok { + t.Fatalf("non-string grok source %s", src) + } + r := fortiRegex(t, g, cfg) + m := r.FindStringSubmatch(str) + if m == nil { + continue + } + for i, p := range g.Patterns { + if p.FieldName != "" { + fortiPut(draft, p.FieldName, m[r.SubexpIndex(fmt.Sprintf("f%d", i))], false) + } + } + case "rename": + for _, p := range s.Rename.From { + if v, ok := fortiGet(draft, p); ok { + fortiPut(draft, s.Rename.To, v, false) + fortiPut(draft, p, nil, true) + break + } + } + case "trim": + for _, p := range s.Trim.Fields { + if v, ok := fortiGet(draft, p); ok { + str, ok := v.(string) + if !ok { + continue + } + switch s.Trim.Function { + case "prefix": + str = strings.TrimPrefix(str, s.Trim.Substring) + case "suffix": + str = strings.TrimSuffix(str, s.Trim.Substring) + default: + t.Fatalf("unsupported trim %s", s.Trim.Function) + } + fortiPut(draft, p, str, false) + } + } + case "add": + if s.Add.Function != "string" { + t.Fatalf("unsupported add function %s", s.Add.Function) + } + fortiPut(draft, s.Add.Params["key"].GetStringValue(), s.Add.Params["value"].AsInterface(), false) + case "delete": + for _, p := range s.Delete.Fields { + fortiPut(draft, p, nil, true) + } + case "kv": + v, ok := fortiGet(draft, s.Kv.Source) + if !ok { + continue + } + // Observed KV output splits quoted multiword values. + // Explicit YAML grok steps rebuild consumed fields afterward. + for _, item := range strings.Split(v.(string), s.Kv.FieldSplit) { + pair := strings.SplitN(item, s.Kv.ValueSplit, 2) + if len(pair) != 2 { + continue + } + key := pair[0] + utils.SanitizeField(&key) + if key != "" { + fortiPut(draft, "log."+key, pair[1], false) + } + } + case "dynamic": + if s.Dynamic.Plugin != "com.utmstack.geolocation" { + t.Fatalf("unsupported dynamic plugin %s", s.Dynamic.Plugin) + } + field := s.Dynamic.Params["source"].GetStringValue() + v, ok := fortiGet(draft, field) + if !ok { + t.Fatalf("missing dynamic source %s", field) + } + ip := net.ParseIP(fmt.Sprint(v)) + if ip == nil || ip.IsUnspecified() { + t.Fatalf("invalid address reaches geolocation: %s", field) + } + // The external geolocation service is not executed. + case "json": + source, ok := fortiGet(draft, s.Json.Source) + if !ok { + continue + } + str, ok := source.(string) + if !ok { + t.Fatalf("JSON source is not a string") + } + var parsed map[string]any + if e := json.Unmarshal([]byte(str), &parsed); e != nil { + t.Fatal(e) + } + for key, value := range fortiSanitizeJSON(parsed) { + fortiPut(draft, "log."+key, value, false) + } + case "cast": + for _, field := range s.Cast.Fields { + if value, ok := fortiGet(draft, field); ok { + switch s.Cast.To { + case "string": + fortiPut(draft, field, utils.CastString(value), false) + case "int": + fortiPut(draft, field, utils.CastInt64(value), false) + default: + t.Fatalf("unsupported cast %s", s.Cast.To) + } + } + } + case "drop": + return "" + default: + t.Fatalf("unsupported filter step %s", kind) + } + } + } + } + b, e := json.Marshal(draft) + if e != nil { + t.Fatal(e) + } + in := string(b) + ev := new(plugins.Event) + if e = utils.StringToProtoMessage(&in, ev); e != nil { + t.Fatal(e) + } + out, e := utils.ProtoMessageToString(ev) + if e != nil { + t.Fatal(e) + } + return *out +} +func fortiRules(t *testing.T) map[string]*plugins.Rule { + t.Helper() + paths, e := filepath.Glob("../../rules/fortinet/fortinet/*.yml") + if e != nil { + t.Fatal(e) + } + out := map[string]*plugins.Rule{} + for _, p := range paths { + b, e := utils.ReadPbYaml(p) + if e != nil { + t.Fatal(e) + } + r := new(plugins.Rule) + if e = protojson.Unmarshal(b, r); e != nil { + t.Fatal(e) + } + r.Normalize() + out[strings.TrimSuffix(filepath.Base(p), ".yml")] = r + } + return out +} + +func fortiSanitizeJSON(input map[string]any) map[string]any { + out := map[string]any{} + for key, value := range input { + utils.SanitizeField(&key) + if nested, ok := value.(map[string]any); ok { + value = fortiSanitizeJSON(nested) + } + out[key] = value + } + return out +} + +func fortiFixtures(t *testing.T) []fortiFixture { + t.Helper() + b, e := os.ReadFile("testdata/fortigate_raw.json") + if e != nil { + t.Fatal(e) + } + var cases []fortiFixture + if e = json.Unmarshal(b, &cases); e != nil { + t.Fatal(e) + } + return cases +} + +var fortiMarkers = map[string]string{ + "fortigate_vpn_brute_force": "vpnAuthFailure", + "ips_critical_severity_events": "ipsCritical", + "dlp_data_exfiltration": "dlp", + "antivirus_outbreak_detection": "virusOutbreak", +} + +func TestFortiGateRawContracts(t *testing.T) { + cfg, rules, cache := fortiConfig(t), fortiRules(t), plugins.NewCELCache("fortigate-raw") + for _, f := range fortiFixtures(t) { + t.Run(f.Name, func(t *testing.T) { + out := fortiParse(t, cfg, f.Raw, f.DataSource, cache) + for field, want := range f.Expected { + got := gjson.Get(out, field) + if !got.Exists() || !reflect.DeepEqual(got.Value(), want) { + t.Errorf("%s got %v want %v", field, got.Value(), want) + } + } + for _, field := range f.Absent { + if gjson.Get(out, field).Exists() { + t.Errorf("unexpected %s", field) + } + } + if gjson.Get(out, "raw").String() != f.Raw { + t.Error("raw changed") + } + expected := map[string]bool{} + for _, n := range f.Matches { + expected[n] = true + } + for name, r := range rules { + yes, e := cache.Eval(r.Where, out) + if e != nil { + t.Fatalf("%s CEL: %v", name, e) + } + if yes != expected[name] { + t.Errorf("%s matched %v want %v", name, yes, expected[name]) + } + if key := fortiMarkers[name]; key != "" { + if (gjson.Get(out, "log.correlationCandidate."+key).String() == "match") != yes { + t.Errorf("%s marker parity", name) + } + } + if yes { + for _, search := range r.Correlation { + for _, term := range search.With { + v := term.Value.GetStringValue() + if strings.HasPrefix(v, "{{.") { + field := strings.TrimSuffix(strings.TrimPrefix(v, "{{."), "}}") + if !gjson.Get(out, field).Exists() { + t.Errorf("%s unresolved %s", name, field) + } + } + } + } + ev := new(plugins.Event) + if e := utils.StringToProtoMessage(&out, ev); e != nil { + t.Fatal(e) + } + if r.Adversary != "origin" { + t.Fatalf("unexpected actor %s", r.Adversary) + } + a := &plugins.Alert{Adversary: ev.Origin, Target: ev.Target, Events: []*plugins.Event{ev}} + wire, e := utils.ProtoMessageToString(a) + if e != nil { + t.Fatal(e) + } + // Indexed lastEvent aliases require the separate alert foundation. + for _, field := range r.GroupBy { + path := strings.Replace(field, "lastEvent.", "events.0.", 1) + if field == "adversary.ip" && ev.GetOrigin().GetIp() != "" && gjson.Get(*wire, path).String() != ev.GetOrigin().GetIp() { + t.Error("actor IP lost") + } + if strings.HasPrefix(field, "lastEvent.log.") && !gjson.Get(*wire, path).Exists() { + t.Errorf("missing grouping %s", field) + } + } + } + } + }) + } +} + +// Optional bounded private evidence: emits only counts/field names and keeps +// customer source material outside the repository. +func TestFortiGatePrivateEvidence(t *testing.T) { + path := os.Getenv("UTM_FORTIGATE_EVIDENCE") + if path == "" { + t.Skip("set UTM_FORTIGATE_EVIDENCE to the private stored-input JSON") + } + b, e := os.ReadFile(path) + if e != nil { + t.Fatal(e) + } + var pack struct { + Documents []struct{ Source map[string]any } + } + if e = json.Unmarshal(b, &pack); e != nil { + t.Fatal(e) + } + if len(pack.Documents) == 0 { + t.Fatal("empty evidence") + } + cfg, rules, cache := fortiConfig(t), fortiRules(t), plugins.NewCELCache("fortigate-private") + matches := map[string]int{} + populated := map[string]int{} + checkedRawStrings := map[string]int{} + for _, doc := range pack.Documents { + raw, ok := doc.Source["raw"].(string) + if !ok { + t.Fatal("raw missing from private evidence") + } + source, _ := doc.Source["dataSource"].(string) + out := fortiParse(t, cfg, raw, source, cache) + stored, err := json.Marshal(doc.Source) + if err != nil { + t.Fatal(err) + } + for _, field := range []string{"dataSource", "action", "protocol", "origin.ip", "target.ip", "origin.port", "target.port", "origin.mac", "origin.bytesSent", "origin.bytesReceived", "origin.packagesSent", "origin.packagesReceived"} { + before, after := gjson.GetBytes(stored, field), gjson.Get(out, field) + if before.Exists() && before.String() != after.String() { + if !after.Exists() && (before.String() == "0" || before.String() == "0.0") { + continue + } + t.Errorf("existing normalized %s changed in private evidence", field) + } + } + for _, field := range []string{"msg", "logdesc"} { + re := regexp.MustCompile(`(?:^|\s)` + field + `=("(?:\\.|[^"\\])*"|[^\s]+)`) + if match := re.FindStringSubmatch(raw); len(match) > 1 { + checkedRawStrings[field]++ + want := strings.TrimSuffix(strings.TrimPrefix(match[1], `"`), `"`) + if gjson.Get(out, "log."+field).String() != want { + t.Errorf("private %s extraction differs from raw", field) + } + } + } + for _, field := range []string{"origin.ip", "target.ip", "origin.host", "origin.user", "origin.group", "target.mac", "origin.operatingSystem", "target.operatingSystem", "target.url", "target.domain", "log.msg", "log.logdesc", "actionResult"} { + if gjson.Get(out, field).Exists() { + populated[field]++ + } + } + for name, rule := range rules { + yes, e := cache.Eval(rule.Where, out) + if e != nil { + t.Fatalf("%s: %v", name, e) + } + if yes { + matches[name]++ + for _, search := range rule.Correlation { + for _, term := range search.With { + v := term.Value.GetStringValue() + if strings.HasPrefix(v, "{{.") { + field := strings.TrimSuffix(strings.TrimPrefix(v, "{{."), "}}") + if !gjson.Get(out, field).Exists() { + t.Errorf("%s unresolved %s", name, field) + } + } + } + } + } + } + } + if len(checkedRawStrings) == 0 { + t.Fatal("no private raw strings were compared") + } + t.Logf("private records=%d checked raw strings=%v modeled field population=%v modeled predicates=%v", len(pack.Documents), checkedRawStrings, populated, matches) +} diff --git a/plugins/alerts/fortigate_history_test.go b/plugins/alerts/fortigate_history_test.go new file mode 100644 index 000000000..167a16d6c --- /dev/null +++ b/plugins/alerts/fortigate_history_test.go @@ -0,0 +1,245 @@ +package main + +// SDK history contracts for synthetic FortiGate events. Extraction uses the +// documented offline model in fortigate_contract_test.go; all search requests, +// placeholder expansion, mapping lookup and count decisions use the pinned SDK. +// The HTTP server below is loopback-only and never queries a customer instance. +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "strings" + "testing" + "time" + + sdkos "github.com/threatwinds/go-sdk/os" + "github.com/threatwinds/go-sdk/plugins" + "github.com/tidwall/gjson" +) + +func TestFortiGateSDKHistory(t *testing.T) { + // The SDK owns a process-wide OpenSearch singleton. Isolate this local mock + // so other technology tests can initialize their own clients in this suite. + if os.Getenv("UTM_FORTIGATE_HISTORY_CHILD") != "1" { + command := exec.Command(os.Args[0], "-test.run=^TestFortiGateSDKHistory$") + command.Env = append(os.Environ(), "UTM_FORTIGATE_HISTORY_CHILD=1") + if out, e := command.CombinedOutput(); e != nil { + t.Fatalf("isolated history test: %v\n%s", e, out) + } + return + } + + cfg, rules, cache := fortiConfig(t), fortiRules(t), plugins.NewCELCache("fortigate-history") + var history []string + var expectedClauses int + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + if strings.HasSuffix(r.URL.Path, "/_mapping") { + // Text fields exercise the SDK's .keyword mapping resolution; IP and + // keyword fields exercise exact mappings without that suffix. + _, _ = io.WriteString(w, `{"v11-log-firewall-fortigate-traffic-test":{"mappings":{"properties":{"@timestamp":{"type":"date"},"dataSource":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"origin":{"properties":{"ip":{"type":"ip"},"user":{"type":"text","fields":{"keyword":{"type":"keyword"}}}}},"log":{"properties":{"devid":{"type":"keyword"},"vd":{"type":"keyword"},"type":{"type":"keyword"},"subtype":{"type":"keyword"},"logid":{"type":"keyword"},"correlationCandidate":{"properties":{"vpnAuthFailure":{"type":"keyword"},"ipsCritical":{"type":"keyword"},"dlp":{"type":"keyword"},"virusOutbreak":{"type":"keyword"}}}}}}}}}`) + return + } + if r.URL.Path != "/v11-log-firewall-fortigate-traffic-*/_search" { + t.Errorf("unexpected path %s", r.URL.Path) + http.Error(w, "unsupported request", http.StatusBadRequest) + return + } + b, e := io.ReadAll(r.Body) + if e != nil { + t.Error(e) + return + } + query := string(b) + clauses := append(gjson.Get(query, "query.bool.must").Array(), gjson.Get(query, "query.bool.filter").Array()...) + if len(clauses) != expectedClauses { + t.Errorf("history clauses: got %d want %d: %s", len(clauses), expectedClauses, query) + } + hits := []map[string]any{} + for _, doc := range history { + yes := true + for _, clause := range clauses { + if term := clause.Get("term"); term.Exists() { + for field, v := range term.Map() { + value := gjson.Get(doc, strings.TrimSuffix(field, ".keyword")) + if !value.Exists() || value.String() != v.Get("value").String() { + yes = false + } + } + } else if span := clause.Get("range"); span.Exists() { + for field, limits := range span.Map() { + stamp, e := time.Parse(time.RFC3339Nano, gjson.Get(doc, field).String()) + if e != nil { + t.Error(e) + } + cutoff, e := time.Parse(time.RFC3339Nano, limits.Get("gte").String()) + if e != nil { + t.Error(e) + } + if stamp.Before(cutoff) { + yes = false + } + } + } else { + t.Errorf("unsupported clause %s", clause.Raw) + yes = false + } + } + if yes { + hits = append(hits, map[string]any{"_id": fmt.Sprint(len(hits)), "_index": "v11-log-firewall-fortigate-traffic-test", "_source": map[string]any{}}) + } + } + _ = json.NewEncoder(w).Encode(map[string]any{"took": 1, "hits": map[string]any{"total": map[string]any{"value": len(hits), "relation": "eq"}, "hits": hits}}) + })) + defer server.Close() + if e := sdkos.Connect([]string{server.URL}, "", ""); e != nil { + t.Fatal(e) + } + mutate := func(doc, field string, value any) string { + var m map[string]any + if e := json.Unmarshal([]byte(doc), &m); e != nil { + t.Fatal(e) + } + fortiPut(m, field, value, value == nil) + b, e := json.Marshal(m) + if e != nil { + t.Fatal(e) + } + return string(b) + } + + // All fixture names refer to fabricated documentation-shaped raw logs. + // Require each named case so a missing fixture cannot silently skip a rule. + fixtures := map[string]fortiFixture{} + for _, fixture := range fortiFixtures(t) { + fixtures[fixture.Name] = fixture + } + cases := []struct { + rule, fixture string + count uint64 + within string + }{ + {"admin_account_compromise", "admin success positive", 5, "15m"}, + {"fortigate_vpn_brute_force", "VPN authentication positive", 10, "15m"}, + {"ips_critical_severity_events", "IPS critical blocked positive", 3, "15m"}, + {"dlp_data_exfiltration", "DLP blocked positive", 3, "1h"}, + {"antivirus_outbreak_detection", "infected-file outbreak positive", 5, "1h"}, + } + for _, test := range cases { + t.Run(test.rule, func(t *testing.T) { + fixture, ok := fixtures[test.fixture] + if !ok { + t.Fatalf("required raw fixture %q missing", test.fixture) + } + rule := rules[test.rule] + if rule == nil || len(rule.Correlation) != 1 { + t.Fatalf("expected exactly one history request for %s", test.rule) + } + search := rule.Correlation[0] + if search.Count != test.count || search.Within != test.within { + t.Fatalf("unexpected threshold %d/%s", search.Count, search.Within) + } + expectedClauses = len(search.With) + 1 // processing-time lower bound + out := fortiParse(t, cfg, fixture.Raw, fixture.DataSource, cache) + if match, e := cache.Eval(rule.Where, out); e != nil || !match { + t.Fatalf("positive raw fixture does not match: %v %v", match, e) + } + parse := func(raw string) string { + return fortiParse(t, cfg, raw, fixture.DataSource, cache) + } + priorRaw := fixture.Raw + if test.rule == "admin_account_compromise" { + priorRaw = strings.NewReplacer("0100032001", "0100032002", "Admin login successful", "Admin login failed", `status="success"`, `status="failed"`).Replace(priorRaw) + } + prior := mutate(parse(priorRaw), "@timestamp", time.Now().Add(-time.Minute).UTC().Format(time.RFC3339Nano)) + if marker := fortiMarkers[test.rule]; marker != "" && gjson.Get(prior, "log.correlationCandidate."+marker).String() != "match" { + t.Fatal("matching raw history did not produce its candidate marker") + } + check := func(name, historical string, count uint64, want bool) { + t.Run(name, func(t *testing.T) { + history = nil + for i := uint64(0); i < count; i++ { + history = append(history, historical) + } + yes, _, e := search.Execute(&out) + if e != nil || yes != want { + t.Fatalf("history result got %v want %v: %v", yes, want, e) + } + }) + } + check("below_threshold", prior, search.Count-1, false) + check("at_threshold", prior, search.Count, true) + duration, e := time.ParseDuration(search.Within) + if e != nil { + t.Fatal(e) + } + check("inside_window", mutate(prior, "@timestamp", time.Now().Add(-duration+10*time.Second).UTC().Format(time.RFC3339Nano)), search.Count, true) + check("expired", mutate(prior, "@timestamp", time.Now().Add(-duration-10*time.Second).UTC().Format(time.RFC3339Nano)), search.Count, false) + for _, change := range []struct{ field, value string }{ + {"origin.ip", "198.51.100.99"}, {"log.devid", "OTHER-FIREWALL"}, {"log.vd", "other-vdom"}, + } { + check("different_"+change.field, mutate(prior, change.field, change.value), search.Count, false) + } + for _, term := range search.With { + if term.Field == "dataSource" { + check("different_ingress", mutate(prior, "dataSource", "other-relay"), search.Count, false) + } + } + if test.rule == "admin_account_compromise" { + check("different_account", mutate(prior, "origin.user", "other-user"), search.Count, false) + check("successful_login_is_not_failed_history", mutate(parse(fixture.Raw), "@timestamp", time.Now().Add(-time.Minute).UTC().Format(time.RFC3339Nano)), search.Count, false) + check("different_event_type", mutate(prior, "log.type", "traffic"), search.Count, false) + check("different_event_subtype", mutate(prior, "log.subtype", "vpn"), search.Count, false) + } else { + marker := "log.correlationCandidate." + fortiMarkers[test.rule] + check("unmarked_history", mutate(prior, marker, nil), search.Count, false) + // Reparse benign raw events so the filter, rather than a test-only + // deletion, establishes why they cannot satisfy a candidate count. + var benignRaws []string + switch test.rule { + case "fortigate_vpn_brute_force": + benignRaws = []string{strings.NewReplacer(`logid="0101039426" `, "", "ssl-login-fail", "ssl-login-succ", "SSL VPN login fail", "SSL VPN login success", "SSL user failed to log in", "SSL user logged in").Replace(fixture.Raw)} + case "ips_critical_severity_events": + benignRaws = []string{strings.ReplaceAll(fixture.Raw, `severity="critical"`, `severity="low"`), strings.ReplaceAll(fixture.Raw, `action="dropped"`, `action="detected"`)} + case "dlp_data_exfiltration": + benignRaws = []string{strings.ReplaceAll(fixture.Raw, `subtype="dlp"`, `subtype="webfilter"`), strings.NewReplacer(`action="block"`, `action="allow"`, "DLP sensor blocked transfer", "ordinary transfer").Replace(fixture.Raw)} + case "antivirus_outbreak_detection": + benignRaws = []string{strings.ReplaceAll(fixture.Raw, `eventtype="infected"`, `eventtype="analytics"`), strings.ReplaceAll(fixture.Raw, `action="blocked"`, `action="passthrough"`)} + } + if len(benignRaws) == 0 { + t.Fatal("missing non-candidate raw history") + } + for i, raw := range benignRaws { + benign := parse(raw) + if match, e := cache.Eval(rule.Where, benign); e != nil || match || gjson.Get(benign, marker).Exists() { + t.Fatalf("benign raw event retains a candidate: %v %v", match, e) + } + if gjson.Get(benign, "origin.ip").String() != gjson.Get(out, "origin.ip").String() { + t.Fatal("benign history must retain the same IP to isolate candidate semantics") + } + check(fmt.Sprintf("non_candidate_raw_%d", i), mutate(benign, "@timestamp", time.Now().Add(-time.Minute).UTC().Format(time.RFC3339Nano)), search.Count, false) + } + } + identityFields := []string{"origin.ip", "log.devid", "log.vd"} + if test.rule == "admin_account_compromise" { + identityFields = append(identityFields, "origin.user") + } + for _, field := range identityFields { + t.Run("missing_"+field, func(t *testing.T) { + missing := mutate(out, field, nil) + if match, e := cache.Eval(rule.Where, missing); e != nil || match { + t.Fatalf("where must reject missing identity before history: %v %v", match, e) + } + // Reproduce the SDK failure that the predicate guard prevents. + if yes, _, e := search.Execute(&missing); e == nil || yes { + t.Error("missing identity must fail placeholder resolution") + } + }) + } + }) + } +} diff --git a/plugins/alerts/testdata/filter-contracts/fortigate.json b/plugins/alerts/testdata/filter-contracts/fortigate.json new file mode 100644 index 000000000..52815a8ea --- /dev/null +++ b/plugins/alerts/testdata/filter-contracts/fortigate.json @@ -0,0 +1,129 @@ +{ + "technology": "FortiGate", + "filters": [ + "filters/fortinet/fortinet.yml" + ], + "rules": [ + "rules/fortinet/fortinet/admin_account_compromise.yml", + "rules/fortinet/fortinet/antivirus_outbreak_detection.yml", + "rules/fortinet/fortinet/dlp_data_exfiltration.yml", + "rules/fortinet/fortinet/fortigate_admin_session_anomaly.yml", + "rules/fortinet/fortinet/fortigate_vpn_brute_force.yml", + "rules/fortinet/fortinet/ips_critical_severity_events.yml", + "rules/fortinet/fortinet/sandbox_malicious_verdict.yml" + ], + "fixtures": [ + { + "name": "FortiGate isolated normalized action accept", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "accept" + }, + "expected": { + "actionResult": "success" + }, + "absent": [], + "rules": {} + }, + { + "name": "FortiGate isolated normalized action deny", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "deny" + }, + "expected": { + "actionResult": "denied" + }, + "absent": [], + "rules": {} + }, + { + "name": "FortiGate isolated normalized action blocked", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "blocked" + }, + "expected": { + "actionResult": "denied" + }, + "absent": [], + "rules": {} + }, + { + "name": "FortiGate isolated normalized action dns", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "dns" + }, + "expected": {}, + "absent": [ + "actionResult" + ], + "rules": {} + }, + { + "name": "FortiGate isolated normalized action ip-conn", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "ip-conn" + }, + "expected": {}, + "absent": [ + "actionResult" + ], + "rules": {} + }, + { + "name": "FortiGate isolated normalized action client-rst", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "client-rst" + }, + "expected": {}, + "absent": [ + "actionResult" + ], + "rules": {} + }, + { + "name": "FortiGate isolated normalized action timeout", + "filter": "fortinet/fortinet.yml", + "input": { + "action": "timeout" + }, + "expected": {}, + "absent": [ + "actionResult" + ], + "rules": {} + }, + { + "name": "FortiGate numeric string protocol 6", + "filter": "fortinet/fortinet.yml", + "input": { + "protocol": "6" + }, + "expected": { + "protocol": "TCP" + }, + "absent": [ + "log.ipProtocolNumber" + ], + "rules": {} + }, + { + "name": "FortiGate numeric string protocol 17", + "filter": "fortinet/fortinet.yml", + "input": { + "protocol": "17" + }, + "expected": { + "protocol": "UDP" + }, + "absent": [ + "log.ipProtocolNumber" + ], + "rules": {} + } + ] +} diff --git a/plugins/alerts/testdata/fortigate_raw.json b/plugins/alerts/testdata/fortigate_raw.json new file mode 100644 index 000000000..3dcbb69f6 --- /dev/null +++ b/plugins/alerts/testdata/fortigate_raw.json @@ -0,0 +1,833 @@ +[ + { + "name": "quoted policy allow and numeric fields", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" subtype=\"forward\" action=\"accept\" srcip=\"192.0.2.10\" dstip=\"198.51.100.20\" srcport=\"49152\" dstport=\"443\" proto=\"6\" sentbyte=\"42\" rcvdbyte=\"84\" sentpkt=\"2\" rcvdpkt=\"3\"", + "expected": { + "origin.ip": "192.0.2.10", + "target.ip": "198.51.100.20", + "origin.port": 49152, + "target.port": 443, + "protocol": "TCP", + "origin.bytesSent": 42, + "origin.bytesReceived": 84, + "origin.packagesSent": "2", + "origin.packagesReceived": "3", + "actionResult": "success" + }, + "absent": [], + "matches": [] + }, + { + "name": "outcome deny", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"deny\"", + "expected": { + "action": "deny", + "actionResult": "denied" + }, + "absent": [], + "matches": [] + }, + { + "name": "outcome blocked", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"blocked\"", + "expected": { + "action": "blocked", + "actionResult": "denied" + }, + "absent": [], + "matches": [] + }, + { + "name": "outcome drop", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"drop\"", + "expected": { + "action": "drop", + "actionResult": "denied" + }, + "absent": [], + "matches": [] + }, + { + "name": "outcome accept", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"accept\"", + "expected": { + "action": "accept", + "actionResult": "success" + }, + "absent": [], + "matches": [] + }, + { + "name": "quoted profile denial overrides policy allow", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"accept\" utmaction=\"block\"", + "expected": { + "actionResult": "denied", + "log.utmaction": "block" + }, + "absent": [], + "matches": [] + }, + { + "name": "no invented result from dns", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" action=\"dns\"", + "expected": { + "action": "dns" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "no invented result from ip-conn", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" action=\"ip-conn\"", + "expected": { + "action": "ip-conn" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "no invented result from unknown", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" action=\"unknown\"", + "expected": { + "action": "unknown" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "no invented result from info", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" action=\"info\"", + "expected": { + "action": "info" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "no invented result from detected", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" action=\"detected\"", + "expected": { + "action": "detected" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "no invented result from analytics", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" action=\"analytics\"", + "expected": { + "action": "analytics" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "documented failed connection dns", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" logid=\"0000000011\" action=\"dns\"", + "expected": { + "actionResult": "failure" + }, + "absent": [], + "matches": [] + }, + { + "name": "documented failed connection ip-conn", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" logid=\"0000000011\" action=\"ip-conn\"", + "expected": { + "actionResult": "failure" + }, + "absent": [], + "matches": [] + }, + { + "name": "closure close", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"close\"", + "expected": { + "connectionStatus": "closed" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "closure client-rst", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"client-rst\"", + "expected": { + "connectionStatus": "closed" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "closure server-rst", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"server-rst\"", + "expected": { + "connectionStatus": "closed" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "closure timeout", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"timeout\"", + "expected": { + "connectionStatus": "closed" + }, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "invalid address retained -", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=- dstip=-", + "expected": { + "log.srcip": "-", + "log.dstip": "-" + }, + "absent": [ + "origin.ip", + "target.ip", + "log.unparsedOriginIp", + "log.unparsedTargetIp" + ], + "matches": [] + }, + { + "name": "invalid address retained not-an-ip", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=not-an-ip dstip=not-an-ip", + "expected": { + "log.srcip": "not-an-ip", + "log.dstip": "not-an-ip" + }, + "absent": [ + "origin.ip", + "target.ip", + "log.unparsedOriginIp", + "log.unparsedTargetIp" + ], + "matches": [] + }, + { + "name": "invalid address retained 0.0.0.0", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=0.0.0.0 dstip=0.0.0.0", + "expected": { + "log.srcip": "0.0.0.0", + "log.dstip": "0.0.0.0" + }, + "absent": [ + "origin.ip", + "target.ip", + "log.unparsedOriginIp", + "log.unparsedTargetIp" + ], + "matches": [] + }, + { + "name": "invalid address retained ::", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=:: dstip=::", + "expected": { + "log.srcip": "::", + "log.dstip": "::" + }, + "absent": [ + "origin.ip", + "target.ip", + "log.unparsedOriginIp", + "log.unparsedTargetIp" + ], + "matches": [] + }, + { + "name": "IPv6 addresses", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=2001:db8::10 dstip=2001:db8::20", + "expected": { + "origin.ip": "2001:db8::10", + "target.ip": "2001:db8::20" + }, + "absent": [], + "matches": [] + }, + { + "name": "native IP takes precedence over CEF alias", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=192.0.2.10 src=203.0.113.10 dstip=198.51.100.20 dst=203.0.113.20", + "expected": { + "origin.ip": "192.0.2.10", + "target.ip": "198.51.100.20" + }, + "absent": [], + "matches": [] + }, + { + "name": "invalid native selection does not fall back to CEF", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcip=- src=203.0.113.10 dstip=- dst=203.0.113.20", + "expected": { + "log.srcip": "-", + "log.dstip": "-" + }, + "absent": [ + "origin.ip", + "target.ip" + ], + "matches": [] + }, + { + "name": "message final field recovered", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"accept\" msg=\"message spans words\"", + "expected": { + "log.msg": "message spans words", + "actionResult": "success" + }, + "absent": [], + "matches": [] + }, + { + "name": "message middle field recovered", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" msg=\"message spans words\" type=\"traffic\" action=\"accept\"", + "expected": { + "log.msg": "message spans words", + "actionResult": "success" + }, + "absent": [], + "matches": [] + }, + { + "name": "quoted embedded keys cannot create standard fields", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" msg=\"srcip=203.0.113.6 dstip=203.0.113.7 srcname=forged action=accept logdesc=Admin login successful\"", + "expected": {}, + "absent": [ + "origin.ip", + "target.ip", + "origin.host", + "action", + "log.logdesc" + ], + "matches": [] + }, + { + "name": "quoted message cannot override real values", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" srcip=192.0.2.10 action=\"deny\" msg=\"srcip=203.0.113.6 action=accept\"", + "expected": { + "origin.ip": "192.0.2.10", + "actionResult": "denied" + }, + "absent": [], + "matches": [] + }, + { + "name": "unterminated quoted identity not promoted", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" srcname=\"incomplete identity srcip=203.0.113.6", + "expected": {}, + "absent": [ + "origin.ip", + "origin.host" + ], + "matches": [] + }, + { + "name": "quoted CEF text is payload", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" action=\"deny\" msg=\"CEF: 0|Fortinet|Fortigate|7|1|traffic|3|src=203.0.113.6 act=accept\"", + "expected": { + "actionResult": "denied" + }, + "absent": [ + "log.cefVersion", + "origin.ip" + ], + "matches": [] + }, + { + "name": "additional observed standard fields", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" srcname=\"client-one\" dstname=\"server-one\" osname=\"UniFi Firmware\" dstosname=\"Server OS\" dstmac=\"00:11:22:33:44:55\" user=\"lab-user\" group=\"Security Team\" policyname=\"Outbound policy\" app=\"Named App\"", + "expected": { + "origin.host": "client-one", + "target.host": "server-one", + "origin.operatingSystem": "UniFi Firmware", + "target.operatingSystem": "Server OS", + "target.mac": "00:11:22:33:44:55", + "origin.user": "lab-user", + "origin.group": "Security Team", + "log.policyname": "Outbound policy", + "log.app": "Named App" + }, + "absent": [], + "matches": [] + }, + { + "name": "unknown identities not promoted", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" user=\"N/A\" group=\"N/A\" srcname=\"unknown\" dstname=\"-\"", + "expected": {}, + "absent": [ + "origin.user", + "origin.group", + "origin.host", + "target.host" + ], + "matches": [] + }, + { + "name": "absolute URL and destination domain", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"webfilter\" hostname=\"service.example\" url=\"https://service.example/path\"", + "expected": { + "target.url": "https://service.example/path", + "target.domain": "service.example" + }, + "absent": [], + "matches": [] + }, + { + "name": "relative URL embedded scheme stays vendor", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"webfilter\" hostname=\"198.51.100.20\" url=\"/redirect/https://service.example/path\"", + "expected": { + "log.url": "/redirect/https://service.example/path" + }, + "absent": [ + "target.url", + "target.domain", + "target.path" + ], + "matches": [] + }, + { + "name": "admin success positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=203.0.113.10 action=\"login\" status=\"success\"", + "expected": { + "actionResult": "success", + "origin.user": "admin-lab", + "log.logdesc": "Admin login successful" + }, + "absent": [], + "matches": [ + "admin_account_compromise", + "fortigate_admin_session_anomaly" + ] + }, + { + "name": "admin failure history record", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032002\" logdesc=\"Admin login failed\" user=\"admin-lab\" srcip=203.0.113.10 action=\"login\" status=\"failed\"", + "expected": { + "actionResult": "failure" + }, + "absent": [], + "matches": [] + }, + { + "name": "local admin not unusual 10.0.0.10", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=10.0.0.10 action=\"login\" status=\"success\"", + "expected": {}, + "absent": [], + "matches": [ + "admin_account_compromise" + ] + }, + { + "name": "local admin not unusual 127.0.0.1", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=127.0.0.1 action=\"login\" status=\"success\"", + "expected": {}, + "absent": [], + "matches": [ + "admin_account_compromise" + ] + }, + { + "name": "local admin not unusual ::1", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=::1 action=\"login\" status=\"success\"", + "expected": {}, + "absent": [], + "matches": [ + "admin_account_compromise" + ] + }, + { + "name": "local admin not unusual fd00::10", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=fd00::10 action=\"login\" status=\"success\"", + "expected": {}, + "absent": [], + "matches": [ + "admin_account_compromise" + ] + }, + { + "name": "local admin not unusual fe80::10", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=fe80::10 action=\"login\" status=\"success\"", + "expected": {}, + "absent": [], + "matches": [ + "admin_account_compromise" + ] + }, + { + "name": "public IPv6 admin positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" logid=\"0100032001\" logdesc=\"Admin login successful\" user=\"admin-lab\" srcip=2001:4860:4860::8888 action=\"login\" status=\"success\"", + "expected": {}, + "absent": [], + "matches": [ + "admin_account_compromise", + "fortigate_admin_session_anomaly" + ] + }, + { + "name": "VPN authentication positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"vpn\" logid=\"0101039426\" action=\"ssl-login-fail\" logdesc=\"SSL VPN login fail\" remip=203.0.113.10 user=\"vpn-user\" group=\"Remote Users\" msg=\"SSL user failed to log in\"", + "expected": { + "origin.ip": "203.0.113.10", + "origin.user": "vpn-user", + "actionResult": "failure" + }, + "absent": [], + "matches": [ + "fortigate_vpn_brute_force" + ] + }, + { + "name": "VPN exact failure ID without action", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"vpn\" logid=\"0101039426\" logdesc=\"SSL VPN login fail\" remip=203.0.113.10 user=\"vpn-user\" group=\"Remote Users\" msg=\"SSL user failed to log in\"", + "expected": { + "origin.ip": "203.0.113.10" + }, + "absent": [], + "matches": [ + "fortigate_vpn_brute_force" + ] + }, + { + "name": "VPN placeholder user allowed peer correlation", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"vpn\" logid=\"0101039426\" action=\"ssl-login-fail\" logdesc=\"SSL VPN login fail\" remip=203.0.113.10 user=\"N/A\" group=\"Remote Users\" msg=\"SSL user failed to log in\"", + "expected": {}, + "absent": [ + "origin.user" + ], + "matches": [ + "fortigate_vpn_brute_force" + ] + }, + { + "name": "routine VPN tunnel shutdown", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"vpn\" action=\"tunnel-down\" logdesc=\"SSL VPN tunnel down\" remip=203.0.113.10", + "expected": {}, + "absent": [ + "origin.ip", + "actionResult" + ], + "matches": [] + }, + { + "name": "IPsec negotiation error is not authentication proof", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"vpn\" logid=\"0101037124\" action=\"negotiate\" status=\"negotiate_error\" remip=203.0.113.10 msg=\"IPsec phase 1 error\"", + "expected": {}, + "absent": [ + "origin.ip" + ], + "matches": [] + }, + { + "name": "IPS critical blocked positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"ips\" eventtype=\"signature\" severity=\"critical\" action=\"dropped\" srcip=203.0.113.10 dstip=192.0.2.20 msg=\"attack signature detected\"", + "expected": { + "actionResult": "denied", + "log.severity": "critical" + }, + "absent": [], + "matches": [ + "ips_critical_severity_events" + ] + }, + { + "name": "IPS low blocked negative", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"ips\" eventtype=\"signature\" severity=\"low\" action=\"dropped\" srcip=203.0.113.10 dstip=192.0.2.20 msg=\"attack signature detected\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "IPS detection alone is not denial", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"ips\" eventtype=\"signature\" severity=\"critical\" action=\"detected\" srcip=203.0.113.10 dstip=192.0.2.20 msg=\"attack signature detected\"", + "expected": {}, + "absent": [ + "actionResult" + ], + "matches": [] + }, + { + "name": "IPS absent actor cannot resolve history", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"ips\" eventtype=\"signature\" severity=\"critical\" action=\"dropped\" dstip=192.0.2.20 msg=\"attack signature detected\"", + "expected": {}, + "absent": [ + "origin.ip" + ], + "matches": [] + }, + { + "name": "DLP blocked positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"dlp\" action=\"block\" srcip=192.0.2.10 dstip=198.51.100.20 profile=\"sensitive-data\" msg=\"DLP sensor blocked transfer\"", + "expected": {}, + "absent": [], + "matches": [ + "dlp_data_exfiltration" + ] + }, + { + "name": "DLP unrelated administrative message", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"event\" subtype=\"system\" srcip=192.0.2.10 msg=\"DLP sensor changed\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "non-DLP sensitive-data message", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"webfilter\" srcip=192.0.2.10 msg=\"sensitive data detected\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "DLP allowed benign", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"dlp\" action=\"allow\" srcip=192.0.2.10 msg=\"ordinary transfer\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "infected-file outbreak positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"virus\" eventtype=\"infected\" action=\"blocked\" srcip=192.0.2.10 dstip=198.51.100.20 virus=\"EICAR_TEST_FILE\"", + "expected": {}, + "absent": [], + "matches": [ + "antivirus_outbreak_detection" + ] + }, + { + "name": "blocked non-malware analytics", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"virus\" eventtype=\"analytics\" action=\"blocked\" srcip=192.0.2.10 dstip=198.51.100.20 virus=\"EICAR_TEST_FILE\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "blocked non-malware oversize", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"virus\" eventtype=\"oversize\" action=\"blocked\" srcip=192.0.2.10 dstip=198.51.100.20 virus=\"EICAR_TEST_FILE\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "blocked non-malware scanerror", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"virus\" eventtype=\"scanerror\" action=\"blocked\" srcip=192.0.2.10 dstip=198.51.100.20 virus=\"EICAR_TEST_FILE\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "blocked non-malware file-type", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"virus\" eventtype=\"file-type\" action=\"blocked\" srcip=192.0.2.10 dstip=198.51.100.20 virus=\"EICAR_TEST_FILE\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "virus target-only cannot prove same source", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"virus\" eventtype=\"infected\" action=\"blocked\" dstip=198.51.100.20 virus=\"EICAR_TEST_FILE\"", + "expected": {}, + "absent": [ + "origin.ip" + ], + "matches": [] + }, + { + "name": "sandbox malicious positive", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"sandbox\" verdict=\"malicious\" filehash=\"lab-file-hash\" srcip=192.0.2.10", + "expected": {}, + "absent": [], + "matches": [ + "sandbox_malicious_verdict" + ] + }, + { + "name": "sandbox clean negative", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"utm\" subtype=\"sandbox\" verdict=\"clean\" srcip=192.0.2.10", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "missing history scope devid", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 vd=\"root\" type=\"event\" subtype=\"vpn\" logid=\"0101039426\" action=\"ssl-login-fail\" logdesc=\"SSL VPN login fail\" remip=203.0.113.10 user=\"vpn-user\" group=\"Remote Users\" msg=\"SSL user failed to log in\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "missing history scope vd", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" type=\"event\" subtype=\"vpn\" logid=\"0101039426\" action=\"ssl-login-fail\" logdesc=\"SSL VPN login fail\" remip=203.0.113.10 user=\"vpn-user\" group=\"Remote Users\" msg=\"SSL user failed to log in\"", + "expected": {}, + "absent": [], + "matches": [] + }, + { + "name": "documented CEF header extension", + "dataSource": "firewall-lab", + "raw": "Dec 27 11:20:48 FGT-LAB CEF: 0|Fortinet|Fortigate|7.4|08192|utm:virus infected blocked|4|deviceExternalId=FGT-LAB FTNTFGTvd=root cat=utm:virus FTNTFGTsubtype=virus FTNTFGTeventtype=infected FTNTFGTlevel=warning src=192.0.2.10 dst=198.51.100.20 spt=49152 dpt=80 act=blocked proto=6 msg=File is infected. FTNTFGTvirus=EICAR_TEST_FILE", + "expected": { + "origin.ip": "192.0.2.10", + "target.ip": "198.51.100.20", + "log.devid": "FGT-LAB", + "log.msg": "File is infected.", + "protocol": "TCP" + }, + "absent": [], + "matches": [ + "antivirus_outbreak_detection" + ] + }, + { + "name": "native without syslog priority", + "dataSource": "firewall-lab", + "raw": "date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" srcip=192.0.2.10 dstip=198.51.100.20 action=\"deny\"", + "expected": { + "origin.ip": "192.0.2.10", + "actionResult": "denied" + }, + "absent": [], + "matches": [] + }, + { + "name": "no PRI with HTML in final message", + "dataSource": "firewall-lab", + "raw": "date=2026-09-17 time=12:34:56 devid=\"FGT-LAB\" vd=\"root\" type=\"traffic\" srcip=192.0.2.10 action=\"deny\" msg=\"\"", + "expected": { + "origin.ip": "192.0.2.10", + "log.type": "traffic", + "actionResult": "denied", + "log.msg": "" + }, + "absent": [ + "log.priority" + ], + "matches": [] + }, + { + "name": "unspecified address representation 0:0:0:0:0:0:0:0", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 devid=\"FGT-LAB\" vd=\"root\" srcip=0:0:0:0:0:0:0:0 dstip=0:0:0:0:0:0:0:0", + "expected": { + "log.srcip": "0:0:0:0:0:0:0:0", + "log.dstip": "0:0:0:0:0:0:0:0" + }, + "absent": [ + "origin.ip", + "target.ip" + ], + "matches": [] + }, + { + "name": "unspecified address representation ::0", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 devid=\"FGT-LAB\" vd=\"root\" srcip=::0 dstip=::0", + "expected": { + "log.srcip": "::0", + "log.dstip": "::0" + }, + "absent": [ + "origin.ip", + "target.ip" + ], + "matches": [] + }, + { + "name": "unspecified address representation ::ffff:0.0.0.0", + "dataSource": "firewall-lab", + "raw": "<189>date=2026-09-17 devid=\"FGT-LAB\" vd=\"root\" srcip=::ffff:0.0.0.0 dstip=::ffff:0.0.0.0", + "expected": { + "log.srcip": "::ffff:0.0.0.0", + "log.dstip": "::ffff:0.0.0.0" + }, + "absent": [ + "origin.ip", + "target.ip" + ], + "matches": [] + } +] diff --git a/rules/fortinet/fortinet/admin_account_compromise.yml b/rules/fortinet/fortinet/admin_account_compromise.yml index 23628f707..48b719905 100644 --- a/rules/fortinet/fortinet/admin_account_compromise.yml +++ b/rules/fortinet/fortinet/admin_account_compromise.yml @@ -25,26 +25,43 @@ description: | 6. Enable multi-factor authentication if not already configured 7. Reset the compromised admin account password if unauthorized access is confirmed where: | - equals("log.logid", "0100032001") && - equals("log.type", "event") && - equals("log.subtype", "system") && - equals("log.logdesc", "Admin login successful") && - exists("origin.ip") && - exists("log.user") + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.logid", "0100032001") && + equals("log.type", "event") && + equals("log.subtype", "system") && + equals("log.logdesc", "Admin login successful") && + exists("origin.ip") && exists("origin.user")) afterEvents: - - indexPattern: v11-log-firewall-fortigate-traffic-* - with: - - field: origin.ip - operator: filter_term - value: '{{.origin.ip}}' - - field: log.logdesc - operator: filter_term - value: 'Admin login failed' - - field: log.user - operator: filter_term - value: '{{.log.user}}' - within: 15m - count: 5 +- indexPattern: v11-log-firewall-fortigate-traffic-* + with: + - field: log.devid + operator: filter_term + value: '{{.log.devid}}' + - field: log.vd + operator: filter_term + value: '{{.log.vd}}' + - field: dataSource + operator: filter_term + value: '{{.dataSource}}' + - field: origin.ip + operator: filter_term + value: '{{.origin.ip}}' + - field: origin.user + operator: filter_term + value: '{{.origin.user}}' + - field: log.type + operator: filter_term + value: event + - field: log.subtype + operator: filter_term + value: system + - field: log.logid + operator: filter_term + value: '0100032002' + within: 15m + count: 5 groupBy: - - adversary.ip - - target.user +- adversary.ip +- adversary.user +- lastEvent.log.devid +- lastEvent.log.vd diff --git a/rules/fortinet/fortinet/antivirus_outbreak_detection.yml b/rules/fortinet/fortinet/antivirus_outbreak_detection.yml index 7e5d7a8ea..cf40c0cbf 100644 --- a/rules/fortinet/fortinet/antivirus_outbreak_detection.yml +++ b/rules/fortinet/fortinet/antivirus_outbreak_detection.yml @@ -14,7 +14,7 @@ references: - https://docs.fortinet.com/document/fortigate/7.4.1/administration-guide/198726/antivirus - https://attack.mitre.org/techniques/T1566/ description: | - Detects potential malware outbreak when multiple antivirus detections occur from the same source or targeting multiple destinations in a short time period. This could indicate a widespread infection attempt or a compromised system attempting to spread malware. + Detects potential malware outbreak when multiple antivirus detections occur from the same source IP on the same appliance and virtual domain in a short time period. This indicates repeated blocked infected-file transfers, not proof of an infected client or successful delivery. Next Steps: 1. Identify the specific malware detected by reviewing the virus field in the alert details @@ -26,24 +26,29 @@ description: | 7. Consider blocking the source IP at the perimeter if it's external 8. Document all affected systems and remediation actions taken where: | - equals("log.type", "utm") && - equals("log.subtype", "virus") && - equals("action", "blocked") && - (exists("origin.ip") || exists("target.ip")) + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.type", "utm") && equals("log.subtype", "virus") && + equals("log.eventtype", "infected") && equals("actionResult", "denied") && + exists("origin.ip")) afterEvents: - - indexPattern: v11-log-firewall-fortigate-traffic-* - with: - - field: log.type - operator: filter_term - value: utm - - field: log.subtype - operator: filter_term - value: virus - - field: action - operator: filter_term - value: blocked - within: 1h - count: 5 +- indexPattern: v11-log-firewall-fortigate-traffic-* + with: + - field: log.devid + operator: filter_term + value: '{{.log.devid}}' + - field: log.vd + operator: filter_term + value: '{{.log.vd}}' + - field: origin.ip + operator: filter_term + value: '{{.origin.ip}}' + - field: log.correlationCandidate.virusOutbreak + operator: filter_term + value: match + within: 1h + count: 5 groupBy: - - adversary.ip - - target.ip +- adversary.ip +- target.ip +- lastEvent.log.devid +- lastEvent.log.vd diff --git a/rules/fortinet/fortinet/dlp_data_exfiltration.yml b/rules/fortinet/fortinet/dlp_data_exfiltration.yml index 730da1f23..d536d3011 100644 --- a/rules/fortinet/fortinet/dlp_data_exfiltration.yml +++ b/rules/fortinet/fortinet/dlp_data_exfiltration.yml @@ -25,22 +25,36 @@ description: | 6. Contact the user if needed to understand the context of the attempted data transfer 7. Consider implementing additional access controls if unauthorized access to suspected where: | - (equals("log.subtype", "dlp") && equals("log.action", "blocked")) || - (contains("log.msg", "DLP sensor")) || - (contains("log.msg", "sensitive data detected")) || - (exists("log.dlp_profile") && equals("log.dlp_action", "block")) || - (contains("log.msg", "Data pattern matched")) + equals("log.type", "utm") && equals("log.subtype", "dlp") && + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (exists("origin.ip") && + ((equals("log.subtype", "dlp") && oneOf("action", ["block", "blocked"])) || + contains("log.msg", "DLP sensor") || + contains("log.msg", "sensitive data detected") || + (exists("log.dlpprofile") && equals("log.dlpaction", "block")) || + contains("log.msg", "Data pattern matched"))) afterEvents: - - indexPattern: v11-log-firewall-fortigate-traffic-* - with: - - field: origin.ip - operator: filter_term - value: '{{.origin.ip}}' - - field: log.subtype - operator: filter_term - value: 'dlp' - within: 1h - count: 3 +- indexPattern: v11-log-firewall-fortigate-traffic-* + with: + - field: log.devid + operator: filter_term + value: '{{.log.devid}}' + - field: log.vd + operator: filter_term + value: '{{.log.vd}}' + - field: dataSource + operator: filter_term + value: '{{.dataSource}}' + - field: origin.ip + operator: filter_term + value: '{{.origin.ip}}' + - field: log.correlationCandidate.dlp + operator: filter_term + value: match + within: 1h + count: 3 groupBy: - - lastEvent.log.dlp_profile - - adversary.ip +- lastEvent.log.profile +- adversary.ip +- lastEvent.log.devid +- lastEvent.log.vd diff --git a/rules/fortinet/fortinet/fortigate_admin_session_anomaly.yml b/rules/fortinet/fortinet/fortigate_admin_session_anomaly.yml index c6a488ff2..4ca1fae79 100644 --- a/rules/fortinet/fortinet/fortigate_admin_session_anomaly.yml +++ b/rules/fortinet/fortinet/fortigate_admin_session_anomaly.yml @@ -32,7 +32,10 @@ where: | !inCIDR("origin.ip", "10.0.0.0/8") && !inCIDR("origin.ip", "172.16.0.0/12") && !inCIDR("origin.ip", "192.168.0.0/16") && - !inCIDR("origin.ip", "127.0.0.0/8") + !inCIDR("origin.ip", "127.0.0.0/8") && + !inCIDR("origin.ip", "::1/128") && + !inCIDR("origin.ip", "fc00::/7") && + !inCIDR("origin.ip", "fe80::/10") groupBy: - - adversary.ip - - adversary.user +- adversary.ip +- adversary.user diff --git a/rules/fortinet/fortinet/fortigate_vpn_brute_force.yml b/rules/fortinet/fortinet/fortigate_vpn_brute_force.yml index 42ffef343..5102fa7ac 100644 --- a/rules/fortinet/fortinet/fortigate_vpn_brute_force.yml +++ b/rules/fortinet/fortinet/fortigate_vpn_brute_force.yml @@ -14,7 +14,7 @@ references: - https://docs.fortinet.com/document/fortigate/7.4.1/fortios-log-message-reference/160372/list-of-log-types-and-subtypes - https://attack.mitre.org/techniques/T1133/ description: | - Detects multiple SSL-VPN or IPsec VPN authentication failures from the same source IP on FortiGate, indicating brute force or credential stuffing attacks against remote access services. + Detects multiple SSL-VPN SSL-VPN authentication failures from the same source IP on FortiGate, indicating brute force or credential stuffing attacks against remote access services. Next Steps: 1. Investigate the source IP and check threat intelligence databases @@ -23,20 +23,35 @@ description: | 4. Consider blocking the source IP or geo-restricting VPN access 5. Enable MFA for VPN access if not already configured 6. Check if the VPN portal is exposed to the internet unnecessarily + + Generic IPsec negotiation errors and routine tunnel shutdowns are not authentication-failure evidence for this rule. where: | - equals("log.type", "event") && - (equals("log.subtype", "vpn") || contains("log.msg", "vpn")) && - (contains("log.msg", ["ssl-login-fail", "tunnel-down", "negotiation-error", "auth-fail"]) || - contains("log.logdesc", ["SSL VPN login fail", "IPsec phase1 negotiation failed", "VPN tunnel down"])) && - exists("origin.ip") + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.type", "event") && equals("log.subtype", "vpn") && + (equals("action", "ssl-login-fail") || equals("log.logid", "0101039426")) && + exists("origin.ip")) afterEvents: - - indexPattern: v11-log-firewall-fortigate-traffic-* - with: - - field: origin.ip - operator: filter_term - value: '{{.origin.ip}}' - within: 15m - count: 10 +- indexPattern: v11-log-firewall-fortigate-traffic-* + with: + - field: log.devid + operator: filter_term + value: '{{.log.devid}}' + - field: log.vd + operator: filter_term + value: '{{.log.vd}}' + - field: dataSource + operator: filter_term + value: '{{.dataSource}}' + - field: origin.ip + operator: filter_term + value: '{{.origin.ip}}' + - field: log.correlationCandidate.vpnAuthFailure + operator: filter_term + value: match + within: 15m + count: 10 groupBy: - - adversary.ip - - adversary.user +- adversary.ip +- adversary.user +- lastEvent.log.devid +- lastEvent.log.vd diff --git a/rules/fortinet/fortinet/ips_critical_severity_events.yml b/rules/fortinet/fortinet/ips_critical_severity_events.yml index 0c49f005d..f08ab9fe5 100644 --- a/rules/fortinet/fortinet/ips_critical_severity_events.yml +++ b/rules/fortinet/fortinet/ips_critical_severity_events.yml @@ -24,15 +24,33 @@ description: | 5. Consider blocking the source IP at the perimeter if multiple critical attacks are detected 6. Update IPS signatures and verify all systems are patched against the detected vulnerability 7. Document the incident and notify the security team if this is part of a larger campaign -where: contains("log.msg", "type=utm") && contains("log.msg", "subtype=ips") && (contains("log.msg", "severity=critical") || contains("log.msg", "severity=high")) && contains("log.msg", "action=blocked") +where: | + exists("log.devid") && !equals("log.devid", "") && exists("log.vd") && !equals("log.vd", "") && + (equals("log.type", "utm") && equals("log.subtype", "ips") && + oneOf("log.severity", ["critical", "high"]) && + equals("actionResult", "denied") && exists("origin.ip")) afterEvents: - - indexPattern: v11-log-firewall-fortigate-traffic-* - with: - - field: origin.ip - operator: filter_term - value: '{{.origin.ip}}' - within: 15m - count: 3 +- indexPattern: v11-log-firewall-fortigate-traffic-* + with: + - field: log.devid + operator: filter_term + value: '{{.log.devid}}' + - field: log.vd + operator: filter_term + value: '{{.log.vd}}' + - field: dataSource + operator: filter_term + value: '{{.dataSource}}' + - field: origin.ip + operator: filter_term + value: '{{.origin.ip}}' + - field: log.correlationCandidate.ipsCritical + operator: filter_term + value: match + within: 15m + count: 3 groupBy: - - adversary.ip - - target.ip +- adversary.ip +- target.ip +- lastEvent.log.devid +- lastEvent.log.vd