Skip to content

Commit 58efc77

Browse files
kryonsxkryonsx
committed
fix: align NetFlow filter and rule contracts
1 parent 6c3af7e commit 58efc77

7 files changed

Lines changed: 250 additions & 9 deletions

File tree

‎filters/audits/netflow.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# NetFlow normalization and rule review
2+
3+
Preserve counters consumed by detection rules, accept extracted counter variants, and align IP/protocol normalization.
4+
5+
This draft targets UTMStack `v11`. It contains 1 filter changes
6+
and 4 rule changes for this technology only. Review covered
7+
1 filter configurations and 12 matching shipped rule files.
8+
Unchanged rules are listed in the regression manifest; they are not duplicated in the diff.
9+
10+
## Contract and validation
11+
12+
- Compared exact standard names/types with go-sdk v1.1.31 and the supplied UTMStack dictionaries.
13+
- Checked documented pipeline ordering, rename/move behavior, open vendor log fields,
14+
event-side versus alert-side fields, and surviving fields used by affected rule predicates/history/grouping.
15+
- Strict SDK configuration decoding and actual CEL compilation pass for this scope.
16+
- 6 synthetic normalization cases pass, including SDK Event conversion and any
17+
trigger predicate assertions recorded in the manifest.
18+
- The scoped alerts module tests and `git diff --check` pass with the shared contract runner applied.
19+
20+
The shared alert-contract PR supplies the reusable Go runner for the manifest in
21+
`plugins/alerts/testdata/filter-contracts/netflow.json`. Apply that support before running `go test ./...` in `plugins/alerts`.
22+
23+
The model starts from synthetic extraction results. It does not run complex grok,
24+
JSON/KV/XML/CSV extraction, time conversion, dynamic plugins, historical OpenSearch
25+
queries, or the closed EventProcessor. Raw vendor logs and resulting alerts must
26+
still be checked in staging before rollout. No customer false-positive reduction
27+
has been measured and no production rollout is included.
28+
29+
The filter previously deleted log.bytes/log.packets that four rules consume. Keep scalar counters and accept the existing extracted totalBytes/totalPackets variants; do not invent totals for multiple-value records.
30+
31+
## References
32+
33+
- [SDK schema](https://github.com/threatwinds/go-sdk/blob/v1.1.31/plugins/plugins.proto)
34+
- [Filter steps](https://github.com/threatwinds/go-sdk/wiki/Filter-Steps-Reference)
35+
- [Standard event schema](https://github.com/threatwinds/go-sdk/wiki/Standard-Event-Schema)
36+
- [Rule implementation](https://github.com/threatwinds/go-sdk/wiki/Implementing-Rules)
37+
38+
`afterEvents`, empty noncapturing grok names, supported numeric strings, and custom
39+
`log.*` fields are accepted. Existing textual protocol casing and vendor action names
40+
are preserved unless a concrete consumer mismatch requires correction.

‎filters/netflow/netflow.yml‎

Lines changed: 66 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1135,8 +1135,6 @@ pipeline:
11351135
# Removing unused fields
11361136
- delete:
11371137
fields:
1138-
- log.bytes
1139-
- log.packets
11401138
- log.inEth
11411139
- log.outEth
11421140
- log.srcPort
@@ -1159,4 +1157,69 @@ pipeline:
11591157
- log.irrelevant8
11601158
- log.irrelevant9
11611159
- log.irrelevant10
1162-
- log.irrelevant11
1160+
- log.irrelevant11
1161+
1162+
# Keep addresses in IP fields and retain other source values under log.
1163+
- rename:
1164+
from:
1165+
- origin.ip
1166+
to: log.unparsedOriginIp
1167+
where: exists("origin.ip") && (!(inCIDR("origin.ip","0.0.0.0/0") || inCIDR("origin.ip","::/0")) || oneOf("origin.ip",["0.0.0.0","::"]))
1168+
- rename:
1169+
from:
1170+
- target.ip
1171+
to: log.unparsedTargetIp
1172+
where: exists("target.ip") && (!(inCIDR("target.ip","0.0.0.0/0") || inCIDR("target.ip","::/0")) || oneOf("target.ip",["0.0.0.0","::"]))
1173+
- add:
1174+
function: string
1175+
params:
1176+
key: protocol
1177+
value: ICMP
1178+
where: equals("protocol",1)
1179+
- add:
1180+
function: string
1181+
params:
1182+
key: protocol
1183+
value: TCP
1184+
where: equals("protocol",6)
1185+
- add:
1186+
function: string
1187+
params:
1188+
key: protocol
1189+
value: UDP
1190+
where: equals("protocol",17)
1191+
- add:
1192+
function: string
1193+
params:
1194+
key: protocol
1195+
value: GRE
1196+
where: equals("protocol",47)
1197+
- add:
1198+
function: string
1199+
params:
1200+
key: protocol
1201+
value: ESP
1202+
where: equals("protocol",50)
1203+
- add:
1204+
function: string
1205+
params:
1206+
key: protocol
1207+
value: AH
1208+
where: equals("protocol",51)
1209+
- add:
1210+
function: string
1211+
params:
1212+
key: protocol
1213+
value: ICMPV6
1214+
where: equals("protocol",58)
1215+
- add:
1216+
function: string
1217+
params:
1218+
key: protocol
1219+
value: SCTP
1220+
where: equals("protocol",132)
1221+
- rename:
1222+
from:
1223+
- protocol
1224+
to: log.ipProtocolNumber
1225+
where: exists("protocol") && greaterOrEqual("protocol",0)

‎plugins/alerts/testdata/filter-contracts/netflow.json‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
{
2+
"technology": "NetFlow",
3+
"filters": [
4+
"filters/netflow/netflow.yml"
5+
],
6+
"rules": [
7+
"rules/netflow/beaconing_behavior_detection.yml",
8+
"rules/netflow/data_exfiltration_indicators.yml",
9+
"rules/netflow/ddos_traffic_patterns.yml",
10+
"rules/netflow/netflow_cryptomining_traffic.yml",
11+
"rules/netflow/netflow_doh_detection.yml",
12+
"rules/netflow/netflow_icmp_tunnel.yml",
13+
"rules/netflow/netflow_internal_scanning.yml",
14+
"rules/netflow/netflow_lateral_movement_smb_rdp.yml",
15+
"rules/netflow/netflow_long_duration_connections.yml",
16+
"rules/netflow/netflow_vpn_unusual_destinations.yml",
17+
"rules/netflow/port_scanning_patterns.yml",
18+
"rules/netflow/tor_usage_detection.yml"
19+
],
20+
"fixtures": [
21+
{
22+
"name": "netflow_valid_source_ip",
23+
"filter": "netflow/netflow.yml",
24+
"input": {
25+
"origin": {
26+
"ip": "2001:db8::7"
27+
}
28+
},
29+
"expected": {
30+
"origin.ip": "2001:db8::7"
31+
},
32+
"absent": [],
33+
"rules": {}
34+
},
35+
{
36+
"name": "netflow_hostname_not_ip",
37+
"filter": "netflow/netflow.yml",
38+
"input": {
39+
"origin": {
40+
"ip": "lab-host"
41+
}
42+
},
43+
"expected": {
44+
"log.unparsedOriginIp": "lab-host"
45+
},
46+
"absent": [
47+
"origin.ip"
48+
],
49+
"rules": {}
50+
},
51+
{
52+
"name": "netflow_numeric_protocol",
53+
"filter": "netflow/netflow.yml",
54+
"input": {
55+
"protocol": 6
56+
},
57+
"expected": {
58+
"protocol": "TCP"
59+
},
60+
"absent": [],
61+
"rules": {}
62+
},
63+
{
64+
"name": "netflow_tor_scalar_counters",
65+
"filter": "netflow/netflow.yml",
66+
"input": {
67+
"origin": {
68+
"ip": "192.0.2.5"
69+
},
70+
"target": {
71+
"ip": "198.51.100.9",
72+
"port": 9001
73+
},
74+
"log": {
75+
"protocol": "6",
76+
"bytes": 2048
77+
}
78+
},
79+
"expected": {
80+
"log.bytes": 2048,
81+
"protocol": "TCP"
82+
},
83+
"absent": [],
84+
"rules": {
85+
"rules/netflow/tor_usage_detection.yml": true
86+
}
87+
},
88+
{
89+
"name": "netflow_tor_extracted_counters",
90+
"filter": "netflow/netflow.yml",
91+
"input": {
92+
"origin": {
93+
"ip": "192.0.2.5"
94+
},
95+
"target": {
96+
"ip": "198.51.100.9",
97+
"port": 9001
98+
},
99+
"log": {
100+
"protocol": "6",
101+
"totalBytes": "2048]\""
102+
}
103+
},
104+
"expected": {
105+
"log.totalBytes": "2048",
106+
"protocol": "TCP"
107+
},
108+
"absent": [],
109+
"rules": {
110+
"rules/netflow/tor_usage_detection.yml": true
111+
}
112+
},
113+
{
114+
"name": "netflow_tor_small_flow",
115+
"filter": "netflow/netflow.yml",
116+
"input": {
117+
"origin": {
118+
"ip": "192.0.2.5"
119+
},
120+
"target": {
121+
"ip": "198.51.100.9",
122+
"port": 9001
123+
},
124+
"log": {
125+
"protocol": "6",
126+
"bytes": 1
127+
}
128+
},
129+
"expected": {
130+
"log.bytes": 1
131+
},
132+
"absent": [],
133+
"rules": {
134+
"rules/netflow/tor_usage_detection.yml": false
135+
}
136+
}
137+
]
138+
}

‎rules/netflow/netflow_icmp_tunnel.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ where: |
2727
exists("origin.ip") &&
2828
exists("target.ip") &&
2929
equals("protocol", "ICMP") &&
30-
greaterThan("log.bytes", 10000)
30+
(greaterThan("log.bytes", 10000) || greaterThan("log.totalBytes", 10000))
3131
afterEvents:
3232
- indexPattern: v11-log-netflow-*
3333
with:

‎rules/netflow/netflow_internal_scanning.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,8 @@ where: |
2828
exists("target.ip") &&
2929
(inCIDR("origin.ip", "10.0.0.0/8") || inCIDR("origin.ip", "172.16.0.0/12") || inCIDR("origin.ip", "192.168.0.0/16")) &&
3030
(inCIDR("target.ip", "10.0.0.0/8") || inCIDR("target.ip", "172.16.0.0/12") || inCIDR("target.ip", "192.168.0.0/16")) &&
31-
lessOrEqual("log.packets", 5) &&
32-
lessThan("log.bytes", 500)
31+
(lessOrEqual("log.packets", 5) || lessOrEqual("log.totalPackets", 5)) &&
32+
(lessThan("log.bytes", 500) || lessThan("log.totalBytes", 500))
3333
afterEvents:
3434
- indexPattern: v11-log-netflow-*
3535
with:

‎rules/netflow/port_scanning_patterns.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@ description: |
2626
where: |
2727
exists("origin.ip") &&
2828
exists("target.ip") &&
29-
lessOrEqual("log.packets", 10) &&
30-
lessThan("log.bytes", 1000) &&
29+
(lessOrEqual("log.packets", 10) || lessOrEqual("log.totalPackets", 10)) &&
30+
(lessThan("log.bytes", 1000) || lessThan("log.totalBytes", 1000)) &&
3131
greaterThan("target.port", 0) &&
3232
equals("protocol", "TCP")
3333
afterEvents:

‎rules/netflow/tor_usage_detection.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ description: |
2727
where: |
2828
exists("origin.ip") && exists("target.ip") &&
2929
oneOf("target.port", [9001, 9030, 9050, 9051]) &&
30-
equals("protocol", "6") && greaterThan("log.bytes", 512)
30+
oneOf("protocol", ["TCP", "tcp", "6"]) && (greaterThan("log.bytes", 512) || greaterThan("log.totalBytes", 512))
3131
afterEvents:
3232
- indexPattern: v11-log-netflow-*
3333
with:

0 commit comments

Comments
 (0)