fix(eset): align JSON parsing and endpoint detection contracts

filters/audits/eset.md

@@ -0,0 +1,134 @@
+# ESET JSON export and detection contracts
+
+This draft targets the official UTMStack v11 repository. It reviews the ESET
+filter and all thirteen existing consumers, with separate inbound/outbound botnet
+handling. The target alerts module pins go-sdk v1.1.31; its protobuf and official
+wiki define the standard. No customer configuration was changed and no alert was
+created during this review.
+
+## Evidence and limits
+
+Fresh read-only counts over the retained ESET source index pattern returned zero
+records on thirty v11 instances. One instance remained unreachable. The query
+failures were not counted as zero; one stale container cache was refreshed before
+its successful retry. No ESET customer raw/normalized pair or deployed filter
+could be compared. This is documented-format and offline contract evidence,
+not a measured recovery of live detections or reduction in false positives.
+
+The substantive primary reference is ESET PROTECT On-Prem 13.1's JSON-export
+specification. The prior 7.0 and 11.0 URLs now redirect to end-of-life notices and
+cannot substantiate historical event variants. ESET explicitly says values vary
+by endpoint application/version and its lists are not exhaustive. The public
+fixtures use fabricated identities and documented field shapes; detection-label
+examples are labeled synthetic, not claimed as observed vendor values.
+
+## Producer corrections
+
+- Recognize bounded bare JSON, the documented ERAServer RFC3164 wrapper and the
+  existing RFC5424 family. Message text containing an embedded header is not an
+  outer envelope. The syslog version is not the ESET event class.
+- Extract the vendor JSON with SDK key sanitization, then expose `log.eventType`
+  from `event_type`. Retain vendor severity, event detail and protected `raw` for
+  investigation; remove the temporary JSON string after extraction. Consumers use structured detection attributes rather than
+  searching a serialized object for attack words.
+- Validate original address fields before standard promotion and geolocation.
+  Invalid and semantically unspecified addresses remain vendor data. IPv6-only
+  reporting endpoints and integer transport-port bounds are covered.
+- Keep the management server's header identity separate from the managed endpoint.
+  Firewall source/destination fields retain their physical roles; the `inbound`
+  boolean determines which side receives the reporting endpoint's host, account
+  and process. Missing direction does not justify pairing a local hostname with
+  a remote IP. Filtered website records describe a local client and remote target.
+- Map documented user, executable, operating-system, object URL and valid SHA-1
+  fields where their roles are established. An Inspect alarm hash is not assumed
+  to identify a file. Static management groups remain vendor fields rather than
+  being mislabeled as security roles.
+- Map unambiguous absolute executable/object paths into directory and basename
+  fields. Keep registry targets, percent-encoded file URIs, remote authorities and
+  URI query/fragment variants under vendor fields. Threat detection name/type map
+  to malware metadata; a bounded firewall CVE signature maps to the destination
+  vulnerability field. `occurred` and the examples' `occured` alias become UTC
+  device time through a guarded temporary field, without reformatting an existing
+  ingress timestamp as vendor text.
+- Preserve native action detail. Explicit blocking means `denied`; audit results
+  describe the audited operation; remediation errors mean `failure`. A handled
+  detection, deleted file or event class does not establish successful malicious
+  execution or a network connection. Vendor severity is retained while standard
+  severity follows Information/Notice, Warning, Error and Critical/Fatal.
+
+## Consumer corrections
+
+The original consumers mixed a numeric header field with event classes, read a
+removed JSON body or severity, and used management-relay identity for endpoint
+history. The revised consumers use the produced canonical class, structured
+vendor detection attributes and standard action/severity fields. All three
+history rules count their own eligible candidates, scoped by collector and a
+namespaced managed-endpoint identity; console login history also scopes the
+account. Ordinary traffic or unrelated audit events cannot satisfy those counts.
+
+Botnet consumers choose the remote side as adversary separately for inbound and
+outbound events without reversing the filter's network fields. Missing direction
+is an explicit coverage gap. The generic network rule excludes that detector
+family to avoid duplicate alerts with contradictory adversary attribution.
+Blocked tampering or exploitation is described as an
+attempt, not proof that protection was disabled or compromise succeeded. Normal
+policy changes and successful console tasks are not evidence of console abuse;
+that consumer detects repeated console authentication failures.
+
+The generic HIPS rule requires a security-specific operation or detector label;
+an ordinary blocked operation under a restrictive policy is insufficient. Specific
+registry and PowerShell rules describe blocked activity, which still requires
+context and policy tuning rather than proving malicious intent.
+
+The heuristic, machine-learning, botnet and behavior labels remain detection
+heuristics over documented fields. Their completeness across product versions is
+not established by this review. An unknown event class remains available under
+vendor fields; unsupported historical classes are not declared impossible.
+
+## Verification
+
+- 155 fabricated raw cases exercise all fourteen consumers with explicit positive
+  and negative expectations, SDK Event conversion and exact history-marker parity.
+  Header, JSON, address, type, calendar, URI and direction boundaries are included.
+  Malformed JSON is a model error control, not proof of the closed engine's error
+  handling. The temporary JSON body is absent and protected raw remains unchanged.
+- Independent SDK/model retests cover opposite-side botnet duplication, an ordinary
+  HIPS policy block, an unrelated agent label, and file-URI query/fragment boundaries.
+- The shared manifest contains two isolated empty/unknown-class negative controls.
+  Its limited normalization model does not execute the anchored extraction used
+  here; raw and positive-consumer proof comes from the standalone source suite.
+
+- Actual SDK history queries pass against an isolated loopback mock for all three
+  consumers: below/at threshold, inside/expired window, unrelated history, cleared
+  forged markers, collector/endpoint/account separation, missing placeholders and
+  UUID-to-host-to-IP fallback. Explicit denied/rejected login results are included.
+- SDK Event/Alert wire-contract checks cover all fourteen positive consumers,
+  produced grouping fields and physical/adversary direction. They assign sides
+  according to rule metadata and do not execute the closed correlation service.
+- The final shared-runner overlay passes 182 test/subtest records, with no failures
+  or skips. `git diff --check` passes. These counts include parent test records and
+  are not unique events or generated alerts.
+
+Stage the filter and consumers together. New history markers require up to one
+hour to populate the longest window. Thresholds count matching indexed documents,
+not underlying occurrences represented by vendor aggregation counts. Endpoint
+identity uses namespaced UUID, hostname, then valid reported IP; the relay hostname
+is never its fallback. Optional detector/file grouping terms can be missing, in
+which case the known endpoint forms a broader group. Shared grouping #2590 remains
+a separate runtime dependency and rollout.
+
+Before production approval, replay native records through the closed collector
+and EventProcessor, verify actual stored field names, history and alert output,
+measure parsing cost and alert volume, and review saved searches that consume
+legacy fields/outcomes. Dynamic geolocation, production history and publication
+were not exercised here. No source-only offline result establishes that an alert
+fired or that every ESET application/version is covered.
+
+## References
+
+- [ESET JSON export fields, semantics and examples](https://help.eset.com/protect_admin/13.1/en-US/events-exported-to-json-format.html)
+- [SDK v1.1.31 protobuf](https://github.com/threatwinds/go-sdk/blob/v1.1.31/plugins/plugins.proto)
+- [SDK field sanitization](https://github.com/threatwinds/go-sdk/blob/v1.1.31/utils/fields.go)
+- [Standard event schema](https://github.com/threatwinds/go-sdk/wiki/Standard-Event-Schema)
+- [Filter steps](https://github.com/threatwinds/go-sdk/wiki/Filter-Steps-Reference)
+- [Rule implementation](https://github.com/threatwinds/go-sdk/wiki/Implementing-Rules)

plugins/alerts/eset_consumers_test.go

@@ -0,0 +1,195 @@
+package main
+
+// SDK Event/Alert wire-contract tests from synthetic ESET JSON. Alert side
+// selection is asserted according to the rule metadata; this is not execution
+// of the closed correlation service and does not prove live alert creation.
+// lastEvent is the indexed alias resolved from events[0] by the companion fix.
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+
+	"github.com/threatwinds/go-sdk/plugins"
+	"github.com/threatwinds/go-sdk/utils"
+	"github.com/tidwall/gjson"
+)
+
+func esetAlertWire(t *testing.T, rule *plugins.Rule, out string) string {
+	t.Helper()
+	event := new(plugins.Event)
+	if err := utils.StringToProtoMessage(&out, event); err != nil {
+		t.Fatal(err)
+	}
+	alert := &plugins.Alert{Events: []*plugins.Event{event}}
+	switch rule.Adversary {
+	case "origin":
+		alert.Adversary, alert.Target = event.Origin, event.Target
+	case "target":
+		alert.Adversary, alert.Target = event.Target, event.Origin
+	default:
+		t.Fatalf("unsupported actor side %q", rule.Adversary)
+	}
+	wire, err := utils.ProtoMessageToString(alert)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return *wire
+}
+
+func esetGroupIdentity(t *testing.T, rule *plugins.Rule, out string) string {
+	t.Helper()
+	wire := esetAlertWire(t, rule, out)
+	var parts []string
+	fields := append(append([]string{}, rule.GroupBy...), rule.DeduplicateBy...)
+	for _, field := range fields {
+		path := strings.Replace(field, "lastEvent.", "events.0.", 1)
+		value := gjson.Get(wire, path)
+		if !value.Exists() || value.String() == "" {
+			// These vendor-specific detector details are documented optional.
+			// The managed endpoint namespace/key and collector remain required.
+			if field == "lastEvent.log.rulename" || field == "lastEvent.log.ruleid" || field == "lastEvent.log.threatname" || field == "lastEvent.log.target" {
+				continue
+			}
+			t.Errorf("positive raw event cannot resolve alert identity field %s", field)
+		}
+		parts = append(parts, field+"="+value.String())
+	}
+	if len(parts) == 0 {
+		t.Fatal("positive rule has no resolved identity fields")
+	}
+	return strings.Join(parts, "\x00")
+}
+
+func TestESETAlertFieldContracts(t *testing.T) {
+	cfg, rules, cache := esetConfig(t), esetRules(t), plugins.NewCELCache("eset-alert-fields")
+	covered := map[string]bool{}
+	for _, fixture := range esetFixtures(t) {
+		if fixture.ParseError || len(fixture.Matches) == 0 {
+			continue
+		}
+		out := esetParse(t, cfg, fixture.Raw, fixture.DataSource, cache)
+		for _, name := range fixture.Matches {
+			rule := rules[name]
+			if rule == nil {
+				t.Fatalf("unknown fixture rule %q", name)
+			}
+			if yes, err := cache.Eval(rule.Where, out); err != nil || !yes {
+				t.Fatalf("%s must match %s before alert checks: %v %v", fixture.Name, name, yes, err)
+			}
+			esetGroupIdentity(t, rule, out)
+			covered[name] = true
+		}
+	}
+	if len(rules) != 14 || len(covered) != len(rules) {
+		t.Fatalf("all14 consumers require raw positive identity coverage: rules=%d covered=%v", len(rules), covered)
+	}
+}
+
+func TestESETDirectionalActorsAndPolicyBoundaries(t *testing.T) {
+	cfg, rules, cache := esetConfig(t), esetRules(t), plugins.NewCELCache("eset-consumer-boundaries")
+	parse := func(m map[string]any) string {
+		b, err := json.Marshal(m)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return esetParse(t, cfg, string(b), "eset-test-relay", cache)
+	}
+	matches := func(out, name string, want bool) {
+		t.Helper()
+		rule := rules[name]
+		if rule == nil {
+			t.Fatalf("rule missing: %s", name)
+		}
+		if yes, err := cache.Eval(rule.Where, out); err != nil || yes != want {
+			t.Fatalf("%s match=%v want=%v: %v", name, yes, want, err)
+		}
+	}
+	base := func(kind string) map[string]any {
+		return map[string]any{"event_type": kind, "hostname": "HOST-A", "source_uuid": "c1c09d6e-24bd-4b45-b1c6-f3c227c9df8e", "ipv4": "192.0.2.10", "severity": "Warning"}
+	}
+	for _, inbound := range []bool{false, true} {
+		event := base("FirewallAggregated_Event")
+		event["inbound"], event["threat_name"], event["event"], event["action"] = inbound, "Botnet.CnC.Generic", "Botnet communication detected", "blocked"
+		event["source_address"], event["target_address"] = "192.0.2.10", "198.51.100.9"
+		event["account"], event["process_name"] = "LAB\\user", `C:\Synthetic\client.exe`
+		name, other := "botnet_communication_attempts", "botnet_inbound_communication_attempts"
+		if inbound {
+			event["source_address"], event["target_address"] = "198.51.100.9", "192.0.2.10"
+			name, other = other, name
+		}
+		out := parse(event)
+		matches(out, name, true)
+		matches(out, other, false)
+		matches(out, "network_attack_detection", false)
+		if gjson.Get(out, "origin.ip").String() != event["source_address"] || gjson.Get(out, "target.ip").String() != event["target_address"] {
+			t.Fatal("physical endpoints were swapped")
+		}
+		wire := esetAlertWire(t, rules[name], out)
+		if gjson.Get(wire, "adversary.ip").String() != "198.51.100.9" || gjson.Get(wire, "target.ip").String() != "192.0.2.10" || gjson.Get(wire, "target.host").String() != "HOST-A" {
+			t.Fatal("botnet alert did not retain remote adversary and managed target")
+		}
+		if gjson.Get(wire, "adversary.host").Exists() {
+			t.Fatal("managed hostname must not be copied to the remote botnet side")
+		}
+		if !inbound {
+			before := esetAlertWire(t, &plugins.Rule{Adversary: "origin"}, out)
+			if gjson.Get(before, "adversary.ip").String() != "192.0.2.10" {
+				t.Fatal("old origin-role control must select the managed client")
+			}
+		}
+		for _, invalid := range []any{nil, "false", "true", 0, 1} {
+			event["inbound"] = invalid
+			neutral := parse(event)
+			matches(neutral, name, false)
+			matches(neutral, other, false)
+			matches(neutral, "network_attack_detection", false)
+			if gjson.Get(neutral, "origin.host").Exists() || gjson.Get(neutral, "target.host").Exists() {
+				t.Fatal("unknown direction must not assign managed hostname to a network side")
+			}
+		}
+		delete(event, "inbound")
+		neutral := parse(event)
+		matches(neutral, name, false)
+		matches(neutral, other, false)
+	}
+	policy := base("HipsAggregated_Event")
+	policy["application"], policy["operation"], policy["target"], policy["action"], policy["rule_name"] = `C:\Synthetic\utility.exe`, "Read file", `C:\Data\ordinary.txt`, "blocked", "Restrictive file access policy"
+	out := parse(policy)
+	matches(out, "host_intrusion_prevention_triggers", false)
+	matches(out, "suspicious_process_behavior", false)
+	// Specific activity rules are intentionally factual policy-block alerts;
+	// they do not claim that these benign examples establish malware.
+	policy["application"] = `C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe`
+	out = parse(policy)
+	matches(out, "suspicious_powershell_activity_blocked", true)
+	matches(out, "host_intrusion_prevention_triggers", false)
+	policy["application"], policy["operation"], policy["target"] = `C:\Synthetic\utility.exe`, "Write registry value", `HKLM\Software\Synthetic`
+	out = parse(policy)
+	matches(out, "registry_modification_attempts_blocked", true)
+	matches(out, "host_intrusion_prevention_triggers", false)
+	policy["operation"], policy["rule_name"] = "Attempt to run a suspicious object", "Suspicious application launch"
+	out = parse(policy)
+	matches(out, "host_intrusion_prevention_triggers", true)
+	inspect := base("EnterpriseInspectorAlert_Event")
+	inspect["rulename"] = "Backup agent disabled"
+	out = parse(inspect)
+	matches(out, "eset_agent_tampering", false)
+	// Incidental tokens in filenames, account names or arbitrary descriptions
+	// must not reclassify an unrelated structured antivirus detection.
+	threat := base("Threat_Event")
+	threat["threat_name"], threat["scanner_id"], threat["action_taken"] = "Synthetic.Other", "Real-time file system protection", "Detected"
+	threat["object_uri"] = "file:///C:/NewHeur-machine-learning-botnet-ransomware.encrypted"
+	threat["username"] = "registry exploit quarantine failed"
+	out = parse(threat)
+	for name := range rules {
+		matches(out, name, false)
+	}
+	audit := base("Audit_Event")
+	audit["domain"], audit["action"], audit["target"], audit["result"], audit["user"] = "Native user", "Login attempt", "AttemptedAdmin", "Failure", "AuditingActor"
+	out = parse(audit)
+	matches(out, "eset_console_abuse", true)
+	wire := esetAlertWire(t, rules["eset_console_abuse"], out)
+	if gjson.Get(wire, "target.user").String() != "AttemptedAdmin" || gjson.Get(wire, "adversary.user").String() != "AuditingActor" {
+		t.Fatal("attempted account and supplied audit actor must remain distinct")
+	}
+}