[Feature] SIEM / log-lake forwarder layer (ROADMAP Phase 3) — design discussion

[s1ns3nz0]

Feature type

New integration (framework / SDK) — adds an outbound log-forwarder layer
alongside the existing ActivityStream and SignedAuditLog tiers. This
is the ROADMAP Phase 3 item
("SIEM integration (Splunk/Datadog/Azure Sentinel)").

This issue intentionally scopes only the foundation slice: the
extension point, the default schema mapper, and one universal HTTPS
transport. Vendor-specific transports (syslog, Splunk HEC, Sentinel DCR)
and detection-content bundles (Sigma rules) are out of scope for this
discussion — they belong in follow-up issues once the foundation lands.

What problem does this solve?

ActivityStream already captures the right events with the right fields
(user_id, risk_score, policy_decision, matched_rules,
owasp_refs, autonomy_level, delegation_chain), and SignedAuditLog
gives them a tamper-evident HMAC chain. But until those events reach the
enterprise SOC, several controls remain checked-in-letter-but-not-in-spirit:

• NIST SP 800-53 AU-2 / AU-6 ("audit events" / "audit review") implies
  centralized review, not per-host JSONL.
• ISMS-P 2.9 (로그관리) / 2.11 (이상행위 분석) and the 2024 Korean
  Financial Services AI guideline require integration with the central
  SOC stack for 통합관제 / UEBA correlation.
• Insider-threat / UEBA signals need to live next to IdP, EDR, DLP
  in the SIEM — local ~/.aigis/ cannot satisfy this.

A forwarder layer closes the integration gap without changing the on-disk
tiers (which remain authoritative).

Proposed solution (foundation slice only)

A new aigis/forwarders/ package with three pieces:

• LogForwarder ABC — bounded background queue, batching, exception
  isolation, Redactor Protocol that runs before the schema mapper so
  PIPA / GDPR data-minimization can strip sensitive fields before they
  leave the process.
• Default schema: ECS 8.x — natively indexed by Elastic / Wazuh,
  DCR-ingestible by Microsoft Sentinel, CIM-derivable for Splunk. Aigis-
  native fields preserved under aigis.* namespace so analysts never
  lose matched_rules, owasp_refs, delegation_chain, autonomy_level.
• HttpJsonForwarder — stdlib-only HTTPS POST sink covering Splunk
  HEC, Sentinel DCR, Elastic _bulk, Datadog Logs, and generic in-VPC
  ingest. NDJSON / array body, optional gzip, exponential backoff,
  4xx-vs-5xx-aware retry.
• ActivityStream.add_forwarder() hook — Tier-4 dispatch after the
  existing local / global / alerts tiers. Default behavior unchanged
  when no forwarder is registered.

Hard invariants:

• A misconfigured or down SIEM cannot stop the agent or corrupt JSONL.
• Zero new required dependencies — pyproject.toml untouched.
• ECS / CEF / OCSF / Splunk CIM are pluggable mappers behind one Protocol.

Open questions for maintainer guidance

I'd like to align on these before writing the merge-targeted PR:

1. Default schema — ECS vs OCSF? ECS has the broader installed base
   today (Elastic, Wazuh, Sentinel-via-DCR, Splunk-via-CIM). OCSF is
   gaining traction (AWS Security Lake, Cisco) but consumer support is
   thinner. Default ECS, ship OCSF as a sibling mapper? Or the reverse?
2. Module location — aigis/forwarders/ vs aigis/middleware/?
   CONTRIBUTING.md documents aigis/middleware/ for integration modules,
   but middleware in the current codebase wraps LLM-provider request /
   response flow, which is semantically different from outbound log
   shipping. New top-level seems cleaner; happy to be wrong.
3. Queue persistence. Current design is an in-memory bounded queue.
   Should the foundation already include an on-disk spool (SQLite WAL) so
   a process crash during a SIEM outage doesn't drop the queued tail, or
   defer it? Cost ~150 LOC + an integrity contract with the JSONL tier.
4. PII redaction defaults. Should details / rule sample text be
   redacted by default when no Redactor is configured? Keeping the data
   preserves analyst signal but adds a PIPA-by-default footgun for KR
   deployments. Lean: keep current (preserve), document the redactor as
   mandatory in compliance contexts.

Reference implementation (for context only)

A working prototype of the foundation slice lives on my fork — purely as
a discussion artifact, not a PR request. It's the simplest way to
make the design above concrete:

• Code: [REFERENCE-ONLY] feat(forwarders): SIEM/log-lake forwarder prototype — supports upstream killertcell428/aigis#98 s1ns3nz0/aigis-kr#1
• RFC: https://github.com/s1ns3nz0/aigis-kr/blob/feat/siem-forwarder/docs/rfcs/0001-siem-forwarder.md
• Tests: 14 new (1726 total pass, 0 fail), stdlib-only, ruff check clean.

I'll rework / move / re-architect whatever direction makes sense after
this discussion before opening a real PR against killertcell428/aigis.

Are you willing to contribute?

• Yes, I'd like to implement this myself (please assign to me)
• I can help review/test follow-up phases

cc / context: https://github.com/killertcell428/aigis/blob/master/ROADMAP.md#L160

[killertcell428]

Before anything else: a small note from the maintainer side.

I'm a Japanese developer and my English isn't strong enough to write a review of this quality unaided, so this reply (and likely my future replies on this thread) is drafted with help from an AI assistant against my own notes. The technical positions and the merge decisions are mine — the English phrasing is not. I wanted to be upfront about that so there are no surprises later, and so you know that if anything reads ambiguously, it's worth asking me to clarify rather than assuming the nuance was deliberate.

And thank you, sincerely, for the way you opened this. Landing the foundation slice on a fork as a discussion artifact rather than as a surprise PR, writing an RFC, scoping vendor transports and Sigma content out explicitly, and asking for maintainer guidance on four specific design questions before touching master — this is the easiest design review I've had on the project so far, and I'd like more contributions to look like this one. The fact that it's grounded in a real ISMS-P / KR financial AI guideline requirement rather than a speculative "wouldn't it be nice" makes it even more welcome.

With that said — onto the substance.

High-level: I'm in favor of the foundation slice you describe, and I'd like to keep this issue scoped to it. Vendor-specific transports — Splunk HEC, Sentinel DCR, syslog — and detection-content bundles, such as Sigma, should each get their own follow-up issue once the extension point lands.

Hard invariants I want preserved — and that your draft already respects

1. Zero new required dependencies. stdlib-only, pyproject.toml untouched.
2. The local JSONL tier remains authoritative. Forwarders are additive; nothing about Tier 1/2/3 changes.
3. A misconfigured or down SIEM must not stop the agent or corrupt JSONL. Exception isolation in the dispatcher is non-negotiable.
4. Vendor-specific transports are out of scope here. Separate issues / PRs.
5. Sigma rule bundles are out of scope. You already noted this.

Answers to the four open questions

1. Default schema → ECS 8.x. Agreed with your reasoning. ECS has the broadest consumer support across Splunk via CIM, Sentinel via DCR, Elastic, and Wazuh, which matches what JP/KR enterprise SOCs are actually running today. OCSF is welcome as a sibling mapper behind the same Protocol — please leave the seam in place but don't ship the OCSF mapper in the foundation PR. We can add it in a follow-up once a real consumer asks for it.

2. Module location → new aigis/forwarders/ package. Agreed. aigis/middleware/ in this repo is reserved for LLM-provider request / response proxies — Anthropic, OpenAI, LangChain, LangGraph, FastAPI. That is semantically different from outbound log shipping, and overloading middleware/ would make CONTRIBUTING.md confusing. I'll update CONTRIBUTING.md to add a "log forwarders" section pointing at the new package as part of the merge.

3. Queue persistence — SQLite spool — defer to a follow-up. Foundation should ship with the in-memory bounded queue only. Persisting the tail across crashes is a real concern, but it introduces an integrity contract with the JSONL tier that I want to design separately, ideally driven by an actual incident report: "we lost N events when the SIEM was down for M hours." Please add a TODO: and a short paragraph in the RFC pointing at that follow-up issue so we don't lose the thread.

4. PII redaction defaults → keep current behavior — preserve fields — and make Redactor mandatory in compliance docs. Stripping details / rule sample text by default would silently erase analyst signal, which is worse than a documented-but-explicit configuration step. For the PIPA / GDPR footgun, I'd rather ship a small "KR preset Redactor" and a "JP/APPI preset" in a follow-up than change the default. Please mention the redactor as a required configuration step in the compliance section of the RFC, with a one-line code sample.

What I'd like to see in the merge-targeted PR

• aigis/forwarders/__init__.py exporting LogForwarder, EcsMapper, HttpJsonForwarder, and the Redactor Protocol.
• ActivityStream.add_forwarder() hook, default behavior unchanged when no forwarder is registered.
• Tests covering:
  • forwarder exception does not propagate to record();
  • bounded queue drops oldest on overflow with a counter exposed for ops;
  • ECS mapping preserves all Aigis-native fields under aigis.*;
  • HttpJsonForwarder retries 5xx but not 4xx;
  • gzip path round-trips.
• A short docs/forwarders.md aimed at SOC engineers: "how to point Aigis at my Splunk HEC / Sentinel DCR / Elastic _bulk endpoint", with a Redactor example.
• CONTRIBUTING.md update covering the new aigis/forwarders/ package.

On assignment

Yes, please consider yourself assigned for the foundation PR. I'd also like to keep you in the loop on the vendor-transport follow-ups, even if you don't end up writing all of them.

One administrative note: please open the PR against master with a clean, rebased branch, and link both this issue and the RFC in the PR body. I'll review on the same invariants listed above plus the usual ruff check / pytest gates.

Thanks again — and apologies in advance for any awkwardness on my side caused by the language barrier. If anything in this reply is unclear, please assume it's a translation artifact and ask.