DEVOP-560: add org-wide daily Shai-Hulud IOC sweep workflow

[srt0422]

Summary

Adds the first workflow under .github/workflows/ in this repo — a scheduled
daily sweep for Shai-Hulud indicators of compromise across every repo in the
allora-network org, plus a rolling GitHub Issue and Slack page on findings.

Closes DEVOP-560.

What ships

• .github/workflows/shai-hulud-sweep.yml — schedule: '7 4 * * *'
  (04:07 UTC daily, off-peak + off-minute) plus workflow_dispatch for manual
  runs. Permissions limited to contents: read + issues: write. Pinned SHAs
  for actions/checkout@v4.2.2 and actions/upload-artifact@v4.4.3, matching
  the convention in allora-network/ci-workflows-private.
• scripts/shai-hulud-ioc-sweep.sh — canonical detection logic, vendored
  verbatim from allora-network/skills@71aeefb (skills/shai-hulud-defense/scripts/).
  See file header for refresh procedure / pinned commit.
• docs/plans/2026-05-25-devop-560-shai-hulud-sweep.md — design notes for
  why we vendor the script, how the rolling issue is maintained, when Slack
  fires, and the GH_ORG_READ_TOKEN follow-up.

Detection coverage (per the vendored script)

• Lockfile entries (npm/pip/Go) matching .github/security/ioc-packages.txt.
• Any .js/.cjs/.mjs file ≤ 2 MB whose SHA-256 matches
  .github/security/ioc-hashes.txt (filename-agnostic — bundle.js rename
  doesn't bypass).
• Persistence: */.github/workflows/shai-hulud*.{yml,yaml} at repo root.
• npm install/postinstall/preinstall lifecycle scripts matching
  node …bundle.js, curl|sh, wget|sh, base64 -d|--decode|-D,
  eval $(…), or npx … bundle.
• Go replace directives: untrusted-host RHS, absolute-path RHS,
  top-level-path mismatch (Scenario C in-org redirect), and local replacements
  (./ / ../) flagged for human review.
• Go workflow env settings (GOSUMDB=off, GONOSUMCHECK, GOINSECURE,
  GOFLAGS=*-insecure) — direct and indirect (vars/secrets/env/inputs).
• Public exfil repos matching ^[Ss]hai-[Hh]ulud under org:allora-network
  AND under each org member (rate-limited).

Outputs

Sweep result	Script exit	Rolling issue	Slack
Clean (no findings)	0	no-op	no-op
Operational (clone_failed / check_skipped / go_local_replace)	2	comment appended (or issue opened with label shai-hulud-sweep)	no-op
IOC-grade	1	comment appended (or issue opened)	paged via ${{ secrets.SLACK_SECURITY_WEBHOOK }}

Forensic evidence (clones of repos that produced IOC findings) is uploaded as
a workflow artifact for 30 days so humans can inspect the matched file without
re-cloning point-in-time evidence.

The workflow never auto-closes the rolling issue; humans drive close/reopen so
triage state survives across daily runs.

Secrets used

• SLACK_SECURITY_WEBHOOK — org secret; payload only delivered on IOC-grade
  findings. No-ops gracefully if unset (warning, not failure).
• GH_ORG_READ_TOKEN — optional org secret. When present, preferred over
  the default GITHUB_TOKEN for org-wide enumeration so private repos and the
  org-members exfil search are covered. When absent, member enumeration emits
  check_skipped operational findings — visible partial coverage, never a
  silent false-clean.

Verification

• actionlint .github/workflows/shai-hulud-sweep.yml — clean (no findings).
• python3 -c 'yaml.safe_load(...)' — parses.
• Manual workflow_dispatch recommended after merge to verify
  gh issue and Slack paths end-to-end against the live org.

Followups (intentionally out-of-scope for this PR)

• Provision GH_ORG_READ_TOKEN (fine-grained PAT or GitHub App token with
  read:org + repo:read) once org-admin signs off — the workflow already
  prefers it when present.
• Quarterly review of the trusted Go module-path allowlist (GO_TRUSTED_HOSTS_RE
  in the script) to keep go_suspicious_replace false-positive rate low.

Made with Cursor

---

Summary by cubic

Adds a daily org‑wide Shai‑Hulud IOC sweep that scans all allora-network repos, updates a rolling issue, and pages Slack on incident‑grade findings with hash‑stamp dedup and a weekly re‑page. Meets DEVOP‑560; latest changes ensure the Slack page is treated as delivered only on HTTP 200 and gate the paged‑at marker on that delivery.

• New Features

  • Added .github/workflows/shai-hulud-sweep.yml (cron 7 4 * * * + workflow_dispatch; minimal perms; serialized concurrency). Pins actions/checkout@v4.2.2 and actions/upload-artifact@v4.4.3.
  • Vendored scripts/shai-hulud-ioc-sweep.sh with SHA‑256 sidecar verification; reads .github/security/ioc-packages.txt and .github/security/ioc-hashes.txt (# schema:v1). Detection covers lockfiles (incl. structured package-lock.json), sub‑2MB JS SHA‑256 hashes, exact Shai‑Hulud persistence workflow filenames, suspicious npm lifecycle scripts, Go replace/env rules, and org/member public exfil search.
  • Outputs: maintains a rolling issue labeled shai-hulud-sweep; Slack via SLACK_SECURITY_WEBHOOK on IOC with dedup gating and weekly re‑page; uploads only findings.json, summary.md, repos.txt (30 days). Added .github/CODEOWNERS to require @allora-network/security (and @allora-network/devops for the workflow).

• Bug Fixes

  • Slack paging: hash‑stamp dedup with weekly re‑page; comment scan filtered to github-actions[bot]; paged‑at marker is now written only after the Slack step returns HTTP 200 and delivered == 'true' (unset webhook cleanly no‑ops and does not mark paged); retries with backoff and Retry‑After; sanitize untrusted text.
  • Rolling issue: single shared “Find rolling issue” step reused across consumers; placeholder summary on pre‑aggregation failure; Slack still fires on IOC even if the issue update fails.
  • Detection correctness: normalize hash IOC list to bare hashes; dedup now extracts and validates IOC_RULES_RE from the detector at runtime (fail‑open on extract/validate failure); exfil search covers both shai-hulud and shai_hulud and fixes OR‑grouping.
  • Safety/robustness: verify the vendored script via a locked‑path SHA‑256 sidecar; restrict artifacts to structured outputs; extend GO_TRUSTED_HOSTS_RE to include cometbft; cap JS hashing to ≤2MB.

^{Written for commit 0717367. Summary will update on new commits.}

[cubic-dev-ai]

cubic analysis

1 issue found

Linked issue analysis

Linked issue: DEVOP-560: Create org-wide daily IOC sweep workflow in .github repo

Status	Acceptance criteria	Notes
✅	.github/workflows/shai-hulud-sweep.yml exists in questa repo	PR adds the workflow file at .github/workflows/shai-hulud-sweep.yml.
✅	Cron: daily at an off-peak local time, off-minute (e.g. '7 4 * * *')	Workflow schedule is set to '7 4 * * *' and includes workflow_dispatch for manual runs.
✅	Iterate all org repos via gh api orgs/allora-network/repos --paginate	The vendored script enumerates repos using gh api --paginate /orgs/$ORG/repos and the workflow runs that script.
✅	For each repo: shallow clone, run sweep checks, report findings	Script uses git clone --depth 1 to shallow-clone each repo, runs the collection/checks, and aggregates findings; workflow uploads artifacts and updates the rolling issue.
✅	Compare against IOC lists at .github/security/ioc-packages.txt and .github/security/ioc-hashes.txt	Workflow passes those two paths to the script; the script validates the schema header and uses both lists in its matching logic.
⚠️	Search the GitHub API for public repos under org members named ^[Ss]hai-[Hh]ulud	Script implements org-scoped and per-member searches and records findings, but per-member enumeration can be limited without a provisioned GH_ORG_READ_TOKEN and will emit check_skipped; member-side search is rate-limited with sleeps. Implementation exists but effective coverage depends on token provisioning and rate limits.
✅	Maintain a single rolling GitHub Issue in .github repo; append new findings	Workflow finds an open issue with label shai-hulud-sweep (oldest first), appends a comment if present, or creates a labeled rolling issue otherwise.
⚠️	If any new finding fires, post to a Slack incoming webhook (SLACK_SECURITY_WEBHOOK org secret)	Workflow posts to the Slack webhook, but only for IOC-grade runs (script exit code 1). Operational findings (exit 2) update the rolling issue but do not page Slack. The step also no-ops if SLACK_SECURITY_WEBHOOK is unset.
✅	Sweep checks — Lockfile entries matching ioc-packages.txt (name@version)	Script builds per-ecosystem needle lists and performs structured and substring matching against npm/pip/Go lockfiles.
✅	Sweep checks — bundle.js files anywhere with SHA-256 matching ioc-hashes.txt	Script collects .js/.cjs/.mjs files ≤ 2 MB and compares SHA-256 against the provided hashes list, emitting ioc_bundle_hash findings.
✅	Sweep checks — .github/workflows/shai-hulud*.yaml / shai-hulud-workflow.yml persistence detection	Script scopes workflow-file scanning to the repo-root .github/workflows directory and flags shai-hulud*.{yml,yaml} as persistence_workflow findings.
✅	Sweep checks — Postinstall patterns: node bundle.js, curl|sh, wget|sh, base64-decode chains in package.json scripts	Script inspects package.json scripts for install/postinstall/preinstall and matches a broad regex covering node …bundle.js, curl|sh, wget|sh, base64 -d/--decode/-D, eval $(…), npx … bundle patterns.
❌	Sweep checks — The webhook.site exfil URL substring	The acceptance criteria list webhook.site substring matching, but I cannot find a specific check for the webhook.site URL substring in the vendored script or workflow; other exfil detection (public-exfil repo name, suspicious curl targets) exists, but no explicit webhook.site pattern match is present.
✅	Workflow uses minimal permissions: contents: read, issues: write	Workflow permissions block sets contents: read and issues: write and no broader permissions are granted.
❌	PR merged (closure of DEVOP-560)	Acceptance requires merging the PR; the PR is open (this is the review), so the 'merged' criterion is not yet satisfied by the current state.

Architecture diagram

sequenceDiagram
    participant Sched as Cron Schedule (04:07 UTC)
    participant Action as GitHub Actions Workflow
    participant Script as shai-hulud-ioc-sweep.sh
    participant GHAPI as GitHub API (REST/GraphQL)
    participant Repos as Org Repos (clone targets)
    participant Issue as Rolling Issue (.github repo)
    participant Slack as Slack Security Webhook
    participant Artifact as Uploaded Artifact

    Note over Sched,Artifact: NEW: Daily org-wide IOC sweep

    alt Scheduled trigger (cron '7 4 * * *')
        Sched->>Action: Trigger workflow
    else Manual trigger (workflow_dispatch)
        Action->>Action: Manual run started
    end

    Action->>Action: concurrency serialization (no cancel-in-progress)
    Action->>Action: Set GH_TOKEN (GH_ORG_READ_TOKEN || GITHUB_TOKEN)
    Action->>Action: Checkout .github repo (IOC lists + script)
    Action->>Script: Run with ORG, IOC package/hash files

    Note over Script: Detection logic (vendored from alla-network/skills)

    Script->>GHAPI: List all repos in org (public + private if token allows)
    GHAPI-->>Script: Repo list

    Script->>GHAPI: List org members (for exfil repo search)
    alt GH_TOKEN has read:org
        GHAPI-->>Script: Member list
    else GH_TOKEN lacks read:org
        Script->>Script: Emit check_skipped operational finding
    end

    loop Per repo in org
        Script->>Repos: git clone (via gh auth credential helper)
        alt Clone succeeds
            Script->>Script: Scan for IOC patterns
            alt IOC package match found
                Script->>Script: finding() - ioc_package_match
            end
            alt IOC hash match found (.js/.cjs/.mjs <= 2MB)
                Script->>Script: finding() - ioc_bundle_hash
            end
            alt Suspicious workflow files found
                Script->>Script: finding() - persistence_workflow
            end
            alt Suspicious npm lifecycle scripts found
                Script->>Script: finding() - suspicious_lifecycle_script
            end
            alt Go replace directives anomalies found
                Script->>Script: finding() - go_suspicious_replace
            end
            alt Go unsafe CI env vars found
                Script->>Script: finding() - go_unsafe_env
            end
            opt Finding detected (non-operational)
                Script->>Script: Preserve clone as forensic evidence
                Script->>Script: Append to dirty-repos list
            end
        else Clone fails
            Script->>Script: finding() - clone_failed (operational)
        end
    end

    loop Check public exfil repos (org scope)
        Script->>GHAPI: Search repos matching ^[Ss]hai-[Hh]ulud under org
        GHAPI-->>Script: Exfil repo list
        alt Exfil repos found
            Script->>Script: finding() - public_exfil_repo
        end
    end

    loop Check public exfil repos (member scope)
        alt GH_TOKEN supports member search
            Script->>GHAPI: Search repos per member
            GHAPI-->>Script: Exfil repo list per member
            alt Exfil repos found
                Script->>Script: finding() - public_exfil_repo_member
            end
        else Token lacks permission
            Script->>Script: finding() - check_skipped (operational)
        end
    end

    Script->>Script: Aggregate findings to findings.ndjson + summary.md
    alt Exit code 0 (clean)
        Script-->>Action: rc=0
    else Exit code 1 (IOC findings)
        Script-->>Action: rc=1
    else Exit code 2 (operational only)
        Script-->>Action: rc=2
    end

    Action->>Artifact: Upload sweep output + forensic evidence clones (30-day retention)

    alt rc == 1 or rc == 2 (findings exist)
        Action->>Issue: Find open issue with label "shai-hulud-sweep"
        alt Existing issue found
            Action->>Issue: Append comment with run summary
        else No existing issue
            Action->>Issue: Create new issue with label + summary
        end
    end

    alt rc == 1 (IOC-grade findings only)
        alt SLACK_SECURITY_WEBHOOK is set
            Action->>Slack: POST run summary (capped at ~2.8 KB)
            Slack-->>Action: 200 OK
        else Webhook not set
            Action->>Action: Log warning, skip (no failure)
        end
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 6 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/shai-hulud-ioc-sweep.sh">

<violation number="1" location="scripts/shai-hulud-ioc-sweep.sh:332">
P1: Persistence workflows with other `shai-hulud*` suffixes now evade this IOC layer, producing a clean result for that indicator. Keep the wildcard coverage and explicitly exempt this repository's legitimate `shai-hulud-sweep.yml` instead of limiting detection to four names.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Persistence workflows with other shai-hulud* suffixes now evade this IOC layer, producing a clean result for that indicator. Keep the wildcard coverage and explicitly exempt this repository's legitimate shai-hulud-sweep.yml instead of limiting detection to four names.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/shai-hulud-ioc-sweep.sh, line 332:

<comment>Persistence workflows with other `shai-hulud*` suffixes now evade this IOC layer, producing a clean result for that indicator. Keep the wildcard coverage and explicitly exempt this repository's legitimate `shai-hulud-sweep.yml` instead of limiting detection to four names.</comment>

<file context>
@@ -301,13 +312,24 @@ while IFS= read -r repo; do
             printf '%s\n' "$path" >> "$WF_FILE"
             case "$bn" in
-              shai-hulud*.yaml|shai-hulud*.yml)
+              shai-hulud.yml|shai-hulud.yaml|shai-hulud-workflow.yml|shai-hulud-workflow.yaml)
                 printf '%s\n' "$path" >> "$PERSIST_FILE" ;;
             esac
</file context>

[srt0422]

Keep the wildcard coverage and explicitly exempt this repository's legitimate shai-hulud-sweep.yml instead of limiting detection to four names.

Leaving open for a security-owner decision — this is a detection-policy trade-off, not a clear defect.

The current explicit four-name list is deliberate (documented in the script comment at scripts/shai-hulud-ioc-sweep.sh:301-321): a shai-hulud* glob false-positives on defensive tooling, which is certain to exist in-org — this repo's shai-hulud-sweep.yml, the skills repo's shai-hulud-defense material, and any org repo that copies the pattern. A guaranteed false IOC page every daily run is the alert-fatigue failure mode this workflow is otherwise engineered to avoid.

The wildcard + exemption alternative catches renamed worm variants but has its own costs: the exemption must be repo-scoped to be safe — a basename-only exemption of shai-hulud-sweep.yml would let a worm evade detection entirely by naming its persistence workflow shai-hulud-sweep.yml in any other org repo — and the exemption list accumulates as more defensive workflows ship. Note also that the four-name list already misses a dropper named shai-hulud-sweep.yml today, so both designs carry name-evasion channels; the difference is detection breadth vs. false-positive and maintenance surface. IOC seed lists (hashes, packages, filenames) are point-in-time by nature and are expected to be refreshed as variants emerge.

[srt0422]

Final cross-provider review (agent-code-review, 3-pass)
verdict: Ready to merge
counts: P0=0 P1=0 P2=0 P3=1
coverage: Pass 1 correctness+security = kimi-k3 (user-mandated Opus-tier substitution; claude-code CLI not invoked); Pass 1 style = z-ai/glm-5.2 (user-mandated Sonnet-tier substitution); Pass 2 = google/gemini-2.5-pro (correctness), deepseek/deepseek-v4-pro (security), meta-llama/llama-4-maverick (style) via OpenRouter HTTP; Pass 3 = meta-llama/llama-4-maverick (correctness), z-ai/glm-5.2 (security, Sonnet-tier substitution), meta-llama/llama-4-scout (style) via OpenRouter HTTP. Every Pass 2/3 finding was source-verified by the orchestrator before entering this report (verification gate).

severity	file:line	category	reviewer	summary	suggested_fix
P3	scripts/shai-hulud-ioc-sweep.sh:328	correctness	correctness-p2 (Gemini; LOW confidence, single-pass, source-verified)	The per-repo `find ...	while IFS= read -r path` walk splits records on newlines, so a filename with an embedded newline would evade the lockfile and JS-hash scans (the line-split path fragments never match the real file). Impact is attenuated: such files cannot be consumed by npm/pip/Go tooling or referenced by package.json lifecycle scripts, so the evasion does not reach an executable artifact.

Refuted false positives (verification gate — documented, not silently dropped):

• security-p2 (DeepSeek): "a repo name containing a newline splits the repo list, truncates cloning, and silently skips later repos" — refuted. GitHub repository names are restricted to ASCII letters, digits, ., -, _; the API rejects names containing newlines, so the premise is impossible. Independently, the while IFS= read -r repo loop continues on any fragment line rather than terminating, so no skip-the-rest behavior exists.
• correctness-p3 (Llama) ×4: (1) "no ORG/GITHUB_ORG validation" — the check exists at scripts/shai-hulud-ioc-sweep.sh:50 ([ -n "$ORG" ] || { usage; exit 2; }). (2) "gh auth token failure unhandled" — the guard exists at scripts/shai-hulud-ioc-sweep.sh:77-80 (exit 2 when unauthenticated). (3) "date -u -d non-portable" — that code runs in the GitHub-hosted ubuntu-latest workflow where GNU date is guaranteed, and already carries a || echo 0 conservative fallback (forces the weekly-repage path). (4) "GO_REPLACE_ALLOWED_FILE not format-validated" — malformed entries degrade to not-allowlisted (fail-closed: mismatches still flagged), and the file is operator-supplied config, not attacker input.