DISCLAIMER: This code is provided as a sample for educational and testing purposes only. Users must perform their own security review and due diligence before deploying any code to production environments. The code provided represents a baseline implementation and may not address all security considerations for your specific environment.
When 5 AI agents collaborate and get it catastrophically wrong, the Blame Graph traces exactly which agent poisoned the well.
GitHub Repository | Companion code for the blog post: Trace cascading decision failures with a blame graph on Amazon OpenSearch (blog link updated on publish)
Researcher → Risk Analyst → Valuation → Macro Economist → Strategist
↓ ↓ ↓ ↓ ↓
[claims] [claims] [claims] [claims] [decision]
↓ ↓ ↓ ↓ ↓
─── BlameTracer ─── InfluenceCalculator ───
↓
OpenSearch (blame graph, claims, checkpoints)
| Component | Purpose |
|---|---|
| Strands Agents SDK | Agent orchestration |
| Amazon Bedrock — Claude Sonnet 4.5 | Agent reasoning (via us.anthropic.claude-sonnet-4-5-20250929-v1:0) |
| Amazon Bedrock — Titan Text Embeddings V2 | Claim embeddings (1024-dim, kNN) |
| Amazon OpenSearch Service (managed) | Blame graph storage, kNN vector search, time-travel checkpoints |
| Python 3.11+ | Runtime |
- AWS account with Bedrock model access enabled for:
us.anthropic.claude-sonnet-4-5-20250929-v1:0(or override viaBEDROCK_MODEL_ID)amazon.titan-embed-text-v2:0
- AWS CLI configured (
aws configure) - Permissions to deploy a managed Amazon OpenSearch Service domain (CloudFormation)
- Python 3.11+
Deploy an Amazon OpenSearch Service domain plus an OpenSearch UI application and set up the demo:
# Full deploy (creates CFN stack, waits for domain ~15-20 min, creates indices)
bash scripts/deploy.sh
# Deploy and immediately run all scenarios
bash scripts/deploy.sh --run
# Load sample data for dashboard exploration
bash scripts/deploy.sh --sample-dataThe deploy script:
- Deploys
infra/opensearch.yaml— a single-nodet3.small.searchdomain (IAM/SigV4 auth) and an OpenSearch UI application that connects to the domain as a data source - Waits for the stack to finish and reads the domain endpoint and application id
- Writes a
.envfile with SigV4 connection settings - Creates a Python venv and installs dependencies
- Creates all 7 OpenSearch indices (
agent-executions,agent-claims,agent-blame,incident-reports,agent-checkpoints,pipeline-outcomes,pipeline-metrics)
The deploy script prints the console link for the OpenSearch UI application. Use
--indices-only to skip the stack and just (re)create indices on an existing domain.
OpenSearch UI admin: by default the application grants dashboard-admin (create/ manage workspaces) to any IAM principal in your account (
AdminUserArn: "*"). This keeps the demo usable under federated/SSO sign-in, where an assumed-role session is not reliably matched by a specific role ARN, so the Create workspace option would otherwise be hidden. Access stays IAM-gated. For shared or production accounts, scope it to a single principal:ADMIN_USER_ARN=arn:aws:iam::<account>:role/<Role> bash scripts/deploy.sh.
OpenSearch UI is an AWS-hosted, single-endpoint interface for Discover, Dev Tools, and dashboards. Unlike OpenSearch Dashboards (which is hosted inside a single domain), OpenSearch UI runs in the AWS Cloud, stays available during domain maintenance, and can connect to multiple data sources. The pipeline still writes its documents to the domain; the OpenSearch UI application reads from the domain as a registered data source.
These run locally, not in the stack. The CloudFormation stack only provisions the OpenSearch domain and the OpenSearch UI application. The scenario runner (
src/main.py) and the Streamlit app below are local processes you start by hand; they use your.envand AWS credentials to talk to the deployed domain and Bedrock. Nothing auto-starts them, and stopping them does not affect the deployed stack.
source .env
source .venv/bin/activate
# Run all 3 scenarios
python src/main.py
# Run a specific scenario
python src/main.py --scenario propagated_hallucination
python src/main.py --scenario silent_dissenter
python src/main.py --scenario stale_dataA local, presenter-friendly front end for the pipeline — separate from the OpenSearch UI application, and not deployed by the stack. Start it manually:
source .env # OpenSearch endpoint + region (written by deploy.sh)
source .venv/bin/activate # the Python env created during deploy
pip install "streamlit>=1.38.0" # first time only, if not already installed
streamlit run src/app.py --server.address localhostThen open http://localhost:8501. Stop it with Ctrl-C (this does not affect the deployed stack). Two modes:
- Prepared scenario — runs 3 realistic failure scenarios with hardcoded agent outputs (reflecting real-world patterns). Bedrock is called for embeddings and influence scoring.
- Custom (live agents) — enter any company/ticker; all 5 agents call Bedrock Claude live (~1–2 minutes). Full end-to-end pipeline with no ground truth.
The History tab browses past runs stored in OpenSearch.
Security: always pass
--server.address localhost. Streamlit's dev server has no authentication and binds all interfaces by default; it uses your ambient AWS credentials and, on live runs, calls Bedrock and writes to OpenSearch.
All config is via environment variables (see src/config.py):
| Variable | Default | Description |
|---|---|---|
OPENSEARCH_AUTH_MODE |
aws |
aws (SigV4 for managed domain) or basic (local dev) |
OPENSEARCH_HOST |
localhost |
OpenSearch domain endpoint |
OPENSEARCH_PORT |
443 (aws) / 9200 (basic) |
Port |
OPENSEARCH_USE_SSL |
true (aws) / false (basic) |
Enable SSL |
OPENSEARCH_AWS_REGION |
us-west-2 |
Region for SigV4 signing |
OPENSEARCH_SERVICE |
es |
es for managed, aoss for serverless |
BEDROCK_MODEL_ID |
us.anthropic.claude-sonnet-4-5-20250929-v1:0 |
LLM model |
EMBEDDING_MODEL_ID |
amazon.titan-embed-text-v2:0 |
Embedding model |
AWS_REGION |
us-west-2 |
AWS region for Bedrock calls |
AGENT_MODEL_ID |
(same as BEDROCK_MODEL_ID) | Override model for live agents |
AGENT_MAX_TOKENS |
1024 |
Max tokens for live agent responses |
| # | Scenario | What Goes Wrong | Root Cause Agent |
|---|---|---|---|
| 1 | Propagated Hallucination | Researcher fabricates a Vantor Pharma partnership — downstream agents amplify it | Researcher (0.67) |
| 2 | Silent Dissenter | Risk Analyst flags FDA danger but gets outweighed by bullish consensus | Strategist (0.42) |
| 3 | Stale Data | Researcher uses 14-day-old data, misses critical FDA safety briefing | Researcher (0.58) |
- Influence Score: Cosine similarity between Agent A's claims and Agent B's cited evidence
- Origination Score: How much of an agent's output is original vs inherited from upstream
- Blame Chain: Backward traversal from failed output, accumulating influence weights
- Dissent Detection: Identifies agents who flagged risk but were outweighed
- Time-Travel: Checkpoint replay — "what if Agent X had different data?"
The demo ships a pre-built dashboard as a saved-objects export
(dashboards/blame-game-dashboard.ndjson). Import it into your OpenSearch UI
application:
- Open the application console page printed by
deploy.sh(theConsoleUrlstack output) and choose Launch application. - Create a workspace (choose the Analytics use case) and associate your
OpenSearch domain (
blame-game-demo) as its data source. (If the Create workspace button isn't shown, relaunch the application in a fresh browser tab — the dashboard-admin mapping can take a minute to propagate after deploy.) - In Manage Workspace → Assets → Import, upload
dashboards/blame-game-dashboard.ndjson. When prompted, map the index patterns to your domain's data source. (The export references a data source namedblame-game-demo; if yours differs, remap it here or regenerate the export withpython dashboards/build_dashboard_ndjson.py --data-source-id <id>.) - Open the Multi-Agent Blame Game — Observability dashboard and set the time range to Last 30 days.
The exported index patterns ship with their field lists pre-baked, so panels resolve immediately. If a panel ever reports "could not locate that index-pattern-field", open the pattern under Manage Workspace → Index patterns and refresh the field list as a fallback.
Load sample data for dashboard exploration:
python scripts/load_sample_data.pyDev Tools queries for manual exploration are in dashboards/devtools_queries.md.
Run them from Dev Tools, found near the bottom of the left navigation panel in
the OpenSearch UI application.
sample-multi-agent-observability-opensearch/
├── README.md
├── LICENSE (MIT-0)
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── .gitignore
├── requirements.txt
├── src/
│ ├── config.py # OpenSearch + Bedrock config (env vars)
│ ├── agents.py # 5 Strands agents with system prompts
│ ├── app.py # Streamlit UI
│ ├── blame_tracer.py # Instruments agents, extracts claims, stores blame graph
│ ├── influence_calculator.py # Cosine similarity, blame chains, backward traversal
│ ├── incident_report.py # Auto-generates incident reports
│ ├── metrics.py # Business + operational metrics collection
│ ├── time_travel.py # Replay from checkpoints
│ ├── guardrails.py # Prevention layer (dissent/source/freshness rules)
│ └── main.py # Orchestrator + 3 demo scenarios
├── scripts/
│ ├── deploy.sh # One-click AWS deploy (domain + OpenSearch UI app)
│ ├── setup_indices.py # Creates 7 OpenSearch indices with mappings
│ ├── load_sample_data.py # Bulk-loads sample data for dashboards
│ └── queries.py # Blame attribution query examples
├── dashboards/
│ ├── blame-game-dashboard.ndjson # Dashboard saved objects (import into OpenSearch UI)
│ ├── build_dashboard_ndjson.py # Regenerates the dashboard saved-objects export
│ ├── sample_data.bulk.ndjson # Sample data for exploration
│ └── devtools_queries.md # Dev Tools queries for manual exploration
├── infra/
│ └── opensearch.yaml # CloudFormation: domain + OpenSearch UI application
└── images/
└── architecture.png
# Delete the CloudFormation stack (removes the OpenSearch domain)
aws cloudformation delete-stack --stack-name blame-game-demo --region us-west-2No other persistent AWS resources are created by this demo. Bedrock is pay-per-request (no cleanup needed).
| Resource | Cost |
|---|---|
| Bedrock (embeddings + LLM per run) | ~$0.50–$2.00 for all 3 scenarios |
Amazon OpenSearch Service (t3.small.search) |
- Author: Smita Singh — Senior Solutions Architect, AWS
- Blog post: Trace cascading decision failures with a blame graph on Amazon OpenSearch (link updated on publish)
- Built with Strands Agents SDK
See CONTRIBUTING for more information.
If you discover a potential security issue in this project, please notify AWS Security via the vulnerability reporting page. Please do not create a public GitHub issue.
This sample uses Amazon Bedrock — Claude Sonnet 4.5 for agent reasoning and incident narratives, and Amazon Titan Text Embeddings V2 for claim-similarity scoring — to demonstrate root-cause attribution in a five-agent investment-research pipeline. It runs on hand-authored, fictional scenarios and is for illustration only.
Transparency
- All agent outputs, influence scores, blame attributions, and incident narratives are AI-generated and labeled as such. The "explainability" and "influence" figures are pragmatic measures defined in this project, not standardized benchmarks.
- Scenarios use fictional companies (e.g. BioGenX, NeuraTech, VaxPrime, Vantor Pharma) with synthetic data. Nothing here is investment advice or usable for real trading decisions.
Human Oversight
- This is an observability and post-mortem tool, not an autonomous trading system. Root-cause attributions are aids for human investigation, not decisions to act on automatically.
- The optional guardrails layer is demo-grade; in production, replace its heuristics with real verification (retrieval/data-source checks, live data feeds, calibrated thresholds) and route high-stakes calls to a human.
Fairness & Bias Mitigation
- Attribution is computed uniformly across all agents from the same signals (claim similarity, influence, dissent). No agent or outcome is privileged by design.
Privacy & Data Handling
- The demo processes only synthetic, fictional data within your own AWS account and OpenSearch domain. No real customer, market, or personal data is used, and no data is sent to third-party services.
Limitations
- Foundation-model outputs may be inaccurate. Scenarios are engineered for reproducibility and a known answer, so results do not generalize to real markets.
- Influence and blame scores are heuristic and depend on the embedding model and chosen thresholds; treat them as directional, not authoritative.
- Model behavior may vary across updates to the underlying foundation model.
Feedback & Improvement
- If you encounter biased, incorrect, or unexpected behavior, please open an issue so it can be investigated.
This library is licensed under the MIT-0 License. See the LICENSE file.