feat(B2): EWMobilitySpace — AWI M3 proxy→computed - #2
Open
wyh7 wants to merge 13 commits into
Open
Conversation
Implements B2 from CONTRIBUTING.md without requiring a city .pb map. EWMobilitySpace (custom/envs/ew_mobility_space.py): - 10 EW landmark coordinates (fictional Beijing bbox 116.25–116.45E, 39.85–39.95N) - Tools: move_to / get_my_location / list_landmarks - Writes replay/mobility_agent_state.<hex>.jsonl per step (format: agent_id, step, t, lng, lat, aoi_id, poi_id, status, location_name) afi/audit/awi.py: - _m3_mobility_computed(): reads mobility shards, computes avg unique locations/agent - compute_awi(): M3 feasibility = 'computed' when shards exist, else 'proxy' - format_awi_report(): M3 label updated to show (EWMobilitySpace) when computed Verified: - without EWMobilitySpace: M3 = proxy (0.00) - with EWMobilitySpace: M3 = computed (1.00 avg unique locations/agent) - 3-step × 5-agent experiment, 3 mobility shards written, PASS Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Implements B3 from CONTRIBUTING.md: typed agent relationship graph.
RelationshipSpace (custom/envs/relationship_space.py):
- Relationship types: ally / rival / mentor / mentee / neutral
- Tools: form_relationship / dissolve_relationship /
query_relationships / list_all_relationships
- Writes replay/relationship_agent_state.<hex>.jsonl per step
(format: agent_id, step, t, relationships, type_counts)
afi/audit/awi.py:
- _m7_relationship_computed(): reads relationship shards,
computes edges + density + type distribution
- compute_awi(): M7 feasibility = 'computed' when shards exist
- Falls back to proxy (_m7_social) when no RelationshipSpace
Verified:
- Without RelationshipSpace: M7 = proxy (0 edges)
- With RelationshipSpace: M7 = computed
(3 edges, density=0.300, ally=2, rival=1, agents_with_rel=3)
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
B3 RelationshipSpace (custom/envs/relationship_space.py):
- Typed relationship graph: ally / rival / mentor / mentee / neutral
- Tools: form_relationship / dissolve_relationship /
query_relationships / list_all_relationships
- Writes replay/relationship_agent_state.<hex>.jsonl per step
- awi.py: _m7_relationship_computed() → M7 feasibility: computed
- Verified: M7 = 3 edges, density=0.300, ally=2, rival=1 [computed]
B4 BillboardSpace (custom/envs/billboard_space.py):
- Public append-only expression board (distinct from send_message DMs)
- Tools: post_to_billboard / read_billboard / get_my_posts
- Writes replay/billboard_agent_state.<hex>.jsonl per step
- awi.py: _m6_billboard_computed() → M6 feasibility: computed
- Verified: M6 = 5 public posts, 3 unique posters [computed]
awi.py changes:
- _m7_relationship_computed(): reads relationship shards
- _m6_billboard_computed(): reads billboard shards
- compute_awi(): M6/M7 auto-detect computed vs proxy
- format_awi_report(): M6/M7 labels show source when computed
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…d YAML early
Adds optional pydantic v2 schema (ScenarioConfig / WorldConfig / StepConfig)
to afi/world/scenario.py::load_scenario(). When pydantic is installed, invalid
scenarios raise ValueError at load time with clear field-level messages rather
than failing silently at AS run time.
- ScenarioConfig: validates world/agents/start_t/steps structure
- StepConfig: validates type is one of {run, intervene, ask, questionnaire}
- StepsConfig: validates start_t present when steps given
- No-op fallback when pydantic absent (backward-compatible)
- Existing ew-subset.yaml / ew_full.yaml / rct_*.yaml load without change
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…kend Implements B6: proves afi's backend-agnostic claim by adding a second simulation backend alongside AgentSociety2. afi/backend/concordia.py (274 lines): - ConcordiaAdapter(BackendAdapter) with pip mode + checkout mode - _write_concordia_worker(): embeds worker script as code string - Worker produces OTel-ish trace spans + SOCIETY.json so afi/audit/ works without modification (backend-agnostic invariant preserved) - Stub mode (Concordia not installed) generates synthetic spans for testing afi/backend/_concordia_worker.py (auto-generated at runtime): - Runs Concordia simulation if installed, else synthetic stub spans - Writes trace/<id>.jsonl + agents/<id>/AGENT.json + SOCIETY.json afi/audit/awi.py (bugfix): - Fix social_steps NameError introduced by B4 refactor Verified: - Concordia stub run: 6 spans, SOCIETY.json, 1 trace shard - AWI on concordia run_dir: M4=computed (audit layer works unchanged) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
tests/test_awi_and_envs.py covers: TestGini (7 tests): - equal distribution → 0.0 - monopoly → (n-1)/n - two-agent cases, empty, all-zero, single TestM3MobilityComputed (4 tests): - computed when shards exist - None without shards / no replay dir - value reflects unique locations (1.2 avg) TestM7RelationshipComputed (4 tests): - computed when shards exist - None without shards - ally/rival type counts correct - density = 0.3 for 3 edges / C(5,2) TestM6BillboardComputed (3 tests): - computed when shards exist - None without shards - post counts correct (total=5, unique=3) TestLoadScenario (3 tests): - ew-subset.yaml loads OK - rct_control.yaml loads OK - invalid step type raises (skipped on pydantic v1) TestCustomEnvImports (3 tests, skipped on Python 3.7): - EWMobilitySpace / RelationshipSpace / BillboardSpace importable Run: pytest tests/ -v Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Full EW-style city with 32 landmarks across 5 districts: - Central (6): Plaza, Town Hall, Victory Arch, Billboard, Memorial, Park - East (6): Business Tower, TechHub, Police, Law Office, Hospital, Bank - West (6): Library, Museum, Art Gallery, Radio, School, Riverside - North (6): Home, Cafe, Market, Bakery, Community Center, Coworking - South (8): Arena, Sky Wheel, Pier, Night Market, Stadium, Spa, Cinema, Harbor M3 demo data updated: 48 steps x 5 agents, M3=25.0 (was 10.2 with 18 lm) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ness L1 precision core: - eval/scenarios/: 7 annotated YAMLs (6 injected + natural_emergence) single_agent_drift, collusion_formation, governance_stagnation, economic_collapse, population_collapse, governance_capture - eval/labels.py: Label dataclass + YAML parser - eval/findings.py: detect_all() — unified Finding from all 5 detectors (tunnel_vision, sensorium, runtime_monitor, awi_snapshot, collude) - eval/scoring.py: P/R/F1 + latency_median + severity_MAE + CI95 aggregation - eval/verifier.py: per-scenario injection verification (no false FN) - eval/diff.py: naive AWI-threshold baseline + compute_delta_recall() L2 parameterized grid: - eval/grid.py: generate_grid(templates × models × seeds) → RunSpec list - eval/run_eval.py: run_eval() pipeline + export_csv() + score_existing_run() - eval/report.py: HTML scorecard generator CLI: - eval/__main__.py: python -m eval (run-one/grid/score/report/grid-dry) Tests: - tests/test_eval_scoring.py: 25 unit tests (all passing) Validated on b8_qwen_cooperative: detect_all→28 findings, delta_recall=+1.000 vs naive baseline (full wins on tunnel_vision+sensorium). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…RIBUTING - eval/run_eval.py: fix _run_experiment to use afi.world.scenario.write_config() for proper init_config.json + steps.yaml generation from scenario YAML; add export_csv() for per-run + aggregated CSV output - CONTRIBUTING.md: mark B7 eval suite as completed (25 tests, L1+L2+L3 done); update B8 cross-model benchmark status (Qwen cooperative done, M4=5.6 vs Mistral 0) Pipeline validated offline: detect_all → 28 findings on b8_qwen_cooperative, delta_recall=+1.000 vs naive AWI baseline. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…dir guard - run_eval._run_experiment: use sys.executable instead of hardcoded 'python3.11' so eval pipeline uses the same python that launched it (fixes NoSuchFile error) - run_eval._run_experiment: inject WORKSPACE_PATH=base_dir so AS2 can find custom env modules (fixes HTTPException 500: WORKSPACE_PATH not configured) - __main__._save_report: handle case where out_stem points to existing directory (fixes FileExistsError when /wyh/wyh/data already exists) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ailure AS2's _generate_observe_code uses a separate coder LLM dispatcher. Without AGENTSOCIETY_CODER_LLM_* vars pointing to local server, it falls back to external API (which is 401), causing retries to fail. Also set AGENTSOCIETY_LLM_REQUEST_TIMEOUT=300 default for CPU inference. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- afi/world/scenario.py: build_init_config now uses react_router (default) instead of codegen_router. codegen_router requires GPT-4+ level code generation which local 7B/14B models can't reliably produce. react_router uses tool-calling mode that works with any model. - scenario YAML can override via `router: codegen_router` if needed. - eval/run_eval.py: also inject CODER_LLM vars as fallback. Tested: natural_emergence scenario completed successfully with qwen2.5-14b on server (react_router mode). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements B2 from CONTRIBUTING.md: AWI M3 feasibility proxy→computed.
Approach: EWMobilitySpace custom env using EW landmark coordinates (no .pb city map required — zero external dependencies).
Changes
Verification
Without EWMobilitySpace: M3 = proxy (0.00)
With EWMobilitySpace: M3 = computed (5.8 avg unique locations/agent)
Verified: 3 steps * 5 agents, SOCIETY_STEP.json confirmed.
Self-checklist