Ehrlich v1.0: Impact Evaluation, Engine Hardening, SSE Reliability, Token Optimization - #1
Merged
Merged
Conversation
Add the 4th scientific domain for causal analysis of social programs. Implements full DDD bounded context with World Bank, WHO GHO, and FRED API clients, 3 tools, domain config, evaluation standards, and 66 tests.
Update all stale numbers in web landing page: - STATS: tools 67->78, dataSources 16->19, domains 3->4, vizTools 12->15, externalAPIs 15->18, boundedContexts 10->11 - Add Impact Evaluation domain card (5 tools, World Bank/WHO/FRED) - Add 3 data sources: World Bank, WHO GHO, FRED - Add 3 impact viz tools to visualization categories - Fix prose: "Three domains" -> "Four domains", API count 15->18 - Update differentiator and pricing tier copy
…ates Core causal inference feature for Impact Evaluation domain: - CausalEstimator port ABC in domain/ports.py - DiDEstimator infrastructure: effect size, SE, p-value, parallel trends test, automated threat assessment, evidence tier classification - estimate_did tool: difference-in-differences with 4 input groups - assess_threats tool: knowledge-based validity assessment for DiD, PSM, RDD, RCT, IV methods - ImpactService: estimate_did() and assess_threats() methods - Domain config: DiD tool examples in experiment_examples - Console: 2 Impact Evaluation template cards (CCT School Enrollment, Health Worker Programs) - Tests: 35 new tests (DiD estimator + tools + e2e registry) - Tool count: 73 -> 75
Impact visualization tools (server + console): - render_program_dashboard: multi-indicator KPI with target tracking - render_geographic_comparison: region bar chart with benchmark line - render_parallel_trends: DiD treatment vs control over time - 3 React chart components + VizRegistry entries - 8 new viz tool tests Doc sync across all files: - CLAUDE.md: 78 tools, 15 viz, 5 impact tools, file tables updated - README.md: 78 tools, new tool rows added - docs/architecture.md: 78 tools - docs/roadmap.md: Phase 13B partial marked DONE - Tool count: 75 -> 78, viz tools: 12 -> 15
Move causal inference from impact/ to analysis/ bounded context, making it available to all domains. Add three new estimators (PSM, RDD, Synthetic Control) alongside the existing DiD, plus cost-effectiveness analysis. Follows prediction/ ports-and-adapters pattern with 4 ISP ABCs. - 6 causal tools tagged "causal" (estimate_did, estimate_psm, estimate_rdd, estimate_synthetic_control, assess_threats, compute_cost_effectiveness) - 2 viz tools (render_rdd_plot, render_causal_diagram) + React components - ThreatAssessment console component for validity threat display - Impact tools trimmed from 5 to 3 (data-fetching only) - 9 template cards (added Education Policy, Health Policy) - 84 total tools, 17 viz tools, all quality gates green
… modules Structural refactoring with zero behavior change. All existing tests pass unchanged. Multi_orchestrator.py dropped from 1783 lines to 1216 lines. Extracted modules (718 new lines total): - diagram_builder.py (139 lines): Excalidraw evidence synthesis diagram - tool_dispatcher.py (155 lines): Tool execution with caching + special handlers - researcher_executor.py (329 lines): Single researcher experiment executor - batch_executor.py (95 lines): Parallel batch experiment executor (2 concurrent) What stays in multi_orchestrator.py: - Main 6-phase orchestration loop (run method) - Director calls (_director_call, _build_literature_context) - Literature survey (_run_literature_survey) - Hypothesis approval gate - Module-level constants (_COMPACT_SCHEMAS, SEARCH_TOOLS, _compact_result) Quality gates: - ruff check: pass - mypy: pass - pytest tests/investigation/test_multi_orchestrator.py: 20/20 pass - pytest tests/investigation/: 183/183 pass (excl. DB-dependent) Updated CLAUDE.md to document the 4 new modules.
- Remove circular import in researcher_executor.py by copying _compact_result - Update roadmap.md checkboxes for completed Phase 14A extractions - Update architecture.md method references to new module locations Result: multi_orchestrator.py decomposed from 1,788 to 1,216 lines.
Upload-first flow: POST /upload returns preview, POST /investigate references file_ids. UploadedFile domain entity with TabularData and DocumentData frozen dataclasses. FileProcessor parses CSV (pandas), XLSX (openpyxl), PDF (pymupdf). PostgreSQL uploaded_files table with JSONB. Uploaded data injected into Director/Researcher prompts as <uploaded_data> XML block. query_uploaded_data tool intercepted by ToolDispatcher for pandas-based filtering. Frontend drag-and-drop FileUpload, DataPreview cards, use-upload mutation hook. 85 tools total.
…lders Split monolithic roadmap.md (1,453 lines) into docs/roadmap/ with 20 phase-numbered files and an index. Moved research/ into docs/research/ with new impact/ subdirectory. Created docs/adr/ for architecture decision records (business-strategy, domain-agnostic, impact-evaluation). Renamed unnumbered phases to phase-10b through phase-10f for consistency. Updated all cross-references across CLAUDE.md, CONTRIBUTING.md, architecture.md, scientific-methodology.md, and internal links.
…stale CHANGELOG CLAUDE.md: 591 -> 231 lines. Removed duplicated tools catalog, data sources table, and domain descriptions (all in README.md). Kept AI-specific content: key patterns, key files, dependency rules, commands. CONTRIBUTING.md: 524 -> 307 lines. Removed duplicated setup/quality gates/ architecture sections, fixed stale data (10->11 contexts, 70->85 tools, SQLite->PostgreSQL). Kept unique contribution guide. CHANGELOG.md: replaced stale unversioned content with pointer to docs/roadmap/README.md which tracks all 18 phases.
…ME.md: add Philosophy section, lead with open-source identity, self-hosting callout - business-strategy.md: add Classification table, rename Pricing Model to Hosted Instance Pricing - phase-12: rename SaaS Infrastructure to Platform Infrastructure (git mv) - roadmap/README.md: update Phase 12 references in table and dependency graph - phase-02: add domain-agnostic evolution note linking to Phase 10B
…hemas from investigation routes - Create registry_factory.py with lru_cache singletons for tool/domain/MCP registration - Create orchestrator_factory.py with Anthropic adapter wiring - Create api/schemas/investigation.py with 6 DTOs and 5 serialization helpers - Eliminate ~70 lines of duplicated entity-to-dict code between _to_detail and _replay_final - Update consumers: app.py, methodology.py, stats.py, test_e2e.py - Fix stale tool count assertions in e2e tests (78 -> 85) Route file: 829 -> 366 lines (56% reduction)
Split prompts.py (1505 lines) into focused modules: - prompts/constants.py: 6 static prompt strings - prompts/director.py: 4 Director-phase builder functions - prompts/builders.py: 6 non-Director builders + _DEFAULT_CATEGORIES Dead code removed: - SCIENTIST_SYSTEM_PROMPT (103 lines, zero imports) - _build_prior_context (20 lines, zero imports) No backward-compat re-exports; consumers import directly from submodules.
…ature_survey - Extract 5 investigation phases into phase_runner.py as async generators - Extract literature survey runner + PICO context builder into literature_survey.py - Move _DirectorResult to phase_runner.py (canonical location) - Keep _compact_result in researcher_executor.py (avoids circular import) - Update SEARCH_TOOLS import in test_prompts.py - Update test_uploaded_file.py to use dispatcher for query handler - Reduce multi_orchestrator.py from 1,221 to ~240 lines - Mark Phase 14A as DONE in roadmap 87/87 tests passing
…side feedback Extract upload limits (MAX_FILE_SIZE=50MB, MAX_FILES=10, ALLOWED_EXTENSIONS) into upload_limits.py domain constants. Enforce max-files in Pydantic schema, remove redundant size check in upload route, add client-side size validation with per-file error feedback and file count indicator in FileUpload.tsx.
…n logic - search_training_literature: MeSH term expansion (6 terms), study type ranking (SR > RCT > cohort), date recency sort, non-human filtering - search_supplement_evidence: GRADE-style ranking (meta-analysis > SR > RCT > observational), retracted paper exclusion, date recency sort - compare_nutrients: per-nutrient delta with winner highlighting, MAR (Mean Adequacy Ratio) score per food via assess_nutrient_adequacy() - Extract _compute_i_squared() with proper inverse-variance-weighted Cochran's Q, replacing inline approximation in analyze_training_evidence 15 new tests (61 total). Zero ruff/mypy violations.
GET /investigate/{id} was public with no auth. Stream and approve had auth
but no ownership check. Now all three verify the caller owns the resource
via _verify_ownership (checks in-memory meta for active, JOIN query for
persisted). Returns 403 Forbidden for non-owners.
…factory Simplified phase result narrowing from compound isinstance+contains to isinstance(dict) alone -- all phase dicts are __phase_result__ sentinels. Added type annotation to tagged_tools list in registry factory. Applied ruff format to investigation application layer.
Reflect the COSS (Commercial Open-Source Software) positioning established in README.md across all landing page components: Hero subtitle, OpenSource section, pricing framing as hosted-instance-only, differentiators, personas, and SEO meta tags. Fix stale tool/data-source/domain counts throughout (67/70/84 → 85 tools, 15/16 → 19 sources, 3 → 4 domains, FTS5 → tsvector).
Dedicated /paper/:id route renders 8 Markdown sections + numbered visualization figures via VizRegistry. Browser-native PDF export via window.print() with @media print CSS -- zero new dependencies. Recharts/Visx SVG charts captured at full vector quality.
…ff, score chart, and cost breakdown Fix finding serialization to include source_type, source_id, evidence_level, confidence. Add 5 new comparison sections: summary, score chart, findings diff, cost, enhanced stats bar.
- Fix license: MIT -> AGPL-3.0-only (matches LICENSE file and README) - Update description to reflect multi-domain scientific discovery engine - Remove unused deps: requests (zero imports), pyarrow (zero imports) - Remove unused dev dep: pillow (zero imports) - Merge [dependency-groups] dev into [project.optional-dependencies] dev
Replace linear hypothesis testing with tree-based exploration. TreeManager scores, selects, deepens, branches, and prunes hypotheses as an exploration tree with configurable max_depth (default: 3).
…nomic indicators Census Bureau (ACS 5-year), BLS (time series), USAspending (awards/grants), College Scorecard (education outcomes), HUD (FMR/income limits), CDC WONDER (mortality/natality via XML), and data.gov (CKAN dataset discovery). Extended search_economic_indicators with BLS and Census sources. 85 -> 90 tools, 19 -> 25 data sources, 87 new tests (995 total).
Defense-in-depth security improvements addressing 26 audit findings: Server: - SecurityHeadersMiddleware (nosniff, DENY, strict-origin, Permissions-Policy) - CORS hardened: explicit methods/headers, wildcard origin warning - Upload isolation: owner-scoped with 30-min TTL, magic byte validation - Prompt injection defense: html.escape() on all uploaded data XML content - Input validation: prompt length, director_tier pattern, SMILES max_length - Error sanitization: generic JWT errors, redacted API keys, safe tool errors - SQL injection prevention: db_name regex validation in _ensure_database() - Approval state guard: 409 if orchestrator not awaiting approval - URL path encoding for external API clients (CompTox, RCSB) - Dead encrypted_api_key column removal Console: - Markdown XSS: rehype-sanitize on all <Markdown> components - BYOK key storage: localStorage -> sessionStorage (tab-scoped) - SSE reconnection: fresh buildSSEUrl() on each reconnect attempt - Production source maps disabled
…nsole hardening Phase 13C: 3 new API clients (INEGIClient, BanxicoClient, DatosGobClient), analyze_program_indicators tool with CREMAA/MIR validation, extended search_economic_indicators and search_open_data with MX sources, 37 new tests (139 impact total), Mexico domain config with CONEVAL/CODESON keywords. Console: PhaseProgress, ClassificationCard, LiteratureSurveyCard, PhaseSection components, Timeline refactor, investigation page layout updates. Phase 13 now fully DONE. Moved MCP Bridge to Phase 16, Stripe to Phase 15. 91 tools, 28 data sources, 4 domains.
- Domain: Added AWAITING_APPROVAL, PAUSED, CANCELLED states.
- Domain: Implemented transition_to() with transition guards.
- PhaseRunner: Removed 5-min auto-approval timeout; now blocks indefinitely.
- Orchestrator: Added cancel() method; updated approval logic.
- API: Added POST /investigate/{id}/cancel endpoint.
- Frontend: Added InvestigationStatus type.
- Frontend: Updated sidebar with status indicators.
- Frontend: Added Cancel button to HypothesisApprovalCard.
- Frontend: Updated SSE handler for cancellation messages.
- Docs: Updated architecture.md and roadmap.
SSE reliability: - Server assigns DB serial id to persisted events, transient events (THINKING, COST_UPDATE, COMPLETED, ERROR) have no id - Client tracks lastEventId and sends on reconnect via query param - Server replays only events after last_event_id (get_events_after) - Fix race condition: subscribe to live queue BEFORE replay, dedup by id - Replay persisted events for FAILED/CANCELLED before ERROR (partial results) Console UX: - Show investigation prompt in header with titleMaxLength hover tooltip - Add cancel button when investigation is active - Refactor CausalDiagram for production use - Improve PromptInput layout Engine hardening: - Remove director thinking settings (use effort only) - Set summarizer max_tokens=4096 - Reformat investigation state machine transitions - Clean up paper generator and phase runner - Fix Mexico integration test assertions
…d beta header - Add token-efficient-tools-2025-02-19 beta header to all Anthropic API calls - Deduplicate papers by DOI (or title+year fallback) across queries in ToolDispatcher - Remove redundant citation field from search_literature/search_citations results - Truncate authors to first 3 in batch search results (get_reference unchanged) - Lower default search limit from 10 to 5 across all literature layers - Fix tool count assertions from 90 to 91 (pre-existing mismatch)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
35 commits spanning the full hackathon build-out (Feb 10-16, 2026):
token-efficient-toolsbeta header. Cross-query paper dedup by DOI. Authors truncated to 3. Default 5 papers/query.Stats: 268 files changed, 36,784 insertions, 7,617 deletions. 1,040 server tests passing. 91 tools. 28 data sources. 4 domains.
Test plan
uv run pytest-- 1,040 passeduv run ruff check-- zero violationsuv run mypy-- zero errorsbun run build && bun run typecheck(console) -- clean