Build a continual scientific research harness

README.md

@@ -79,7 +79,7 @@ Bring-your-own-key usage is always free and is never gated — Atlas only meters
 
 ## How it works
 
-OpenScience runs a local server that hosts the workspace UI, the agent runtime, the complete default skill library, and the tool layer. The agent plans with a research harness, calls tools (shell, editor, LSP, MCP servers, scientific connectors, and skills), and streams its work back to the browser. Models are routed per request, so you can switch between providers or run local models without changing anything else. Sessions, skills, artifacts, and provenance are stored on disk. Atlas adds optional managed models, credential sync, research graphs, library search, and cloud publishing after login.
+OpenScience runs a local server that hosts the workspace UI, the agent runtime, the complete default skill library, and the tool layer. The agent plans with a research harness, calls tools (shell, editor, LSP, MCP servers, scientific connectors, and skills), and streams its work back to the browser. Models are routed per request, so you can switch between providers or run local models without changing anything else. Sessions, skills, artifacts, and provenance are stored on disk. Atlas adds optional managed models, credential sync, research graphs, library search, and cloud publishing after login. The verifier-grounded product architecture is documented in [Scientific research harness](docs/HARNESS.md).
 
 | Path                 | Contents                                                     |
 | -------------------- | ------------------------------------------------------------ |

backend/cli/skills/physics/simulator-validation/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: simulator-validation
+description: Validate an ODE, PDE, CFD, materials, molecular, or physics simulator with executable convergence, residual, invariant, and reference checks. Use before trusting a numerical result, comparing solvers, or making a benchmark or scientific claim from simulated data.
+---
+
+# Simulator Validation
+
+Treat simulation as a numerical experiment with a falsifiable validation contract, not as a picture generator.
+
+## Select the smallest credible simulator
+
+Choose from the problem structure and installed capability:
+
+- analytic, symbolic, or low-dimensional ODE: SymPy and SciPy;
+- structured finite differences or finite volumes: NumPy/SciPy or FiPy;
+- unstructured finite elements and multiphysics: FEniCSx/DOLFINx;
+- spectral PDEs: Dedalus or a documented spectral implementation;
+- production CFD: OpenFOAM or SU2 when its model and mesh support are required;
+- atomistic/material workflows: ASE or pymatgen as workflow layers plus the declared physical engine;
+- molecular dynamics: OpenMM, GROMACS, or LAMMPS according to force field and scale.
+
+Check the actual executable/import and capture its exact version. Do not silently replace an unavailable solver with a different physical model.
+
+## Freeze the problem
+
+Record equations, coefficients, units or nondimensionalization, domain and geometry, material regions, initial and boundary conditions, scheme and formal order, mesh/timestep sequence, linear/nonlinear solvers, tolerances, stopping rules, and random seeds. Hash the effective simulator configuration.
+
+Choose at least one reference:
+
+- analytic solution;
+- manufactured solution with derived source term;
+- trusted benchmark solution;
+- independently implemented solver; or
+- known limiting/asymptotic result.
+
+## Run a refinement study
+
+Use at least three systematically refined levels. Evaluate the same quantity and norm on every level. Capture a validation JSON:
+
+```json
+{
+  "simulator":{"name":"solver","version":"1.2.3","command":"solver case.yaml","configSHA256":"64-hex"},
+  "expectedOrder":2,
+  "orderTolerance":0.3,
+  "maxResidual":1e-8,
+  "invariantTolerances":{"mass_drift":1e-6},
+  "levels":[
+    {"label":"coarse","h":0.1,"error":0.01,"residual":1e-9,"invariants":{"mass_drift":2e-7}},
+    {"label":"medium","h":0.05,"error":0.0025,"residual":2e-9,"invariants":{"mass_drift":3e-7}},
+    {"label":"fine","h":0.025,"error":0.000625,"residual":3e-9,"invariants":{"mass_drift":4e-7}}
+  ]
+}
+```
+
+Validate it:
+
+```bash
+python scripts/validate_convergence.py validation.json --output validation-report.json
+```
+
+The script exits nonzero unless resolution decreases, error decreases, median observed order meets tolerance, every residual passes, and every declared invariant deviation stays bounded.
+
+## Adversarial validation
+
+Also test applicable properties:
+
+- timestep and solver-tolerance sensitivity;
+- conservation, positivity, symmetry, maximum principle, or boundedness;
+- coordinate, sign, and unit conventions;
+- stiffness, shocks, singularities, mesh distortion, or chaotic sensitivity;
+- domain truncation and boundary reflection;
+- independent implementation or clean replay for the headline result.
+
+Keep failed levels and nonconvergent runs. A small residual alone does not establish discretization accuracy, and visual agreement is not a convergence test.
+
+## Report
+
+Publish simulator/version, configuration hash, level table, error norm, observed orders, residuals, invariant deviations, reference identity, artifacts, compute, and the validator report. Do not claim physical fidelity beyond the validated model regime.

backend/cli/skills/physics/simulator-validation/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Simulator Validation"
+  short_description: "Validate convergence and physical invariants"
+  default_prompt: "Use $simulator-validation to validate a simulator with convergence, residual, and invariant checks."

backend/cli/skills/physics/simulator-validation/scripts/validate_convergence.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""Validate a simulator refinement study and emit machine-readable checks."""
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+
+def require(condition: bool, message: str) -> None:
+    if not condition:
+        raise ValueError(message)
+
+
+def write(path: Path, payload: dict) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
+    try:
+        with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+            json.dump(payload, handle, indent=2, sort_keys=True)
+            handle.write("\n")
+        os.replace(temporary, path)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", type=Path)
+    parser.add_argument("--output", type=Path)
+    args = parser.parse_args()
+    source = args.input.read_bytes()
+    data = json.loads(source)
+    require(isinstance(data, dict), "validation input must be an object")
+    simulator = data.get("simulator")
+    require(isinstance(simulator, dict), "simulator identity is required")
+    for field in ("name", "version", "command"):
+        require(isinstance(simulator.get(field), str) and simulator[field], f"simulator.{field} is required")
+    config_hash = simulator.get("configSHA256")
+    require(isinstance(config_hash, str) and len(config_hash) == 64, "simulator.configSHA256 must be 64 hex characters")
+    require(all(character in "0123456789abcdef" for character in config_hash), "simulator.configSHA256 must be lowercase hex")
+
+    expected = data.get("expectedOrder")
+    tolerance = data.get("orderTolerance")
+    maximum = data.get("maxResidual")
+    require(isinstance(expected, (int, float)) and not isinstance(expected, bool) and expected > 0, "expectedOrder must be positive")
+    require(isinstance(tolerance, (int, float)) and not isinstance(tolerance, bool) and tolerance >= 0, "orderTolerance must be nonnegative")
+    require(isinstance(maximum, (int, float)) and not isinstance(maximum, bool) and maximum >= 0, "maxResidual must be nonnegative")
+    invariants = data.get("invariantTolerances", {})
+    require(isinstance(invariants, dict), "invariantTolerances must be an object")
+    require(all(isinstance(key, str) and key for key in invariants), "invariant names must be non-empty strings")
+    require(
+        all(isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) and value >= 0 for value in invariants.values()),
+        "invariant tolerances must be finite and nonnegative",
+    )
+
+    levels = data.get("levels")
+    require(isinstance(levels, list) and len(levels) >= 3, "at least three refinement levels are required")
+    parsed = []
+    for index, level in enumerate(levels):
+        require(isinstance(level, dict), f"level {index} must be an object")
+        label = level.get("label")
+        require(isinstance(label, str) and label, f"level {index} needs a label")
+        values = {key: level.get(key) for key in ("h", "error", "residual")}
+        require(
+            all(isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) for value in values.values()),
+            f"level {label} contains a non-finite numeric value",
+        )
+        require(values["h"] > 0 and values["error"] > 0 and values["residual"] >= 0, f"level {label} has invalid h/error/residual")
+        observed = level.get("invariants", {})
+        require(isinstance(observed, dict), f"level {label} invariants must be an object")
+        require(set(observed) == set(invariants), f"level {label} must report every declared invariant")
+        require(
+            all(isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) and value >= 0 for value in observed.values()),
+            f"level {label} has an invalid invariant deviation",
+        )
+        parsed.append({"label": label, **values, "invariants": observed})
+
+    resolution = all(left["h"] > right["h"] for left, right in zip(parsed, parsed[1:]))
+    monotone = all(left["error"] > right["error"] for left, right in zip(parsed, parsed[1:]))
+    orders = [
+        math.log(left["error"] / right["error"]) / math.log(left["h"] / right["h"])
+        for left, right in zip(parsed, parsed[1:])
+    ]
+    median = sorted(orders)[len(orders) // 2] if len(orders) % 2 else sum(sorted(orders)[len(orders) // 2 - 1 : len(orders) // 2 + 1]) / 2
+    residual = all(level["residual"] <= maximum for level in parsed)
+    invariant_status = {
+        name: all(level["invariants"][name] <= limit for level in parsed) for name, limit in invariants.items()
+    }
+    checks = {
+        "resolution_decreases": resolution,
+        "error_decreases": monotone,
+        "observed_order": median >= expected - tolerance,
+        "residual_bound": residual,
+        **{f"invariant:{name}": status for name, status in invariant_status.items()},
+    }
+    passed = all(checks.values())
+    report = {
+        "schemaVersion": 1,
+        "passed": passed,
+        "inputSHA256": hashlib.sha256(source).hexdigest(),
+        "simulator": simulator,
+        "observedOrders": orders,
+        "medianObservedOrder": median,
+        "requiredOrder": expected - tolerance,
+        "checks": checks,
+        "levels": parsed,
+    }
+    if args.output:
+        write(args.output, report)
+    print(json.dumps(report, sort_keys=True))
+    return 0 if passed else 1
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main())
+    except (OSError, ValueError, json.JSONDecodeError) as error:
+        print(json.dumps({"error": str(error)}), file=sys.stderr)
+        raise SystemExit(2)

backend/cli/skills/research/active-failure-audit/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: active-failure-audit
+description: Build and run a blinded, commitment-bound active evaluation over a costly hidden probe pool. Use when full evaluation is too expensive, when diverse failure discovery matters, or when a benchmark needs uncertainty-aware sample selection without exposing hidden cases to the agent.
+---
+
+# Active Failure Audit
+
+Use an evaluator-owned probe pool to estimate loss and search for diverse failures under a fixed budget. Keep this workflow outside the candidate-producing agent session.
+
+## Freeze the protocol
+
+Before evaluating, pin:
+
+- the exact run or candidate artifact SHA-256;
+- `performance`, `failure`, or `hybrid` mode;
+- the probe budget and minimum sample count;
+- the loss definition and frozen failure threshold;
+- the precision tolerance and abstention threshold;
+- the feature representation, strata, and evaluation weights; and
+- an optional target number of distinct failures.
+
+Do not tune these fields after seeing outcomes.
+
+## Build opaque probe commitments
+
+Prepare evaluator-private JSONL with one object per hidden case:
+
+```json
+{"id":"case-17","hidden":{"prompt":"...","target":"..."},"features":[0.2,-1.1,0.4],"stratum":"long-tail","weight":1,"priorLoss":0.5}
+```
+
+Run:
+
+```bash
+python scripts/build_probe_manifest.py private-probes.jsonl public-manifest.json
+```
+
+The script validates one shared finite feature dimension, unique IDs, and unique hidden-case bytes. It emits only opaque IDs, numeric features, strata, weights, prior loss, and SHA-256 commitments. Keep the private JSONL outside the agent workspace. Do not use the generated manifest if its validation fails.
+
+## Run the audit
+
+1. Bind the audit configuration in the immutable harness contract.
+2. Initialize `/harness/audits` with the evaluator capability, frozen subject artifact, and generated `probes` array.
+3. Request one `/selection` at a time. Resolve the returned commitment to the private case inside the evaluator boundary.
+4. Evaluate the frozen artifact. Submit loss, threshold-consistent failure label, and observable evidence to `/observations`.
+5. Resume through `/status` after interruption. A pending selection is idempotent.
+6. Stop only when the persisted state reports a terminal reason.
+
+Never send hidden text, targets, or expected outputs to OpenScience. Treat numeric features as fixed side information, not agent-generated descriptions of benchmark answers.
+
+## Interpret the result
+
+Report posterior mean loss, standard deviation, 95% interval, discovered failures, stratum coverage, sample count, pool fingerprint, artifact hash, and stop reason. Preserve `abstain: true` whenever the minimum sample count or uncertainty requirement is unmet.
+
+An active-audit estimate is not an official benchmark score. Attach its immutable receipt to a separately authenticated evaluation before using it as evidence.
+
+## Failure checks
+
+Reject the audit if:
+
+- probe bytes, feature vectors, weights, or thresholds changed;
+- the evaluator capability or artifact hash does not match;
+- selected cases cannot be resolved back to their commitments;
+- failure labels contradict the frozen threshold;
+- one observation is overwritten; or
+- only favorable strata or failure types are reported.

backend/cli/skills/research/active-failure-audit/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Active Failure Audit"
+  short_description: "Build blinded probes for active evaluation"
+  default_prompt: "Use $active-failure-audit to prepare an opaque committed probe pool and an uncertainty-aware audit protocol."

backend/cli/skills/research/active-failure-audit/scripts/build_probe_manifest.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""Build a public active-audit manifest without copying hidden probe content."""
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+
+def fail(message: str) -> None:
+    raise ValueError(message)
+
+
+def canonical(value: object) -> bytes:
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", type=Path)
+    parser.add_argument("output", type=Path)
+    parser.add_argument("--force", action="store_true")
+    args = parser.parse_args()
+
+    if args.output.exists() and not args.force:
+        fail(f"output already exists: {args.output}")
+
+    rows = []
+    for number, line in enumerate(args.input.read_text(encoding="utf-8").splitlines(), 1):
+        if not line.strip():
+            continue
+        try:
+            item = json.loads(line)
+        except json.JSONDecodeError as error:
+            fail(f"line {number} is not valid JSON: {error.msg}")
+        if not isinstance(item, dict):
+            fail(f"line {number} must be an object")
+        if "hidden" not in item:
+            fail(f"line {number} is missing hidden content")
+        identifier = item.get("id")
+        features = item.get("features")
+        stratum = item.get("stratum")
+        if not isinstance(identifier, str) or not identifier or len(identifier) > 240:
+            fail(f"line {number} has an invalid id")
+        if not isinstance(stratum, str) or not stratum or len(stratum) > 120:
+            fail(f"line {number} has an invalid stratum")
+        if not isinstance(features, list) or not 1 <= len(features) <= 32:
+            fail(f"line {number} must contain 1 to 32 features")
+        if any(isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) for value in features):
+            fail(f"line {number} contains a non-finite numeric feature")
+        weight = item.get("weight", 1)
+        prior = item.get("priorLoss", 0.5)
+        if isinstance(weight, bool) or not isinstance(weight, (int, float)) or not 0 < weight <= 1000:
+            fail(f"line {number} has an invalid weight")
+        if isinstance(prior, bool) or not isinstance(prior, (int, float)) or not 0 <= prior <= 1:
+            fail(f"line {number} has an invalid priorLoss")
+        rows.append(
+            {
+                "id": identifier,
+                "commitment": hashlib.sha256(canonical(item["hidden"])).hexdigest(),
+                "features": features,
+                "stratum": stratum,
+                "weight": weight,
+                "priorLoss": prior,
+            }
+        )
+
+    if len(rows) < 2:
+        fail("at least two non-empty probes are required")
+    if len({row["id"] for row in rows}) != len(rows):
+        fail("probe ids must be unique")
+    if len({row["commitment"] for row in rows}) != len(rows):
+        fail("hidden probe commitments must be unique")
+    if len({len(row["features"]) for row in rows}) != 1:
+        fail("all probes must share one feature dimension")
+
+    probes = sorted(rows, key=lambda row: row["id"])
+    payload = {
+        "schemaVersion": 1,
+        "manifestSHA256": hashlib.sha256(canonical(probes)).hexdigest(),
+        "probes": probes,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    descriptor, temporary = tempfile.mkstemp(prefix=f".{args.output.name}.", dir=args.output.parent)
+    try:
+        with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+            json.dump(payload, handle, indent=2, ensure_ascii=False)
+            handle.write("\n")
+        os.replace(temporary, args.output)
+    finally:
+        if os.path.exists(temporary):
+            os.unlink(temporary)
+    print(json.dumps({"probes": len(probes), "manifestSHA256": payload["manifestSHA256"]}))
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        raise SystemExit(main())
+    except (OSError, ValueError) as error:
+        print(json.dumps({"error": str(error)}), file=sys.stderr)
+        raise SystemExit(2)