Skip to content

Add model discovery integration tests with managed config - #655

Closed
andy-xu-db wants to merge 6 commits into
codex/model-discovery-test-stackfrom
andy/model-discovery-integration-tests
Closed

andy-xu-db wants to merge 6 commits into
codex/model-discovery-test-stackfrom
andy/model-discovery-integration-tests

Conversation

@andy-xu-db

@andy-xu-db andy-xu-db commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add Tests-table Cases 1–12, where managed config exists, as the first model-discovery test PR.
  • Cover 12 named tests and 24 configured/fresh executions across dedicated Claude Code and Codex managed modules.
  • Fetch the workspace's real CodingAgentConfig once per agent module, derive one MPS-backed test variant, and reuse that mode-0600 stub across every isolated test session.
  • Verify that the admin catalog wins even when developer discovery is disabled and that conflicting provider/model-location overrides fail without changing agent state.

Managed workspace

The managed jobs use the workspace configured by the GitHub Actions secret E2E_ADMIN_WORKSPACE. The runner mints a short-lived bearer with E2E_ADMIN_SP_CLIENT_ID and E2E_ADMIN_SP_CLIENT_SECRET; it does not select a developer Databricks profile or use a PAT.

Each module calls GET /api/ai-gateway/v2/coding-agent-configs once and keeps the first config, matching production selection. The test variant changes only the selected agent's model source and removes its incompatible static defaults:

  • Claude Code: main.default.ci_e2e_anthropic_mps
  • Codex: main.default.ci_e2e_openai_mps

The stub is outside the uploaded artifact root and reused through UCODE_MANAGED_CONFIG_STUB. test_ug_configure_managed.py remains un-stubbed so the real fetch/cache/wire contract is still covered.

Cases

Cases Scope Executions
1–4 managed source after configure and with developer discovery disabled 8
5–12 conflicting provider/location overrides, configured + fresh 16

Each test function includes its Tests-table case number (test_case_01_... through test_case_12_...).

Stack

Verification

  • Head: 76d42794a3061efe256f32dc3f5a6b5d7a644168.
  • CI run 35284235166: both managed-config lanes and both full-journey lanes passed.
  • tests/test_integration_contract.py: 4 passed.
  • Managed discovery collection: 24 executions; complete integration collection: 79.
  • Ruff, formatting, and diff checks passed.
  • No local Databricks profile was selected.

The run's separate Claude agent-launch shard fails because system.ai.claude-opus-5 rejects Claude Code's advisor_20260301 tool type. That path is outside this PR's managed-discovery files; aggregate checks inherit the same failure.

@andy-xu-db
andy-xu-db changed the base branch from main to codex/model-discovery-test-stack September 16, 2026 08:44
@andy-xu-db andy-xu-db changed the title Add parent schema and MPS discovery integration matrix Add model discovery integration tests with managed config Sep 16, 2026
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-tests branch from b9006d2 to 1fda02d Compare September 16, 2026 15:03
@andy-xu-db
andy-xu-db changed the base branch from codex/model-discovery-test-stack to andy/model-discovery-integration-unmanaged September 16, 2026 15:03
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-unmanaged branch from 9751ad8 to 88b4d6f Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-tests branch from 1fda02d to d895788 Compare September 16, 2026 18:32
@andy-xu-db
andy-xu-db changed the base branch from andy/model-discovery-integration-unmanaged to codex/model-discovery-test-stack September 16, 2026 20:06
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-tests branch from d895788 to b7837b7 Compare September 16, 2026 20:06
@andy-xu-db
andy-xu-db force-pushed the andy/model-discovery-integration-tests branch from 8cc5eb8 to ecd7c40 Compare September 17, 2026 21:37
@andy-xu-db
andy-xu-db force-pushed the codex/model-discovery-test-stack branch from 2c0b3df to a3ee267 Compare September 17, 2026 21:37
@andy-xu-db andy-xu-db closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant