Skip to content

Add Codex Azure OpenAI and Microsoft Foundry MPS test coverage - #707

Open
masonc08 wants to merge 3 commits into
mainfrom
masonc08/codex-azure-mps-e2e
Open

masonc08 wants to merge 3 commits into
mainfrom
masonc08/codex-azure-mps-e2e

Conversation

@masonc08

@masonc08 masonc08 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

What

Adds test coverage for routing Codex through an azure_openai Model Provider Service, backed by main.ucode.ci_azure_openai_mps on the CI workspace (allowlist: gpt-5-nano, gpt-5.6-luna reasoning-capable; gpt-4.1-mini non-reasoning).

e2e — TestAzureOpenAiMps (direct Responses-API probes, no agent binary; runs in the gateway shard, independent of #685):

  • reasoning-capable models route with reasoning.effort, and a session can switch between them
  • a non-reasoning allow-listed model (gpt-4.1-mini) rejects reasoning.effort with 400 — the parity invariant: a model that wouldn't work with Codex natively must not be papered over by the gateway
  • the same model serves a plain (no-reasoning) request
  • an off-allowlist model (gpt-4o) is refused with 403

integration — test_ug_configure_codex_azure_openai_mps: the real ug CLI selects the Azure MPS in the configure picker and completes a file task on gpt-5-nano. Requires #685 (adds azure_openai to Codex's provider-type allowlist); until #685 lands, the codex full lane is red.

Plus --codex-azure-provider / --codex-azure-provider-model in run_integration.py and matching conftest fixtures.

Why

Fills the Codex coverage gaps: no live azure_openai coverage, no reasoning-model behavior, and thin OpenAI-MPS coverage generally.

Not covered (deliberately)

Bare ug codex --provider <mps> (no --model) can default to a non-working model — #621 pins the first catalog model, and the MPS discovery call (/ai-gateway/codex/v1/models) carries no per-model reasoning capability, so ucode can't pick a reasoning-capable default. Known limitation, not addressed here.

Test

ruff + py_compile clean; integration contract checks pass. Direct-probe behavior verified live against the MPS (reasoning models 200, gpt-4.1-mini + reasoning.effort 400, gpt-4o 403).

Microsoft Foundry

Foundry speaks the same OpenAI dialect as Azure OpenAI, so test_ug_configure_codex_foundry_mps reuses the Azure CUJ through a shared _run_codex_openai_dialect_mps_cuj helper — the only difference is which MPS the picker selects. Adds --codex-foundry-provider / --codex-foundry-provider-model and matching fixtures. Requires a CI Foundry MPS (flag default main.ucode.ci_foundry_mps) and #685 (adds microsoft_foundry to Codex's allowlist); until the MPS exists the foundry lane errors, as the suite has no capability skips.

This pull request and its description were written by Isaac.

@masonc08
masonc08 force-pushed the masonc08/codex-azure-mps-e2e branch 2 times, most recently from f592398 to 203fd1b Compare September 18, 2026 14:30
@masonc08 masonc08 changed the title Add Codex Azure OpenAI MPS test coverage Add Codex Azure OpenAI and Microsoft Foundry MPS test coverage Sep 18, 2026
masonc08 and others added 3 commits September 18, 2026 16:30
Cover routing Codex through an azure_openai Model Provider Service, backed by
main.ucode.ci_azure_openai_mps (allowlist: gpt-5-nano, gpt-5.6-luna reasoning;
gpt-4.1-mini non-reasoning).

- e2e TestAzureOpenAiMps: direct Responses-API probes through the gateway MPS
  (no agent binary; gateway shard, independent of #685) — reasoning-capable
  models route and are switchable, a non-reasoning allow-listed model rejects
  reasoning.effort (parity: what fails natively fails here), the same model
  serves without reasoning, and an off-allowlist model is refused.
- integration test_ug_configure_codex_azure_openai_mps: the real ug CLI picks
  the Azure MPS in the configure picker and completes a file task on gpt-5-nano.
  Requires #685 (adds azure_openai to Codex's provider-type allowlist).
- run_integration.py / conftest.py: --codex-azure-provider(-model) plumbing.

Co-authored-by: Isaac <no-reply@databricks.com>
Foundry speaks the same OpenAI dialect as Azure OpenAI, so extract the
configure-picker + launch CUJ into a shared helper and add a thin
test_ug_configure_codex_foundry_mps wrapper reusing it. Mirror the Azure
flags/fixtures as --codex-foundry-provider / --codex-foundry-provider-model.

Requires #685 (adds microsoft_foundry to Codex's allowlist) and a CI Foundry MPS.

Co-authored-by: Isaac <no-reply@databricks.com>
The shared-helper refactor left the azure/foundry test wrappers with one-line
docstrings, which test_integration_contract flags (every integration test must
describe Scenario + Expected). Restore both.

Co-authored-by: Isaac <no-reply@databricks.com>
@masonc08
masonc08 force-pushed the masonc08/codex-azure-mps-e2e branch from 04286f3 to ba50bb4 Compare September 18, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant