An all-zero probability map returns option #1 as a successful answer

[JYeswak]

What happened

With system-one-adapter 0.2.1 (also 0.2.0) in llm_answer_mode="probabilities", a model
output whose Choice probabilities are all 0 comes back as a normal, successful answer: choice
is the first criterion, confidence is 0.0, and with normalize_probabilities=True every option
reads 1/n. Nothing is raised and no corrective retry is spent. The Score twin returns the middle
level. The only trace is response.debug["probability_errors"][<question id>] == 1.0.

This happens on real inputs. Claude Haiku 4.5 through provider="anthropic" with
structured_outputs=True returns all-zero maps. One raw content[0].text (stop reason
end_turn, 10 options):

{"answers":{"intent":{"activate my card":0,"age limit":0,"apple pay or google pay":0,"atm support":0,"automatic top up":0,"balance not updated after bank transfer":0,"balance not updated after cheque or cash deposit":0,"beneficiary not allowed":0,"cancel transfer":0,"card about to expire":0}}}

Measured with one 10-option Choice question over Banking77 test messages: 58 of 70 calls came back
all-zero on 14 messages that had come back uniform in an earlier run (5 calls each), against 1 of
70 on 14 other messages with the same labels. All 59 answers were the first option at confidence 0.

Repro

No key and no model call. bash, macOS arm64, Python 3.14 (resolved by uv 0.9):

mkdir -p /tmp/allzero && cd /tmp/allzero
cat > repro.py <<'EOF'
import json

from typesafe_sdk import Choice

from system_one_adapter import SystemOneAdapterClient
from system_one_adapter.providers import ProviderResult

LABELS = ["refund", "shipping", "billing"]


class CannedProvider:
    """A custom provider returning one fixed model output."""

    model_name = "canned"

    def __init__(self, distribution):
        self.text = json.dumps({"answers": {"topic": distribution}})
        self.calls = 0

    def request(self, messages, *, schema, structured):
        self.calls += 1
        return ProviderResult(text=self.text, input_tokens=0, output_tokens=0)

    def translate_error(self, error):
        return error


question = {"topic": Choice(instructions="Ticket topic", criteria=dict.fromkeys(LABELS))}

for label, distribution, normalize in [
    ("model returned all zeros", dict.fromkeys(LABELS, 0.0), True),
    ("model returned all zeros", dict.fromkeys(LABELS, 0.0), False),
    ("model returned 1/3 each", dict.fromkeys(LABELS, 1 / 3), True),
]:
    provider = CannedProvider(distribution)
    with SystemOneAdapterClient(
        structured_outputs=True,
        llm_answer_mode="probabilities",
        normalize_probabilities=normalize,
        n_retry_malformed_structure=2,
    ) as client:
        response = client.system_one({"ticket": "x"}, question, model=provider)
    answer = response.answers["topic"]
    print(
        f"{label:26} normalize={normalize!s:5} calls={provider.calls} "
        f"choice={answer.choice!r} confidence={answer.confidence} "
        f"probabilities={ {k: round(v, 3) for k, v in answer.probabilities.items()} } "
        f"probability_errors={response.debug['probability_errors']}"
    )
EOF
uv run --no-project --with system-one-adapter==0.2.1 python repro.py

Output (typesafe-sdk 0.7.1 resolved):

model returned all zeros   normalize=True  calls=1 choice='refund' confidence=0.0 probabilities={'refund': 0.333, 'shipping': 0.333, 'billing': 0.333} probability_errors={'topic': 1.0}
model returned all zeros   normalize=False calls=1 choice='refund' confidence=0.0 probabilities={'refund': 0.0, 'shipping': 0.0, 'billing': 0.0} probability_errors={'topic': 1.0}
model returned 1/3 each    normalize=True  calls=1 choice='refund' confidence=0.0 probabilities={'refund': 0.333, 'shipping': 0.333, 'billing': 0.333} probability_errors={}

Expected vs observed

- expected: a model output that assigns no probability to any option is not returned as an
-           answer (it is surfaced as an error, or counts against n_retry_malformed_structure)
+ observed: choice='refund' (option #1), confidence=0.0, calls=1, no exception, in both
+           normalize_probabilities modes; answers identical to a model-asserted uniform

Workarounds tried

1. normalize_probabilities=False: still choice='refund', confidence=0.0, no exception
   (repro line 2); only the probabilities stay 0.0.
2. n_retry_malformed_structure=2: one provider call (calls=1). Each value is inside [0, 1],
   so the output validates and no retry happens.
3. Reading response.answers only: identical to a model-asserted uniform (repro lines 1 and 3).
   Only debug["probability_errors"] or debug["original_probabilities"] tell them apart.
4. Score has the same shape: an all-zero map over 3 levels returns score=1.0 (the middle level)
   at confidence=0.0 with normalize_probabilities True or False, because the score is
   always computed from a rescaled copy.

Root cause

All citations at commit e1d4cc9 (tag v0.2.1), src/system_one_adapter/:

• _utils/probability_normalization.py:100-106: the model's values are read as-is; a zero total
  gives error = 1.0, and with normalization on, rescale_probabilities is called.
• _utils/probability_normalization.py:70-72: rescale_probabilities has a zero-total branch that
  returns 1/len(probabilities) for every option. Its docstring (:58) names this fallback; the
  README describes the option as "Rescale invalid LLM probability distributions to sum to 1",
  and a zero vector has no scale to preserve.
• _client.py:159: max(answers, key=probabilities.__getitem__) returns the first option on a
  tie, so an all-zero or uniform vector always answers option Review fixes and cassette-backed provider tests #1.
• _utils/confidence_metrics.py:22-24 with :27-32: confidence normalizes a zero total to uniform
  and reports 0.0.
• _client.py:141-142: the Score path always rescales (rescale_probabilities) before taking the
  expected value, so the substituted uniform becomes the middle level regardless of the option.
• _schema.py:25: Probability = Annotated[float, Field(ge=0, le=1)] bounds each value but not
  the total, so _client.py:222-231 (_decode_or_correct) accepts the output and the corrective
  retry never triggers.

I see why the uniform fallback exists: it keeps the arithmetic total-free. The trouble is where
it lands: "the model gave no option any probability" becomes "the model picked option #1, weakly".

Independent confirmation

Live numbers: one model and provider, macOS arm64. The repro is provider-free, on the published
wheel.

Why this matters

Choice is the routing primitive. A caller reading choice routes every such message to whichever
option is listed first, and accuracy computed from choice counts it as that option. Score reports
a mid-scale value the model never gave. A confidence gate does catch these rows, but cannot tell
them from a model-asserted uniform without reading debug.

This is the same shape as #38, where an incomplete provider output was returned as a successful
evaluation and v0.2.1 now rejects it.

Constructive ask

Make the zero-total case distinguishable outside debug. One option that reuses existing
machinery: treat a probability map with no mass as malformed output, so it raises
TypeSafeAPIResponseValidationError by default and a caller who sets
n_retry_malformed_structure gets a corrective retry. You may prefer a different shape; the
ask is only that choice not name an option the model gave zero probability.

Out of scope

• Rescaling of non-zero invalid totals (e.g. 0.6 to 1.0); that behaviour looks right.
• The model's own tendency to return all zeros; that is provider behaviour and outside this repo.
• llm_answer_mode="discrete"; not exercised here.

Duplicate check

Searched this repo with gh issue list --repo typesafe-ai/system-one-adapter-python --state all --search for uniform, zero probabilities, normalize, all zero, rescale,
confidence 0, and probability sum, and read the full issue and PR lists (8 issues and
32 PRs as of 2026-09-24). No duplicate found; the nearest is #38 (truncated output accepted as success), a
different mechanism in the provider layer.

[aoright]

I have investigated this and verified the behavior.

Cause

As identified, Probability = Annotated[float, Field(ge=0, le=1)] in _schema.py validates that individual values lie in [0, 1], but does not constrain the total mass across the discrete outcomes of a Choice or Score question. When an LLM outputs an all-zero mapping:

1. _decode_or_correct accepts the output model without error, bypassing corrective retries.
2. During answer construction in _client.py:
   • For Choice, max(answers, key=probabilities.__getitem__) returns option Review fixes and cassette-backed provider tests #1 due to Python's tie-breaking semantics.
   • For Score, rescale_probabilities substitutes a uniform vector 1.0 / len(probabilities), calculating a mid-scale expected value.
3. In contrast, for Noul, a probability of 0.0 is semantically valid (signaling False / no).

Solution

A clean solution is to equip the dynamic ProbabilityMap{index} models with an after model validator that rejects all-zero probability distributions while keeping provider JSON schemas clean:

class _BaseProbabilityMap(BaseModel):
    """Base model for choice and score probability distributions.

    Enforces that at least one outcome has a non-zero probability so that an
    all-zero distribution is treated as malformed structure rather than
    silently defaulting to option #1.
    """

    model_config = _OUTPUT_CONFIG

    @model_validator(mode="after")
    def _validate_nonzero_probability(self) -> Self:
        if not any(getattr(self, name) > 0 for name in type(self).model_fields):
            raise ValueError("All probabilities are zero; at least one option must have a non-zero probability.")
        return self

In _create_llm_answer_type_for_question:

    return create_model(
        f"ProbabilityMap{index}",
        __base__=_BaseProbabilityMap,
        __doc__=_build_llm_output_question_description(question, llm_answer_mode),
        **probability_fields,
    )

Impact & Behavior

• When n_retry_malformed_structure > 0: An all-zero distribution fails model validation and queues a corrective retry with the prompt detailing the validation failure.
• When n_retry_malformed_structure == 0: Raises TypeSafeAPIResponseValidationError.
• Provider JSON schema output (create_raw_output_schema) remains strictly clean and unmodified (since model validators are purely client-side validation logic).
• Preserves Noul behavior where 0.0 is valid.

The complete patch with unit tests in tests/test_schema.py and retry/rejection tests in tests/test_client_with_fake_model.py is available on branch:
https://github.com/aoright/system-one-adapter-python/tree/fix/reject-all-zero-probabilities (commit 6002682).
All 432 tests pass.