fix: exponential slowdown with cross-referencing oneOf/anyOf schemas

[adamrimon]

Reasons for making this change

Fixes #4990

Schemas where definitions cross-reference each other through oneOf/anyOf can freeze the browser. This PR fixes two sources of exponential work.

1. Shallow validation in getFirstMatchingOption (getFirstMatchingOption.ts)

getFirstMatchingOption calls validator.isValid() for each option. When properties contain oneOf/anyOf/$ref, the validator recursively checks all nested branches, growing exponentially slower with depth.

This function only needs to check structural shape (right property names and types), not which nested branch matches best. That's handled later by scoring.

Fixed by replacing properties containing oneOf/anyOf/allOf/$ref with {} before validation. This can only widen matches, never miss the correct one.

2. Bounded recursion in calculateIndexScore (getClosestMatchingOption.ts)

calculateIndexScore called getClosestMatchingOption for nested oneOf/anyOf properties, which scores all N options at each level, resulting in O(N^D) work.

Replaced with getFirstMatchingOption (picks the first match, O(N)) followed by recursive calculateIndexScore on that single match. Complexity drops from O(N^D) to O(N² × D), bounded by form data depth.

Also fixes a pre-existing bug where a $ref that resolved to a oneOf/anyOf would return early without evaluating the options. The $ref is now resolved inline and falls through to the oneOf/anyOf handling.

Trade-offs

Both fixes reduce matching precision in a narrow edge case: when two options are structurally identical (same property names, same top-level types) and differ only in nested oneOf/anyOf content.

In practice, the scoring phase still recurses into nested oneOf/anyOf (only the validation in getFirstMatchingOption is shallow), so it resolves these ambiguities by checking types, const, and default values at every level. Schemas that use a discriminator field are unaffected entirely.

Checklist

• I'm adding or updating code
  • I've added and/or updated tests. I've run npx nx run-many --target=build --exclude=@rjsf/docs && npm run test:update to update snapshots, if needed.
  • I've updated docs if needed
  • I've updated the changelog with a description of the PR

[x0k]

Here are a few thoughts:

• Instead of limiting recursion only for oneOf/anyOf, you could introduce a new option that allows users to control the recursion depth of calculateIndexScore in general (depth increases when traversing into nested objects), or allow replacing calculateIndexScore with a custom implementation.
• validator.isValid(stripRecursiveProperties(augmentedSchema as S), will most likely break precompiled validators.
• Also fixes a pre-existing bug - it would be good to add a test.
• As a potential optimization, you could eliminate the use of JUNK_OPTION. For this, extract an isOptionMatching function from getFirstMatchingOption (example).

[heath-freenome]

Here are a few thoughts:

• Instead of limiting recursion only for oneOf/anyOf, you could introduce a new option that allows users to control the recursion depth of calculateIndexScore in general (depth increases when traversing into nested objects), or allow replacing calculateIndexScore with a custom implementation.
• validator.isValid(stripRecursiveProperties(augmentedSchema as S), will most likely break precompiled validators.
• Also fixes a pre-existing bug - it would be good to add a test.
• As a potential optimization, you could eliminate the use of JUNK_OPTION. For this, extract an isOptionMatching function from getFirstMatchingOption (example).

@x0k another fine analysis. Thanks for the input. And yes, the validator.isValid() changes is likely to break precompiled validators unless the precompilation logic is forced through this code path for all potential options.

@adamrimon You are doing great work! Love the willingness to tackle some hard problems. Let me know if you need some support addressing the points raised above.