Skip to content

[benchmark]: Add multi-retriever benchmarking, Nova 2 Lite support, and PGA bio/stat split - #416

Open
oussamahansal wants to merge 4 commits into
mainfrom
benchmark-multi-retriever-nova2
Open

[benchmark]: Add multi-retriever benchmarking, Nova 2 Lite support, and PGA bio/stat split#416
oussamahansal wants to merge 4 commits into
mainfrom
benchmark-multi-retriever-nova2

Conversation

@oussamahansal

@oussamahansal oussamahansal commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds multi-retriever benchmark infrastructure, fixes Nova 2 Lite extraction compatibility,
and introduces PGA bio/stat split evaluation.

Changes

  • Fix null handling in batch extractors for Nova 2 Lite (returns None for some docs)
  • Add topic-beam-chunk_only and topic_beam_search retrievers to benchmark harness
  • Fix topic-beam-chunk_only to use SemanticGuidedChunkRetriever (was silently failing)
  • Add 3K token budget (per_topic_cap) and increase max_search_results from 5 to 10
  • Add PGA bio/stat split evaluation support (BENCHMARK_DATASET env var)
  • Add multi-retriever loop (run_all_retrievers.sh) for single-command full benchmark
  • Fix boto3/botocore version pins to use ranges for notebook compatibility

Problem

Related issue (if any): #

Testing

  • Unit tests added/updated
  • Integration tests added (as appropriate)
  • Existing tests pass (pytest)
  • Tested manually (describe below)

Checklist

  • Code follows existing style and conventions
  • License headers present on new files
  • Documentation updated (if applicable)
  • No breaking changes (or clearly documented)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@github-actions

Copy link
Copy Markdown

Lexical Graph Coverage Report: The coverage is at 61.9% (target: 80%). Download the HTML report here.

@mykola-pereyma mykola-pereyma left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few questions:

  1. Three concerns bundled in one PR — The Nova 2 Lite null fix is a library-level change, the benchmark infra is test tooling, and the boto3 range is dependency management. Would it be possible to split into separate PRs? Makes review/revert cleaner if something regresses.

  2. See inline comment on on-start.sh about the boto3 version range width.

  3. See inline comment on retriever_factory.py about the max_search_results scope change.

  4. batch_topic_extractor_sync.py is missing a newline at end of file — minor but may trigger linting.

Comment thread integration-tests/test-scripts/on-start.sh Outdated
@github-actions

Copy link
Copy Markdown

Lexical Graph Coverage Report: The coverage is at 61.9% (target: 80%). Download the HTML report here.

@github-actions

Copy link
Copy Markdown

Lexical Graph Coverage Report: The coverage is at 61.9% (target: 80%). Download the HTML report here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants