Skip to content

Fix two NameErrors in the e2e-rag reference implementation - #2664

Open
David-Wu1119 wants to merge 1 commit into
mlcommons:masterfrom
David-Wu1119:fix/e2e-rag-undefined-names
Open

Fix two NameErrors in the e2e-rag reference implementation#2664
David-Wu1119 wants to merge 1 commit into
mlcommons:masterfrom
David-Wu1119:fix/e2e-rag-undefined-names

Conversation

@David-Wu1119

Copy link
Copy Markdown

Summary

Two undefined names in the e2e-rag reference implementation, both flagged by ruff --select F821.

1. evaluation.pyhas_reranker is never defined. The reranking step tests availability inline, but the verbose-output branch further down uses a name that does not exist:

if not no_rerank and hasattr(
        rag_db, '_reranker_model') and rag_db._reranker_model is not None:
    ...
# ~90 lines later
if not no_rerank and has_reranker:      # NameError

So retrieve_and_evaluate(..., verbose=True) raises NameError: name 'has_reranker' is not defined whenever reranking actually ran. Fixed by binding the availability check once and using it in both places, which also removes the duplicated attribute chain:

has_reranker = (
    getattr(rag_db, '_reranker_model', None) is not None
)
if not no_rerank and has_reranker:

2. reference_SUT_datasetup.pylog is used before it exists. The ImportError handler for BeautifulSoup logs a warning, but log is only created after the try/except:

try:
    from bs4 import BeautifulSoup
    ...
except ImportError:
    HAVE_HTML = False
    log.warning("BeautifulSoup not available - HTML processing disabled")   # NameError

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("DatasetupSUT")

The point of that handler is to degrade gracefully via HAVE_HTML, but on a machine without bs4 the module fails to import at all with NameError: name 'log' is not defined — so the fallback it implements can never be reached. Fixed by moving the logger setup above the import attempt.

Verification

ruff --select F821 e2e-rag/ is clean afterwards, and both files parse. The second one is the more consequential of the two: it is the difference between "HTML processing disabled" and the reference SUT not importing.

evaluation.py branched on an undefined `has_reranker` when printing
reranked results, so verbose output raised NameError whenever reranking
had run. Bind it once from the same condition the reranking step uses
and share it, rather than repeating the attribute chain.

reference_SUT_datasetup.py called log.warning() from the ImportError
handler for BeautifulSoup, but `log` is only created after that
try/except. A missing bs4 therefore raised NameError instead of the
intended warning, so the graceful HAVE_HTML fallback never worked. Move
the logging setup above the import attempt.
@David-Wu1119
David-Wu1119 requested review from a team as code owners September 3, 2026 15:00
Copilot AI lite review requested due to automatic review settings September 3, 2026 15:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@David-Wu1119
You can retrigger this bot by commenting recheck in this Pull Request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants