Repository navigation
test(e2e): track example.com's reworded body in fetch asserts - #72
Merged
Merged
Conversation
example.com dropped its <h1> and reworded its paragraph, so test_fetch_single_url and test_fetch_live failed on every platform looking for "# Example Domain". Assert on the new body text instead, kept in one EXAMPLE_BODY constant. The same change silently neutered test_fetch_prompt: its negative assert looked for the old "illustrative examples" wording, so it passed even if the daemon dropped `prompt`. It now uses EXAMPLE_BODY. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the <h1> gone from example.com the markdown no longer contains the
page title, so "Answer with the exact page title" failed 4/5 locally
("The page content does not contain a title.") and once in CI. Ask for
a fact the paragraph states instead (6/6 locally).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nightly E2E (run 36529843973) failed on all three platforms:
test_fetch_single_urlandtest_fetch_liveasserted"# Example Domain" in content.example.com itself changed: it no longer has the
<h1>in the body, and the paragraph now reads "This domain is for use in documentation examples without needing permission. This is not a service, avoid relying on it for testing and monitoring purposes." Both cached and--livefetches return this. The backend still keeps headings on other pages (e.g.# Example Domainson iana.org), so this is not a converter regression.EXAMPLE_BODYconstant, so the next rewording is a one-line fix.test_fetch_prompt: its "full page not returned" check looked for the old "illustrative examples" wording, so it passed even if the daemon droppedprompt. It now usesEXAMPLE_BODY.test_fetch_prompton a fact the body states. With the<h1>gone the title is no longer in the markdown, so "Answer with the exact page title" passed 1/5 locally (the model replied "The page content does not contain a title.") and failed in CI — the same trap test(e2e): pin fetch --prompt on a fact the page actually states #59 fixed for the domain name. The new prompt ("What does the page say not to rely on it for?") returned "Testing and monitoring purposes" 6/6.Test plan
Follow-up idea: example.com now explicitly asks not to be used for testing — content-sensitive asserts could move to a fixture page we control.
🤖 Generated with Claude Code