A clause is not a concept - #45
Merged
Merged
Conversation
Thirty-eight of the stress corpus's 482 concepts were clause fragments, and the programme asked questions about them: "What did they report for figure shows?" One paper had "figure shows", "shows the XRD pattern" and "scan shows the xrd" among its twenty-seven concepts, another had "see Figure" and "see Table". Both guards were written in the past tense. PHRASE_BOUNDARY_VERBS is thirty-five entries and every one of them is a past participle, so a phrase opening or closing on a finite verb went straight through. CLAUSE_LIKE -- the rule whose own docstring says "a finite verb makes a phrase a clause rather than a term" -- had the auxiliaries and nothing else, so a verb in the *middle* was invisible to both: the boundary check cannot see it and CLAUSE_LIKE did not know the word. The lead family is deliberately absent from CLAUSE_LIKE. In a battery corpus "lead" is the metal, and a rule that loses "lead acid battery" to catch "leads to capacity fade" is a bad trade. Clause fragments across the corpus fall from 38 to 8, concepts from 482 to 442, and SEG-03 from 27 to 23 -- a segment reported as introducing five new terms when two of them were the same noun phrase cut in different places was the working-memory rule being broken by arithmetic rather than by content. The eight that remain are reduced relatives: "electrodes cycled with LiPF6", "electrode cycled in EC". They are at least comprehensible as terms, and the judgement that separates them from "cycled electrode surface" is not one a word list can make. What this does not fix is the overlap: forty-three pairs where one concept is a sub-phrase of another, "metal counter" beside "metal counter electrodes" beside "sodium metal counter electrodes". _drop_redundant already decides those on the ratio of their mention counts, and measuring the corpus says the ratio cannot: the truncation "metal counter" sits at 0.75, between "thin film" and "specific surface", which are both real terms. What separates them is that a truncation ends on a modifier rather than a head noun, which is a judgement about English that a regex does not have. Left alone rather than guessed at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jepegit
force-pushed
the
concept-fragments
branch
from
September 9, 2026 07:02
d0f42d1 to
0726c8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
38 of the corpus's 482 concepts were clause fragments, and the programme asked questions about them:
One paper had
figure shows,shows the XRD patternandscan shows the xrdamong its 27 concepts. Another hadsee Figureandsee Table. Alsobecomes amorphous,containing species,consistent with the presence,peaks characteristic,capacity associated,against sodium metal counter,discharged to mv,above mv.Both guards were written in the past tense
PHRASE_BOUNDARY_VERBSis 35 entries and every one is a past participle —measured,observed,shown,reported… So a phrase opening or closing on a finite verb went straight through.CLAUSE_LIKE, whose own docstring says "a finite verb makes a phrase a clause rather than a term", had the auxiliaries and nothing else. A verb in the middle was therefore invisible to both: the boundary check cannot reach it, andCLAUSE_LIKEdid not know the word. Hencescan shows the xrd.The lead family is deliberately absent from
CLAUSE_LIKE. In a battery corpus "lead" is the metal, and a rule that loseslead acid batteryto catch "leads to capacity fade" is a bad trade. Tested.Corpus
A segment reported as introducing five new terms, when two of them were the same noun phrase cut in different places, was the working-memory rule being broken by arithmetic rather than by content.
The eight that remain are reduced relatives —
electrodes cycled with LiPF6,electrode cycled in EC. They are at least comprehensible as terms, and the judgement separating them fromcycled electrode surfaceis not one a word list can make.What this does not fix, and why I stopped
43 pairs where one concept is a sub-phrase of another:
metal counterbesidemetal counter electrodesbesidesodium metal counter electrodes._drop_redundantalready decides these on the ratio of their mention counts (drop the shorter at ≥0.8, the longer at <0.35). Measuring every surviving pair in the corpus says the ratio cannot separate them:What distinguishes them is that a truncation ends on a modifier rather than a head noun —
metal counter,a-Si thin,Si hollow,TOF-SIMS depth— which is a judgement about English that a regex does not have. I tried two window-level discounts before measuring this; one was too weak (it compared against a single containing phrase instead of their union) and the positional one cut Irisarri from 32 concepts to 5, takingirreversible capacityandgraphitization degreewith it. Reverted both. Left alone rather than guessed at — this is the shape of thingdocs/PLAN-ai.mdassigns to the critic role.🤖 Generated with Claude Code