Repository navigation
feat(dataset): add DiagnosticIQ creation tools - #662
Open
DevinDeSilva wants to merge 5 commits into
Open
DevinDeSilva wants to merge 5 commits into
DevinDeSilva wants to merge 5 commits into
Conversation
Signed-off-by: Devin De Silva <desild@rpi.edu>
DevinDeSilva
marked this pull request as draft
October 24, 2026 18:18
Signed-off-by: Devin De Silva <desild@rpi.edu>
…and Verbose variant Complete the DiagnosticIQ transfer so every artifact the dataset depends on, other than the source documents themselves, is produced by code in this repository. No source data is included; all inputs and outputs live under ASSETOPSBENCH_DATASET_DIR. Source artifacts (new notebooks, transferred with path-only changes): - source_rules_extraction: rule PDF -> extracted/extracted_data.json - source_rules_consistency_check: -> extracted/observations.rules.json - source_rules_formatting: adds asset, splits observations (in place) - source_rules_formalizing: -> extracted/TreeStruct.pkl - source_asset_descriptions: -> extracted/equipment_info/gre_desc.json - source_test_rules_conversion: -> test_datasets/TEST_* inputs Three fixes were needed for these to run on a fresh kernel: drop an empty create_tree_node stub (SyntaxError), quote the self-referencing Node annotation (NameError), and enable the observation split that had been commented out after its one-off in-place run. Question formatting: the post-creation step from question_structuring.ipynb is now dataset_utils.outputs.question_formatting and runs in the DiagnosticIQ, Test, Pert (after perturbing) and Verbose notebooks. DiagnosticIQ Verbose: verbose_dataset_creation.ipynb combines the example selection and LLM paraphrasing steps; dataset_utils.watsonx keeps the original mistral-large text-generation call and decoding parameters, with ibm-watsonx-ai in the dataset-creation group. Also: - Name notebook outputs after the paper (DiagnosticIQ, DiagnosticIQPro, DiagnosticIQPert, DiagnosticIQVerbose) so the chain connects, and save the base dataset instead of leaving the save commented out. - load_pickle maps __main__.Node, which TreeStruct.pkl actually references. - ADIQDataset records loc relative to the dataset workspace; a Path loc made save() and save_dataset() fail on metadata.json. - Document the generic pipeline and the configuration of the V3.1 release. Signed-off-by: Devin De Silva <desild@rpi.edu> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Drop unused fitz/nltk imports and the nltk stopwords download from the DiagnosticIQ notebooks - Remove the now-unused nltk dependency from the dataset-creation group - Document the test command and the embedding library versions used for V3.1 Signed-off-by: Devin De Silva <desild@rpi.edu> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rename the eleven DiagnosticIQ notebooks in notebook/ so they are distinguishable from the other AssetOpsBench notebooks, and update the references in the docs and notebook markdown. Signed-off-by: Devin De Silva <desild@rpi.edu> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DevinDeSilva
marked this pull request as ready for review
October 26, 2026 10:02
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the code that builds the DiagnosticIQ benchmark (DiagnosticIQ, Pro, Pert and Verbose) from expert rule documents, transferred from the DiagnosticIQ code base. No source data is included: the rule documents, all derived artifacts and the generated datasets live in an external workspace named by
ASSETOPSBENCH_DATASET_DIR, and the notebooks contain no outputs.src/dataset_utils/: question generation (creation), configuration, dataset reader/writer, question formatting (outputs.question_formatting), and the watsonx.ai call used for Verbose.notebook/diagnosticiq_source_*.ipynb: the steps that turn the source documents into the generation inputs (extracted_data.json→observations.rules.json→TreeStruct.pkl, asset descriptions, test rules).notebook/diagnosticiq_*_dataset_creation.ipynb: DiagnosticIQ, Pro, Pert, Verbose and a small test-rule dataset.docs/dataset-creation.md: the pipeline, workspace layout, and the configuration used for the V3.1 release.The generation algorithms were not rewritten. With the same inputs,
create_datasetproduces byte-identical output to the original code. The source notebooks needed three fixes to run on a fresh kernel; they are listed in the commit message.Type of Change
Testing & Validation
uv run pytest src/dataset_utils/tests.extracted_data.json, the source notebooks reproduceobservations.rules.json,TreeStruct.pkl(all 120 rule trees) and the test-rule inputs used for the released data. With the V3.1 configuration, generation reproduces the released composition (6,690 questions: 5,180 selection, 1,510 elimination). The Verbose notebook was run against a stub of the watsonx.ai client to check the request parameters.Checklist
src/dataset_utils.🤖 Generated with Claude Code