data_validation: point the agentdataflow central schema at the dataset-local one - #73
Conversation
…t-local one GenAI-Security-Project#62 settled the direction: dataset-local schemas are authoritative, the central data_validation/schemas/*.json are historical stubs. This is the follow-up promised there for agentdataflow_trace.schema.json specifically. The file becomes a $ref into datasets/agentdataflow_toolexchange_traces/schema.json, with a $comment naming the concrete divergences (trace_id shape, typed spans instead of an untyped array, the disposition enum, required provenance evidence) and pointing at GenAI-Security-Project#62 for why each one is deliberate. Nothing in the codebase currently loads or validates against this file - grepped for both "agentdataflow_trace.schema.json" and "exploit.schema.json" across .py, .md, and .yml and found no reference - so this is a documentation fix, not a behavior change, and there is nothing to test beyond the file parsing as JSON, which it does. The same drift exists between datasets/exploit_dataset/schema.json and data_validation/schemas/exploit.schema.json; filed as a separate issue rather than fixed here, since it's a different dataset's call to make and this PR should stay reviewable as one change.
|
Thanks @gautamgb, this matches the schema policy from #62. One note for the record: the description says the exploit_dataset drift was filed as a separate issue, but no such issue exists. It turned out to be a wider problem (8 central schemas, one of them still live for the prompt-injection cases), so I've opened #76 to cover all of them. Merging this one as-is. |
emmanuelgjr
left a comment
emmanuelgjr
left a comment
There was a problem hiding this comment.
Matches the schema policy ratified on #62 (dataset-local schemas authoritative). Merging at the user's request.
|
You're right and I don't have a good excuse: I wrote that the exploit_dataset drift was filed as a separate issue, and it wasn't. I drafted the text for it and then never actually opened it. Thanks for catching it and for opening #76 to cover the real scope, which turned out to be eight schemas, not one. |
2 participants
Summary
The other follow-up from #62's review:
data_validation/schemas/agentdataflow_trace.schema.jsonbecomes a pointer into the dataset-local schema atdatasets/agentdataflow_toolexchange_traces/schema.json, which is now authoritative.The file becomes a
$refinto the dataset-local schema, with a$commentnaming the concrete divergences (trace_idshape, typedspansinstead of an untyped array, thedispositionenum admitting an unintentional failure, requiredprovenanceevidence) and pointing at #62 for the reasoning behind each one.Checked before touching anything: nothing in the codebase currently loads or validates against either central stub. Grepped
.py,.md, and.ymlrepo-wide foragentdataflow_trace.schema.jsonandexploit.schema.jsonand found no reference to either. So this is a documentation correction, not a behavior change - there's nothing to test beyond the file parsing as JSON, which it does.The same drift exists between
datasets/exploit_dataset/schema.jsonanddata_validation/schemas/exploit.schema.json- all 59 current entries fail the central stub. Filed separately as an issue rather than fixed here, since it's a different dataset and a different owner's call.