diff --git a/fuzzing/graphql-validation/.gitignore b/fuzzing/graphql-validation/.gitignore new file mode 100644 index 00000000000..583c5558d9c --- /dev/null +++ b/fuzzing/graphql-validation/.gitignore @@ -0,0 +1,10 @@ +node_modules/ +package-lock.json +# generated corpora and results +*.jsonl +out.json +*-diff.json +*-err.txt +# build output +HcValidationRunner/bin/ +HcValidationRunner/obj/ diff --git a/fuzzing/graphql-validation/FINDINGS.md b/fuzzing/graphql-validation/FINDINGS.md new file mode 100644 index 00000000000..e1bd845e67e --- /dev/null +++ b/fuzzing/graphql-validation/FINDINGS.md @@ -0,0 +1,182 @@ +# Differential validation findings: graphql-js vs HotChocolate + +Method: validate the same documents against the same schema +([schema.graphql](./schema.graphql)) with the spec reference validator +(graphql-js v17.0.1, `validate(schema, parse(q), specifiedRules)`) and +HotChocolate's full default rule set +(`DocumentValidatorBuilder.New().AddDefaultRules()`), then diff. Prime signal: +**reference reports invalid, HotChocolate reports valid**. + +Corpus: 36 curated seeds + ~95 hand-written hard cases ([hard.json](./hard.json), +[hard2.json](./hard2.json), [hard3.json](./hard3.json) for interfaces/unions) + +~125k AST-mutation fuzz documents (seeded, reproducible) whose mutators span value +coercion, field merging, directive locations, fragment spreads/cycles, variable +usage/positions/types, operation structure, and inline-fragment type conditions +on abstract types. HotChocolate matched graphql-js except for the four findings +below. Interface/union *structural* validation is solid — fragment-spread +possibility, fields-on-abstract-types, and even `OverlappingFieldsCanBeMerged`'s +`sameResponseShape` across mutually-exclusive union members all match graphql-js +in both directions (the new gaps are on the `__typename` meta-field). No `hc_extra` +(HC stricter than the spec) was seen. + +Related HotChocolate issues: searched `ChilliCream/graphql-platform` for `Int` +range / `ValuesOfCorrectType` / input-value / scalar-literal validation — **no +matching issue found**; both findings appear unreported. (The `>4 directives -> +Parsing aborted` behaviour seen in `parse_disagree` is an intentional parser DoS +guard: `ParserOptions.Default.MaxAllowedDirectives = 4`, not a validation gap.) + +--- + +## Finding 1 — `Int` literals are not range-checked (localized miss) + +HotChocolate validates the *kind* of an `Int` literal (it rejects `[1,2]`, +`{a:1}`, `"x"` for `Int`) but not the 32-bit signed range the GraphQL `Int` +scalar mandates. graphql-js rejects out-of-range integer literals during +validation; HotChocolate accepts them. + +```graphql +{ complicatedArgs { intArgField(intArg: 2147483648) } } # Int32 max + 1 +{ complicatedArgs { intArgField(intArg: -2147483649) } } # Int32 min - 1 +{ complicatedArgs { intArgField(intArg: 99999999999999999999) } } +{ complicatedArgs { multipleReqs(req1: 1, req2: 2147483648) } } # non-null arg +{ complicatedArgs { nestedArgField(nested: { required: 2147483648 }) } } # nested input field +{ complicatedArgs { nestedArgField(nested: { required: 1, ints: [1, 2147483648] }) } } # nested list item +query ($v: Int = 9999999999) { complicatedArgs { intArgField(intArg: $v) } } # variable default +query ($v: NestedInput = { required: 2147483648 }) { complicatedArgs { nestedArgField(nested: $v) } } +``` + +graphql-js: `Int cannot represent non 32-bit signed integer value: `. + +- Boundary is exactly Int32: `2147483647` / `-2147483648` accepted by both; one + past the edge diverges. +- Holds at every literal position: direct arg, non-null arg, nested input field, + nested list element, variable default, and nested variable default. +- **Localized**: it does *not* suppress sibling errors. HotChocolate still reports + a bad enum / unused variable / unknown field alongside the un-flagged Int. +- This is a validation-completeness gap (the document is declared valid when the + spec says otherwise); execution-time coercion would still fail. + +## Finding 2 — a list value where an input object is expected silently aborts validation (error-masking bypass) + +When an argument expecting an **input object** is given a **list** literal, +HotChocolate reports the **entire document as valid** and **suppresses every other +validation error in it**. + +```graphql +# reference: 2 errors (list-for-input + bad enum). HotChocolate: VALID, 0 errors. +{ complicatedArgs { complexArgField(complexArg: []) enumArgField(enumArg: NOPE) } } + +# reference: 2 errors (unused var + list-for-input). HotChocolate: VALID, 0 errors. +query ($u: Int) { complicatedArgs { complexArgField(complexArg: []) } } + +# also masked: unknown field, string-for-Int, etc., whenever any input-object arg gets a list. +``` + +Scope (precisely): the trigger is **(list value) x (input-object type)** only. +Both empty `[]` and non-empty `[{ ... }]` lists fire it. It is NOT triggered by a +scalar given for an input object (`complexArg: 5`), an object given for a list +(`stringListArg: {a:1}`), or a list given for a scalar field inside an input +object (`{ stringField: [] }`) — those each produce a normal, localized error. + +### Root cause + +`src/HotChocolate/Core/src/Validation/Rules/ValueVisitor.cs:305-325`: + +```csharp +protected override ISyntaxVisitorAction Enter(ListValueNode node, DocumentValidatorContext context) +{ + if (context.Types.TryPeek(out var type)) + { + if (type.NamedType().IsLeafType()) { return Enter((IValueNode)node, context); } // scalar: validated + if (type.IsListType()) { context.Types.Push(type.ElementType()); return Continue; } + } + context.UnexpectedErrorsDetected = true; // list for an INPUT OBJECT type + return Break; // <-- aborts traversal, NO ReportError +} +``` + +For a list value whose expected type is an input object (neither leaf nor list) +the visitor sets `UnexpectedErrorsDetected` and returns `Break` **without calling +`context.ReportError(...)`**. `Break` aborts the validation traversal, so no error +is recorded for the malformed value and subsequent nodes/rules are not visited, +leaving the document reported valid. The scalar mismatch path +(`Enter(IValueNode)`, line 335) correctly calls `ReportError`; the list path does +not. Likely fix: report a "value is not of the correct type" error (as the scalar +path does) instead of `Break`-ing. + +The same defect exists a second time in `VariableVisitor.Enter(ListValueNode)` +(`src/HotChocolate/Core/src/Validation/Rules/VariableVisitor.cs:283-293`): a list +value for a non-list type (e.g. an input-object-typed variable default such as +`$v: ComplexInput = []`) returns `Break` with no `ReportError`. Both list visitors +need the same fix. + +### Why it matters + +A client can append `anyInputObjectArg: []` to an otherwise-invalid (or +abuse-shaped) document and have HotChocolate's validator pass it as valid, +bypassing all other validation rules. Worth confirming downstream behaviour +(execution/coercion) and reporting upstream. + +## Finding 3 — `ScalarLeafs` not enforced on `__typename` under abstract types + +A selection set on the `__typename` meta-field is invalid (`__typename: String!` +is a leaf). HotChocolate catches it when the parent is an object/root type but +**not when the parent is a union or interface**: + +| query | graphql-js | HotChocolate | +| --- | --- | --- | +| `{ dog { __typename { x } } }` (object) | invalid | invalid | +| `{ __typename { x } }` (root) | invalid | invalid | +| `{ catOrDog { __typename { x } } }` (union) | invalid | **VALID** | +| `{ pet { __typename { x } } }` (interface) | invalid | **VALID** | + +So `__typename`'s leaf-selection check is skipped specifically when it is selected +on an abstract type (`FieldSelectionsRule` special-cases the `__typename` meta-field). + +## Finding 4 — directive *arguments* are not validated on `__typename` + +HotChocolate validates a directive's existence and location on `__typename` +(`@nosuchdir`, `@onQuery` are rejected) and validates directive arguments on +ordinary fields, but it does **not** validate the *arguments* of directives +applied to `__typename`: + +| query | graphql-js | HotChocolate | +| --- | --- | --- | +| `{ dog { __typename @skip } }` (missing required `if`) | invalid | **VALID** | +| `{ dog { __typename @skip(if: 5) } }` (wrong arg type) | invalid | **VALID** | +| `{ dog { __typename @skip(wrong: true) } }` (unknown arg) | invalid | **VALID** | +| `{ dog { name @skip } }` (control: ordinary field) | invalid | invalid | + +Holds for `__typename` on every parent kind (object, union, interface, root). + +### Root cause + +`src/HotChocolate/Core/src/Validation/Rules/ArgumentVisitor.cs:48-64`: the +`__typename` branch validates the field's own arguments (it has none) and then +`return Skip`, so the visitor never descends into the field's **directives** and +their arguments go unvalidated. Ordinary fields fall through to the `Continue` +branch (line 73), which validates their directive arguments. Fix: validate the +directive arguments of `__typename` instead of `Skip`-ping past them. + +--- + +## Reproduce + +```bash +cd fuzzing/graphql-validation && npm install +dotnet build -c Release HcValidationRunner/HcValidationRunner.csproj + +# the two findings, minimally: +printf '%s\n' \ + '"{ complicatedArgs { intArgField(intArg: 2147483648) } }"' \ + '"{ complicatedArgs { complexArgField(complexArg: []) enumArgField(enumArg: NOPE) } }"' > /tmp/repro.jsonl +node node-runner.mjs schema.graphql /tmp/repro.jsonl +dotnet HcValidationRunner/bin/Release/net10.0/HcValidationRunner.dll schema.graphql /tmp/repro.jsonl +# graphql-js: both invalid. HotChocolate: both valid. + +# full fuzz: +node fuzz.mjs --count 30000 --seed 3 +node node-runner.mjs schema.graphql corpus.jsonl > node-results.jsonl +dotnet HcValidationRunner/bin/Release/net10.0/HcValidationRunner.dll schema.graphql corpus.jsonl > hc-results.jsonl +node diff.mjs corpus.jsonl node-results.jsonl hc-results.jsonl out.json +``` diff --git a/fuzzing/graphql-validation/HcValidationRunner/HcValidationRunner.csproj b/fuzzing/graphql-validation/HcValidationRunner/HcValidationRunner.csproj new file mode 100644 index 00000000000..9af4e3d4c4a --- /dev/null +++ b/fuzzing/graphql-validation/HcValidationRunner/HcValidationRunner.csproj @@ -0,0 +1,31 @@ + + + + + Exe + net10.0 + disable + enable + false + + $(NoWarn);CS1591;NU1605;NU1608;NU1701 + false + false + false + false + false + false + preview + true + + + + + + + + diff --git a/fuzzing/graphql-validation/HcValidationRunner/Program.cs b/fuzzing/graphql-validation/HcValidationRunner/Program.cs new file mode 100644 index 00000000000..65462ed2a85 --- /dev/null +++ b/fuzzing/graphql-validation/HcValidationRunner/Program.cs @@ -0,0 +1,274 @@ +// HotChocolate validation runner for the differential validation fuzzer. +// +// Usage: HcValidationRunner +// +// schema.graphql : GraphQL SDL, loaded schema-first. +// corpus.jsonl : one JSON-encoded query string per non-empty line. +// +// For every corpus line (index i from 0) exactly one JSON object is written to +// stdout, aligned by i: +// {"i":,"parseError":,"valid":,"errorCount":,"messages":[<=5 strings]} +// +// Mirrors the graphql-js reference runner (node-runner.mjs) so the two outputs +// can be diffed line-for-line. All diagnostics go to stderr; stdout is results only. + +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using HotChocolate; +using HotChocolate.Language; +using HotChocolate.Resolvers; +using HotChocolate.Types; +using HotChocolate.Validation; + +// AddDefaultRules lives in the Microsoft.Extensions.DependencyInjection namespace. +using Microsoft.Extensions.DependencyInjection; + +if (args.Length < 2) +{ + Console.Error.WriteLine("usage: HcValidationRunner "); + return 2; +} + +var schemaPath = args[0]; +var corpusPath = args[1]; + +// Compact JSON, exact key names matching the JS reference runner. Non-ASCII +// characters in error messages are escaped conservatively (default encoder). +var jsonOptions = new JsonSerializerOptions +{ + DefaultIgnoreCondition = JsonIgnoreCondition.Never, + WriteIndented = false +}; + +if (!File.Exists(schemaPath)) +{ + Console.Error.WriteLine($"schema file not found: {schemaPath}"); + return 2; +} + +if (!File.Exists(corpusPath)) +{ + Console.Error.WriteLine($"corpus file not found: {corpusPath}"); + return 2; +} + +// --- Build schema + validator ONCE --------------------------------------- + +ISchemaDefinition schema; +try +{ + var sdl = File.ReadAllText(schemaPath); + schema = BuildSchema(sdl); +} +catch (Exception ex) +{ + Console.Error.WriteLine("FATAL: failed to load schema:"); + Console.Error.WriteLine(ex); + return 1; +} + +DocumentValidator validator; +try +{ + validator = DocumentValidatorBuilder.New().AddDefaultRules().Build(); +} +catch (Exception ex) +{ + Console.Error.WriteLine("FATAL: failed to build validator:"); + Console.Error.WriteLine(ex); + return 1; +} + +// --- Stream the corpus ---------------------------------------------------- + +// Buffered stdout; UTF-8 without BOM; '\n' line endings to match the JS runner. +var stdout = new StreamWriter( + Console.OpenStandardOutput(), + new UTF8Encoding(encoderShouldEmitUTF8Identifier: false)) +{ + NewLine = "\n", + AutoFlush = false +}; + +var i = 0; +using (stdout) +using (var reader = new StreamReader(corpusPath, Encoding.UTF8)) +{ + string line; + while ((line = reader.ReadLine()) is not null) + { + if (line.Trim().Length == 0) + { + // Skip blank lines without consuming an index, matching the JS runner + // which filters empty lines before enumerating. + continue; + } + + var record = Evaluate(i, line, schema, validator); + stdout.WriteLine(JsonSerializer.Serialize(record, jsonOptions)); + i++; + } + + stdout.Flush(); +} + +return 0; + +// --- Helpers -------------------------------------------------------------- + +static Result Evaluate(int i, string line, ISchemaDefinition schema, DocumentValidator validator) +{ + // 1) Decode the corpus line (a JSON-encoded string) into the raw query. + string query; + try + { + query = JsonSerializer.Deserialize(line); + } + catch (Exception) + { + return new Result + { + I = i, + ParseError = true, + Valid = false, + ErrorCount = 0, + Messages = ["CORPUS_LINE_NOT_JSON"] + }; + } + + if (query is null) + { + return new Result + { + I = i, + ParseError = true, + Valid = false, + ErrorCount = 0, + Messages = ["CORPUS_LINE_NULL"] + }; + } + + // 2) Parse the GraphQL document. + DocumentNode document; + try + { + document = Utf8GraphQLParser.Parse(query); + } + catch (SyntaxException ex) + { + return new Result + { + I = i, + ParseError = true, + Valid = false, + ErrorCount = 0, + Messages = [FirstLine(ex.Message)] + }; + } + catch (Exception ex) + { + // Any other parser failure is still a parse error for diffing purposes. + return new Result + { + I = i, + ParseError = true, + Valid = false, + ErrorCount = 0, + Messages = [FirstLine(ex.Message)] + }; + } + + // 3) Validate against the full default rule set. + try + { + var result = validator.Validate(schema, document); + var messages = new List(5); + foreach (var error in result.Errors) + { + if (messages.Count == 5) + { + break; + } + + messages.Add(error.Message); + } + + return new Result + { + I = i, + ParseError = false, + Valid = !result.HasErrors, + ErrorCount = result.Errors.Count, + Messages = messages.ToArray() + }; + } + catch (Exception ex) + { + // Validation itself threw (should be rare). Treat as invalid, signal -1. + return new Result + { + I = i, + ParseError = false, + Valid = false, + ErrorCount = -1, + Messages = ["VALIDATE_THREW: " + FirstLine(ex.Message)] + }; + } +} + +static ISchemaDefinition BuildSchema(string sdl) +{ + // Validation-only schema: + // - StrictValidation=false: relax spec-completeness checks; we only need a + // schema the validator can read, not an executable one. + // - EnableDefer/EnableStream: match graphql-js v17 so @defer/@stream are known. + // - .Use(next => next): a no-op field middleware so fields have a pipeline and + // the schema builds without resolvers. + // - AnyType("GeoPoint"): bind the SDL's custom scalar to a permissive scalar + // that accepts any literal, mirroring graphql-js's treatment of a custom + // scalar with no parseLiteral (no input-literal validation). Without a + // binding the type initializer cannot resolve references to GeoPoint. + return SchemaBuilder.New() + .AddDocumentFromString(sdl) + .AddType(new AnyType("GeoPoint")) + .Use(next => next) + .ModifyOptions(o => + { + o.StrictValidation = false; + o.EnableDefer = true; + o.EnableStream = true; + }) + .Create(); +} + +static string FirstLine(string message) +{ + if (string.IsNullOrEmpty(message)) + { + return message; + } + + var idx = message.IndexOfAny(['\r', '\n']); + return idx < 0 ? message : message[..idx]; +} + +// --- Output shape --------------------------------------------------------- + +internal sealed class Result +{ + [JsonPropertyName("i")] + public int I { get; init; } + + [JsonPropertyName("parseError")] + public bool ParseError { get; init; } + + [JsonPropertyName("valid")] + public bool Valid { get; init; } + + [JsonPropertyName("errorCount")] + public int ErrorCount { get; init; } + + [JsonPropertyName("messages")] + public string[] Messages { get; init; } = []; +} diff --git a/fuzzing/graphql-validation/README.md b/fuzzing/graphql-validation/README.md new file mode 100644 index 00000000000..fef4d92a0a4 --- /dev/null +++ b/fuzzing/graphql-validation/README.md @@ -0,0 +1,48 @@ +# GraphQL validation differential fuzzer + +Differentially fuzz GraphQL document **validation** between the spec reference +implementation (graphql-js) and HotChocolate, to find documents that graphql-js +rejects as invalid but HotChocolate accepts (HC missed a spec violation). + +See [FINDINGS.md](./FINDINGS.md) for what it has turned up. + +## Pieces + +| File | Role | +| --- | --- | +| `schema.graphql` | Shared schema (graphql-js validation harness, adapted). Loaded by both engines. | +| `seeds.json` | 36 curated known-invalid documents (verified against graphql-js). | +| `hard.json` | 35 hand-written hard cases targeting rules implementations classically diverge on. | +| `fuzz.mjs` | AST-mutation fuzzer. Mutates valid bases + seeds into parseable-but-likely-invalid docs targeting specific rules. | +| `node-runner.mjs` | Reference runner: `graphql-js` `validate(schema, parse(q), specifiedRules)`. | +| `HcValidationRunner/` | HotChocolate runner: `DocumentValidatorBuilder.New().AddDefaultRules()` against the schema-first schema. | +| `diff.mjs` | Compares the two result files; reports `hc_missed` / `hc_extra` / `parse_disagree`. | + +## I/O contract + +- **Corpus** (`corpus.jsonl`): one JSON-encoded query string per line. +- **Results** (`*-results.jsonl`): one record per corpus line, aligned by index: + `{ "i", "parseError", "valid", "errorCount", "messages": [..] }`. +- **Diff categories**: `hc_missed` (reference invalid, HC valid — the prime signal), + `hc_extra` (reference valid, HC invalid), `parse_disagree` (one parser errored, the other didn't). + +## Run + +```bash +npm install # graphql@17 +dotnet build -c Release HcValidationRunner/HcValidationRunner.csproj + +node fuzz.mjs --count 15000 --seed 2 +node node-runner.mjs schema.graphql corpus.jsonl > node-results.jsonl +dotnet HcValidationRunner/bin/Release/net10.0/HcValidationRunner.dll schema.graphql corpus.jsonl > hc-results.jsonl +node diff.mjs corpus.jsonl node-results.jsonl hc-results.jsonl out.json +``` + +## Parity notes + +- Reference: graphql-js **v17.0.1**, `specifiedRules`. HotChocolate: full default rule set. +- HC schema-first build enables defer/stream and registers `GeoPoint` as a permissive + scalar (`AnyType`) so a custom scalar with no `parseLiteral` matches graphql-js behaviour. +- The schema omits `@oneOf` to keep both parsers identical. +- graphql-js v17 has defer/stream + max-introspection-depth rules; if comparing against a + HotChocolate that lacks any of these, exclude them from `specifiedRules` to avoid noise. diff --git a/fuzzing/graphql-validation/diff.mjs b/fuzzing/graphql-validation/diff.mjs new file mode 100644 index 00000000000..5167cd379e7 --- /dev/null +++ b/fuzzing/graphql-validation/diff.mjs @@ -0,0 +1,92 @@ +// Compare reference (graphql-js) vs HotChocolate validation results. +// Usage: node diff.mjs [out.json] +// +// Categories (both parse OK unless noted): +// hc_missed : reference INVALID, HotChocolate VALID -> HC missed a spec violation (PRIME signal) +// hc_extra : reference VALID, HotChocolate INVALID -> HC stricter than the spec +// parse_disagree: one parser errored and the other did not +import { readFileSync, writeFileSync } from 'node:fs'; + +const [corpusPath, nodePath, hcPath, outPath] = process.argv.slice(2); +if (!corpusPath || !nodePath || !hcPath) { + console.error('usage: node diff.mjs [out.json]'); + process.exit(2); +} + +const readJsonl = (p) => + readFileSync(p, 'utf8') + .split('\n') + .filter((l) => l.trim().length > 0) + .map((l) => JSON.parse(l)); + +const corpus = readFileSync(corpusPath, 'utf8') + .split('\n') + .filter((l) => l.trim().length > 0) + .map((l) => JSON.parse(l)); + +const nodeBy = new Map(readJsonl(nodePath).map((r) => [r.i, r])); +const hcBy = new Map(readJsonl(hcPath).map((r) => [r.i, r])); + +const buckets = { hc_missed: [], hc_extra: [], parse_disagree: [] }; +let compared = 0; +let agree = 0; + +for (const [i, n] of nodeBy) { + const h = hcBy.get(i); + if (!h) continue; + compared++; + const query = corpus[i]; + + if (n.parseError !== h.parseError) { + buckets.parse_disagree.push({ i, query, node: summarize(n), hc: summarize(h) }); + continue; + } + if (n.parseError && h.parseError) { + agree++; + continue; // both rejected at parse time; not a validation signal + } + if (!n.valid && h.valid) { + buckets.hc_missed.push({ i, query, refErrors: n.messages }); + } else if (n.valid && !h.valid) { + buckets.hc_extra.push({ i, query, hcErrors: h.messages }); + } else { + agree++; + } +} + +function summarize(r) { + return { parseError: r.parseError, valid: r.valid, errorCount: r.errorCount, messages: r.messages }; +} + +const report = { + summary: { + compared, + agree, + hc_missed: buckets.hc_missed.length, + hc_extra: buckets.hc_extra.length, + parse_disagree: buckets.parse_disagree.length, + }, + hc_missed: buckets.hc_missed, + hc_extra: buckets.hc_extra, + parse_disagree: buckets.parse_disagree, +}; + +if (outPath) { + writeFileSync(outPath, JSON.stringify(report, null, 2)); +} + +// Console summary + the prime signal. +console.log('=== differential validation summary ==='); +console.log(JSON.stringify(report.summary, null, 2)); +const show = (title, arr, fmt) => { + if (!arr.length) return; + console.log(`\n=== ${title} (${arr.length}) ===`); + for (const x of arr.slice(0, 30)) console.log(fmt(x)); + if (arr.length > 30) console.log(`... and ${arr.length - 30} more (see ${outPath ?? 'out.json'})`); +}; +show('HC MISSED (reference invalid, HC valid)', buckets.hc_missed, (x) => + ` #${x.i} ${JSON.stringify(x.query)}\n ref: ${x.refErrors.join(' | ')}`); +show('HC EXTRA (reference valid, HC invalid)', buckets.hc_extra, (x) => + ` #${x.i} ${JSON.stringify(x.query)}\n hc: ${x.hcErrors.join(' | ')}`); +show('PARSE DISAGREE', buckets.parse_disagree, (x) => + ` #${x.i} ${JSON.stringify(x.query)}\n ref=${JSON.stringify(x.node)} hc=${JSON.stringify(x.hc)}`); diff --git a/fuzzing/graphql-validation/fuzz.mjs b/fuzzing/graphql-validation/fuzz.mjs new file mode 100644 index 00000000000..8d1c057d515 --- /dev/null +++ b/fuzzing/graphql-validation/fuzz.mjs @@ -0,0 +1,149 @@ +// AST-mutation fuzzer. Produces parseable-but-likely-invalid documents that target +// specific validation rules, so the differential focuses on VALIDATION (not syntax). +// Usage: node fuzz.mjs [--count 5000] [--seed 1] [--seeds seeds.json] [--out corpus.jsonl] +import { readFileSync, writeFileSync } from 'node:fs'; +import { parse, print, visit, Kind } from 'graphql'; + +const args = process.argv.slice(2); +const opt = (name, def) => { const i = args.indexOf(name); return i >= 0 ? args[i + 1] : def; }; +const N = parseInt(opt('--count', '5000'), 10); +const seedArg = opt('--seed', null); +const seedsPath = opt('--seeds', new URL('./seeds.json', import.meta.url).pathname); +const outPath = opt('--out', new URL('./corpus.jsonl', import.meta.url).pathname); + +let rng = Math.random; +if (seedArg !== null) { + let s = (parseInt(seedArg, 10) >>> 0) || 1; + rng = () => { s = (s + 0x6d2b79f5) | 0; let t = Math.imul(s ^ (s >>> 15), 1 | s); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; +} +const ri = (n) => Math.floor(rng() * n); +const pick = (a) => a[ri(a.length)]; +const rstr = (p) => p + ri(1e6).toString(36); +const NAME = (v) => ({ kind: Kind.NAME, value: v }); + +const BASE = [ + '{ dog { name barkVolume } }', + '{ dog { name(surname: true) nickname } }', + 'query Q($c: Boolean) { dog { isHouseTrained(atOtherHomes: $c) } }', + 'query Q($cmd: DogCommand) { dog { doesKnowCommand(dogCommand: $cmd) } }', + '{ complicatedArgs { multipleReqs(req1: 1, req2: 2) } }', + '{ complicatedArgs { enumArgField(enumArg: BROWN) } }', + '{ complicatedArgs { complexArgField(complexArg: { requiredField: true }) } }', + '{ complicatedArgs { nestedArgField(nested: { required: 1, child: { requiredField: true }, ints: [1, 2] }) } }', + '{ complicatedArgs { stringListArgField(stringListArg: ["a", "b"]) } }', + '{ catOrDog { ... on Dog { name barkVolume } ... on Cat { meows } } }', + 'fragment F on Dog { name barkVolume } { dog { ...F } }', + '{ human { name pets { name } relatives { name } } }', + 'mutation { createDog(name: "rex") { name } }', + 'subscription { newMessage }', + '{ dog @onField { name } }', + 'query Named @onQuery { dog { name } }', + '{ pet { name ... on Dog { barkVolume } ... on Cat { meowsVolume } } }', + '{ catOrDog { ... on Dog { name barkVolume tag } ... on Cat { nickname meowsVolume tag } } }', + '{ dogOrHuman { __typename ... on Dog { name } ... on Human { name relatives { name } } } }', + '{ pet { name ... on Canine { mother { name } father { name } } } }', + '{ catOrDog { __typename ... on Dog { x: name } ... on Cat { x: nickname } } }', + '{ dog { name @complex(arg: "x") } }', +]; + +const seeds = JSON.parse(readFileSync(seedsPath, 'utf8')).map((s) => s.query); + +function countKind(doc, kind) { let n = 0; visit(doc, { [kind]: { enter() { n++; } } }); return n; } +function replaceNth(doc, kind, target, fn) { + let cur = 0, changed = false; + const nd = visit(doc, { [kind]: { enter(node, key, parent, path, ancestors) { + if (cur++ === target) { const r = fn(node, { key, parent, path, ancestors }); if (r !== undefined) { changed = true; return r; } } + } } }); + return changed ? nd : null; +} +function mutKind(doc, kind, fn) { const c = countKind(doc, kind); if (!c) return null; return replaceNth(doc, kind, ri(c), fn); } + +const EDGE_VALUES = [ + { kind: Kind.INT, value: '2147483648' }, + { kind: Kind.INT, value: '-2147483649' }, + { kind: Kind.INT, value: '99999999999999999999' }, + { kind: Kind.FLOAT, value: '1e400' }, + { kind: Kind.FLOAT, value: '1.5' }, + { kind: Kind.STRING, value: 'x', block: false }, + { kind: Kind.BOOLEAN, value: true }, + { kind: Kind.NULL }, + { kind: Kind.ENUM, value: 'NOPE' }, + { kind: Kind.LIST, values: [] }, + { kind: Kind.OBJECT, fields: [] }, +]; + +const fieldMutators = [ + (d) => mutKind(d, Kind.FIELD, (n) => ({ ...n, name: NAME(rstr('zfield_')) })), + (d) => mutKind(d, Kind.ARGUMENT, (n) => ({ ...n, name: NAME(rstr('zarg_')) })), + (d) => mutKind(d, Kind.FIELD, (n) => (n.arguments && n.arguments.length ? { ...n, arguments: [] } : undefined)), + (d) => mutKind(d, Kind.ENUM, (n) => ({ ...n, value: rstr('ZENUM_').toUpperCase() })), + (d) => mutKind(d, Kind.ARGUMENT, (n) => ({ ...n, value: { kind: Kind.INT, value: String(ri(999)) } })), + (d) => mutKind(d, Kind.ARGUMENT, (n) => ({ ...n, value: { kind: Kind.STRING, value: rstr('s_'), block: false } })), + // edge-value injection: surfaces ValuesOfCorrectType range/coercion gaps (e.g. Int overflow). + (d) => mutKind(d, Kind.ARGUMENT, (n) => ({ ...n, value: pick(EDGE_VALUES) })), + (d) => mutKind(d, Kind.OBJECT_FIELD, (n) => ({ ...n, value: pick(EDGE_VALUES) })), + (d) => mutKind(d, Kind.VARIABLE, (n) => ({ ...n, name: NAME(rstr('undef_')) })), + (d) => mutKind(d, Kind.FIELD, (n) => ({ ...n, directives: [...(n.directives || []), { kind: Kind.DIRECTIVE, name: NAME(rstr('zdir_')) }] })), + (d) => mutKind(d, Kind.FIELD, (n) => ({ ...n, directives: [...(n.directives || []), { kind: Kind.DIRECTIVE, name: NAME('onField') }, { kind: Kind.DIRECTIVE, name: NAME('onField') }] })), + (d) => mutKind(d, Kind.INLINE_FRAGMENT, (n) => ({ ...n, typeCondition: { kind: Kind.NAMED_TYPE, name: NAME(pick(['Cat', 'Human', 'Int', 'GeoPoint', 'NoSuchType'])) } })), + (d) => mutKind(d, Kind.FRAGMENT_DEFINITION, (n) => ({ ...n, typeCondition: { kind: Kind.NAMED_TYPE, name: NAME(pick(['Cat', 'Int', 'NoSuchType'])) } })), + (d) => mutKind(d, Kind.FIELD, (n) => (n.selectionSet ? undefined : { ...n, selectionSet: { kind: Kind.SELECTION_SET, selections: [{ kind: Kind.FIELD, name: NAME(rstr('sub_')) }] } })), + (d) => mutKind(d, Kind.FIELD, (n) => (n.selectionSet ? { ...n, selectionSet: undefined } : undefined)), + (d) => mutKind(d, Kind.OPERATION_DEFINITION, (n) => ({ ...n, variableDefinitions: [...(n.variableDefinitions || []), { kind: Kind.VARIABLE_DEFINITION, variable: { kind: Kind.VARIABLE, name: NAME(rstr('unused_')) }, type: { kind: Kind.NAMED_TYPE, name: NAME('Int') } }] })), + // OverlappingFieldsCanBeMerged: alias the first two fields in a set to the same name. + (d) => mutKind(d, Kind.SELECTION_SET, (n) => { + const fs = n.selections.filter((s) => s.kind === Kind.FIELD); + if (fs.length < 2) return undefined; + const a = { ...fs[0], alias: NAME('m') }, b = { ...fs[1], alias: NAME('m') }; + return { ...n, selections: n.selections.map((s) => (s === fs[0] ? a : s === fs[1] ? b : s)) }; + }), + // directive used in the wrong location (known directive, illegal spot). + (d) => mutKind(d, Kind.OPERATION_DEFINITION, (n) => ({ ...n, directives: [...(n.directives || []), { kind: Kind.DIRECTIVE, name: NAME('onField') }] })), + (d) => mutKind(d, Kind.FIELD, (n) => ({ ...n, directives: [...(n.directives || []), { kind: Kind.DIRECTIVE, name: NAME(pick(['onQuery', 'deprecated', 'skip'])) }] })), + // variable type changes -> position / coercion mismatches. + (d) => mutKind(d, Kind.VARIABLE_DEFINITION, (n) => ({ ...n, type: n.type.kind === Kind.NON_NULL_TYPE ? n.type.type : n.type })), + (d) => mutKind(d, Kind.VARIABLE_DEFINITION, (n) => ({ ...n, type: { kind: Kind.NAMED_TYPE, name: NAME(pick(['String', 'Boolean', 'Float', 'DogCommand', 'ComplexInput'])) } })), + // inject an inline fragment with a varied (often impossible / wrong-kind) type condition. + (d) => mutKind(d, Kind.SELECTION_SET, (n) => ({ + ...n, + selections: [...n.selections, { + kind: Kind.INLINE_FRAGMENT, + typeCondition: { kind: Kind.NAMED_TYPE, name: NAME(pick(['Dog', 'Cat', 'Human', 'Pet', 'Mammal', 'Canine', 'CatOrDog', 'DogOrHuman', 'GeoPoint', 'DogCommand', 'ComplexInput', 'NoSuchType'])) }, + selectionSet: { kind: Kind.SELECTION_SET, selections: [{ kind: Kind.FIELD, name: NAME(pick(['name', 'barkVolume', 'meows', 'tag', 'nickname', '__typename'])) }] }, + }], + })), +]; + +const docMutators = [ + (d) => { const ops = d.definitions.filter((x) => x.kind === Kind.OPERATION_DEFINITION && x.name); if (!ops.length) return null; return { ...d, definitions: [...d.definitions, pick(ops)] }; }, + (d) => { const anon = { kind: Kind.OPERATION_DEFINITION, operation: 'query', selectionSet: { kind: Kind.SELECTION_SET, selections: [{ kind: Kind.FIELD, name: NAME('dog'), selectionSet: { kind: Kind.SELECTION_SET, selections: [{ kind: Kind.FIELD, name: NAME('name') }] } }] } }; return { ...d, definitions: [...d.definitions, anon] }; }, + (d) => { const frag = { kind: Kind.FRAGMENT_DEFINITION, name: NAME(rstr('Unused_')), typeCondition: { kind: Kind.NAMED_TYPE, name: NAME('Dog') }, selectionSet: { kind: Kind.SELECTION_SET, selections: [{ kind: Kind.FIELD, name: NAME('name') }] } }; return { ...d, definitions: [...d.definitions, frag] }; }, + (d) => { const sp = { kind: Kind.FRAGMENT_SPREAD, name: NAME(rstr('Missing_')) }; let done = false; const nd = visit(d, { [Kind.SELECTION_SET]: { enter(n) { if (!done) { done = true; return { ...n, selections: [...n.selections, sp] }; } } } }); return done ? nd : null; }, +]; + +const ALL = [...fieldMutators, ...docMutators]; +const tryParse = (s) => { try { return parse(s); } catch { return null; } }; + +const corpus = new Set(); +for (const q of [...seeds, ...BASE]) corpus.add(q); + +let attempts = 0; +while (corpus.size < N && attempts < N * 20) { + attempts++; + const base = pick([...BASE, ...seeds]); + let doc = tryParse(base); + if (!doc) continue; + const rounds = 1 + ri(2); + for (let r = 0; r < rounds; r++) { + let nd = null; + try { nd = pick(ALL)(doc); } catch { nd = null; } + if (nd) doc = nd; + } + let printed; + try { printed = print(doc); } catch { continue; } + if (printed.trim().length) corpus.add(printed); +} + +const arr = [...corpus].slice(0, N); +writeFileSync(outPath, arr.map((q) => JSON.stringify(q)).join('\n') + '\n'); +console.error(`wrote ${arr.length} queries to ${outPath} (attempts=${attempts})`); diff --git a/fuzzing/graphql-validation/hard.json b/fuzzing/graphql-validation/hard.json new file mode 100644 index 00000000000..3b85d76be05 --- /dev/null +++ b/fuzzing/graphql-validation/hard.json @@ -0,0 +1,37 @@ +[ + { "rule": "OverlappingFieldsCanBeMerged", "query": "{ dog { x: nickname x: barkVolume } }" }, + { "rule": "OverlappingFieldsCanBeMerged", "query": "{ dog { doesKnowCommand(dogCommand: SIT) doesKnowCommand(dogCommand: HEEL) } }" }, + { "rule": "OverlappingFieldsCanBeMerged", "query": "{ dog { name name(surname: true) } }" }, + { "rule": "OverlappingFieldsCanBeMerged", "query": "{ pet { ... on Dog { name(surname: true) } ... on Cat { name } } }" }, + { "rule": "OverlappingFieldsCanBeMerged", "query": "{ dog { mother { x: name } father { x: nickname } } }" }, + { "rule": "VariablesInAllowedPosition", "query": "query ($v: Int) { complicatedArgs { nonNullIntArgField(nonNullIntArg: $v) } }" }, + { "rule": "VariablesInAllowedPosition", "query": "query ($v: [String]) { complicatedArgs { stringListNonNullArgField(stringListNonNullArg: $v) } }" }, + { "rule": "VariablesInAllowedPosition", "query": "query ($v: String) { complicatedArgs { intArgField(intArg: $v) } }" }, + { "rule": "ValuesOfCorrectType-null-for-nonnull", "query": "{ complicatedArgs { nonNullIntArgField(nonNullIntArg: null) } }" }, + { "rule": "ValuesOfCorrectType-string-for-int", "query": "{ complicatedArgs { intArgField(intArg: \"nope\") } }" }, + { "rule": "ValuesOfCorrectType-int-overflow", "query": "{ complicatedArgs { intArgField(intArg: 99999999999999999999) } }" }, + { "rule": "ValuesOfCorrectType-bool-for-float", "query": "{ complicatedArgs { floatArgField(floatArg: true) } }" }, + { "rule": "ValuesOfCorrectType-float-for-id", "query": "{ complicatedArgs { idArgField(idArg: 1.5) } }" }, + { "rule": "ValuesOfCorrectType-string-for-enum", "query": "{ complicatedArgs { enumArgField(enumArg: \"BROWN\") } }" }, + { "rule": "ValuesOfCorrectType-int-for-enum", "query": "{ complicatedArgs { enumArgField(enumArg: 1) } }" }, + { "rule": "ValuesOfCorrectType-missing-required-input-field", "query": "{ complicatedArgs { complexArgField(complexArg: { intField: 1 }) } }" }, + { "rule": "ValuesOfCorrectType-nested-wrong-type", "query": "{ complicatedArgs { complexArgField(complexArg: { requiredField: true, intField: \"x\" }) } }" }, + { "rule": "ValuesOfCorrectType-unknown-input-field", "query": "{ complicatedArgs { complexArgField(complexArg: { requiredField: true, bogusField: 1 }) } }" }, + { "rule": "UniqueInputFieldNames", "query": "{ complicatedArgs { complexArgField(complexArg: { requiredField: true, requiredField: false }) } }" }, + { "rule": "ValuesOfCorrectType-list-nullitem", "query": "{ complicatedArgs { stringListNonNullArgField(stringListNonNullArg: [\"a\", null]) } }" }, + { "rule": "SingleFieldSubscriptions-multi-root", "query": "subscription { newDog { name } newHuman { name } }" }, + { "rule": "SingleFieldSubscriptions-introspection", "query": "subscription { __typename }" }, + { "rule": "KnownDirectives-field-dir-on-query", "query": "query @onField { dog { name } }" }, + { "rule": "KnownDirectives-query-dir-on-field", "query": "{ dog @onQuery { name } }" }, + { "rule": "KnownDirectives-deprecated-in-query", "query": "{ dog { name @deprecated } }" }, + { "rule": "ProvidedRequiredArguments-skip-missing-if", "query": "{ dog { name @skip } }" }, + { "rule": "PossibleFragmentSpreads-inline", "query": "{ dog { ... on Cat { meows } } }" }, + { "rule": "PossibleFragmentSpreads-named", "query": "fragment catFrag on Cat { meows } { dog { ...catFrag } }" }, + { "rule": "DeferStream-defer-on-mutation-root", "query": "mutation { ... @defer { createDog(name: \"x\") { name } } }" }, + { "rule": "DeferStream-stream-on-nonlist", "query": "{ dog { name @stream } }" }, + { "rule": "DeferStream-defer-on-subscription", "query": "subscription { ... @defer { newMessage } }" }, + { "rule": "ScalarLeafs-subselection-on-scalar", "query": "{ dog { barkVolume { x } } }" }, + { "rule": "FieldsOnCorrectType-on-interface", "query": "{ pet { barkVolume } }" }, + { "rule": "KnownArgumentNames-on-directive", "query": "{ dog { name @skip(iff: true) } }" }, + { "rule": "Anonymous-with-named", "query": "{ dog { name } } query Other { cat { name } }" } +] diff --git a/fuzzing/graphql-validation/hard2.json b/fuzzing/graphql-validation/hard2.json new file mode 100644 index 00000000000..683cd216b4d --- /dev/null +++ b/fuzzing/graphql-validation/hard2.json @@ -0,0 +1,19 @@ +[ + { "rule": "nested-int-overflow", "query": "{ complicatedArgs { nestedArgField(nested: { required: 2147483648 }) } }" }, + { "rule": "nested-list-int-overflow", "query": "{ complicatedArgs { nestedArgField(nested: { required: 1, ints: [1, 2147483648] }) } }" }, + { "rule": "nested-list-for-input", "query": "{ complicatedArgs { nestedArgField(nested: { required: 1, child: [] }) } }" }, + { "rule": "nested-list-of-input-contains-list", "query": "{ complicatedArgs { nestedArgField(nested: { required: 1, children: [{ requiredField: true }, []] }) } }" }, + { "rule": "nested-missing-required", "query": "{ complicatedArgs { nestedArgField(nested: { child: { requiredField: true } }) } }" }, + { "rule": "default-string-for-int", "query": "query ($v: Int = \"s\") { complicatedArgs { intArgField(intArg: $v) } }" }, + { "rule": "default-list-for-input", "query": "query ($v: ComplexInput = []) { complicatedArgs { complexArgField(complexArg: $v) } }" }, + { "rule": "default-bad-enum", "query": "query ($v: FurColor = NOPE) { complicatedArgs { enumArgField(enumArg: $v) } }" }, + { "rule": "default-int-overflow", "query": "query ($v: Int = 2147483648) { complicatedArgs { intArgField(intArg: $v) } }" }, + { "rule": "default-missing-required-input-field", "query": "query ($v: ComplexInput = { intField: 1 }) { complicatedArgs { complexArgField(complexArg: $v) } }" }, + { "rule": "default-nested-int-overflow", "query": "query ($v: NestedInput = { required: 2147483648 }) { complicatedArgs { nestedArgField(nested: $v) } }" }, + { "rule": "float-infinity", "query": "{ complicatedArgs { floatArgField(floatArg: 1e400) } }" }, + { "rule": "float-from-string", "query": "{ complicatedArgs { floatArgField(floatArg: \"NaN\") } }" }, + { "rule": "VALID-int-for-float", "query": "{ complicatedArgs { floatArgField(floatArg: 3) } }" }, + { "rule": "VALID-list-coercion-single", "query": "{ complicatedArgs { stringListArgField(stringListArg: \"single\") } }" }, + { "rule": "VALID-id-int", "query": "{ complicatedArgs { idArgField(idArg: 12345) } }" }, + { "rule": "id-bool-bad", "query": "{ complicatedArgs { idArgField(idArg: true) } }" } +] diff --git a/fuzzing/graphql-validation/hard3.json b/fuzzing/graphql-validation/hard3.json new file mode 100644 index 00000000000..a155656b50e --- /dev/null +++ b/fuzzing/graphql-validation/hard3.json @@ -0,0 +1,25 @@ +[ + { "rule": "INVALID-field-on-union", "query": "{ catOrDog { name } }" }, + { "rule": "INVALID-field-on-union2", "query": "{ dogOrHuman { name } }" }, + { "rule": "INVALID-union-no-subselection", "query": "{ catOrDog }" }, + { "rule": "INVALID-object-field-on-interface", "query": "{ pet { barkVolume } }" }, + { "rule": "INVALID-interface-frag-field-missing", "query": "{ dog { ... on Pet { barkVolume } } }" }, + { "rule": "INVALID-impossible-inline-object-object", "query": "{ dog { ... on Cat { meows } } }" }, + { "rule": "INVALID-impossible-inline-nonimpl-on-interface", "query": "{ pet { ... on Human { relatives { name } } } }" }, + { "rule": "INVALID-impossible-inline-nonmember-on-union", "query": "{ catOrDog { ... on Human { name } } }" }, + { "rule": "INVALID-impossible-named-spread", "query": "fragment df on Dog { barkVolume } { cat { ...df } }" }, + { "rule": "INVALID-inline-on-scalar", "query": "{ dog { ... on GeoPoint { x } } }" }, + { "rule": "INVALID-inline-on-enum", "query": "{ catOrDog { ... on DogCommand { x } } }" }, + { "rule": "INVALID-merge-conflict-across-union-types", "query": "{ catOrDog { ... on Dog { tag } ... on Cat { tag } } }" }, + { "rule": "INVALID-merge-conflict-alias-across-union", "query": "{ catOrDog { ... on Dog { x: tag } ... on Cat { x: nickname } } }" }, + { "rule": "INVALID-merge-conflict-interface-vs-impl", "query": "{ pet { ... on Dog { x: tag } name x: name } }" }, + { "rule": "INVALID-spread-union-on-disjoint-object", "query": "{ human { ... on CatOrDog { __typename } } }" }, + { "rule": "VALID-typename-on-union", "query": "{ catOrDog { __typename } }" }, + { "rule": "VALID-fragments-on-union-members", "query": "{ catOrDog { ... on Dog { name barkVolume } ... on Cat { name meowsVolume } } }" }, + { "rule": "VALID-mutually-exclusive-alias-same-shape", "query": "{ catOrDog { ... on Dog { x: name } ... on Cat { x: nickname } } }" }, + { "rule": "VALID-mutually-exclusive-alias-same-int", "query": "{ catOrDog { ... on Dog { x: barkVolume } ... on Cat { x: meowsVolume } } }" }, + { "rule": "VALID-interface-field-plus-inline", "query": "{ pet { name ... on Dog { barkVolume } } }" }, + { "rule": "VALID-interface-inline-on-interface", "query": "{ pet { ... on Canine { name } } }" }, + { "rule": "VALID-interface-inline-on-object", "query": "{ dog { ... on Mammal { mother { name } } } }" }, + { "rule": "VALID-union-inline-on-overlapping-interface", "query": "{ pet { ... on CatOrDog { __typename } } }" } +] diff --git a/fuzzing/graphql-validation/node-runner.mjs b/fuzzing/graphql-validation/node-runner.mjs new file mode 100644 index 00000000000..19e41d59a25 --- /dev/null +++ b/fuzzing/graphql-validation/node-runner.mjs @@ -0,0 +1,57 @@ +// graphql-js (reference) validation runner. +// Usage: node node-runner.mjs > node-results.jsonl +// Input : corpus.jsonl, one JSON-encoded query string per line. +// Output: one JSON record per line, aligned by index `i`: +// { i, parseError, valid, errorCount, messages: [..] } +import { readFileSync } from 'node:fs'; +import { buildSchema, parse, validate, specifiedRules } from 'graphql'; + +const [schemaPath, corpusPath] = process.argv.slice(2); +if (!schemaPath || !corpusPath) { + console.error('usage: node node-runner.mjs '); + process.exit(2); +} + +const schema = buildSchema(readFileSync(schemaPath, 'utf8')); +const lines = readFileSync(corpusPath, 'utf8').split('\n').filter((l) => l.trim().length > 0); + +const results = lines.map((line, i) => { + const rec = { i, parseError: false, valid: true, errorCount: 0, messages: [] }; + let query; + try { + query = JSON.parse(line); + } catch { + rec.parseError = true; + rec.valid = false; + rec.messages = ['CORPUS_LINE_NOT_JSON']; + return rec; + } + + let doc; + try { + doc = parse(query); + } catch (e) { + rec.parseError = true; + rec.valid = false; + rec.messages = [String(e.message)]; + return rec; + } + + let errors; + try { + errors = validate(schema, doc, specifiedRules); + } catch (e) { + // validate throws e.g. on too many errors; treat as invalid. + rec.valid = false; + rec.errorCount = -1; + rec.messages = ['VALIDATE_THREW: ' + e.message]; + return rec; + } + + rec.errorCount = errors.length; + rec.valid = errors.length === 0; + rec.messages = errors.slice(0, 5).map((er) => er.message); + return rec; +}); + +process.stdout.write(results.map((r) => JSON.stringify(r)).join('\n') + '\n'); diff --git a/fuzzing/graphql-validation/package.json b/fuzzing/graphql-validation/package.json new file mode 100644 index 00000000000..777ecd6a224 --- /dev/null +++ b/fuzzing/graphql-validation/package.json @@ -0,0 +1,9 @@ +{ + "name": "gql-validation-fuzz", + "private": true, + "type": "module", + "description": "Differential validation fuzzer: graphql-js (reference) vs HotChocolate.", + "dependencies": { + "graphql": "^17.0.1" + } +} diff --git a/fuzzing/graphql-validation/schema.graphql b/fuzzing/graphql-validation/schema.graphql new file mode 100644 index 00000000000..08659c59816 --- /dev/null +++ b/fuzzing/graphql-validation/schema.graphql @@ -0,0 +1,133 @@ +# Shared schema for differential validation fuzzing (graphql-js vs HotChocolate). +# Adapted from the graphql-js validation test harness. Must load identically in +# graphql-js `buildSchema` and HotChocolate schema-first. Avoids @oneOf to keep +# both parsers happy; custom scalar + custom directives are declared explicitly. + +schema { + query: Query + mutation: Mutation + subscription: Subscription +} + +directive @onField on FIELD +directive @onQuery on QUERY +directive @complex(arg: String) repeatable on FIELD | FRAGMENT_SPREAD + +scalar GeoPoint + +enum DogCommand { + SIT + HEEL + DOWN +} + +enum FurColor { + BROWN + BLACK + TAN + SPOTTED + NO_FUR + UNKNOWN +} + +interface Mammal { + mother: Mammal + father: Mammal +} + +interface Pet { + name(surname: Boolean): String +} + +interface Canine implements Mammal { + name(surname: Boolean): String + mother: Canine + father: Canine +} + +type Dog implements Pet & Mammal & Canine { + name(surname: Boolean): String + nickname: String + barkVolume: Int + barks: Boolean + tag: Int + doesKnowCommand(dogCommand: DogCommand): Boolean + isHouseTrained(atOtherHomes: Boolean = true): Boolean + isAtLocation(x: Int, y: Int): Boolean + distanceFrom(loc: GeoPoint): Float + mother: Dog + father: Dog +} + +type Cat implements Pet { + name(surname: Boolean): String + nickname: String + meows: Boolean + meowsVolume: Int + tag: String + furColor: FurColor +} + +union CatOrDog = Cat | Dog + +union DogOrHuman = Dog | Human + +type Human { + name(surname: Boolean): String + pets: [Pet] + relatives: [Human]! +} + +input ComplexInput { + requiredField: Boolean! + nonNullField: Boolean! = false + intField: Int + stringField: String + booleanField: Boolean + stringListField: [String] +} + +input NestedInput { + required: Int! + child: ComplexInput + children: [ComplexInput!] + ints: [Int!] +} + +type ComplicatedArgs { + intArgField(intArg: Int): String + nonNullIntArgField(nonNullIntArg: Int!): String + stringArgField(stringArg: String): String + booleanArgField(booleanArg: Boolean): String + enumArgField(enumArg: FurColor): String + floatArgField(floatArg: Float): String + idArgField(idArg: ID): String + stringListArgField(stringListArg: [String]): String + stringListNonNullArgField(stringListNonNullArg: [String!]): String + complexArgField(complexArg: ComplexInput): String + nestedArgField(nested: NestedInput): String + multipleReqs(req1: Int!, req2: Int!): String + nonNullFieldWithDefault(arg: Int! = 0): String + multipleOpts(opt1: Int = 0, opt2: Int = 0): String + multipleOptAndReq(req1: Int!, req2: Int!, opt1: Int = 0, opt2: Int = 0): String +} + +type Query { + human(id: ID): Human + dog: Dog + cat: Cat + pet: Pet + catOrDog: CatOrDog + dogOrHuman: DogOrHuman + complicatedArgs: ComplicatedArgs +} + +type Mutation { + createDog(name: String!): Dog +} + +type Subscription { + newMessage: String + newDog: Dog + newHuman: Human +} diff --git a/fuzzing/graphql-validation/seeds.json b/fuzzing/graphql-validation/seeds.json new file mode 100644 index 00000000000..d953a6084cf --- /dev/null +++ b/fuzzing/graphql-validation/seeds.json @@ -0,0 +1,146 @@ +[ + { + "rule": "FieldsOnCorrectTypeRule", + "query": "{ dog { meowVolume } }" + }, + { + "rule": "FieldsOnCorrectTypeRule", + "query": "fragment f on Dog { unknown_field }" + }, + { + "rule": "ScalarLeafsRule", + "query": "{ dog { barkVolume { sinceWhen } } }" + }, + { + "rule": "ScalarLeafsRule", + "query": "{ human }" + }, + { + "rule": "KnownArgumentNamesRule", + "query": "{ dog { doesKnowCommand(command: SIT) } }" + }, + { + "rule": "ProvidedRequiredArgumentsRule", + "query": "{ complicatedArgs { multipleReqs(req1: 1) } }" + }, + { + "rule": "ProvidedRequiredArgumentsRule", + "query": "{ complicatedArgs { nonNullIntArgField } }" + }, + { + "rule": "ValuesOfCorrectTypeRule", + "query": "{ complicatedArgs { intArgField(intArg: \"abc\") } }" + }, + { + "rule": "ValuesOfCorrectTypeRule", + "query": "{ complicatedArgs { enumArgField(enumArg: GREEN) } }" + }, + { + "rule": "ValuesOfCorrectTypeRule", + "query": "{ complicatedArgs { complexArgField(complexArg: { intField: 1 }) } }" + }, + { + "rule": "NoUndefinedVariablesRule", + "query": "query ($a: Int) { complicatedArgs { intArgField(intArg: $b) } }" + }, + { + "rule": "NoUnusedVariablesRule", + "query": "query ($a: Int, $b: Int) { complicatedArgs { intArgField(intArg: $a) } }" + }, + { + "rule": "NoUnusedFragmentsRule", + "query": "query { dog { name } } fragment Unused on Dog { barkVolume }" + }, + { + "rule": "KnownFragmentNamesRule", + "query": "{ dog { ...UnknownFragment } }" + }, + { + "rule": "NoFragmentCyclesRule", + "query": "fragment a on Dog { ...b } fragment b on Dog { ...a }" + }, + { + "rule": "NoFragmentCyclesRule", + "query": "fragment self on Human { relatives { ...self } }" + }, + { + "rule": "UniqueOperationNamesRule", + "query": "query Foo { dog { name } } query Foo { cat { name } }" + }, + { + "rule": "LoneAnonymousOperationRule", + "query": "{ dog { name } } query Named { cat { name } }" + }, + { + "rule": "UniqueFragmentNamesRule", + "query": "fragment F on Dog { name } fragment F on Cat { name } { dog { ...F } }" + }, + { + "rule": "FragmentsOnCompositeTypesRule", + "query": "fragment scalarFrag on Boolean { __typename }" + }, + { + "rule": "FragmentsOnCompositeTypesRule", + "query": "{ dog { ... on DogCommand { __typename } } }" + }, + { + "rule": "PossibleFragmentSpreadsRule", + "query": "fragment c on Cat { __typename } { dog { ...c } }" + }, + { + "rule": "UniqueArgumentNamesRule", + "query": "{ dog { isHouseTrained(atOtherHomes: true, atOtherHomes: false) } }" + }, + { + "rule": "UniqueDirectivesPerLocationRule", + "query": "{ dog { name @skip(if: true) @skip(if: false) } }" + }, + { + "rule": "KnownDirectivesRule", + "query": "{ dog { name @unknownDirective } }" + }, + { + "rule": "KnownDirectivesRule", + "query": "query @skip(if: true) { dog { name } }" + }, + { + "rule": "KnownDirectivesRule (@onField in wrong location)", + "query": "query @onField { dog { name } }" + }, + { + "rule": "KnownOperationTypesRule", + "query": "subscription { dog { name } cat { name } }" + }, + { + "rule": "VariablesAreInputTypesRule", + "query": "query ($d: Dog) { dog { name } }" + }, + { + "rule": "UniqueVariableNamesRule", + "query": "query ($x: Int, $x: Int) { complicatedArgs { intArgField(intArg: $x) } }" + }, + { + "rule": "UniqueInputFieldNamesRule", + "query": "{ complicatedArgs { complexArgField(complexArg: { requiredField: true, intField: 1, intField: 2 }) } }" + }, + { + "rule": "VariablesInAllowedPositionRule", + "query": "query ($s: String) { complicatedArgs { intArgField(intArg: $s) } }" + }, + { + "rule": "ExecutableDefinitionsRule", + "query": "type Extra { id: ID } { dog { name } }" + }, + { + "rule": "KnownTypeNamesRule", + "query": "fragment f on NotAType { __typename } { dog { ...f } }" + }, + { + "rule": "OverlappingFieldsCanBeMergedRule", + "query": "{ dog { x: name x: barkVolume } }" + }, + { + "rule": "OverlappingFieldsCanBeMergedRule", + "query": "{ dog { doesKnowCommand(dogCommand: SIT) doesKnowCommand(dogCommand: HEEL) } }" + } +]