Skip to content

github: Batch graphql requests to avoid resource limit errors#271

Open
f0rmiga wants to merge 1 commit into
Skydio:mainfrom
f0rmiga:batch-graphql-requests
Open

github: Batch graphql requests to avoid resource limit errors#271
f0rmiga wants to merge 1 commit into
Skydio:mainfrom
f0rmiga:batch-graphql-requests

Conversation

@f0rmiga

@f0rmiga f0rmiga commented Jul 1, 2026

Copy link
Copy Markdown

Problem

With a large enough topic stack, revup upload fails outright with GitHub graphql errors — intermittently Resource limits for this query exceeded or Timeout on validation of query.

Two call sites build a single request that grows linearly (or worse) with stack size:

  • Github.query_everything() aliases one pullRequests(headRefName:) lookup per head ref into one query. Since every topic in a relative chain also adds its relative branch to the lookup list, a 41-topic stack produces ~80+ aliases, each nesting reviews, assignees, labels, timeline items, and comments.
  • create_pull_requests() / update_pull_requests() alias one mutation per PR into one request. These hit the limit even sooner: each PR's body and review-graph comment lists the whole relative chain, so the payload grows quadratically with stack size.

Fix

Batch the requests instead of aliasing everything into one:

  • query_everything() looks up head refs 10 per query (MAX_HEAD_REFS_PER_QUERY); users/labels/teams ride along with the first batch. Results are merged in order, so callers are unaffected.
  • create_pull_requests() / update_pull_requests() send mutations 5 PRs per request (MAX_PRS_PER_MUTATION), smaller because of the body/comment payloads.

The public methods keep their signatures (the original bodies became _impl methods), and stacks below the batch sizes send byte-identical requests to before, so there is no behavior change for small stacks. Batches are sequential; they could be issued concurrently with asyncio.gather if you prefer — happy to change it.

Verification

  • Reproduced on a 41-topic relative-chain stack in a large monorepo: the single-query form fails reproducibly during real uploads, while the batched form completes and updated all 41 PRs.
  • Stress-tested query_everything() read-only against 2,173 head refs (218 batches) without errors, and smoke-tested this branch's code against the real API.
  • make lint (mypy, ruff, pylint) and make test (201 passed) are clean.

Github rejects graphql requests that are too large or expensive with
errors like "Resource limits for this query exceeded" or "Timeout on
validation of query". query_everything() aliases one pullRequests
lookup per head ref into a single query, and create/update_pull_requests
alias one mutation per PR, so a large enough topic stack makes uploads
fail entirely. Mutations hit the limit even sooner because each PR body
and review-graph comment lists the whole relative chain, making the
payload grow quadratically with stack size.

Batch the head ref lookups 10 per query and the PR mutations 5 per
request, merging results in order. Stacks below the batch sizes send
byte-identical requests to before.

Verified against a 41-topic stack in a large monorepo where uploads
previously failed: the single query reproducibly errors while the
batched version succeeds, including a 2173-ref stress run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant