Skip to content

@gustavo/feat/queries scraping - #2

Merged
gustavobftorres merged 3 commits into
mainfrom
@gustavo/feat/queries-scraping
Mar 27, 2026
Merged

@gustavo/feat/queries scraping#2
gustavobftorres merged 3 commits into
mainfrom
@gustavo/feat/queries-scraping

Conversation

@gustavobftorres

@gustavobftorres gustavobftorres commented Mar 27, 2026

Copy link
Copy Markdown
Collaborator

Is this linked to an existing issue?
N/A

Fill out the following table describing your edits:

Original Updated Change Reasoning
Balancer workspace queries queries.yml + imported SQL files under balancer/ Bulk imported and categorized existing Balancer query IDs from workspace export; updated manifest categories to match folder placement Establish a local source-of-truth and structured repo layout before Phase 4 dependency work
scripts/validate.py (pre-fix) scripts/validate.py (current) Fixed validator path handling so it scans repository SQL files correctly and reports real consistency results Reliability fix required to unblock validation of large import sets
queries.yml (pre-cleanup) queries.yml (current) Removed 4 manifest IDs with no corresponding SQL files (4600099, 4600227, 4605635, 5875552) Resolve hard manifest/file mismatches and get consistency checks to zero errors
scripts/pull_from_dune.py (pre-hardening) scripts/pull_from_dune.py (current) Added skip-and-continue behavior for inaccessible query IDs with explicit success/failure summary Prevent full-run aborts during ingestion when API returns non-query payloads

Provide any other context or screenshots that explain or justify the changes above:

  • This PR is intended as a Phase 3 checkpoint before dependency resolution.
  • Validation now scans successfully and reports 0 errors (remaining unresolved dependencies are warnings to be handled in Phase 4).
  • Important: merging SQL changes to main will trigger Sync to Dune; use temporary workflow disable and/or [skip ci] on merge if Dune updates should be avoided for this checkpoint.

Checklist:

  • SQL file name follows descriptive_name_{queryId}.sql convention
  • File includes -- part of a query repo header
  • Query ID is listed in queries.yml
  • SQLFluff passes locally (sqlfluff lint <file>) (full-repo pass has known legacy debt; changed-files strategy used)
  • If adding a new query: created on dune.com first, then added to repo
  • If adding new Jinja parameters: updated the relevant .sqlfluff context

@github-actions

github-actions Bot commented Mar 27, 2026

Copy link
Copy Markdown

Are you sure you want to use the legacy prices.usd table? Consider using prices.minute (docs)

Caused by:

@github-actions

github-actions Bot commented Mar 27, 2026

Copy link
Copy Markdown

This query reads from a partitioned table without a block_date filter. Adding WHERE block_date >= ... enables partition pruning and significantly reduces cost. (efficient queries guide)

Caused by:

@gustavobftorres
gustavobftorres merged commit a88bfbf into main Mar 27, 2026
1 of 2 checks passed
@gustavobftorres
gustavobftorres deleted the @gustavo/feat/queries-scraping branch March 27, 2026 17:56
gustavobftorres added a commit that referenced this pull request Apr 15, 2026
@gustavobftorres
Merge pull request #2 from gustavobftorres/@gustavo/feat/queries-scra…
ac8f7a6 
…ping

@gustavo/feat/queries scraping
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gustavobftorres