Skip to content

fix(tesseract): generate the time series in SQL on Snowflake - #11785

Open
waralexrom wants to merge 2 commits into
masterfrom
tesseract-snowflake-generated-time-series
Open

fix(tesseract): generate the time series in SQL on Snowflake#11785
waralexrom wants to merge 2 commits into
masterfrom
tesseract-snowflake-generated-time-series

Conversation

@waralexrom

Copy link
Copy Markdown
Member

Summary

On Snowflake, any measure with a rolling_window queried at a granularity but
without a dateRange failed with Date range is required for time series
(reported on v1.7.26 with CUBEJS_TESSERACT_SQL_PLANNER=true). SnowflakeQuery
defined no generated_time_series_select, so the planner had no way to derive the
series bounds from the data and rejected the query. This adds that template, so
Snowflake joins the seven dialects that already generate the series in SQL.

Changes

  • SnowflakeQuery: generated_time_series_select and
    generated_time_series_with_cte_range_source.
    Snowflake has no
    generate_series, and GENERATOR only takes a literal row count; a recursive
    CTE is out because Snowflake requires an explicit column list for one, which the
    planner cannot emit (it renders time_series AS (...)). ARRAY_GENERATE_RANGE
    does accept arbitrary expressions for its bounds, so the series covers whatever
    range the query turns out to need instead of a fixed number of rows — the only
    ceiling is the maximum size of a single ARRAY value.
  • SnowflakeQuery.intervalAndMinimalTimeUnit override. The series steps with
    DATEADD, which takes a time unit and a row number, so it needs the unit the
    interval is actually expressed in. The inherited diffTimeUnitForInterval
    degrades WEEK to DAY and QUARTER to MONTH, which would produce seven or
    three times as many periods as asked for. This method feeds nothing but the
    generated-time-series templates, so the override is contained to Snowflake.
  • Driver tests un-skipped. The three tesseractSkip entries in
    fixtures/snowflake.json covering rolling window by 2 day / by 2 month / YTD without date range are removed, with the snapshots added. They stay skipped on
    the legacy planner, which still builds the series outside the database.
  • Docs. The rolling_window warning claimed Tesseract lifts the date-range
    requirement outright; it now says the data source has to be able to generate the
    series, lists which ones can, and notes that custom granularities on BigQuery,
    MSSQL and Snowflake still need the range.

Notes on the shape of the emitted SQL

Two things differ from the existing dialect templates and are easy to get wrong:

  • Snowflake folds unquoted identifiers to upper case, so the two output columns
    are emitted as AS "date_from" / AS "date_to" and the range CTE is read as
    time_series_get_range."min_date". The Postgres, MSSQL and Databricks templates
    get away with bare names.
  • The series is built as timestamp_ntz to match the time dimension
    (CONVERT_TIMEZONE(...)::timestamp_ntz) and the non-generated series
    (date_from::timestamp). timeStampCast returns ::timestamp_tz, so the
    template deliberately does not use the date_from / date_to parameters.

Known limitation

ARRAY_GENERATE_RANGE's row number is a count of whole time units, which is all
DATEADD can step by. A custom granularity's interval carries no such count, so
supportGeneratedSeriesForCustomTd stays off and those queries keep asking for an
explicit date range — the same position BigQuery and MSSQL are in today.

Testing

New unit test packages/cubejs-schema-compiler/test/unit/generated-time-series.test.ts
(23 cases). On the unfixed code 8 of them failed, all Snowflake, all with
Date range is required for time series; Postgres passed throughout. With the fix
all 23 pass. It covers, for both Postgres and Snowflake:

  • a rolling window with no date range at every predefined granularity
    (second through year);
  • a rolling window with an explicit date range, unchanged;
  • a bounded (trailing: 30 day) window with no date range;
  • Snowflake stepping the series by week / quarter rather than by their
    smallest time unit;
  • Snowflake quoting date_from / date_to;
  • a custom granularity still requiring the date range, and still planning with one.

Neighbouring unit suites are green: base-query, dialect-intervals,
like-filter-escaping, mssql-query, postgres-query, oracle-query.

The driver-test snapshots are predicted, not recorded — I have no Snowflake
credentials, so the generated SQL was never executed against a real Snowflake, and
its syntax was checked against the Snowflake docs only. The expected rows are the
ones Postgres, Databricks and Trino already record for these three tests byte for
byte, and all 16 rolling-window snapshots Snowflake already carries match Postgres
exactly. drivers-tests triggers on packages/cubejs-schema-compiler/** and runs
snowflake with the planner both on and off, so that run is what actually proves
this; if a snapshot is off, its diff will say exactly how.

Unrelated defect noticed on the way

MSSQL has the same minimal_time_unit problem this PR works around for Snowflake:
its template does DATEADD({{ minimal_time_unit }}, 1, ...), so a week series
steps by day and a quarter series by month. Confirmed by dumping the SQL. Left
alone here — happy to file it separately.

waralexrom and others added 2 commits September 6, 2026 16:01
A rolling window asked for at some granularity but with no date range has
to derive the bounds of its time series from the data. The planner only
does that where the dialect defines `generated_time_series_select`, so on
Snowflake every such query was rejected with "Date range is required for
time series".

Snowflake has no generate_series, and GENERATOR only takes a literal row
count. ARRAY_GENERATE_RANGE accepts arbitrary expressions for its bounds,
so the series covers whatever range the query turns out to need. Its row
number is a count of whole time units, which is all DATEADD can step by,
so custom granularities keep asking for an explicit date range.

`intervalAndMinimalTimeUnit` is overridden because the DATEADD step needs
the unit the interval is actually expressed in, while the inherited one
degrades WEEK to DAY and QUARTER to MONTH.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…esseract

Snowflake now generates the time series in SQL, so the three driver tests
that ask for a rolling window at some granularity without a date range no
longer have to be skipped on the native planner. They stay skipped on the
legacy planner, which still builds the series outside the database.

The expected rows are the ones Postgres, Databricks and Trino already
record for these tests byte for byte, and every rolling-window snapshot
Snowflake already carries matches Postgres exactly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@waralexrom
waralexrom requested review from a team and keydunov as code owners September 6, 2026 14:20
@mintlify

mintlify Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
cubed3 🟢 Ready View Preview Sep 6, 2026, 2:21 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions github-actions Bot added the javascript Pull requests that update Javascript code label Sep 6, 2026
@claude

claude Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Claude finished @waralexrom's task in 4m 12s —— View job


Review: approach is right; 1 medium correctness risk in the emitted SQL, 3 low. 4 inline comments posted.

Full review
  • Gather context (diff, planner-side template call sites, sibling dialects)
  • Review SnowflakeQuery changes
  • Review tests & fixtures
  • Review docs
  • Post review

What's good

The diagnosis and the shape of the fix hold up against the planner side. Two things I checked specifically and they're right:

  • generated_time_series_with_cte_range_source receives min_name/max_name unquoted (plan.rs:595-608 passes them raw, unlike time_series_get_range which quotes them at plan.rs:399), while the range CTE aliases them through SelectBuilderquote_identifier"min_date". So the {{ range_source }}."{{ min_name }}" quoting in the template is necessary and correct, and the bare {{ min_name }} that Postgres/MSSQL get away with would have failed on Snowflake. Easy to get wrong; good catch.
  • supportGeneratedSeriesForCustomTd staying off is the right call given ARRAY_GENERATE_RANGE's row count is unit-scalar — time_series.rs:82 gates on is_predefined_granularity(), so custom granularities keep the old path.

Findings

# Severity Where Issue
1 Medium SnowflakeQuery.ts:177-184 DATEDIFF counts boundary crossings, not whole units, so an unaligned start/min_date yields one row past the range end vs. Postgres. Most exposed on the CTE path, where the bounds come from MIN()/MAX() over raw data — the exact path the un-skipped driver tests take.
2 Low SnowflakeQuery.ts:97-101 new RegExp(u, 'i') substring-matches ('1 millisecond'SECOND); || 'year' silently mislabels instead of throwing as diffTimeUnitForInterval does; the interval count is dropped while both templates ignore {{ granularity }}.
3 Low SnowflakeQuery.ts:165-176 12-line comment mostly restating the commit message; the one rationale a future edit could undo (timestamp_ntz vs timeStampCast) isn't in it.
4 Low generated-time-series.test.ts:71-87 toContain('time_series') passes on the non-generated path too — the assertions amount to "did not throw".

Other notes (no action required)

  • Blast radius is wider than the bug. Defining these templates switches every Snowflake time-series query onto the generated path, including ones that already worked with an explicit dateRange. That's inherent to how supports_generated_time_series is gated and not avoidable, but it means the 16 existing rolling-window snapshots are load-bearing regression coverage here, not just the 3 new ones.
  • ARRAY_GENERATE_RANGE ceiling. The array is materialised, so a data-derived range at second granularity is one element per second (~31M for a year) and will hit Snowflake's single-ARRAY size limit with an opaque error rather than degrading. Fine as a trade for the fix, but it's a runtime failure mode the other dialects don't have — worth a line in the docs section this PR already touches, next to the custom-granularity caveat.
  • Snapshots are predicted, not recorded — as the PR body says. Finding 1 is a plausible mechanism for them to come back off by one trailing bucket, so the drivers-tests Snowflake run (planner on and off) really is the gate here.
  • The MSSQL minimal_time_unit defect noted in the PR body is real and matches what I see at MssqlQuery.ts:349-366. Agreed it belongs in a separate issue.
· branch `tesseract-snowflake-generated-time-series`

Comment on lines +177 to +184
templates.statements.generated_time_series_select = 'SELECT series_date AS "date_from",\n' +
'DATEADD(MILLISECOND, -1, DATEADD({{ minimal_time_unit }}, 1, series_date)) AS "date_to"\n' +
'FROM (SELECT DATEADD({{ minimal_time_unit }}, series_index.value::int, {{ start }}::timestamp_ntz) AS series_date\n' +
'FROM TABLE(FLATTEN(input => ARRAY_GENERATE_RANGE(0, DATEDIFF({{ minimal_time_unit }}, {{ start }}::timestamp_ntz, {{ end }}::timestamp_ntz) + 1))) AS series_index) AS series';
templates.statements.generated_time_series_with_cte_range_source = 'SELECT series_date AS "date_from",\n' +
'DATEADD(MILLISECOND, -1, DATEADD({{ minimal_time_unit }}, 1, series_date)) AS "date_to"\n' +
'FROM (SELECT DATEADD({{ minimal_time_unit }}, series_index.value::int, {{ range_source }}."{{ min_name }}") AS series_date\n' +
'FROM {{ range_source }}, LATERAL FLATTEN(input => ARRAY_GENERATE_RANGE(0, DATEDIFF({{ minimal_time_unit }}, {{ range_source }}."{{ min_name }}", {{ range_source }}."{{ max_name }}") + 1)) AS series_index) AS series';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Row count can overshoot the range end. Snowflake's DATEDIFF counts unit boundaries crossed, not whole elapsed units, so when start / min_date is not aligned to the unit boundary DATEDIFF(...) + 1 yields one row more than Postgres' generate_series(start, end, interval) does, and the extra row starts after end.

Concretely, month granularity with min_date = 2024-01-15, max_date = 2024-02-05:

  • DATEDIFF(month, '2024-01-15', '2024-02-05') = 1ARRAY_GENERATE_RANGE(0, 2) → series 2024-01-15, 2024-02-15
  • Postgres: generate_series('2024-01-15','2024-02-05','1 month')2024-01-15 only

The CTE-range variant is the more exposed of the two, since min_date/max_date come straight from MIN()/MAX() over the data and are almost never boundary-aligned — which is exactly the path the three un-skipped driver tests take. That is also a plausible way for the predicted snapshots to come back with one extra trailing bucket.

Cheap guard — bound the series explicitly instead of relying on DATEDIFF being exact:

  • generated_time_series_select: append \nWHERE series_date <= {{ end }}::timestamp_ntz after AS series.
  • generated_time_series_with_cte_range_source: the outer scope can't see range_source, so the filter has to go on the inner select, repeating the expression:
    ... AS series_index WHERE DATEADD({{ minimal_time_unit }}, series_index.value::int, {{ range_source }}."{{ min_name }}") <= {{ range_source }}."{{ max_name }}"

With that in place + 1 stays a safe over-allocation rather than an exact count.

Comment on lines +97 to +101
public override intervalAndMinimalTimeUnit(interval: string): [string, string] {
const unit = INTERVAL_UNITS.find(u => new RegExp(u, 'i').test(interval));

return [interval, unit || 'year'];
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three things about this override worth tightening:

  1. new RegExp(u, 'i') is an unanchored substring match, so '1 millisecond' matches second (milli**second**) and returns SECOND — a silently wrong unit rather than an error. A word-boundary pattern (new RegExp(\b${u}s?\b, 'i')) or interval.toLowerCase().includes(...) with millisecond handled explicitly avoids it.
  2. The || 'year' fallback turns an unrecognised interval into a plausible-looking series instead of failing. The inherited diffTimeUnitForInterval throws in that case; matching it would be safer here, since a wrong unit produces silently wrong data rather than an error.
  3. The count is dropped: '2 week'['2 week', 'week'], and both templates step by 1 {{ minimal_time_unit }} and ignore {{ granularity }} entirely, so the series would come out at the wrong period. Unreachable today because supportGeneratedSeriesForCustomTd is false — but nothing here records that dependency, so flipping that flag on later would silently produce a wrong series. Throwing unless the interval is exactly 1 <unit> would make the coupling enforced rather than documented.

Comment on lines +165 to +176
// Snowflake has no generate_series, and GENERATOR only takes a literal row
// count. ARRAY_GENERATE_RANGE does accept arbitrary expressions for its
// bounds, so the series covers whatever range the query turns out to need;
// the only ceiling is the maximum size of a single ARRAY value.
//
// Its row number is a count of whole time units, which is all DATEADD can
// step by. A custom granularity's interval carries no such count, so
// `supportGeneratedSeriesForCustomTd` stays off and those queries keep
// asking for an explicit date range.
//
// Snowflake folds unquoted identifiers to upper case while the planner reads
// the series back by lower-case name, so the two output columns are quoted.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a 12-line comment over two template strings, and most of it re-states the PR description rather than protecting the code below. The load-bearing sentences are the two that a later edit could otherwise undo: why the columns are quoted (identifier folding) and why timestamp_ntz rather than timeStampCast — the latter isn't stated here at all, which is the one a future reader is most likely to "fix" by swapping in {{ date_from }}/{{ date_to }}. The dialect survey (no generate_series, GENERATOR takes a literal) and the custom-granularity paragraph belong in the commit message, which already carries them. Suggest trimming to ~3 lines and moving the timestamp_ntz rationale in.

Comment on lines +71 to +87
it.each(PREDEFINED_GRANULARITIES)('plans a rolling window at %s granularity with no date range', async (granularity) => {
const sql = await buildSql(QueryClass, { granularity });

expect(sql).toContain('time_series');
});

it('keeps planning a rolling window with an explicit date range', async () => {
const sql = await buildSql(QueryClass, { dateRange: ['2024-01-01', '2024-12-31'] });

expect(sql).toContain('time_series');
});

it('plans a bounded rolling window with no date range', async () => {
const sql = await buildSql(QueryClass, { measure: 'events.rolling_30d_users' });

expect(sql).toContain('time_series');
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toContain('time_series') is satisfied by the non-generated path too, so these cases really only assert "did not throw". Since the whole point is that the series is now built in SQL, asserting the generator is present would pin the behaviour: e.g. ARRAY_GENERATE_RANGE for Snowflake and generate_series for Postgres, keyed off the dialect in GENERATING_DIALECTS. Worth it especially for the rolling_30d_users case, where a regression back to the legacy series would still pass as written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript Pull requests that update Javascript code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant