fix(tesseract): generate the time series in SQL on Snowflake - #11785
fix(tesseract): generate the time series in SQL on Snowflake#11785waralexrom wants to merge 2 commits into
Conversation
A rolling window asked for at some granularity but with no date range has to derive the bounds of its time series from the data. The planner only does that where the dialect defines `generated_time_series_select`, so on Snowflake every such query was rejected with "Date range is required for time series". Snowflake has no generate_series, and GENERATOR only takes a literal row count. ARRAY_GENERATE_RANGE accepts arbitrary expressions for its bounds, so the series covers whatever range the query turns out to need. Its row number is a count of whole time units, which is all DATEADD can step by, so custom granularities keep asking for an explicit date range. `intervalAndMinimalTimeUnit` is overridden because the DATEADD step needs the unit the interval is actually expressed in, while the inherited one degrades WEEK to DAY and QUARTER to MONTH. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…esseract Snowflake now generates the time series in SQL, so the three driver tests that ask for a rolling window at some granularity without a date range no longer have to be skipped on the native planner. They stay skipped on the legacy planner, which still builds the series outside the database. The expected rows are the ones Postgres, Databricks and Trino already record for these tests byte for byte, and every rolling-window snapshot Snowflake already carries matches Postgres exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
|
Claude finished @waralexrom's task in 4m 12s —— View job Review: approach is right; 1 medium correctness risk in the emitted SQL, 3 low. 4 inline comments posted.Full review
What's goodThe diagnosis and the shape of the fix hold up against the planner side. Two things I checked specifically and they're right:
Findings
Other notes (no action required)
|
| templates.statements.generated_time_series_select = 'SELECT series_date AS "date_from",\n' + | ||
| 'DATEADD(MILLISECOND, -1, DATEADD({{ minimal_time_unit }}, 1, series_date)) AS "date_to"\n' + | ||
| 'FROM (SELECT DATEADD({{ minimal_time_unit }}, series_index.value::int, {{ start }}::timestamp_ntz) AS series_date\n' + | ||
| 'FROM TABLE(FLATTEN(input => ARRAY_GENERATE_RANGE(0, DATEDIFF({{ minimal_time_unit }}, {{ start }}::timestamp_ntz, {{ end }}::timestamp_ntz) + 1))) AS series_index) AS series'; | ||
| templates.statements.generated_time_series_with_cte_range_source = 'SELECT series_date AS "date_from",\n' + | ||
| 'DATEADD(MILLISECOND, -1, DATEADD({{ minimal_time_unit }}, 1, series_date)) AS "date_to"\n' + | ||
| 'FROM (SELECT DATEADD({{ minimal_time_unit }}, series_index.value::int, {{ range_source }}."{{ min_name }}") AS series_date\n' + | ||
| 'FROM {{ range_source }}, LATERAL FLATTEN(input => ARRAY_GENERATE_RANGE(0, DATEDIFF({{ minimal_time_unit }}, {{ range_source }}."{{ min_name }}", {{ range_source }}."{{ max_name }}") + 1)) AS series_index) AS series'; |
There was a problem hiding this comment.
Row count can overshoot the range end. Snowflake's DATEDIFF counts unit boundaries crossed, not whole elapsed units, so when start / min_date is not aligned to the unit boundary DATEDIFF(...) + 1 yields one row more than Postgres' generate_series(start, end, interval) does, and the extra row starts after end.
Concretely, month granularity with min_date = 2024-01-15, max_date = 2024-02-05:
DATEDIFF(month, '2024-01-15', '2024-02-05') = 1→ARRAY_GENERATE_RANGE(0, 2)→ series2024-01-15,2024-02-15- Postgres:
generate_series('2024-01-15','2024-02-05','1 month')→2024-01-15only
The CTE-range variant is the more exposed of the two, since min_date/max_date come straight from MIN()/MAX() over the data and are almost never boundary-aligned — which is exactly the path the three un-skipped driver tests take. That is also a plausible way for the predicted snapshots to come back with one extra trailing bucket.
Cheap guard — bound the series explicitly instead of relying on DATEDIFF being exact:
generated_time_series_select: append\nWHERE series_date <= {{ end }}::timestamp_ntzafterAS series.generated_time_series_with_cte_range_source: the outer scope can't seerange_source, so the filter has to go on the inner select, repeating the expression:
... AS series_index WHERE DATEADD({{ minimal_time_unit }}, series_index.value::int, {{ range_source }}."{{ min_name }}") <= {{ range_source }}."{{ max_name }}"
With that in place + 1 stays a safe over-allocation rather than an exact count.
| public override intervalAndMinimalTimeUnit(interval: string): [string, string] { | ||
| const unit = INTERVAL_UNITS.find(u => new RegExp(u, 'i').test(interval)); | ||
|
|
||
| return [interval, unit || 'year']; | ||
| } |
There was a problem hiding this comment.
Three things about this override worth tightening:
new RegExp(u, 'i')is an unanchored substring match, so'1 millisecond'matchessecond(milli**second**) and returnsSECOND— a silently wrong unit rather than an error. A word-boundary pattern (new RegExp(\b${u}s?\b, 'i')) orinterval.toLowerCase().includes(...)withmillisecondhandled explicitly avoids it.- The
|| 'year'fallback turns an unrecognised interval into a plausible-looking series instead of failing. The inheriteddiffTimeUnitForIntervalthrows in that case; matching it would be safer here, since a wrong unit produces silently wrong data rather than an error. - The count is dropped:
'2 week'→['2 week', 'week'], and both templates step by1 {{ minimal_time_unit }}and ignore{{ granularity }}entirely, so the series would come out at the wrong period. Unreachable today becausesupportGeneratedSeriesForCustomTdis false — but nothing here records that dependency, so flipping that flag on later would silently produce a wrong series. Throwing unless the interval is exactly1 <unit>would make the coupling enforced rather than documented.
| // Snowflake has no generate_series, and GENERATOR only takes a literal row | ||
| // count. ARRAY_GENERATE_RANGE does accept arbitrary expressions for its | ||
| // bounds, so the series covers whatever range the query turns out to need; | ||
| // the only ceiling is the maximum size of a single ARRAY value. | ||
| // | ||
| // Its row number is a count of whole time units, which is all DATEADD can | ||
| // step by. A custom granularity's interval carries no such count, so | ||
| // `supportGeneratedSeriesForCustomTd` stays off and those queries keep | ||
| // asking for an explicit date range. | ||
| // | ||
| // Snowflake folds unquoted identifiers to upper case while the planner reads | ||
| // the series back by lower-case name, so the two output columns are quoted. |
There was a problem hiding this comment.
This is a 12-line comment over two template strings, and most of it re-states the PR description rather than protecting the code below. The load-bearing sentences are the two that a later edit could otherwise undo: why the columns are quoted (identifier folding) and why timestamp_ntz rather than timeStampCast — the latter isn't stated here at all, which is the one a future reader is most likely to "fix" by swapping in {{ date_from }}/{{ date_to }}. The dialect survey (no generate_series, GENERATOR takes a literal) and the custom-granularity paragraph belong in the commit message, which already carries them. Suggest trimming to ~3 lines and moving the timestamp_ntz rationale in.
| it.each(PREDEFINED_GRANULARITIES)('plans a rolling window at %s granularity with no date range', async (granularity) => { | ||
| const sql = await buildSql(QueryClass, { granularity }); | ||
|
|
||
| expect(sql).toContain('time_series'); | ||
| }); | ||
|
|
||
| it('keeps planning a rolling window with an explicit date range', async () => { | ||
| const sql = await buildSql(QueryClass, { dateRange: ['2024-01-01', '2024-12-31'] }); | ||
|
|
||
| expect(sql).toContain('time_series'); | ||
| }); | ||
|
|
||
| it('plans a bounded rolling window with no date range', async () => { | ||
| const sql = await buildSql(QueryClass, { measure: 'events.rolling_30d_users' }); | ||
|
|
||
| expect(sql).toContain('time_series'); | ||
| }); |
There was a problem hiding this comment.
toContain('time_series') is satisfied by the non-generated path too, so these cases really only assert "did not throw". Since the whole point is that the series is now built in SQL, asserting the generator is present would pin the behaviour: e.g. ARRAY_GENERATE_RANGE for Snowflake and generate_series for Postgres, keyed off the dialect in GENERATING_DIALECTS. Worth it especially for the rolling_30d_users case, where a regression back to the legacy series would still pass as written.
Summary
On Snowflake, any measure with a
rolling_windowqueried at a granularity butwithout a
dateRangefailed withDate range is required for time series(reported on v1.7.26 with
CUBEJS_TESSERACT_SQL_PLANNER=true).SnowflakeQuerydefined no
generated_time_series_select, so the planner had no way to derive theseries bounds from the data and rejected the query. This adds that template, so
Snowflake joins the seven dialects that already generate the series in SQL.
Changes
SnowflakeQuery:generated_time_series_selectandgenerated_time_series_with_cte_range_source. Snowflake has nogenerate_series, andGENERATORonly takes a literal row count; a recursiveCTE is out because Snowflake requires an explicit column list for one, which the
planner cannot emit (it renders
time_series AS (...)).ARRAY_GENERATE_RANGEdoes accept arbitrary expressions for its bounds, so the series covers whatever
range the query turns out to need instead of a fixed number of rows — the only
ceiling is the maximum size of a single
ARRAYvalue.SnowflakeQuery.intervalAndMinimalTimeUnitoverride. The series steps withDATEADD, which takes a time unit and a row number, so it needs the unit theinterval is actually expressed in. The inherited
diffTimeUnitForIntervaldegrades
WEEKtoDAYandQUARTERtoMONTH, which would produce seven orthree times as many periods as asked for. This method feeds nothing but the
generated-time-series templates, so the override is contained to Snowflake.
tesseractSkipentries infixtures/snowflake.jsoncoveringrolling window by 2 day / by 2 month / YTD without date rangeare removed, with the snapshots added. They stay skipped onthe legacy planner, which still builds the series outside the database.
rolling_windowwarning claimed Tesseract lifts the date-rangerequirement outright; it now says the data source has to be able to generate the
series, lists which ones can, and notes that custom granularities on BigQuery,
MSSQL and Snowflake still need the range.
Notes on the shape of the emitted SQL
Two things differ from the existing dialect templates and are easy to get wrong:
are emitted as
AS "date_from"/AS "date_to"and the range CTE is read astime_series_get_range."min_date". The Postgres, MSSQL and Databricks templatesget away with bare names.
timestamp_ntzto match the time dimension(
CONVERT_TIMEZONE(...)::timestamp_ntz) and the non-generated series(
date_from::timestamp).timeStampCastreturns::timestamp_tz, so thetemplate deliberately does not use the
date_from/date_toparameters.Known limitation
ARRAY_GENERATE_RANGE's row number is a count of whole time units, which is allDATEADDcan step by. A custom granularity's interval carries no such count, sosupportGeneratedSeriesForCustomTdstays off and those queries keep asking for anexplicit date range — the same position BigQuery and MSSQL are in today.
Testing
New unit test
packages/cubejs-schema-compiler/test/unit/generated-time-series.test.ts(23 cases). On the unfixed code 8 of them failed, all Snowflake, all with
Date range is required for time series; Postgres passed throughout. With the fixall 23 pass. It covers, for both Postgres and Snowflake:
(second through year);
trailing: 30 day) window with no date range;week/quarterrather than by theirsmallest time unit;
date_from/date_to;Neighbouring unit suites are green:
base-query,dialect-intervals,like-filter-escaping,mssql-query,postgres-query,oracle-query.The driver-test snapshots are predicted, not recorded — I have no Snowflake
credentials, so the generated SQL was never executed against a real Snowflake, and
its syntax was checked against the Snowflake docs only. The expected rows are the
ones Postgres, Databricks and Trino already record for these three tests byte for
byte, and all 16 rolling-window snapshots Snowflake already carries match Postgres
exactly.
drivers-teststriggers onpackages/cubejs-schema-compiler/**and runssnowflakewith the planner both on and off, so that run is what actually provesthis; if a snapshot is off, its diff will say exactly how.
Unrelated defect noticed on the way
MSSQL has the same
minimal_time_unitproblem this PR works around for Snowflake:its template does
DATEADD({{ minimal_time_unit }}, 1, ...), so aweekseriessteps by day and a
quarterseries by month. Confirmed by dumping the SQL. Leftalone here — happy to file it separately.