release_branch/v2.6.0 by fivetran-savage · Pull Request #194 · fivetran/dbt_fivetran_log

fivetran-savage · 2026-05-21T22:29:35Z

PR Overview

Package version introduced in this PR:

2.6.0

This PR addresses the following Issue/Feature(s):

GA-1020307

Summary of changes:

Adds stg_fivetran_platform__connector_sdk_log to package.
Adds net-new end model, fivetran_platform__errors_and_warnings, that allows users to track severity levels of errors and warnings for both sdk and standard connections.
Adds fivetran_platform__transformation_run_log to the package to provide data on number of models run per transformation, whether the run succeeded/failed/partially succeeded, run duration, as well as the associated destination, connection(s), and full run log.
Adds fivetran_platform__sync_metrics model, which returns one record per completed ssync_stats log event, combining each sync's extract, process, load timing and volume statistics with its total duration and total records modified, enriched with connection and destination details
Adds stg_fivetran_platform__audit_trail to the package, which records user actions performed within your Fivetran account (what was changed, by whom, and how). Only available for customers on the Enterprise plan and above.
Adds fivetran_platform__audit_trail_enriched, which enriches each audit trail event with the names of the primary and secondary resources involved (resolved for CONNECTION, DESTINATION, ACCOUNT, and USER resources) and the details of the user who performed the action. Only available for customers on the Enterprise plan and above.
Adds the optional fivetran_platform_log_start_date variable to limit the amount of data included in log-based models. When set to a date (YYYY-MM-DD), stg_fivetran_platform__log filters to records on or after that date.
Changes materializes stg_fivetran_platform__log as table.

Submission Checklist

Alignment meeting with the reviewer (if needed)
- Timeline and validation requirements discussed
Provide validation details:
- Validation Steps: Check for unintentional effects (e.g., add/run consistency & integrity tests)
- Testing Instructions: Confirm the change addresses the issue(s)
- Focus Areas: Complex logic or queries that need extra attention
Merge any relevant open PRs into this PR

Changelog

Draft changelog for PR
Final changelog for release review

* Buildkite improvements FY27 Q2 (#192) * Buildkite improvements * Update dbt_project.yml --------- Co-authored-by: Catherine Fritz <111930712+fivetran-catfritz@users.noreply.github.com> * feature/add_errors_and_warnings_model * update_model * Update integration_tests/ci/test_scenarios.yml Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> --------- Co-authored-by: Fivetran Maintainer Bot <129092423+fivetran-data-model-bot@users.noreply.github.com> Co-authored-by: Catherine Fritz <111930712+fivetran-catfritz@users.noreply.github.com>

* feature/new_fivetran_log_end_model * add_consistancy_integrity_tests * revert_integration_yml * fix_syntax_for_sql_server * fix_syntax * remove_status_column * add_unnest_macro * revert_integration_yml * change_cte_names * update_syntax * Generate dbt docs via GitHub Actions * Update number of materialized models in README * update_connection_ids * update_macro * add_unknown_transformation_status * Update column types for dbt_project.yml --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* feature/add_audit_trail_events * add_audit_trail_end_model * dedup_connection_id

* feature/build_sync_statistics_and_duration_model Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix_boolean_for_sql_server * add_consistency_tests --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

amypeterson27 · 2026-07-14T22:29:14Z

hey @fivetran-savage , @fivetran-joemarkiewicz pointed me to this PR.

we were looking at how we could add sync details into this QDM, which it seems you were already working toward! there may be some changes that @fivetran-arakovskii will be making as part of https://fivetran.atlassian.net/browse/RD-1007740 so we should stay aligned here

fivetran-savage · 2026-07-14T22:46:55Z

@amypeterson27 thanks for the heads up! Most relevant to the fivetran_platform__sync_metrics model. Main thing to keep open lines of communication on is if there are going to be new fields added that might be useful to bring into the model, or if any field names are changing. This is a quarterly project for me, so looking to close it out in the next couple of weeks but definitely open to a meeting if you have specific requests as to metrics you'd like to see in the model that aren't there already.

…ns in transformation_run_log Retains the non-lookback changes originally bundled in cd2e8a0: - Add event_id to the fivetran_platform__errors_and_warnings surrogate key (and matching yml description); drop the stale "Excludes INFO-level events" README note. - Deduplicate connections in fivetran_platform__transformation_run_log by connection_id, preferring non-deleted / most recent set_up_at. - Add fivetran_platform_audit_trail_identifier to integration test vars. - DECISIONLOG formatting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

amypeterson27 · 2026-07-21T17:57:11Z

Hey @fivetran-savage — I wanted to flag a question about table-level coverage.

The current fivetran_platform__sync_metrics model gives us one row per completed sync, which is great. But customers are also asking for per-table granularity (rows extracted/loaded per table per sync).
The data should be available from two sources in the log table:

Loaded rows per table: records_modified events already fire per table and are already being parsed in your model for the total row_count — so this seems close
Extracted rows per table: will come from extract_summary.objects[] once the Donkey field additions land (that's the work tracked in RD-1007740)

The main question for you: is a second model — something like fivetran_platform__table_sync_metrics — feasible, given the JSON array unnesting required for extract_summary.objects[]? Or if this needs to wait for the next version, that's probably okay too. Would love to jump on a quick call if helpful.

* add_lookback_window_variable Adds the optional fivetran_platform_lookback_window_months variable to limit log history in log-based models. Filters stg_fivetran_platform__log_tmp (now materialized as a table) by time_stamp, with docs in README, CHANGELOG, and the quickstart supported_vars. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * update_changelog * update_seed_data * update_macro * update_changelog * Update CHANGELOG.md * update_variable_placement * update_documentation * update_variable * Apply suggestions from code review Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com>

fivetran-joemarkiewicz

@fivetran-savage great work on this PR. A few comments before approving.

fivetran-joemarkiewicz · 2026-07-24T21:47:48Z

+            {%- if var('fivetran_platform_using_user', true) %},
+            primary_user.email
+            {%- endif %}
+        ) as primary_resource_name,


I noticed the TEAM resource type always has a null value for these primary and secondary values. This is expected since we don't bring in TEAM yet.

Let's document this, but create a FR for us to bring in TEAM in the future.

fivetran-joemarkiewicz · 2026-07-28T20:13:19Z

+  `fivetran_platform__connection_status` 
+  `fivetran_platform__schema_changelog` 
+  `fivetran_platform__audit_user_activity` 
+  `fivetran_platform__connection_daily_events`
+  `fivetran_platform__audit_table`.


Suggested change

`fivetran_platform__connection_status`

`fivetran_platform__schema_changelog`

`fivetran_platform__audit_user_activity`

`fivetran_platform__connection_daily_events`

`fivetran_platform__audit_table`.

- `fivetran_platform__connection_status`

- `fivetran_platform__schema_changelog`

- `fivetran_platform__audit_user_activity`

- `fivetran_platform__connection_daily_events`

- `fivetran_platform__audit_table`

fivetran-joemarkiewicz · 2026-07-29T15:07:37Z

+supported_vars:
+  fivetran_platform_log_start_date:
+    type: string
+    description: "Earliest date (YYYY-MM-DD) of log history to include when building log-based models. When unset, all log history is used. Because it is a fixed date, a full refresh always reloads from this start date and never loses history. Affects `fivetran_platform__connection_status`, `fivetran_platform__schema_changelog`, `fivetran_platform__audit_user_activity`, `fivetran_platform__connection_daily_events`, and `fivetran_platform__audit_table`."


The newly added models in this PR are also impacted by the variable. Since this list is becoming exhaustive, it may be best to instead list which models are not impacted. It looks like it's just two: fivetran_platform__mar_table_history, fivetran_platform__usage_history

fivetran-joemarkiewicz · 2026-07-29T15:20:45Z

+  - name: "audit trail enabled"
+    vars:
+      fivetran_platform_using_audit_trail: true
+    include_incremental: false
+
+  - name: "connector sdk log enabled"
+    vars:
+      fivetran_platform_using_connector_sdk_log: true


In one of these let's also set the fivetran_platform_log_start_date variable to a relevant date in the seed data so we can continually test that functionality going forward as well.

These are both tables that many customers won't have enabled. Does that matter? I'm inclined to add another test on a table that more customers will have.

fivetran-joemarkiewicz · 2026-07-29T15:45:14Z

Let's add a decisionlog for the variable materialization of the log staging model based on setting the start date variable.

fivetran-joemarkiewicz · 2026-07-30T18:26:13Z

+    - `stg_fivetran_platform__transformation_runs` has no fixed default. It checks your schema for the `transformation_runs` table at runtime. If the table exists, the staging model and the `paid_model_runs`, `free_model_runs`, and `total_model_runs` fields in `fivetran_platform__usage_history` are populated. If it does not, the staging model builds empty and those fields are null.
+    - `fivetran_platform__transformation_run_log` defaults to `false` and does not build. The runtime check does not apply here, so set this variable to `true` to build the model.
+
+  Setting the variable explicitly overrides both layers. `true` requires the `transformation_runs` table to exist, and `false` disables the staging model and its downstream fields even when the table does exist.


Was this meant to be a bulleted item? If so, let's add that. If not, let's fix the indent

It's a continuation of the paragraph started with "transformation_runs has no fixed default" but it looks odd when rendered, so I'll move all the bullets below the paragraph.

fivetran-joemarkiewicz · 2026-07-30T18:27:03Z

+  `fivetran_platform__connection_status` 
+  `fivetran_platform__schema_changelog` 
+  `fivetran_platform__audit_user_activity` 
+  `fivetran_platform__connection_daily_events`
+  `fivetran_platform__audit_table`


Can we have this be a bulleted list so it renders better in markdown. Also, I believe we're missing the new models that are also impacted.

fivetran-savage and others added 8 commits May 21, 2026 15:21

release_branch/v2.6.0

76c9e42

Merge branch 'main' into release_branch/v2.6.0

472fb47

feature/add_audit_trail_events (#197)

872d6fb

* feature/add_audit_trail_events * add_audit_trail_end_model * dedup_connection_id

add_surrogate_keys

cc546ba

Feature/build sync statistics and duration model (#198)

e1340b3

* feature/build_sync_statistics_and_duration_model Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix_boolean_for_sql_server * add_consistency_tests --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

update changelog and readme for v2.6.0 release

2ba76f4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fivetran-savage force-pushed the release_branch/v2.6.0 branch from 43b3ed1 to eda9246 Compare July 20, 2026 17:51

fivetran-savage marked this pull request as ready for review July 22, 2026 19:17

fivetran-savage requested a review from fivetran-joemarkiewicz July 22, 2026 19:17

fivetran-savage self-assigned this Jul 22, 2026

fivetran-savage added the docs:ready Triggers the docs generator workflow. label Jul 22, 2026

github-actions Bot and others added 2 commits July 22, 2026 19:19

Generate dbt docs via GitHub Actions

c2b6309

add_surrogate_key

06465bf

fivetran-joemarkiewicz requested changes Jul 24, 2026

View reviewed changes

fivetran-savage removed the docs:ready Triggers the docs generator workflow. label Jul 27, 2026

fivetran-savage added 3 commits July 27, 2026 12:29

incorporate_review_suggestions

0e8356b

update_macros

c3bca57

update_yml

2b4cd0a

fivetran-joemarkiewicz requested changes Jul 29, 2026

View reviewed changes

fivetran-savage added 3 commits July 29, 2026 10:31

update_run_log

f934df5

update_ci_test

73b0407

add_event_id_to_model

c4b7e50

fivetran-joemarkiewicz approved these changes Jul 30, 2026

View reviewed changes

update_readme

9acf3c4

Conversation

fivetran-savage commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Overview

Submission Checklist

Changelog

Uh oh!

amypeterson27 commented Jul 14, 2026

Uh oh!

fivetran-savage commented Jul 14, 2026

Uh oh!

amypeterson27 commented Jul 21, 2026

Uh oh!

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fivetran-savage commented May 21, 2026 •

edited

Loading