README.md

Deployment Pipelines - Demo & Overview

Costa Rica

brown9804

Last updated: 2025-04-21

---

Lakehouse Schema and Deployment Pipelines

List of References (Click to expand)

• Table deletes, updates, and merges - DeltaTables
• DeltaLake - DeltaTables Releases
• DeltaTable execute() example
• Create deployment rules
• Notebook source control and deployment
• Fabric: Understand the deployment process
• Incremental refresh in Dataflow Gen2 - Microsoft Fabric
• Dataflow Gen2 refresh - Microsoft Fabric
• Announcing Public Preview: Incremental Refresh in Dataflow Gen2
• Choose the best Fabric CI/CD workflow option for you

Table of Contents (Click to expand)

• Overview
• Demo
  • Create a Workspace
  • Create a Lakehouse
  • Create a New Semantic Model
  • Auto-Generate Report with Copilot
  • Create a Deployment Pipeline
  • Deploy to Production
• How to refresh the data

Overview

Note

Ownership and Deployment: In Microsoft Fabric, only the owner of an artifact can deploy it. Non-owner admins do not have the permissions to deploy artifacts. You can create a service principal to act like an owner, allowing it to deploy artifacts on behalf of the actual owner. This ensures that deployment tasks can be automated and managed efficiently without compromising security.

Deployment Rules for Notebooks and Pipelines: You can add deployment rules for notebooks and pipelines in Microsoft Fabric. Deployment rules can be configured to manage different stages (development, test, production) and to change content settings during deployment. For notebooks, you can set rules to specify the default lakehouse, which helps avoid manual changes to source and destination references post-deployment.

Process Overview:

1. Pipeline Structure: You define the deployment pipeline structure, which typically includes stages like development, test, and production. Each stage can have its own workspace.
2. Adding Content: Content can be added to a pipeline stage by assigning a workspace or deploying content from one stage to another. When you deploy content, Microsoft Fabric copies the metadata, reports, dashboards, and semantic models to the target stage.
3. Content Cloning: During deployment, the content from the source stage is cloned to the target stage. This includes the structure and semantic models but not the actual data within the tables. This means that while the structure and connections are maintained, the data itself needs to be refreshed or reloaded in the new stage.
4. Autobinding: Microsoft Fabric tries to maintain connections between items during deployment. For example, if a report is connected to a semantic model, this connection is preserved in the target stage.

Important

Specifics for Lakehouse: For lakehouses, the deployment process typically includes the structure and metadata but not the actual data tables. This is why you might see the structure and semantic models deployed, but the tables themselves need to be manually refreshed or reloaded in the target environment.

Deployment Rules: You can set deployment rules to manage different stages and change content settings during deployment. For example, you can specify default lakehouses for notebooks to avoid manual changes post-deployment.

Demo

Create a Workspace

1. Navigate to the Microsoft Fabric portal.

2. Click on Create a new workspace.

   

3. Name your workspace (e.g., Analytics Dev).

   

4. Ensure that your workspace is allocated to the appropriate Fabric Capacity:

   

5. You can repeat this same process to create the production environment. You may host your workspace in the same capacity or in a different Fabric Capacity. Name your workspace (e.g., Analytics Prod).

   

   Dev Workspace	Prod Workspace

Create a Lakehouse

1. Within your new workspace, select New item and choose Lakehouse.

   

2. Name your lakehouse (e.g., Sales_Data_Lakehouse).

   

3. Click on Get data, and Upload files:

   

4. Load data into the tables:

   

5. The information loaded will be under Tables/dbo section, if you are using the lakehouse schema:

   

Create a New Semantic Model

1. Use the New semantic model option to create a Power BI semantic model.

2. Select specific tables and views from your lakehouse.

3. Save the semantic model to your workspace for sharing, data modeling, and reporting.

   

4. At this point, you should see something similar like following:

   

Auto-Generate Report with Copilot

Note

Ensure you have the necessary permissions and settings enabled for Copilot to access and use your data.

1. Use Copilot to create a report based on your dataset. In your workspace, click on ... over your semantic model and select Auto-create report:

   

2. The outcome will depend on the information you provide in this case. Copilot automatically generated this report for me; you can continue editing:

   

3. Remember to click on Save:

   

Create a Deployment Pipeline

1. Go back to the Microsoft Fabric portal and select Deployment Pipelines.

   

2. Create a new pipeline and name it (e.g., Dev to Prod Pipeline).

   

3. You can customize your stages. For this case, we will only have a development workspace and a production workspace:

   

4. Add your Analytics Dev workspace as the source, click on Assign a workspace:

   

5. Add the Production Workspace as the target.

   

Deploy to Production

1. Use the deployment pipeline to move your lakehouse, dataset, and report from the Analytics Dev to the Analytics Prod workspace.

   

2. Verify that everything is working correctly in the production environment.

   

How to refresh the data

To ensure that data is refreshed and reloaded in the new stage of your deployment in Microsoft Fabric, you can use the following methods:

Type of Refresh	Description	Details
Scheduled Refresh	Automatically refreshes data at specified intervals to keep it up-to-date. This allows you to automatically refresh the data at specified intervals, ensuring that the data in the new stage is always up-to-date.	- Can be configured to run multiple times a day. - Ensures data in the new stage is always current.
On-Demand Refresh	Allows immediate data refresh, triggered manually or programmatically.	- Can be done through workspace list or lineage views. - Can be triggered via a pipeline containing a dataflow activity.
Incremental Refresh	Refreshes only the data that has changed since the last refresh, improving efficiency. Click here to understand more about incremental refresh	- Evaluate Changes: Checks for changes in the data source based on a DateTime column. - Retrieve Data: Only changed data is retrieved and loaded. - Replace Data: Updated data is processed and replaced.

Steps to Set Up Incremental Refresh:

1. Create or Open a Dataflow: Start by creating a new Dataflow Gen2 or opening an existing one.
2. Configure the Query: Ensure your query includes a DateTime column that can be used to filter the data.
3. Enable Incremental Refresh: Right-click the query and select Incremental Refresh. Configure the settings, such as the DateTime column and the time range for data extraction.
4. Publish the Dataflow: Once configured, publish the dataflow. The dataflow will then automatically refresh the data incrementally based on the settings.

Refresh Date: 2025-10-15