Total Visitors
From 30b282ccba5c2ce502efde71cae9770f42ef0144 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Tue, 15 Apr 2025 14:12:17 -0600
Subject: [PATCH 02/13] demo + overview
---
Deployment-Pipelines/README.md | 204 +++++++++++++++++++++++++++++++++
1 file changed, 204 insertions(+)
create mode 100644 Deployment-Pipelines/README.md
diff --git a/Deployment-Pipelines/README.md b/Deployment-Pipelines/README.md
new file mode 100644
index 0000000..2b1bc3c
--- /dev/null
+++ b/Deployment-Pipelines/README.md
@@ -0,0 +1,204 @@
+# Deployment Pipelines - Demo \& Overview
+
+Costa Rica
+
+[](https://github.com/)
+[brown9804](https://github.com/brown9804)
+
+Last updated: 2025-01-29
+
+------------------------------------------
+
+> Lakehouse Schema and Deployment Pipelines
+
+
+
+ List of References (Click to expand)
+
+- [Table deletes, updates, and merges - DeltaTables](https://docs.delta.io/latest/delta-update.html#-merge-in-streaming)
+- [DeltaLake - DeltaTables Releases](https://github.com/delta-io/delta/releases)
+- [DeltaTable execute() example](https://github.com/delta-io/delta/blob/master/examples/python/quickstart.py#L50-L56)
+- [Create deployment rules](https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/create-rules?tabs=new)
+- [Notebook source control and deployment](https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-source-control-deployment)
+- [Fabric: Understand the deployment process](https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/understand-the-deployment-process)
+- [Incremental refresh in Dataflow Gen2 - Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh)
+- [Dataflow Gen2 refresh - Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-refresh)
+- [Announcing Public Preview: Incremental Refresh in Dataflow Gen2](https://blog.fabric.microsoft.com/en-us/blog/announcing-public-preview-incremental-refresh-in-dataflows-gen2/)
+- [Choose the best Fabric CI/CD workflow option for you](https://learn.microsoft.com/en-us/fabric/cicd/manage-deployment)
+
+
+
+
+Table of Contents (Click to expand)
+
+- [Overview](#overview)
+- [Demo](#demo)
+ - [Create a Workspace](#create-a-workspace)
+ - [Create a Lakehouse](#create-a-lakehouse)
+ - [Create a New Semantic Model](#create-a-new-semantic-model)
+ - [Auto-Generate Report with Copilot](#auto-generate-report-with-copilot)
+ - [Create a Deployment Pipeline](#create-a-deployment-pipeline)
+ - [Deploy to Production](#deploy-to-production)
+- [How to refresh the data](#how-to-refresh-the-data)
+
+
+
+
+## Overview
+
+> [!Note]
+> `Ownership and Deployment`: In Microsoft Fabric, only the owner of an artifact can deploy it. Non-owner admins do not have the permissions to deploy artifacts. You can create a service principal to act like an owner, allowing it to deploy artifacts on behalf of the actual owner. This ensures that deployment tasks can be automated and managed efficiently without compromising security.
+> `Deployment Rules for Notebooks and Pipelines`: You can add `deployment rules` for notebooks and pipelines in Microsoft Fabric. Deployment rules `can be configured to manage different stages (development, test, production)` and to change content settings during deployment. For notebooks, you can set rules to specify the default lakehouse, which helps avoid manual changes to source and destination references post-deployment.
+
+Process Overview:
+
+1. **Pipeline Structure**: You define the deployment pipeline structure, which typically includes stages like development, test, and production. Each stage can have its own workspace.
+2. **Adding Content**: Content can be added to a pipeline stage by assigning a workspace or deploying content from one stage to another. When you deploy content, Microsoft Fabric copies the metadata, reports, dashboards, and semantic models to the target stage.
+3. **Content Cloning**: During deployment, the content from the source stage is cloned to the target stage. This includes the structure and semantic models but `not the actual data` within the tables. This means that while the structure and connections are maintained, the data itself needs to be refreshed or reloaded in the new stage.
+4. **Autobinding**: Microsoft Fabric tries to maintain connections between items during deployment. For example, if a report is connected to a semantic model, this connection is preserved in the target stage.
+
+> [!IMPORTANT]
+> `Specifics for Lakehouse:` For lakehouses, the deployment process typically `includes the structure and metadata but not the actual data tables`. This is why you might see the structure and semantic models deployed, but the tables themselves need to be manually refreshed or reloaded in the target environment.
+> `Deployment Rules:` You can set deployment rules to manage different stages and change content settings during deployment. For example, you can specify default lakehouses for notebooks to avoid manual changes post-deployment.
+
+## Demo
+
+
+
+
+
+### Create a Workspace
+
+1. Navigate to the Microsoft Fabric portal.
+2. Click on `Create a new workspace`.
+
+
+
+3. Name your workspace (e.g., `Analytics Dev`).
+
+
+
+4. Ensure that your workspace is allocated to the appropriate Fabric Capacity:
+
+
+
+5. You can repeat this same process to create the production environment. You may host your workspace in the same capacity or in a different Fabric Capacity. Name your workspace (e.g., `Analytics Prod`).
+
+
+
+ | Dev Workspace | Prod Workspace |
+ | --- | --- |
+ |
|
|
+
+### Create a Lakehouse
+
+1. Within your new workspace, select `New item` and choose `Lakehouse`.
+
+
+
+2. Name your lakehouse (e.g., `Sales_Data_Lakehouse`).
+
+
+
+3. Click on `Get data`, and `Upload files`:
+
+
+
+4. Load data into the tables:
+
+
+
+
+
+5. The information loaded will be under Tables/dbo section, if you are using the lakehouse schema:
+
+
+
+### Create a New Semantic Model
+
+1. Use the `New semantic model` option to create a Power BI semantic model.
+2. Select specific tables and views from your lakehouse.
+3. Save the semantic model to your workspace for sharing, data modeling, and reporting.
+
+
+
+4. At this point, you should see something similar like following:
+
+
+
+
+### Auto-Generate Report with Copilot
+
+> [!NOTE]
+> Ensure you have the necessary permissions and settings enabled for Copilot to access and use your data.
+
+1. Use Copilot to create a report based on your dataset. In your workspace, click on `...` over your semantic model and select `Auto-create report`:
+
+
+
+2. The outcome will depend on the information you provide in this case. Copilot automatically generated this report for me; you can continue editing:
+
+
+
+3. Remember to click on `Save`:
+
+
+
+### Create a Deployment Pipeline
+
+1. Go back to the Microsoft Fabric portal and select `Deployment Pipelines`.
+
+
+
+2. Create a new pipeline and name it (e.g., `Dev to Prod Pipeline`).
+
+
+
+3. You can customize your stages. For this case, we will only have a development workspace and a production workspace:
+
+
+
+4. Add your `Analytics Dev` workspace as the source, click on `Assign a workspace`:
+
+
+
+5. Add the `Production Workspace` as the target.
+
+
+
+
+
+### Deploy to Production
+
+1. Use the deployment pipeline to move your lakehouse, dataset, and report from the `Analytics Dev` to the `Analytics Prod` workspace.
+
+
+
+
+
+2. Verify that everything is working correctly in the production environment.
+
+
+
+
+
+## How to refresh the data
+
+> To ensure that data is refreshed and reloaded in the new stage of your deployment in Microsoft Fabric, you can use the following methods:
+
+| **Type of Refresh** | **Description** | **Details** |
+|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Scheduled Refresh** | Automatically refreshes data at specified intervals to keep it up-to-date. This allows you to automatically refresh the data at specified intervals, ensuring that the data in the new stage is always up-to-date. | - Can be configured to run multiple times a day.
- Ensures data in the new stage is always current. |
+| **On-Demand Refresh** | Allows immediate data refresh, triggered manually or programmatically. | - Can be done through workspace list or lineage views.
- Can be triggered via a pipeline containing a dataflow activity. |
+| **Incremental Refresh** | Refreshes only the data that has changed since the last refresh, improving efficiency. Click [here to understand more about incremental refresh](https://github.com/brown9804/MicrosoftCloudEssentialsHub/blob/main/0_Azure/2_AzureAnalytics/0_Fabric/demos/11_PBi_IncreRefresh.md) | - **Evaluate Changes**: Checks for changes in the data source based on a DateTime column.
- **Retrieve Data**: Only changed data is retrieved and loaded.
- **Replace Data**: Updated data is processed and replaced. |
+
+Steps to Set Up Incremental Refresh:
+1. **Create or Open a Dataflow**: Start by creating a new Dataflow Gen2 or opening an existing one.
+2. **Configure the Query**: Ensure your query includes a DateTime column that can be used to filter the data.
+3. **Enable Incremental Refresh**: Right-click the query and select Incremental Refresh. Configure the settings, such as the DateTime column and the time range for data extraction.
+4. **Publish the Dataflow**: Once configured, publish the dataflow. The dataflow will then automatically refresh the data incrementally based on the settings.
+
+
+
Total Visitors
+
+
From b9b68a9a8848ad220578829d6f7730a2b6f0d9fd Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
Date: Tue, 15 Apr 2025 20:12:29 +0000
Subject: [PATCH 03/13] Update last modified date in Markdown files
---
Deployment-Pipelines/README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Deployment-Pipelines/README.md b/Deployment-Pipelines/README.md
index 2b1bc3c..d5ca894 100644
--- a/Deployment-Pipelines/README.md
+++ b/Deployment-Pipelines/README.md
@@ -5,7 +5,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-01-29
+Last updated: 2025-04-15
------------------------------------------
From b16b5e550281bf2ebca36274448580fce28289f4 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Tue, 15 Apr 2025 14:13:00 -0600
Subject: [PATCH 04/13] demo samples readme
---
Deployment-Pipelines/samples/README.md | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
create mode 100644 Deployment-Pipelines/samples/README.md
diff --git a/Deployment-Pipelines/samples/README.md b/Deployment-Pipelines/samples/README.md
new file mode 100644
index 0000000..d654a7c
--- /dev/null
+++ b/Deployment-Pipelines/samples/README.md
@@ -0,0 +1,18 @@
+# Demo Samples
+
+Costa Rica
+
+[](https://github.com/)
+[brown9804](https://github.com/brown9804)
+
+Last updated: 2025-01-29
+
+------------------------------------------
+
+> In this folder, you'll find some sample files that we'll be using throughout the demo.
+> `These files will help us illustrate the process of creating a workspace, setting up a lakehouse, defining a dataset schema, generating reports with Copilot, creating a semantic model, and deploying through a pipeline`.
+
+
+
Total Visitors
+
+
+List of References (Click to expand)
+
+- [What is Microsoft Fabric Git integration?](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/intro-to-git-integration?tabs=azure-devops)
+- [Get started with Git integration - Guided examples](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#connect-a-workspace-to-a-git-repo)
+- [Basic concepts in Git integration (git roles, permissions, connect and sync workspace](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process?tabs=Azure%2Cazure-devops#permissions)
+- [Best practices for lifecycle management in Fabric](https://learn.microsoft.com/en-us/fabric/cicd/best-practices-cicd)
+- [End-to-end tutorials in Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/get-started/end-to-end-tutorials)
+- [Microsoft Fabric Samples - Git repo](https://github.com/microsoft/fabric-samples)
+
+
+
+
+Table of Content (Click to expand)
+
+- [Wiki](#wiki)
+- [Content](#content)
+- [Connect a workspace to a Git repo](#connect-a-workspace-to-a-git-repo)
+- [Connecting to a workspace Already linked to GitHub](#connecting-to-a-workspace-already-linked-to-github)
+- [Commit changes to git](#commit-changes-to-git)
+- [Update workspace from Git](#update-workspace-from-git)
+- [Disconnect a workspace from Git](#disconnect-a-workspace-from-git)
+- [Considerations and limitations](#considerations-and-limitations)
+
+
+
+https://github.com/user-attachments/assets/64f099a1-b749-47a6-b723-fa1cb5c575a3
+
+## Connect a workspace to a Git repo
+
+> Only a workspace admin has the capability to connect a workspace to a repository. However, once the connection is established, anyone with the appropriate permission can work within the workspace. If you are not an admin, please request assistance from your admin to connect. To link a workspace to an Azure or GitHub repository, [follow these steps](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#connect-a-workspace-to-a-git-repo).
+
+## Connecting to a workspace (Already linked to GitHub)
+
+> [For detailed instructions, please click here](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#connect-to-a-workspace). If the workspace is already integrated with GitHub, please refer to the instructions for [Connecting to a shared workspace](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process?tabs=Azure%2Cazure-devops#connect-to-a-shared-workspace).
+
+## Commit changes to git
+
+> [See detailed instructions here](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#commit-changes-to-git). After connecting to a Git folder, proceed to edit your workspace as normal. Saved changes will be stored in the workspace only. When ready, you may commit your changes to the Git branch or revert to the previous state. For more information, refer to [commits](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process?tabs=Azure%2Cazure-devops#commit).
+
+## Update workspace from Git
+
+> [Check out detailed instructions here](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#update-workspace-from-git). When a new change is committed to the connected Git branch, a notification will appear in the relevant workspace. Please use the Source control panel to pull the latest changes, merges, or reverts into the workspace and update live items. For further details, refer to [updating](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process?tabs=Azure%2Cazure-devops#update).
+
+## Disconnect a workspace from Git
+
+> Only a workspace admin has the authority to disconnect a workspace from a Git Repo. If you are not an admin, please contact your admin for assistance with disconnecting. If you are an admin and wish to disconnect your repo. [Click here for more information about it](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#disconnect-a-workspace-from-git)
+
+## Considerations and limitations
+
+> Click here to read more about [considerations and limitations](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#considerations-and-limitations).
+
+
+
Total Visitors
+
+