Skip to content

Commit 65701ac

Browse files
brown9804brown9804
authored
moved
1 parent daf0dd1 commit 65701ac

1 file changed

Lines changed: 132 additions & 0 deletions

File tree

  • Workloads-Specific/RealTimeIntelligence/FabricActivatorRulePipeline

‎Workloads-Specific/RealTimeIntelligence/FabricActivatorRulePipeline/README.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# Microsoft Fabric: Automating Pipeline Execution with Activator
2+
3+
Costa Rica
4+
5+
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
6+
[brown9804](https://github.com/brown9804)
7+
8+
Last updated: 2025-04-21
9+
10+
----------
11+
12+
> This process shows how to set up Microsoft Fabric Activator to automate workflows by detecting file creation events in a storage system and triggering another pipeline to run. <br/>
13+
>
14+
> 1. **First Pipeline**: The process starts with a pipeline that ends with a `Copy Data` activity. This activity uploads data into the `Lakehouse`. <br/>
15+
> 2. **Event Stream Setup**: An `Event Stream` is configured in Activator to monitor the Lakehouse for file creation or data upload events. <br/>
16+
> 3. **Triggering the Second Pipeline**: Once the event is detected (e.g., a file is uploaded), the Event Stream triggers the second pipeline to continue the workflow.
17+
18+
<details>
19+
<summary><b>List of References </b> (Click to expand)</summary>
20+
21+
- [Activate Fabric items](https://learn.microsoft.com/en-us/fabric/real-time-intelligence/data-activator/activator-trigger-fabric-items)
22+
- [Create a rule in Fabric Activator](https://learn.microsoft.com/en-us/fabric/real-time-intelligence/data-activator/activator-create-activators)
23+
24+
</details>
25+
26+
<details>
27+
<summary><b>List of Content </b> (Click to expand)</summary>
28+
29+
- [Set Up the First Pipeline](#set-up-the-first-pipeline)
30+
- [Configure Activator to Detect the Event](#configure-activator-to-detect-the-event)
31+
- [Set Up the Second Pipeline](#set-up-the-second-pipeline)
32+
- [Define the Rule in Activator](#define-the-rule-in-activator)
33+
- [Test the Entire Workflow](#test-the-entire-workflow)
34+
- [Troubleshooting If Needed](#troubleshooting-if-needed)
35+
36+
</details>
37+
38+
> [!NOTE]
39+
> This code generates random data with fields such as id, name, age, email, and created_at, organizes it into a PySpark DataFrame, and saves it to a specified Lakehouse path using the Delta format. Click here to see the [example script](./GeneratesRandomData.ipynb)
40+
41+
<https://github.com/user-attachments/assets/95206bf3-83a7-42c1-b501-4879df22ef7d>
42+
43+
## Set Up the First Pipeline
44+
45+
1. **Create the Pipeline**:
46+
- In [Microsoft Fabric](https://app.fabric.microsoft.com/), create the first pipeline that performs the required tasks.
47+
- Add a `Copy Data` activity as the final step in the pipeline.
48+
49+
2. **Generate the Trigger File**:
50+
- Configure the `Copy Data` activity to create a trigger file in a specific location, such as `Azure Data Lake Storage (ADLS)` or `OneLake`.
51+
- Ensure the file name and path are consistent and predictable (e.g., `trigger_file.json` in a specific folder).
52+
3. **Publish and Test**: Publish the pipeline and test it to ensure the trigger file is created successfully.
53+
54+
<https://github.com/user-attachments/assets/798a3b12-c944-459d-9e77-0112b5d82831>
55+
56+
## Configure Activator to Detect the Event
57+
58+
> [!TIP]
59+
> Event options:
60+
61+
<https://github.com/user-attachments/assets/282fae9b-e1c6-490d-bd23-9ed9bdf6105d>
62+
63+
1. **Set Up an Event**:
64+
- Create a new event to monitor the location where the trigger file is created (e.g., ADLS or OneLake). Click on `Real-Time`:
65+
66+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/e1ce1f83-a8f6-4a3c-94dc-749e370d8079" />
67+
68+
- Choose the appropriate event type, such as `File Created`.
69+
70+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/3a21abd7-0ff4-428f-a3a1-5e387314c1f5" />
71+
72+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/94e5556b-5d56-4a42-9edd-83b514e7c953" />
73+
74+
- Add a source:
75+
76+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/9709a690-f3b5-453b-b3d9-c67d4b1a9465" />
77+
78+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/8dcadd23-4abb-47ee-82ca-f3868cb818e1" />
79+
80+
<https://github.com/user-attachments/assets/43a9654b-e8d0-44da-80b9-9f528483fa3b>
81+
82+
2. **Test Event Detection**:
83+
- Save the event and test it by manually running the first pipeline to ensure Activator detects the file creation.
84+
- Check the **Event Details** screen in Activator to confirm the event is logged.
85+
86+
<https://github.com/user-attachments/assets/6b21194c-54b4-49de-9294-1bf78b1e5acd>
87+
88+
## Set Up the Second Pipeline
89+
90+
1. **Create the Pipeline**:
91+
- In Microsoft Fabric, create the second pipeline that performs the next set of tasks.
92+
- Ensure it is configured to accept external triggers.
93+
2. **Publish the Pipeline**: Publish the second pipeline and ensure it is ready to be triggered.
94+
95+
<https://github.com/user-attachments/assets/5b630579-a0ec-4d5b-b973-d9b4fdd8254c>
96+
97+
## Define the Rule in Activator
98+
99+
1. **Setup the Activator**:
100+
101+
<https://github.com/user-attachments/assets/7c88e080-d5aa-4920-acd6-94c2e4ae0568>
102+
103+
2. **Create a New Rule**:
104+
- In `Activator`, create a rule that responds to the event you just configured.
105+
- Set the condition to match the event details (e.g., file name, path, or metadata).
106+
3. **Set the Action**:
107+
- Configure the rule to trigger the second pipeline.
108+
- Specify the pipeline name and pass any required parameters.
109+
3. **Save and Activate**:
110+
- Save the rule and activate it.
111+
- Ensure the rule is enabled and ready to respond to the event.
112+
113+
<https://github.com/user-attachments/assets/5f139eeb-bab0-4d43-9f22-bbe44503ed75>
114+
115+
## Test the Entire Workflow
116+
117+
1. **Run the First Pipeline**: Execute the first pipeline and verify that the trigger file is created.
118+
2. **Monitor Activator**: Check the `Event Details` and `Rule Activation Details` in Activator to ensure the event is detected and the rule is activated.
119+
3. **Verify the Second Pipeline**: Confirm that the second pipeline is triggered and runs successfully.
120+
121+
<https://github.com/user-attachments/assets/0a1dab70-2317-4636-b0be-aa0cb301b496>
122+
123+
## Troubleshooting (If Needed)
124+
125+
- If the second pipeline does not trigger:
126+
1. Double-check the rule configuration in Activator.
127+
2. Review the logs in Activator for any errors or warnings.
128+
129+
<div align="center">
130+
<h3 style="color: #4CAF50;">Total Visitors</h3>
131+
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
132+
</div>

0 commit comments

Comments
 (0)