Skip to content

HowMonitorChanges.md

Latest commit

 

History

History
133 lines (94 loc) · 7.42 KB

HowMonitorChanges.md

File metadata and controls

133 lines (94 loc) · 7.42 KB

Demostration: How to Track Pipeline Modifications

Costa Rica

GitHub brown9804

Last updated: 2025-04-21

List of References (Click to expand)
Table of Content (Click to expand)

How to create a Data Factory in Azure

  1. Log in to Azure Portal: Open your web browser and go to the Azure Portal. Enter your credentials to log in.

  2. Search for Data Factory: Use the search bar at the top to search for Data Factory and select Data Factory from the results.

    image

  3. Create a New Data Factory:

    • Click on the + Create button.
    • In the "Basics" tab, fill in the required fields:
      • Subscription: Select your Azure subscription.
      • Resource Group: Select an existing resource group or create a new one.
      • Region: Choose the region where you want to deploy the Data Factory.
      • Name: Enter a unique name for your Data Factory.
      • Version: Select V2 (the latest version).

  4. Configure Git (Optional): If you want to configure Git for source control, you can do so in the Git configuration tab. This step is optional and can be skipped if not needed.

Note

Or later (crucial for source control or auditing):

image

  1. Review and Create:

    • Click on the Review + create button.

    • Review your settings and click Create once the validation passes.

      image

  2. Wait for Deployment: The deployment process will take a few minutes. Once it's complete, you will see a notification.

  3. Access Data Factory: After the deployment is complete, click on the Go to resource button to access your new Data Factory.

  4. Launch Data Factory Studio: In the Data Factory resource page, click on the Launch Studio tile to launch the Data Factory Studio where you can start creating pipelines and other data integration tasks.

    image

Create a pipeline

  1. Log in to Azure Portal: Open your web browser and go to the Azure Portal. Enter your credentials to log in.

  2. Go to Data Factory: Use the search bar at the top to search for Data Factory and select your Data Factory instance from the list.

  3. Launch Data Factory Studio: In the Data Factory resource page, click on the Launch Studio tile to launch the Data Factory Studio where you can start creating pipelines and other data integration tasks.

    image

  4. Create a New Pipeline:

    • Click on the New next to Pipelines in the tree view.

    • Select Pipeline from the dropdown menu.

      image

  5. Add Activities to the Pipeline:

    • In the pipeline canvas, click on the Activities pane on the left.

    • Drag and drop the desired activities (e.g., Copy Data, Data Flow) onto the pipeline canvas.

      image

  6. Configure Activities:

    • Click on each activity on the canvas to configure its properties.
    • For example, if you are using a Copy Data activity, you will need to specify the source and destination datasets.

  7. Set Up Linked Services:

    • Linked services are used to define the connection information for data sources and destinations.

    • Go to the Manage tab on the left, then click on Linked services.

    • Click on the + New button to create a new linked service and configure the connection details.

      image

  8. Create Datasets:

    • Datasets represent the data structures within the data stores.

    • Go to the Author tab, then click on Datasets.

    • Click on the + (plus) icon to create a new dataset and configure its properties.

      image

  9. Validate the Pipeline: Click on the Validate button at the top of the pipeline canvas to check for any errors or missing configurations.

  10. Publish the Pipeline: Once validation is successful, click on the Publish All button to save and publish your pipeline.

  11. Trigger the Pipeline: Click on Trigger now to run the pipeline immediately, or configure a trigger for scheduled runs.

    image

  12. Monitor Pipeline Runs: In the Monitor tab, you can view the status of pipeline runs, check for any errors, and review the execution details.

    image

How to see who modified a pipeline

  1. Log in to Azure Portal: Open your web browser and go to the Azure Portal. Enter your credentials to log in.

  2. Go to Azure Data Factory: Once logged in, use the search bar at the top to search for Data Factory and select your Data Factory instance from the list.

  3. Open the Activity Log:

    • In the Data Factory resource page, look for the Activity log option in the left-hand menu under the Monitoring section.
    • Click on Activity log to open the log view.

  4. View Activity Log Details:

    • In the Activity Log, you will see a list of events related to your Data Factory.
    • You can see columns such as Operation Name, Status, Event Initiated By, Time,Subscription, and more.

  5. Filter and Search:

    • Use the filters at the top to narrow down the events by time range, resource group, resource, and more.
    • You can also use the search bar to find specific events or operations.

  6. Review Event Details: Click on any event in the list to view more detailed information about that event, including the JSON payload with additional properties.

    image
Total views

Refresh Date: 2025-07-22