Merge pull request #54 from microsoft/AIFoundry-FDPUpdate

AI foundry fdp update

Author: Prajwal-Microsoft

CHANGELOG.md

@@ -2,6 +2,22 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.2] - 2025-05-13
+### Added
+- Add new project module leveraging the new cognitive services/projects type
+- Add BYO service connections for search, storage and CosmosDB to project (based on feature flag selection)
+- new infrastructure drawing
+
+### Changed
+- Revise Cognitive Services module to leverage new preview api to leverage new FDP updates
+- Update AI Search CMK enforcement value to 'disabled'
+- Update and add private endpoints for cognitive services project subtype
+- Update and add required roles and scopes to cognitive services and ai search modules
+- Update md to show changes
+
+### Deprecated
+- Remove the modules deploying AML hub and project.
+
 
 ## [1.1] - 2025-04-30
 ### Added

README.md

@@ -4,21 +4,24 @@
 
 ## Overview
 
-This is a foundational deployment solution for deploying an AI hub and project into an isolated environment (vNet) within Azure. The deployed features follow Microsoft's Well-Architected Framework [WAF](https://learn.microsoft.com/en-us/azure/well-architected/) to establish an isolated infrastructure for AI Foundry, intended to assist in moving from a Proof of Concept state to a production-ready application. 
+<span style="font-size: 3em;">🚀</span> **New: Updated deployment to match Foundry release at Build 2025!**
+This new update has been tested in the EastUS2 region successfully.
+This is a foundational solution for deploying an AI Foundry account ([Cognitive Services accountKind = 'AIServices'](https://review.learn.microsoft.com/en-us/azure/templates/microsoft.cognitiveservices/2025-04-01-preview/accounts?branch=main&pivots=deployment-language-bicep)) and project ([cognitiveServices/projects](https://review.learn.microsoft.com/en-us/azure/templates/microsoft.cognitiveservices/2025-04-01-preview/accounts/projects?branch=main&pivots=deployment-language-bicep)) into an isolated environment (vNet) within Azure. The deployed features follow Microsoft's Well-Architected Framework [WAF](https://learn.microsoft.com/en-us/azure/well-architected/) to establish an isolated infrastructure for AI Foundry, intended to assist in moving from a Proof of Concept state to a production-ready application. 
 
-This template leverages Azure Verified Modules (AVM) and the Azure Developer CLI (AZD) to provision a WAF-aligned infrastructure for AI application development. This infrastructure includes AI Foundry elements, a virtual network (VNET), private endpoints, Key Vault, a storage account, and additional, optional WAF-aligned resources (such as Cosmos DB and SQL Server) that can be leveraged with Foundry developed projects.
+This template leverages Azure Verified Modules (AVM) and the Azure Developer CLI (AZD) to provision a WAF-aligned infrastructure for AI application development. This infrastructure includes AI Foundry elements, a virtual network (VNET), private endpoints, Key Vault, a storage account, and additional, optional WAF-aligned resources (such as AI Search, Cosmos DB and SQL Server) that can be leveraged with Foundry developed projects.
 
 The following deployment automates our recommended configuration to protect your data and resources; using Microsoft Entra ID role-based access control, a managed network, and private endpoints. We recommend disabling public network access for Azure OpenAI resources, Azure AI Search resources, and storage accounts (which will occur when deploying those optional services within this workflow). Using selected networks with IP rules isn't supported because the services' IP addresses are dynamic.
 
-AI Foundry has two network isolation aspects, this repository will automate:
-1. Configuring the network isolation of the Azure AI Foundry hub and project managed compute (compute instance, serverless compute, managed online endpoint) [Configure Managed Network](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/configure-managed-network)
-2. Configuring the virtual network, private end points and private link services to isolate resources to connect to the hub and project in a secure way. [Secure Data Playground](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/secure-data-playground)
+This repository will automate:
+1. Configuring the virtual network, private end points and private link services to isolate resources connecting to the account and project in a secure way. [Secure Data Playground](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/secure-data-playground)
+2. Deploying and configuring the network isolation of the Azure AI Foundry account and project sub-resource within the virtual network, and with all services configured behind private end points. 
+
 
 
 ## Architecture
 The diagram below illustrates the capabilities included in the template.
 
-![Network Isolation Infrastructure](./img/Architecture/Deploy-AI-App-in-Prod-Architecture_final.png)
+![Network Isolation Infrastructure](./img/Architecture/FDParch.png)
 
 | Diagram Step      | Description     |
 | ------------- | ------------- |
@@ -31,12 +34,12 @@ The diagram below illustrates the capabilities included in the template.
 ## Features
 
 ### What solutions does this enable? 
-- Deploys AI hub and AI project into a virtual network with all dependent services connected via private end points. 
+- Deploys an AI Foundry account and project leveraging the latest AI Foundry updates announced at Build 2025, into a virtual network with all dependent services connected via private end points. 
 
 - Configures AI Foundry, adhering to the best practices outlined in the Well Architected Framework.
 
 - Provides the ability to [add additional Azure services during deployment](docs/add_additional_services.md), configured to connect via isolation to enrich your AI project.
-    (API Management, CosmosDB, Azure SQL DB)
+    (AI Search, API Management, CosmosDB, Azure SQL DB)
 
 -  <span style="font-size: 3em;">🚀</span> **New**: 
 Offers ability to [start with an existing Azure AI Project](docs/transfer_project_connections.md) which will provision dependent Azure resources based on the Project's established connections within AI Foundry.
@@ -49,8 +52,7 @@ Offers ability to [start with an existing Azure AI Project](docs/transfer_projec
 3. The solution ensures secure access to the private VNET through a jump-box VM with Azure Bastion. By default, Bastion does not require an inbound NSG rule for network traffic. However, if your environment enforces specific policy rules, you can resolve access issues by entering your machine's IP address in the `allowedIpAddress` parameter when prompted during deployment. If not specified, all IP addresses are allowed to connect to Azure Bastion. 
 4. If deploying from your [local environment](docs/local_environment_steps.md), install the [Azure CLI (AZ)](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and the [Azure Developer CLI (AZD)](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/install-azd?tabs=winget-windows%2Cbrew-mac%2Cscript-linux&pivots=os-windows).
 5. If deploying via [GitHub Codespaces](docs/github_code_spaces_steps.md) - requires the user to be on a GitHub Team or Enterprise Cloud plan.
-6. If leveraging [One-click deployment](#quick-deploy).
-7. If leveraging [GitHub Actions](docs/github_actions_steps.md).
+6. If leveraging [GitHub Actions](docs/github_actions_steps.md).
 
 ### Check Azure OpenAI Quota Availability  
 
@@ -110,7 +112,7 @@ This template has [Managed Identity](https://learn.microsoft.com/entra/identity/
 
 ## Resources
 
-- [Azure AI Foundry documentation](https://learn.microsoft.com/en-us/azure/ai-studio/)
+- [Azure AI Foundry documentation](https://learn.microsoft.com/en-us/azure/ai-foundry/)
 - [Azure Well Architecture Framework documentation](https://learn.microsoft.com/en-us/azure/well-architected/)
 - [Azure OpenAI Service - Documentation, quickstarts, API reference - Azure AI services | Microsoft Learn](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data)
 - [Azure AI Content Understanding documentation](https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/)

docs/Required_roles_scopes_resources.md

@@ -17,8 +17,7 @@ Be sure these resource providers are registered in your Azure subscription. To r
 |Azure Log Analytics|Microsoft.OperationalInsights|/workspaces|An Azure Log Analytics workspace used to collect diagnostics|
 |Azure Key Vault|Microsoft.KeyVault|/vaults|An Azure Key Vault instance associated with the Azure AI Foundry Hub|
 |Azure Storage Account|Microsoft.Storage|/storageAccounts|An Azure Storage instance associated with the Azure AI Foundry Hub|
-|Azure Container Registry|Microsoft.ContainerRegistry|/registries|An Azure Container Registry instance associated with the Azure AI Foundry Hub|
-|Azure AI Hub / Project|Microsoft.MachineLearningServices|/workspaces|An Azure AI Studio Hub and Project (Azure ML Workspace of kind ‘hub’ and ‘project’)|
+|Azure Container Registry|Microsoft.ContainerRegistry|/registries|An Azure Container Registry instance associated with the Azure AI Foundry Account|
 |Azure AI Services|Microsoft.CognitiveServices|/accounts|An Azure AI Services as the model-as-a-service endpoint provider including GPT-4o and ADA Text Embeddings model deployments|
 |Azure Virtual Network|Microsoft.Network|/virtualNetworks|A bring-your-own (BYO) virtual network hosting a virtual machine to connect to Azure AI Foundry which will be behind a private endpoint when in network isolation mode. |
 |Bastion Host|Microsoft.Network||A Bastion Host defined in the BYO virtual network that provides RDP connectivity to the jumpbox virtual machine|

img/Architecture/Architecture.png

@@ -8,9 +8,6 @@ param name string
 @description('Specifies the location for all the Azure resources. Defaults to the location of the resource group.')
 param location string
 
-@description('Optional. Specifies the connections to be created for the Azure AI Hub workspace. The connections are used to connect to other Azure resources and services.')
-param connections connectionType[] = []
-
 @description('Optional. Specifies the OpenAI deployments to create.')
 param aiModelDeployments deploymentsType[] = []
 
@@ -82,6 +79,21 @@ param translatorEnabled bool = false
 @description('Whether to include Azure Document Intelligence in the deployment.')
 param documentIntelligenceEnabled bool = false
 
+@description('Optional. A collection of rules governing the accessibility from specific network locations.')
+param networkAcls object ={
+  defaultAction: 'Deny'
+  bypass: 'AzureServices' // ✅ Allows trusted Microsoft services
+  // virtualNetworkRules: [
+  //   {
+  //     id: networkIsolation ? network.outputs.vmSubnetName : ''
+  //     ignoreMissingVnetServiceEndpoint: true
+  //   }
+  // ]
+}
+
+@description('Name of the first project')
+param projectName string = '${take(name, 8)}proj'
+
 var defaultTags = {
   'azd-env-name': name
 }
@@ -170,7 +182,7 @@ module containerRegistry 'modules/containerRegistry.bicep' = if (acrEnabled) {
 module storageAccount 'modules/storageAccount.bicep' = {
   name: take('${name}-storage-account-deployment', 64)
   params: {
-    name: 'st${name}${resourceToken}'
+    storageName: 'st${name}${resourceToken}'
     location: location
     networkIsolation: networkIsolation
     virtualNetworkResourceId: networkIsolation ? network.outputs.virtualNetworkId : ''
@@ -190,7 +202,7 @@ module storageAccount 'modules/storageAccount.bicep' = {
       }
     ], searchEnabled ? [
       {
-        principalId: aiSearch.outputs.systemAssignedMIPrincipalId
+        principalId: searchEnabled ? aiSearch.outputs.systemAssignedMIPrincipalId : ''
         principalType: 'ServicePrincipal'
         roleDefinitionIdOrName: 'Storage Blob Data Contributor'
       }
@@ -206,6 +218,7 @@ module cognitiveServices 'modules/cognitive-services/main.bicep' = {
     resourceToken: resourceToken
     location: location
     networkIsolation: networkIsolation
+    networkAcls: networkAcls
     virtualNetworkResourceId: networkIsolation ? network.outputs.virtualNetworkId : ''
     virtualNetworkSubnetResourceId: networkIsolation ? network.outputs.vmSubnetId : ''
     logAnalyticsWorkspaceResourceId: logAnalyticsWorkspace.outputs.resourceId
@@ -221,6 +234,21 @@ module cognitiveServices 'modules/cognitive-services/main.bicep' = {
   }
 }
 
+// // Add the new FDP cognitive services module
+module project 'modules/ai-foundry-project/main.bicep' = {
+  name: '${name}prj'
+  params: {
+    cosmosDBname: cosmosDbEnabled? cosmosDb.outputs.cosmosDBname : ''
+    cosmosDbEnabled: cosmosDbEnabled
+    searchEnabled: searchEnabled
+    name: projectName
+    location: location
+    storageName: storageAccount.outputs.storageName
+    aiServicesName: cognitiveServices.outputs.aiServicesName
+    nameFormatted:  searchEnabled ? aiSearch.outputs.name : ''
+    }
+}
+
 module aiSearch 'modules/aisearch.bicep' = if (searchEnabled) {
   name: take('${name}-ai-search-deployment', 64)
   params: {
@@ -230,6 +258,7 @@ module aiSearch 'modules/aisearch.bicep' = if (searchEnabled) {
     virtualNetworkResourceId: networkIsolation ? network.outputs.virtualNetworkId : ''
     virtualNetworkSubnetResourceId: networkIsolation ? network.outputs.vmSubnetId : ''
     logAnalyticsWorkspaceResourceId: logAnalyticsWorkspace.outputs.resourceId
+    userObjectId: userObjectId
     roleAssignments: union(empty(userObjectId) ? [] : [
       {
         principalId: userObjectId
@@ -259,7 +288,7 @@ module virtualMachine './modules/virtualMachine.bicep' = if (networkIsolation)
     vmNicName: toLower('nic-vm-${name}-jump')
     vmSize: vmSize
     vmSubnetId: network.outputs.vmSubnetId
-    storageAccountName: storageAccount.outputs.name
+    storageAccountName: storageAccount.outputs.storageName
     storageAccountResourceGroup: resourceGroup().name
     imagePublisher: 'MicrosoftWindowsDesktop'
     imageOffer: 'Windows-11'
@@ -282,72 +311,6 @@ module virtualMachine './modules/virtualMachine.bicep' = if (networkIsolation)
   dependsOn: networkIsolation ? [storageAccount] : []
 }
 
-module aiHub 'modules/ai-foundry/hub.bicep' = {
-  name: take('${name}-ai-hub-deployment', 64)
-  params: {
-    name: 'hub-${name}'
-    location: location
-    networkIsolation: networkIsolation
-    virtualNetworkResourceId: networkIsolation ? network.outputs.virtualNetworkId : ''
-    virtualNetworkSubnetResourceId: networkIsolation ? network.outputs.vmSubnetId : ''
-    logAnalyticsWorkspaceResourceId: logAnalyticsWorkspace.outputs.resourceId
-    appInsightsResourceId: applicationInsights.outputs.resourceId
-    containerRegistryResourceId: acrEnabled ? containerRegistry.outputs.resourceId : null
-    keyVaultResourceId: keyvault.outputs.resourceId
-    storageAccountResourceId: storageAccount.outputs.resourceId
-    roleAssignments:empty(userObjectId) ? [] : [
-      {
-        roleDefinitionIdOrName: 'f6c7c914-8db3-469d-8ca1-694a8f32e121' // ML Data Scientist Role
-        principalId: userObjectId
-        principalType: 'User'
-      }
-    ]
-    connections: concat(
-      cognitiveServices.outputs.connections,
-      connections,
-      searchEnabled ? [
-      {
-        name: aiSearch.outputs.name
-        value: null
-        category: 'CognitiveSearch'
-        target: 'https://${aiSearch.outputs.name}.search.windows.net/'
-        connectionProperties: {
-          authType: 'AAD'
-        }
-        isSharedToAll: true
-        metadata: {
-          ApiType: 'Azure'
-          ResourceId: aiSearch.outputs.resourceId
-        }
-      }] : [])
-    tags: allTags
-  }
-}
-
-module aiProject 'modules/ai-foundry/project.bicep' = {
-  name: take('${name}-ai-project-deployment', 64)
-  params: {
-    name: 'proj-${name}'
-    location: location
-    hubResourceId: aiHub.outputs.resourceId
-    networkIsolation: networkIsolation
-    logAnalyticsWorkspaceResourceId: logAnalyticsWorkspace.outputs.resourceId
-    roleAssignments: union(empty(userObjectId) ? [] : [
-      {
-        roleDefinitionIdOrName: 'f6c7c914-8db3-469d-8ca1-694a8f32e121' // ML Data Scientist Role
-        principalId: userObjectId
-        principalType: 'User'
-      }
-    ], [
-      {
-        roleDefinitionIdOrName: 'f6c7c914-8db3-469d-8ca1-694a8f32e121' // ML Data Scientist Role
-        principalId: cognitiveServices.outputs.aiServicesSystemAssignedMIPrincipalId
-        principalType: 'ServicePrincipal'
-      }
-    ])
-    tags: allTags
-  }
-}
 
 module apim 'modules/apim.bicep' = if (apiManagementEnabled) {
   name: take('${name}-apim-deployment', 64)
@@ -395,23 +358,23 @@ module sqlServer 'modules/sqlServer.bicep' = if (sqlServerEnabled) {
 }
 
 import { sqlDatabaseType, databasePropertyType, deploymentsType } from 'modules/customTypes.bicep'
-import { connectionType } from 'br/public:avm/res/machine-learning-services/workspace:0.10.1'
+
 
 output AZURE_KEY_VAULT_NAME string = keyvault.outputs.name
 output AZURE_AI_SERVICES_NAME string = cognitiveServices.outputs.aiServicesName
 output AZURE_AI_SEARCH_NAME string = searchEnabled ? aiSearch.outputs.name : ''
-output AZURE_AI_HUB_NAME string = aiHub.outputs.name
-output AZURE_AI_PROJECT_NAME string = aiHub.outputs.name
+output AZURE_AI_HUB_NAME string = cognitiveServices.outputs.aiServicesName
+output AZURE_AI_PROJECT_NAME string = project.outputs.projectName
 output AZURE_BASTION_NAME string = networkIsolation ? network.outputs.bastionName : ''
 output AZURE_VM_RESOURCE_ID string = networkIsolation ? virtualMachine.outputs.id : ''
 output AZURE_VM_USERNAME string = servicesUsername
 output AZURE_APP_INSIGHTS_NAME string = applicationInsights.outputs.name
 output AZURE_CONTAINER_REGISTRY_NAME string = acrEnabled ? containerRegistry.outputs.name : ''
 output AZURE_LOG_ANALYTICS_WORKSPACE_NAME string = logAnalyticsWorkspace.outputs.name
-output AZURE_STORAGE_ACCOUNT_NAME string = storageAccount.outputs.name
+output AZURE_STORAGE_ACCOUNT_NAME string = storageAccount.outputs.storageName
 output AZURE_API_MANAGEMENT_NAME string = apiManagementEnabled ? apim.outputs.name : ''
 output AZURE_VIRTUAL_NETWORK_NAME string = networkIsolation ?  network.outputs.virtualNetworkName : ''
 output AZURE_VIRTUAL_NETWORK_SUBNET_NAME string =networkIsolation ?  network.outputs.vmSubnetName : ''
 output AZURE_SQL_SERVER_NAME string = sqlServerEnabled ? sqlServer.outputs.name : ''
 output AZURE_SQL_SERVER_USERNAME string = sqlServerEnabled ? servicesUsername : ''
-output AZURE_COSMOS_ACCOUNT_NAME string = cosmosDbEnabled ? cosmosDb.outputs.name : ''
+output AZURE_COSMOS_ACCOUNT_NAME string = cosmosDbEnabled ? cosmosDb.outputs.cosmosDBname : ''