diff --git a/modules/adb-data-storage-vnet-ncc-public-endpoint/adb-data-storage-vnet-ncc-private-endpoint/README.md b/modules/adb-data-storage-vnet-ncc-public-endpoint/adb-data-storage-vnet-ncc-private-endpoint/README.md new file mode 100644 index 00000000..63368778 --- /dev/null +++ b/modules/adb-data-storage-vnet-ncc-public-endpoint/adb-data-storage-vnet-ncc-private-endpoint/README.md @@ -0,0 +1,133 @@ +# Secure Storage Connectivity to both Classic and Serverless compute using private endpoints + +This module deploys an Azure Databricks workspace with VNet injection and connects it to an Azure Storage account (used for Unity Catalog external storage and DBFS) over **private endpoints** from both classic (VNet-injected) compute and serverless compute. + +It is the private-endpoint variant of the sibling [`adb-data-storage-vnet-ncc-public-endpoint`](../adb-data-storage-vnet-ncc-public-endpoint) module: instead of allowing the workspace VNet subnets and serverless NCC subnets through storage network (service-endpoint) rules, all storage access flows through private endpoints and private DNS. + +## Module content + +This module can be used to deploy the following: + +* A resource group containing a virtual network with private and public subnets, a Private Link subnet, network security groups, a Databricks access connector and a VNet-injected Databricks workspace (assigned to a metastore). +* A network connectivity configuration (NCC) for the workspace, with **private endpoint rules** (`databricks_mws_ncc_private_endpoint_rule`) so that serverless compute reaches the storage account privately. +* A second resource group containing the data storage account and container, a user-assigned identity and a Databricks access connector. +* **Azure private endpoints** for the storage account (blob and dfs, for both the data and DBFS storage), with private DNS zones and VNet links, and approval of the private endpoint connections. +* Databricks storage credential, external location and catalog backed by the storage container, plus grants. + +## How to use + +> **Note** +> You can customize this module by adding, deleting or updating the Azure resources to adapt it to your requirements. + +1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources). +2. Configure the `azurerm`, `azapi` and account-level/workspace-level `databricks` providers (see [providers.tf](providers.tf)). +3. Provide values for the variables defined in [variables.tf](variables.tf) via a `terraform.tfvars` file. +4. Run `terraform init`. +5. Run `terraform apply`. + + +## Requirements + +| Name | Version | +|------|---------| +| [azapi](#requirement\_azapi) | 2.0.1 | +| [azurerm](#requirement\_azurerm) | >=4.31.0 | +| [databricks](#requirement\_databricks) | >=1.81.1 | + +## Providers + +| Name | Version | +|------|---------| +| [azapi](#provider\_azapi) | 2.0.1 | +| [azurerm](#provider\_azurerm) | >=4.31.0 | +| [databricks.accounts](#provider\_databricks.accounts) | >=1.81.1 | +| [databricks.workspace](#provider\_databricks.workspace) | >=1.81.1 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [azapi_update_resource.approve_storage_private_endpoint_connection_blob](https://registry.terraform.io/providers/Azure/azapi/2.0.1/docs/resources/update_resource) | resource | +| [azapi_update_resource.approve_storage_private_endpoint_connection_dbfs_blob](https://registry.terraform.io/providers/Azure/azapi/2.0.1/docs/resources/update_resource) | resource | +| [azapi_update_resource.approve_storage_private_endpoint_connection_dbfs_dfs](https://registry.terraform.io/providers/Azure/azapi/2.0.1/docs/resources/update_resource) | resource | +| [azapi_update_resource.approve_storage_private_endpoint_connection_dfs](https://registry.terraform.io/providers/Azure/azapi/2.0.1/docs/resources/update_resource) | resource | +| [azurerm_databricks_access_connector.ac](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_access_connector) | resource | +| [azurerm_databricks_access_connector.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_access_connector) | resource | +| [azurerm_databricks_workspace.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_workspace) | resource | +| [azurerm_network_security_group.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_security_group) | resource | +| [azurerm_private_dns_zone.blob](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone) | resource | +| [azurerm_private_dns_zone.dfs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone) | resource | +| [azurerm_private_dns_zone_virtual_network_link.blobdnszonevnetlink](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone_virtual_network_link) | resource | +| [azurerm_private_dns_zone_virtual_network_link.dfsdnszonevnetlink](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone_virtual_network_link) | resource | +| [azurerm_private_endpoint.data_blob](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_endpoint) | resource | +| [azurerm_private_endpoint.data_dfs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_endpoint) | resource | +| [azurerm_private_endpoint.dbfs_blob](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_endpoint) | resource | +| [azurerm_private_endpoint.dbfs_dfs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_endpoint) | resource | +| [azurerm_resource_group.storage_rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group) | resource | +| [azurerm_resource_group.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group) | resource | +| [azurerm_role_assignment.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource | +| [azurerm_storage_account.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account) | resource | +| [azurerm_storage_account_network_rules.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account_network_rules) | resource | +| [azurerm_storage_container.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_container) | resource | +| [azurerm_subnet.plsubnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet) | resource | +| [azurerm_subnet.private](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet) | resource | +| [azurerm_subnet.public](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet) | resource | +| [azurerm_subnet_network_security_group_association.private](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet_network_security_group_association) | resource | +| [azurerm_subnet_network_security_group_association.public](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet_network_security_group_association) | resource | +| [azurerm_user_assigned_identity.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/user_assigned_identity) | resource | +| [azurerm_virtual_network.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/virtual_network) | resource | +| [databricks_catalog.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog) | resource | +| [databricks_external_location.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/external_location) | resource | +| [databricks_grants.admins_browse_access](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) | resource | +| [databricks_grants.grant_catalog_access](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) | resource | +| [databricks_metastore_assignment.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_assignment) | resource | +| [databricks_mws_ncc_binding.ncc_binding](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_ncc_binding) | resource | +| [databricks_mws_ncc_private_endpoint_rule.dbfs_blob](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_ncc_private_endpoint_rule) | resource | +| [databricks_mws_ncc_private_endpoint_rule.dbfs_dfs](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_ncc_private_endpoint_rule) | resource | +| [databricks_mws_ncc_private_endpoint_rule.storage_blob](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_ncc_private_endpoint_rule) | resource | +| [databricks_mws_ncc_private_endpoint_rule.storage_dfs](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_ncc_private_endpoint_rule) | resource | +| [databricks_mws_network_connectivity_config.ncc](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_network_connectivity_config) | resource | +| [databricks_storage_credential.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/storage_credential) | resource | +| [azapi_resource_list.list_storage_private_endpoint_connection](https://registry.terraform.io/providers/Azure/azapi/2.0.1/docs/data-sources/resource_list) | data source | +| [azapi_resource_list.list_storage_private_endpoint_connection_dbfs](https://registry.terraform.io/providers/Azure/azapi/2.0.1/docs/data-sources/resource_list) | data source | +| [azurerm_storage_account.dbfs_storage_account](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/storage_account) | data source | +| [databricks_metastore.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/metastore) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [azure\_region](#input\_azure\_region) | Variable for the Azure region where resources will be deployed | `string` | `""` | no | +| [azure\_subscription\_id](#input\_azure\_subscription\_id) | Variable for the Azure subscription ID used for authentication | `string` | `""` | no | +| [catalog\_privileges](#input\_catalog\_privileges) | List of Privileges to catalog (grant to principal\_name) | `list(string)` | 
[
  "BROWSE"
]
| no | +| [cidr\_block](#input\_cidr\_block) | VPC CIDR block range | `string` | `"10.20.0.0/23"` | no | +| [data\_storage\_account](#input\_data\_storage\_account) | ADLS Storage account Name | `string` | `""` | no | +| [data\_storage\_account\_rg](#input\_data\_storage\_account\_rg) | ADLS Storage account resource group | `string` | `""` | no | +| [databricks\_account\_id](#input\_databricks\_account\_id) | Your Databricks Account ID | `string` | `""` | no | +| [databricks\_calalog](#input\_databricks\_calalog) | Name of catalog in metastore | `string` | `""` | no | +| [databricks\_host](#input\_databricks\_host) | Databricks Account URL | `string` | `""` | no | +| [databricks\_metastore](#input\_databricks\_metastore) | Databricks UC Metastore | `string` | `""` | no | +| [dbfs\_storage\_account](#input\_dbfs\_storage\_account) | Variable for the name of the storage account for DBFS | `string` | `""` | no | +| [default\_storage\_firewall\_enabled](#input\_default\_storage\_firewall\_enabled) | Disallow public access to default storage account | `bool` | `false` | no | +| [name\_prefix](#input\_name\_prefix) | Variable for the prefix used in naming resources | `string` | `""` | no | +| [network\_security\_group\_rules\_required](#input\_network\_security\_group\_rules\_required) | Variable to control whether network security group rules are required | `string` | `"AllRules"` | no | +| [pl\_subnets\_cidr](#input\_pl\_subnets\_cidr) | Variable for the CIDR block of the Private Link subnets | `string` | `"10.20.1.0/27"` | no | +| [principal\_name](#input\_principal\_name) | Name of principal to grant access to catalog | `string` | `""` | no | +| [private\_subnets\_cidr](#input\_private\_subnets\_cidr) | Variable for the CIDR block of the private subnet for cluster containers | `string` | `"10.20.0.0/25"` | no | +| [public\_network\_access\_enabled](#input\_public\_network\_access\_enabled) | Allow public access to frontend workspace web UI | `bool` | `true` | no | +| [public\_subnets\_cidr](#input\_public\_subnets\_cidr) | Variable for the CIDR block of the public subnet for cluster hosts | `string` | `"10.20.0.128/25"` | no | +| [rg\_name](#input\_rg\_name) | Variable for the name of the resource group to create | `string` | `""` | no | +| [storage\_account\_allowed\_ips](#input\_storage\_account\_allowed\_ips) | Variable for the list of allowed IP addresses for the storage account | `list(string)` | `[]` | no | +| [subnet\_service\_endpoints](#input\_subnet\_service\_endpoints) | Variable for service endpoints to enable on subnets | `list(string)` | `[]` | no | +| [tags](#input\_tags) | Variable for tags to apply to resources for organization and billing | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [databricks\_host](#output\_databricks\_host) | Output the URL of the Databricks workspace | + diff --git a/modules/adb-data-storage-vnet-ncc-public-endpoint/adb-data-storage-vnet-ncc-public-endpoint/README.md b/modules/adb-data-storage-vnet-ncc-public-endpoint/adb-data-storage-vnet-ncc-public-endpoint/README.md new file mode 100644 index 00000000..149a6f48 --- /dev/null +++ b/modules/adb-data-storage-vnet-ncc-public-endpoint/adb-data-storage-vnet-ncc-public-endpoint/README.md @@ -0,0 +1,110 @@ +# Secure Storage Connectivity to both Classic and Serverless compute using service endpoints + +This module deploys an Azure Databricks workspace with VNet injection and connects it to an Azure Storage account (used for Unity Catalog external storage and DBFS) using **service endpoints**, allowing access from both classic (VNet-injected) compute and serverless compute while keeping the storage account off the public internet. + +It is the public-endpoint (service-endpoint) variant of the sibling [`adb-data-storage-vnet-ncc-private-endpoint`](../adb-data-storage-vnet-ncc-private-endpoint) module: rather than routing storage traffic through Azure private endpoints and private DNS, it restricts the storage account's network rules so that only the workspace VNet subnets and the serverless NCC subnets are permitted. + +## Module content + +This module can be used to deploy the following: + +* A resource group containing a virtual network with private and public subnets, network security groups, a Databricks access connector and a VNet-injected Databricks workspace (assigned to a metastore). +* A network connectivity configuration (NCC) bound to the workspace, from which the serverless storage subnets are read and added to the storage account's network rules. +* A second resource group containing the data storage account and container, a user-assigned identity and a Databricks access connector. +* **Storage account network rules** (`azurerm_storage_account_network_rules`) that allow connections only from the workspace VNet subnets and the serverless NCC subnets. +* Databricks storage credential, external location and catalog backed by the storage container, plus grants. + +## How to use + +> **Note** +> You can customize this module by adding, deleting or updating the Azure resources to adapt it to your requirements. + +1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources). +2. Configure the `azurerm` and account-level/workspace-level `databricks` providers (see [providers.tf](providers.tf)). +3. Provide values for the variables defined in [variables.tf](variables.tf) via a `terraform.tfvars` file. +4. Run `terraform init`. +5. Run `terraform apply`. + + +## Requirements + +| Name | Version | +|------|---------| +| [azurerm](#requirement\_azurerm) | >=4.31.0 | +| [databricks](#requirement\_databricks) | >=1.81.1 | + +## Providers + +| Name | Version | +|------|---------| +| [azurerm](#provider\_azurerm) | >=4.31.0 | +| [databricks.accounts](#provider\_databricks.accounts) | >=1.81.1 | +| [databricks.workspace](#provider\_databricks.workspace) | >=1.81.1 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [azurerm_databricks_access_connector.ac](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_access_connector) | resource | +| [azurerm_databricks_access_connector.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_access_connector) | resource | +| [azurerm_databricks_workspace.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/databricks_workspace) | resource | +| [azurerm_network_security_group.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/network_security_group) | resource | +| [azurerm_resource_group.storage_rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group) | resource | +| [azurerm_resource_group.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group) | resource | +| [azurerm_role_assignment.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource | +| [azurerm_storage_account.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account) | resource | +| [azurerm_storage_account_network_rules.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account_network_rules) | resource | +| [azurerm_storage_container.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_container) | resource | +| [azurerm_subnet.private](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet) | resource | +| [azurerm_subnet.public](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet) | resource | +| [azurerm_subnet_network_security_group_association.private](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet_network_security_group_association) | resource | +| [azurerm_subnet_network_security_group_association.public](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/subnet_network_security_group_association) | resource | +| [azurerm_user_assigned_identity.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/user_assigned_identity) | resource | +| [azurerm_virtual_network.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/virtual_network) | resource | +| [databricks_catalog.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog) | resource | +| [databricks_external_location.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/external_location) | resource | +| [databricks_grants.admins_browse_access](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) | resource | +| [databricks_grants.grant_catalog_access](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) | resource | +| [databricks_metastore_assignment.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_assignment) | resource | +| [databricks_mws_ncc_binding.ncc_binding](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_ncc_binding) | resource | +| [databricks_mws_network_connectivity_config.ncc](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_network_connectivity_config) | resource | +| [databricks_storage_credential.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/storage_credential) | resource | +| [databricks_metastore.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/metastore) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [azure\_region](#input\_azure\_region) | Variable for the Azure region where resources will be deployed | `string` | `""` | no | +| [azure\_subscription\_id](#input\_azure\_subscription\_id) | Variable for the Azure subscription ID used for authentication | `string` | `""` | no | +| [catalog\_privileges](#input\_catalog\_privileges) | List of Privileges to catalog (grant to principal\_name) | `list(string)` | 
[
  "BROWSE"
]
| no | +| [cidr\_block](#input\_cidr\_block) | VPC CIDR block range | `string` | `"10.20.0.0/23"` | no | +| [data\_storage\_account](#input\_data\_storage\_account) | ADLS Storage account Name | `string` | `""` | no | +| [data\_storage\_account\_rg](#input\_data\_storage\_account\_rg) | ADLS Storage account resource group | `string` | `""` | no | +| [databricks\_account\_id](#input\_databricks\_account\_id) | Your Databricks Account ID | `string` | `""` | no | +| [databricks\_calalog](#input\_databricks\_calalog) | Name of catalog in metastore | `string` | `""` | no | +| [databricks\_host](#input\_databricks\_host) | Databricks Account URL | `string` | `""` | no | +| [databricks\_metastore](#input\_databricks\_metastore) | Databricks UC Metastore | `string` | `""` | no | +| [dbfs\_storage\_account](#input\_dbfs\_storage\_account) | Variable for the name of the storage account for DBFS | `string` | `""` | no | +| [default\_storage\_firewall\_enabled](#input\_default\_storage\_firewall\_enabled) | Disallow public access to default storage account | `bool` | `false` | no | +| [name\_prefix](#input\_name\_prefix) | Variable for the prefix used in naming resources | `string` | `""` | no | +| [network\_security\_group\_rules\_required](#input\_network\_security\_group\_rules\_required) | Variable to control whether network security group rules are required | `string` | `"AllRules"` | no | +| [principal\_name](#input\_principal\_name) | Name of principal to grant access to catalog | `string` | `""` | no | +| [private\_subnets\_cidr](#input\_private\_subnets\_cidr) | Variable for the CIDR block of the private subnet for cluster containers | `string` | `"10.20.0.0/25"` | no | +| [public\_network\_access\_enabled](#input\_public\_network\_access\_enabled) | Allow public access to frontend workspace web UI | `bool` | `true` | no | +| [public\_subnets\_cidr](#input\_public\_subnets\_cidr) | Variable for the CIDR block of the public subnet for cluster hosts | `string` | `"10.20.0.128/25"` | no | +| [rg\_name](#input\_rg\_name) | Variable for the name of the resource group to create | `string` | `""` | no | +| [storage\_account\_allowed\_ips](#input\_storage\_account\_allowed\_ips) | Variable for the list of allowed IP addresses for the storage account | `list(string)` | `[]` | no | +| [subnet\_service\_endpoints](#input\_subnet\_service\_endpoints) | Variable for service endpoints to enable on subnets | `list(string)` | `[]` | no | +| [tags](#input\_tags) | Variable for tags to apply to resources for organization and billing | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [databricks\_host](#output\_databricks\_host) | Output the URL of the Databricks workspace | + diff --git a/modules/aws-databricks-workspace/README.md b/modules/aws-databricks-workspace/README.md new file mode 100644 index 00000000..0d53d5fd --- /dev/null +++ b/modules/aws-databricks-workspace/README.md @@ -0,0 +1,67 @@ +# AWS Databricks Workspace + +This module creates a Databricks workspace on AWS (E2 platform) on top of networking and storage that you provide. It wires together the account-level building blocks required by a workspace: + +* `databricks_mws_credentials` — registers the cross-account IAM role Databricks uses to manage resources in your AWS account. +* `databricks_mws_storage_configurations` — registers the root S3 bucket (DBFS root) for the workspace. +* `databricks_mws_networks` — registers your existing VPC, private subnets and security groups (customer-managed VPC). +* `databricks_mws_workspaces` — creates the workspace itself, combining the credentials, storage and network configuration. + +Because it consumes an existing VPC, subnets, security groups, cross-account role and root bucket, this module is meant to be composed with a networking/IAM/storage layer (for example the [`aws-databricks-base-infra`](../aws-databricks-base-infra) module) rather than used on its own. + +## How to use + +1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources). +2. Provide values for the required variables (`region`, `vpc_id`, `security_group_ids`, `vpc_private_subnets`, `databricks_account_id`, `cross_account_role_arn`, `root_storage_bucket`). +3. Configure the `databricks` provider at account level with your account credentials. +4. Run `terraform init`. +5. Run `terraform apply`. + + +## Requirements + +| Name | Version | +|------|---------| +| [databricks](#requirement\_databricks) | >=1.24.1 | + +## Providers + +| Name | Version | +|------|---------| +| [databricks](#provider\_databricks) | >=1.24.1 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [databricks_mws_credentials.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_credentials) | resource | +| [databricks_mws_networks.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_networks) | resource | +| [databricks_mws_storage_configurations.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_storage_configurations) | resource | +| [databricks_mws_workspaces.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [cross\_account\_role\_arn](#input\_cross\_account\_role\_arn) | (Required) AWS cross account role ARN that will be used for the Databricks workspace | `string` | n/a | yes | +| [databricks\_account\_id](#input\_databricks\_account\_id) | (Required) Databricks Account ID | `string` | n/a | yes | +| [region](#input\_region) | (Required) AWS region where the assets will be deployed | `string` | n/a | yes | +| [root\_storage\_bucket](#input\_root\_storage\_bucket) | (Required) AWS root storage bucket | `string` | n/a | yes | +| [security\_group\_ids](#input\_security\_group\_ids) | (Required) List of VPC network security group IDs | `list(string)` | n/a | yes | +| [vpc\_id](#input\_vpc\_id) | (Required) AWS VPC ID | `string` | n/a | yes | +| [vpc\_private\_subnets](#input\_vpc\_private\_subnets) | (Required) AWS VPC Subnets where the Databricks workspace will be deployed | `list(string)` | n/a | yes | +| [prefix](#input\_prefix) | (Optional) Prefix for use in the generated names | `string` | `"demo"` | no | +| [tags](#input\_tags) | (Optional) Optional tags to add to created resources | `map(string)` | `{}` | no | +| [workspace\_name](#input\_workspace\_name) | (Optional) Workspace Name for this module - if none are provided, the prefix will be used to name the workspace via coalesce() | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [databricks\_host](#output\_databricks\_host) | n/a | +| [databricks\_workspace\_id](#output\_databricks\_workspace\_id) | n/a | + diff --git a/modules/aws/aws-infra/modules/hub-networking/README.md b/modules/aws/aws-infra/modules/hub-networking/README.md new file mode 100644 index 00000000..e89bff4c --- /dev/null +++ b/modules/aws/aws-infra/modules/hub-networking/README.md @@ -0,0 +1,93 @@ +# Hub Networking + +Submodule of [`aws-infra`](../../README.md) that implements a hub-and-spoke network topology with centralized egress inspection for Databricks workloads. It creates: + +* **Transit Gateway** — an `aws_ec2_transit_gateway` with dedicated hub and spoke route tables, VPC attachments, and routes that steer all spoke egress through the hub VPC. +* **Hub VPC** — an `aws_vpc` with public, private and firewall subnets across the provided availability zones, an internet gateway, NAT gateway and associated route tables. +* **Network Firewall** — an optional `aws_networkfirewall_firewall` with FQDN allow-list, network-level allow rules and a default deny, so outbound traffic from the spoke VPC is inspected and filtered before reaching the internet. + +Spoke traffic is routed to the Transit Gateway, into the hub VPC, through the Network Firewall, and out via the NAT/internet gateway — giving a single, centrally controlled egress point. + +This is an internal submodule consumed by the `aws-infra` module and is not intended to be deployed standalone. It expects an existing spoke VPC (its ID, CIDR, private subnets and route tables are passed in as variables). + +## How to use + +This submodule is called by the parent [`aws-infra`](../../README.md) module. To use it directly, pass the spoke VPC details and hub configuration via the variables documented below and provide the `aws` provider. + + +## Requirements + +No requirements. + +## Providers + +| Name | Version | +|------|---------| +| [aws](#provider\_aws) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [aws_ec2_transit_gateway.main](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway) | resource | +| [aws_ec2_transit_gateway_route.hub_to_spoke](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route) | resource | +| [aws_ec2_transit_gateway_route.spoke_default_to_hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route) | resource | +| [aws_ec2_transit_gateway_route.spoke_to_hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route) | resource | +| [aws_ec2_transit_gateway_route_table.hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route_table) | resource | +| [aws_ec2_transit_gateway_route_table.spoke](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route_table) | resource | +| [aws_ec2_transit_gateway_route_table_association.hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route_table_association) | resource | +| [aws_ec2_transit_gateway_route_table_association.spoke](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_route_table_association) | resource | +| [aws_ec2_transit_gateway_vpc_attachment.hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_vpc_attachment) | resource | +| [aws_ec2_transit_gateway_vpc_attachment.spoke](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_vpc_attachment) | resource | +| [aws_eip.hub_nat](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eip) | resource | +| [aws_internet_gateway.hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/internet_gateway) | resource | +| [aws_nat_gateway.hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/nat_gateway) | resource | +| [aws_networkfirewall_firewall.main](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/networkfirewall_firewall) | resource | +| [aws_networkfirewall_firewall_policy.main](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/networkfirewall_firewall_policy) | resource | +| [aws_networkfirewall_rule_group.allow_fqdns](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/networkfirewall_rule_group) | resource | +| [aws_networkfirewall_rule_group.allow_network](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/networkfirewall_rule_group) | resource | +| [aws_networkfirewall_rule_group.deny_all](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/networkfirewall_rule_group) | resource | +| [aws_route.hub_public_to_firewall](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route) | resource | +| [aws_route.hub_public_to_igw](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route) | resource | +| [aws_route.hub_public_to_spoke](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route) | resource | +| [aws_route.private_to_tgw](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route) | resource | +| [aws_route_table.hub_firewall](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table) | resource | +| [aws_route_table.hub_private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table) | resource | +| [aws_route_table.hub_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table) | resource | +| [aws_route_table_association.hub_firewall](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table_association) | resource | +| [aws_route_table_association.hub_private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table_association) | resource | +| [aws_route_table_association.hub_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table_association) | resource | +| [aws_security_group.hub_default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group) | resource | +| [aws_subnet.hub_firewall](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet) | resource | +| [aws_subnet.hub_private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet) | resource | +| [aws_subnet.hub_public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet) | resource | +| [aws_vpc.hub](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [availability\_zones](#input\_availability\_zones) | List of availability zones | `list(string)` | n/a | yes | +| [hub\_vpc\_cidr](#input\_hub\_vpc\_cidr) | CIDR block for the hub VPC | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix for resource names | `string` | n/a | yes | +| [region](#input\_region) | AWS region | `string` | n/a | yes | +| [spoke\_private\_subnet\_ids](#input\_spoke\_private\_subnet\_ids) | IDs of the spoke VPC private subnets | `list(string)` | n/a | yes | +| [spoke\_route\_table\_ids](#input\_spoke\_route\_table\_ids) | IDs of the spoke VPC private route tables | `list(string)` | n/a | yes | +| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR block of the spoke (main) VPC | `string` | n/a | yes | +| [spoke\_vpc\_id](#input\_spoke\_vpc\_id) | ID of the spoke (main) VPC | `string` | n/a | yes | +| [allowed\_fqdns](#input\_allowed\_fqdns) | List of FQDNs to allow through the firewall | `list(string)` | `[]` | no | +| [allowed\_network\_rules](#input\_allowed\_network\_rules) | List of network-level rules (IP, protocol, port) | 
list(object({
    protocol         = string
    source_ip        = string
    destination_ip   = string
    destination_port = string
  }))
| `[]` | no | +| [common\_tags](#input\_common\_tags) | Common tags to apply to all resources | `map(string)` | `{}` | no | +| [enable\_dns\_support](#input\_enable\_dns\_support) | Enable DNS support on the Transit Gateway | `bool` | `true` | no | +| [enable\_firewall](#input\_enable\_firewall) | Enable Network Firewall in the hub VPC | `bool` | `true` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [hub\_vpc\_id](#output\_hub\_vpc\_id) | ID of the hub VPC | + diff --git a/modules/gcp-unity-catalog/README.md b/modules/gcp-unity-catalog/README.md new file mode 100644 index 00000000..907f0a38 --- /dev/null +++ b/modules/gcp-unity-catalog/README.md @@ -0,0 +1,67 @@ +# GCP Unity Catalog + +This module provisions Unity Catalog for a Databricks workspace on Google Cloud and the GCS storage that backs it. It creates: + +* `google_storage_bucket` — a GCS bucket used as the storage root for Unity Catalog data. +* `databricks_metastore` and `databricks_metastore_assignment` — a regional metastore, assigned to the target workspace. +* `databricks_storage_credential` — a storage credential backed by a Databricks-managed GCP service account. +* `google_storage_bucket_iam_member` — grants the storage credential's service account read/admin access on the bucket. +* `databricks_external_location` — an external location pointing at the bucket, using the storage credential. +* `databricks_catalog` — a catalog whose storage root is the external location. + +The module expects an existing Databricks workspace on GCP and uses two `databricks` provider configurations (account-level and workspace-level via the `databricks.workspace` alias). + +## How to use + +1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources). +2. Configure two `databricks` providers: the default (account-level) and a `databricks.workspace` alias pointing at the target workspace, plus the `google` provider. +3. Provide values for the required variables (`databricks_workspace_url`, `databricks_workspace_id`, `google_region`, `google_project`, `prefix`, `metastore_name`, `catalog_name`). +4. Run `terraform init`. +5. Run `terraform apply`. + + +## Requirements + +No requirements. + +## Providers + +| Name | Version | +|------|---------| +| [databricks](#provider\_databricks) | n/a | +| [databricks.workspace](#provider\_databricks.workspace) | n/a | +| [google](#provider\_google) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [databricks_catalog.main](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog) | resource | +| [databricks_external_location.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/external_location) | resource | +| [databricks_metastore.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore) | resource | +| [databricks_metastore_assignment.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_assignment) | resource | +| [databricks_storage_credential.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/storage_credential) | resource | +| [google_storage_bucket.ext_bucket](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource | +| [google_storage_bucket_iam_member.unity_cred_admin](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource | +| [google_storage_bucket_iam_member.unity_cred_reader](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [catalog\_name](#input\_catalog\_name) | Name to assign to default catalog | `string` | n/a | yes | +| [databricks\_workspace\_id](#input\_databricks\_workspace\_id) | The unique identifier of the Databricks workspace in which resources will be managed. | `any` | n/a | yes | +| [databricks\_workspace\_url](#input\_databricks\_workspace\_url) | The URL of the Databricks workspace to which resources will be deployed (e.g., https://.gcp.databricks.com). | `any` | n/a | yes | +| [google\_project](#input\_google\_project) | The Google Cloud project ID where the Databricks workspace and associated resources will be created. | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | Google Cloud region where the resources will be created | `string` | n/a | yes | +| [metastore\_name](#input\_metastore\_name) | Name to assign to regional metastore | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix to use in generated resources name | `string` | n/a | yes | + +## Outputs + +No outputs. +