Adding terraform and documentation for end-users

Author: chapmanc

customer-managed/azure/privatelink/README.md

@@ -0,0 +1,244 @@
+# Redpanda Azure private link
+
+
+### Getting Started
+
+#### Required tools:
+- rpk
+- terraform
+- Azure CLI
+- Curl
+- jq
+
+Once all tools are installed make sure you have logged into to the azure CLI
+
+```
+az login
+```
+
+## Provisioning Redpanda with Private Link
+
+To provision a redpanda with private link there is currently only API support with UX support coming soon. 
+
+### Environment setup
+
+1. To start set the following env variables:
+    ```
+    export API_ENDPOINT=https://api.cloud.redpanda.com
+    export RPK_CLOUD_AUTH_URL=https://auth.prd.cloud.redpanda.com
+    export RPK_CLOUD_AUTH_AUDIENCE=cloudv2-production.redpanda.cloud
+    ```
+
+2. Next obtaion a client id and secret by going to Organizations in the Cloud UI. Navigate to Clients. Select or create a client you would like to use to create the network and cluster. You can copy the ID and secret to your clipboard by clicking the “Copy ID” and “Copy secret” buttons.
+    ```
+    export CLOUD_CLIENT_ID=<client-ID>
+    export CLOUD_CLIENT_SECRET=<client-secret>
+    ```
+
+3. Set the Azure subscriptions you would like to use:
+
+    ```
+    # Where the redpanda cluster will be provisioned
+    AZURE_SUBSCRIPTION_ID=60fc0bed-3072-4c53-906a-d130a934d520
+
+    # Where you will connect to the private link service fronting redpanda
+    AZURE_PRIVATE_LINK_SUBSCRIPTION_ID=1b88eb19-4c80-4edd-870d-461e83ddcbb5
+    ```
+
+### Provisioning
+
+1. Obtain an auth token
+    ```
+    export AUTH_TOKEN=`curl -s --request POST \
+        --url "${RPK_CLOUD_AUTH_URL}/oauth/token" \
+        --header 'content-type: application/x-www-form-urlencoded' \
+        --data grant_type=client_credentials \
+        --data client_id=$CLOUD_CLIENT_ID \
+        --data client_secret=$CLOUD_CLIENT_SECRET \
+        --data audience=$RPK_CLOUD_AUTH_AUDIENCE | jq -r '.access_token'`
+
+    # MAKE SURE THIS IS POUPULATED
+    echo $AUTH_TOKEN
+
+
+    declare -a HEADERS=(
+        "-H" "Content-Type: application/json"
+        "-H" "Authorization: Bearer $AUTH_TOKEN"
+    )
+
+    ```
+
+1. Find the Resource Group where we will create the redpanda cluster
+
+    ```
+    RESOURCE_GROUP_ID=`curl -s -X GET "${HEADERS[@]}" \
+    $API_ENDPOINT/v1beta2/resource-groups | jq -r ".resource_groups[0].id"`
+
+    # NOTE: this should have output
+    echo $RESOURCE_GROUP_ID
+
+    ```
+
+1. Create the network that will be used by the redpanda cluster
+    ```
+    NETWORK_POST_BODY=`cat << EOF
+    {
+    "cidr_block": "10.0.0.0/20",
+    "cloud_provider": "CLOUD_PROVIDER_AZURE",
+    "cluster_type": "TYPE_BYOC",
+    "name": "azure-byoc-network",
+    "resource_group_id": "$RESOURCE_GROUP_ID",
+    "region": "uksouth"
+    }
+    EOF`
+
+    NETWORK_ID=`curl -s -X POST "${HEADERS[@]}" \
+    -d "$NETWORK_POST_BODY" $API_ENDPOINT/v1beta2/networks |
+    jq -r '.operation.metadata.network_id'`
+
+    # NOTE: this should have output
+    echo $NETWORK_ID
+    ```
+1. Create the redpanda cluster with the allowed subscriptions we set before. NOTE: you can add as many other subscriptions as you would like under `allowed_subscriptions`
+
+    ```
+    CLUSTER_POST_BODY=`cat << EOF
+    {
+    "cloud_provider": "CLOUD_PROVIDER_AZURE",
+    "connection_type": "CONNECTION_TYPE_PRIVATE",
+    "name": "azure-byoc-pl",
+    "resource_group_id": "$RESOURCE_GROUP_ID",
+    "network_id": "$NETWORK_ID",
+    "region": "uksouth",
+    "throughput_tier": "tier-1-azure-beta",
+    "type": "TYPE_BYOC",
+    "zones": ["uksouth-az1",  "uksouth-az2", "uksouth-az3"],
+    "redpanda_version": "24.1",
+    "azure_private_link": { 
+        "allowed_subscriptions": ["$AZURE_PRIVATE_LINK_SUBSCRIPTION_ID"],
+        "enabled": true,
+        "connect_console": true
+    }
+    }
+    EOF`
+
+    RP_ID=`curl -s -X POST "${HEADERS[@]}" \
+    -d "$CLUSTER_POST_BODY" $API_ENDPOINT/v1beta2/clusters | jq -r '.operation.metadata.cluster_id'`
+
+    # NOTE: this should have output
+    echo $RP_ID
+    ```
+1. Run the rpk apply to create the infrastructure for the cluster. This will take about 45 minutes to complete. 
+    ```
+    rpk login --save --client-id=$CLOUD_CLIENT_ID --client-secret=$CLOUD_CLIENT_SECRET
+
+    rpk cloud byoc azure apply --redpanda-id=$RP_ID --subscription-id=$AZURE_SUBSCRIPTION_ID
+    
+    ```
+
+## Connecting to Redpanda Private Link
+
+Once the cluster has been created we can now run terraform to connect to the created cluster with private link. This terraform will provision a vm along with a private link endpoint that connects to the redpanda cluster. 
+
+
+
+1. First fetch credentials to the Redpanda cluster:
+
+    ```
+    az login 
+    az account set --subscription $AZURE_SUBSCRIPTION_ID
+    az aks get-credentials -g rg-rpcloud-$RP_ID -n aks-rpcloud-$RP_ID
+    secret=$(kubectl get secret redpanda-superusers -n redpanda -o jsonpath='{.data.users\.txt}' | base64 --decode)
+
+    IFS=':' read -r user password sasl <<< "$secret"
+
+    export USER=$user
+    export PASS=$password
+
+    ```
+1. Fetch outputs from the redpanda cluster that we will use as inputs to our terraform
+    ```
+    DNS_RECORD=`curl -s -X GET "${HEADERS[@]}" \
+    $API_ENDPOINT/v1beta2/clusters/$RP_ID | jq -r ".cluster.azure_private_link.status.dns_a_record"`
+
+    PRIVATE_SERVICE_ID=`curl -s -X GET "${HEADERS[@]}" \
+    $API_ENDPOINT/v1beta2/clusters/$RP_ID | jq -r ".cluster.azure_private_link.status.service_id"`
+
+    CONSOLE_URL=`curl -s -X GET "${HEADERS[@]}" \
+    $API_ENDPOINT/v1beta2/clusters/$RP_ID | jq -r ".cluster.redpanda_console.url"`
+
+    echo $DNS_RECORD
+    echo $PRIVATE_SERVICE_ID
+    echo $CONSOLE_URL
+
+    ```
+1. Next make sure you are logged into the subscription that was allowed in the creation request: 
+    ```
+    az login
+    az account set --subscription $AZURE_PRIVATE_LINK_SUBSCRIPTION_ID
+    ```
+1. Write our terraform variables out to a file so we can pick it up and use it
+    ```
+    export MYGROUP=your-rg-group
+    cat << EOF > terraform.tfvars
+    region = "uksouth"
+    resource_group_name = "$MYGROUP"
+    rp_endpoint_service_id = "$PRIVATE_SERVICE_ID"
+    rp_id = "$RP_ID"
+    rp_domain = "$DNS_RECORD"
+    rp_node_count = 3
+    EOF
+    ```
+1. Initialize the terraform and create the needed resources
+    ```
+    cd terraform
+    terraform init 
+    terraform apply -var-file="terraform.tfvars"
+
+    ```
+
+1. When the terraform completes copy and paste the following block. This will give us another block we can copy once we have ssh'd into the vm.
+    ```
+    VM_IP=$(terraform output -raw instance_public_ip)
+    KEY_PATH=$(terraform output -raw private_key_file_path)
+
+    cat << EOF 
+    Copy and paste this block after sshing in:
+
+    export USER=$USER
+    export PASS=$PASS
+    export CONSOLE_URL=$CONSOLE_URL
+    export BROKERS=brokers.${DNS_RECORD}:30292
+
+    EOF
+
+    ```
+
+1. Next ssh in and copy the output of the previous command into the vm to setup the environment variables needed to connect
+    ```
+    ssh -i $KEY_PATH redpanda@$VM_IP
+    
+    $ <paste in output from previous command>
+    ```
+
+1. Finally you now can connect to the redpanda cluster over the private link endpoint
+    ```
+    auth="-X user=$USER -X pass=$PASS -X sasl.mechanism=SCRAM-SHA-512 -X brokers=$BROKERS  --tls-enabled"
+
+    rpk cluster info $auth
+    rpk topic create mytopic $auth
+
+    cat <<EOF | rpk topic $auth produce mytopic  -f '%k,%v\n'
+    key1,value1
+    key2,value2
+    key3,value4
+    EOF
+
+    rpk topic consume mytopic $auth
+
+    ```
+1. Once  you have completed testing and want to tear down the infrastructure you can run
+    ```
+    terraform destroy -var-file="terraform.tfvars"
+
+    ```

customer-managed/azure/privatelink/terraform/files/files/install-rpk.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+if command -v rpk &> /dev/null
+then
+    echo "found rpk, skip the installation"
+    exit 0
+fi
+
+cd /tmp; curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
+export DEBIAN_FRONTEND=noninteractive
+sudo apt-get update
+# Need NEEDRESTART_MODE=a for not having the prompt to restart services
+sudo NEEDRESTART_MODE=a apt install -y unzip
+sudo unzip /tmp/rpk-linux-amd64.zip -d /usr/local/bin/

customer-managed/azure/privatelink/terraform/main.tf

@@ -0,0 +1,68 @@
+terraform {
+  required_providers {
+    azurerm = {
+      source  = "hashicorp/azurerm"
+      version = "=3.98.0"
+    }
+  }
+
+}
+
+resource "azurerm_resource_group" "rg" {
+  name     = var.resource_group_name
+  location = var.region
+}
+
+resource "azurerm_virtual_network" "vnet" {
+  name                = "${var.resource_prefix}_vnet"
+  address_space       = [var.vnet_cidr]
+  location            = var.region
+  resource_group_name = azurerm_resource_group.rg.name
+}
+
+resource "azurerm_subnet" "subnet" {
+  name                 = "${var.resource_prefix}_subnet"
+  resource_group_name  = azurerm_resource_group.rg.name
+  virtual_network_name = azurerm_virtual_network.vnet.name
+  address_prefixes     = [var.vnet_subnet] # Adjust the subnet CIDR block as needed
+}
+
+resource "azurerm_private_endpoint" "service-endpoint" {
+  name                = "${var.resource_prefix}_service_endpoint"
+  location            = var.region
+  resource_group_name = azurerm_resource_group.rg.name
+  subnet_id           = azurerm_subnet.subnet.id
+
+
+  private_service_connection {
+    name                           = "${var.resource_prefix}_service_connection"
+    private_connection_resource_id = var.rp_endpoint_service_id
+    is_manual_connection           = false
+  }
+  private_dns_zone_group {
+    name                 = "example-dns-zone-group"
+    private_dns_zone_ids = [azurerm_private_dns_zone.example.id]
+  }
+}
+
+resource "azurerm_private_dns_zone" "example" {
+  name                = var.rp_domain
+  resource_group_name = azurerm_resource_group.rg.name
+}
+
+resource "azurerm_private_dns_a_record" "example" {
+  name                = "*"
+  zone_name           = azurerm_private_dns_zone.example.name
+  resource_group_name = azurerm_resource_group.rg.name
+  ttl                 = 300
+  records             = [azurerm_private_endpoint.service-endpoint.private_service_connection[0].private_ip_address]
+}
+
+resource "azurerm_private_dns_zone_virtual_network_link" "example" {
+  name                  = "${var.resource_prefix}_example_link"
+  resource_group_name   = azurerm_resource_group.rg.name
+  private_dns_zone_name = azurerm_private_dns_zone.example.name
+  virtual_network_id    = azurerm_virtual_network.vnet.id
+}
+
+

customer-managed/azure/privatelink/terraform/output.tf

@@ -0,0 +1,65 @@
+locals {
+  kafka_api_seed_nlb_listener_port       = 30292
+  schema_registry_seed_nlb_listener_port = 30081
+  rp_proxy_seed_nlb_listener_port        = 30282
+
+  kafka_api_node_nlb_listener_port = 32092
+  rp_proxy_node_nlb_listener_port  = 31082
+
+  console_port = 443
+
+  resource_prefix = (var.resource_prefix) == "" ? "" : replace(var.resource_prefix, "_", "-")
+
+
+  cert_file = format("/tmp/%s.pem", azurerm_public_ip.public_ip.ip_address)
+
+}
+
+
+output "instance_public_ip" {
+  value = azurerm_public_ip.public_ip.ip_address
+}
+
+output "rp_kafka_api_node_endpoints" {
+  description = "Redpanda Kafka service urls"
+  value = var.resource_prefix == "" ? {
+    for i in range(var.rp_node_count) : format("kafka_api_node%d_endpoint", i) => format("%s:%d", var.rp_domain, local.kafka_api_node_nlb_listener_port + i)
+  } : null
+}
+
+output "rp_rp_proxy_node_endpoints" {
+  description = "Redpanda proxy service urls"
+  value = var.resource_prefix == "" ? {
+    for i in range(var.rp_node_count) : format("redpanda_proxy_node%d_endpoint", i) => format("%s:%d", var.rp_domain, local.rp_proxy_node_nlb_listener_port + i)
+  } : null
+}
+
+output "rp_seed_endpoints" {
+  description = "Redpanda service seed urls"
+  value = var.resource_prefix == "" ? {
+    "kafka_api_seed" : format("%s:%d", var.rp_domain, local.kafka_api_seed_nlb_listener_port)
+    "schema_registry_seed" : format("%s:%d", var.rp_domain, local.schema_registry_seed_nlb_listener_port)
+    "redpanda_proxy_seed" : format("%s:%d", var.rp_domain, local.rp_proxy_seed_nlb_listener_port)
+  } : null
+}
+
+output "ssh_private_key" {
+  description = "SSH key to EC instance"
+  value       = tls_private_key.key.private_key_pem
+  sensitive   = true
+}
+
+output "private_key_file_path" {
+  value = abspath(local_file.private_key.filename)
+}
+
+output "ssh_to_ec2_commands" {
+  description = "SSH to EC2 instance commands"
+  # Terraform go code has issue to unmarshall the output. So no output if running inside the certification tests.
+  value = var.resource_prefix == "" ? {
+    "get_ssh_key_command" : <<EOF
+    cat terraform.tfstate | jq .outputs.ssh_private_key.value | sed 's/"//g' | awk '{gsub(/\\n/,"\n")}1' > ${local.cert_file}; chmod 600 ${local.cert_file}
+    EOF
+    "ssh_command" : format("ssh -i %s redpanda@%s", local.cert_file, azurerm_public_ip.public_ip.ip_address)
+  } : null
+}