Skip to content

Commit a434349

Browse files
frenchfrywpepperfrenchfrywpepper
committed
Initial upload of BYOVPC AWS code
This terraform content and the corresponding README are replacing the pre-existing Google Drive content: https://docs.google.com/document/d/1mUfRTDecvM5rv8y2D8yMpkMmLypmZp3stavA48pxF18/edit
1 parent d1cba2f commit a434349

23 files changed

Lines changed: 3052 additions & 0 deletions

‎.pre-commit-config.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
repos:
2+
- repo: https://github.com/gruntwork-io/pre-commit
3+
rev: v0.1.23
4+
hooks:
5+
- id: tflint
6+
- id: terraform-validate
7+
- id: terraform-fmt

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Developer Setup
2+
3+
Install [tflint](https://github.com/terraform-linters/tflint)

‎customer-managed/aws/README.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
This document contains customer instructions for early access to the Redpanda BYOVPC AWS feature, for proof of concept
2+
purposes.
3+
4+
# Prerequisites
5+
6+
1. Your Redpanda representative must have enabled the BYOVPC feature flag for your organization.
7+
2. Familiarity with the [Redpanda Public API](https://redpanda-api-documentation.netlify.app/). (e.g. How to
8+
authenticate and create a cluster using the API.)
9+
3. Access to an AWS project in which you’d like to create your cluster.
10+
4. [Minimum permissions](https://github.com/redpanda-data/cloud-examples/customer-managed/aws/terraform/iam_rpk_user.tf) in that
11+
AWS project. (Note that the link to the minimum permissions is provided simply as documentation, we do not anticipate
12+
that you would create the role in that file, it is provided only as a way to communicate the actions that will be
13+
required of the employee running rpk byoc apply.)
14+
15+
# Create VPC and other sensitive resources
16+
17+
The [terraform code](https://github.com/redpanda-data/cloud-examples/customer-managed/aws/terraform) contains the
18+
specifications for what resources are expected to exist and how they are expected to be configured.
19+
20+
This terraform code will create the following resources:
21+
* IAM Roles
22+
* IAM Instance Profiles
23+
* IAM Role Policy Attachments
24+
* IAM Policies
25+
* Elastic IP
26+
* Internet Gateway
27+
* Route
28+
* VPC
29+
* Subnets
30+
* VPC Endpoint
31+
* Route Tables
32+
* Route Table Associations
33+
* VPC Endpoint Route Table Associations
34+
* Security Groups
35+
* Security Group Rules
36+
* S3 Buckets
37+
* S3 Bucket Server-Side Encryption Configurations
38+
* S3 Bucket Versioning
39+
* S3 Bucket Ownership Controls
40+
* DynamoDB Table
41+
42+
Some of these resources may already exist in your AWS project in which case you can remove that item from the terraform
43+
and modify references to use a data block instead of the resource.
44+
45+
In particular the resources in the
46+
[iam_rpk_user.tf](https://github.com/redpanda-data/cloud-examples/customer-managed/aws/terraform/iam_rpk_user.tf) file
47+
are not intended to be created but are only provided to serve as documentation of the minimum viable permissions for the
48+
employee who will run rpk byoc apply.
49+
50+
This terraform module outputs a number of ARN values, these will be used as inputs for the Public API calls.
51+
52+
# Create Redpanda Cloud Network using Public API
53+
54+
After [authenticating](https://redpanda-api-documentation.netlify.app/#auth) to the Public API, issue a create network
55+
request using the outputs from the terraform in the previous step.
56+
57+
```shell
58+
curl -X POST "https://api.redpanda.com/v1beta2/networks" \
59+
-H "accept: application/json"\
60+
-H "content-type: application/json" \
61+
-H "authorization: Bearer $PUBLIC_API_TOKEN" \
62+
-d @- << EOF
63+
{
64+
"name":"<name you choose for your network resource>",
65+
"resource_group_id": "<Resource group ID of the network>",
66+
"cloud_provider":"CLOUD_PROVIDER_AWS",
67+
"region":"<region>",
68+
"cluster_type":"TYPE_BYOC",
69+
"customer_managed_resources": {
70+
"aws": {
71+
"management_bucket": {
72+
"arn": "<management_bucket_arn from outputs>"
73+
},
74+
"dynamodb_table": {
75+
"arn": "<dynamodb_table_arn from outputs>"
76+
},
77+
"private_subnets": {
78+
"arns": [<private_subnet_ids from outputs>]
79+
},
80+
"public_subnets": {
81+
"arns": [<public_subnet_ids from outputs>]
82+
},
83+
"vpc": {
84+
"arn": "<vpc_arn from outputs>"
85+
}
86+
}
87+
}
88+
}
89+
EOF
90+
```
91+
92+
The create network request will return a resource_id, you will need this in the create cluster step next. Example:
93+
94+
```shell
95+
{
96+
"operation": {
97+
"id":"cpas8k6r4up5li18auh0",
98+
"metadata": {
99+
"@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateNetworkMetadata",
100+
"network_id":"cpb338gekjj5i1cpj3t0"
101+
},
102+
"state":"STATE_IN_PROGRESS",
103+
"started_at":"2024-05-28T19:33:54.631Z",
104+
"type":"TYPE_CREATE_NETWORK",
105+
"resource_id":"cpb338gekjj5i1cpj3t0"
106+
}
107+
}
108+
```
109+
110+
# Create Redpanda Cloud Cluster using Public API
111+
112+
Issue a create cluster request using the outputs from the terraform in the prior step.
113+
114+
```shell
115+
curl -X POST "https://api.redpanda.com/v1beta2/clusters" \
116+
-H "accept: application/json"\
117+
-H "content-type: application/json" \
118+
-H "authorization: Bearer $PUBLIC_API_TOKEN" \
119+
-d @- << EOF
120+
{
121+
"cloud_provider":"CLOUD_PROVIDER_AWS",
122+
"connection_type":"CONNECTION_TYPE_PRIVATE",
123+
"name":"<name of cluster>",
124+
"resource_group_id":"<Resource group ID of the network>",
125+
"network_id":"<resource_id of network from previous step>",
126+
"region":"<region>",
127+
"throughput_tier":"<throughput tier>",
128+
"type":"TYPE_BYOC",
129+
"zones":["use2-az1","use2-az2","use2-az3"],
130+
"redpanda_version": "<redpanda version>",
131+
"customer_managed_resources": {
132+
"aws": {
133+
"agent_instance_profile": {
134+
"arn": "<agent_instance_profile_arn from outputs>"
135+
},
136+
"connectors_node_group_instance_profile": {
137+
"arn": "<connectors_node_group_instance_profile_arn from outputs>"
138+
},
139+
"redpanda_node_group_instance_profile": {
140+
"arn": "<redpanda_node_group_instance_profile_arn from outputs>"
141+
},
142+
"utility_node_group_instance_profile": {
143+
"arn": "<utility_node_group_instance_profile_arn from outputs>"
144+
},
145+
"console_secrets_manager_role": {
146+
"arn": "<console_secrets_manager_role_arn from outputs>"
147+
},
148+
"connectors_security_group": {
149+
"arn": "<connectors_security_group_arn from outputs>"
150+
},
151+
"redpanda_cloud_storage_manager_role": {
152+
"arn": "<redpanda_cloud_storage_manager_role_arn from outputs>"
153+
},
154+
"connectors_secrets_manager_role": {
155+
"arn": "<connectors_secrets_manager_role_arn from outputs>"
156+
},
157+
"node_security_group": {
158+
"arn": "<node_security_group_arn from outputs>"
159+
},
160+
"utility_security_group": {
161+
"arn": "<utility_security_group_arn from outputs>"
162+
},
163+
"redpanda_agent_security_group": {
164+
"arn": "<redpanda_agent_security_group_arn from outputs>"
165+
},
166+
"redpanda_node_group_security_group": {
167+
"arn": "<redpanda_node_group_security_group_arn from outputs>"
168+
},
169+
"cluster_security_group": {
170+
"arn": "<cluster_security_group_arn from outputs>"
171+
},
172+
"k8s_cluster_role": {
173+
"arn": "<k8s_cluster_role_arn from outputs>"
174+
},
175+
"cloud_storage_bucket": {
176+
"arn": "<cloud_storage_bucket_arn from outputs>"
177+
}
178+
}
179+
},
180+
# <This aws_private_link section is optional, see https://docs.redpanda.com/current/deploy/deployment-option/cloud/aws-privatelink/ for more information>
181+
"aws_private_link": {
182+
"enabled": true,
183+
"allowed_principals": [<allowed principals>],
184+
"connect_console": <true|false>
185+
},
186+
# <This cloud_provider_tags section is optional. If you want to provide a unique identifier to your cluster that can also be used in IAM conditionals you would need to provide that value here. At this time tags may not be deleted or modified after cluster creation.>
187+
"cloud_provider_tags": { "<key>": "<value>" }
188+
}
189+
EOF
190+
```
191+
192+
The create cluster request will return a resource_id, you will need this in the next step. Example:
193+
194+
```shell
195+
{
196+
"operation": {
197+
"id":"cpas8k6r4up5li18auhg",
198+
"metadata": {
199+
"@type":"type.googleapis.com/redpanda.api.controlplane.v1beta2.CreateClusterMetadata",
200+
"cluster_id":"cpb33c8ekjj5i1cpj3v0"
201+
},
202+
"state":"STATE_IN_PROGRESS",
203+
"started_at":"2024-05-28T19:34:09.501Z",
204+
"type":"TYPE_CREATE_CLUSTER",
205+
"resource_id":"cpb33c8ekjj5i1cpj3v0"
206+
}
207+
}
208+
```
209+
210+
# Run RPK Cloud BYOC AWS Apply
211+
212+
Next you will run `rpk cloud byoc aws apply` to create the initial resources required for creating a redpanda cluster in
213+
your cloud. Primarily this consists of an autoscaling group and an agent VM. Once the agent VM is created it will handle
214+
the remaining provisioning steps.
215+
216+
For this step you must have the at least those permissions mentioned in the prerequisites.
217+
218+
The following resources will be created during this step:
219+
* S3 Objects
220+
* Launch Template
221+
* Autoscaling Group
222+
223+
```shell
224+
rpk cloud login \
225+
--save \
226+
--client-id='<client-id>’ \
227+
--client-secret='<client-secret>' \
228+
--no-profile
229+
230+
rpk cloud byoc aws apply \
231+
--redpanda-id='<resource_id of cluster from previous step>'
232+
```
233+
234+
RPK will output its progress. Example:
235+
236+
```shell
237+
2024-05-28T20:38:34.020Z INFO .rpk.managed-byoc aws/apply.go:112 Reconciling agent infrastructure...
238+
2024-05-28T20:38:34.200Z INFO .rpk.managed-byoc cli/cli.go:194 Running apply {"provisioner": "redpanda-bootstrap"}
239+
2024-05-28T20:39:09.104Z INFO .rpk.managed-byoc cli/cli.go:204 Finished apply {"provisioner": "redpanda-bootstrap"}
240+
2024-05-28T20:39:09.104Z INFO .rpk.managed-byoc cli/cli.go:194 Running apply {"provisioner": "redpanda-network"}
241+
2024-05-28T20:41:41.962Z INFO .rpk.managed-byoc cli/cli.go:204 Finished apply {"provisioner": "redpanda-network"}
242+
2024-05-28T20:41:41.962Z INFO .rpk.managed-byoc cli/cli.go:194 Running apply {"provisioner": "redpanda-agent"}
243+
2024-05-28T20:42:40.767Z INFO .rpk.managed-byoc cli/cli.go:204 Finished apply {"provisioner": "redpanda-agent"}
244+
2024-05-28T20:42:40.768Z INFO .rpk.managed-byoc aws/apply.go:158 The Redpanda cluster is deploying. This can take up to 45 minutes. View status at https://cloud.redpanda.com/clusters/cpb4074p81130pqt8gjg/overview.
245+
```
246+
247+
# Wait for cluster creation to complete
248+
249+
At this point the agent VM is running and will handle the remaining setup steps. This can take up to 45 minutes. When it
250+
is complete the cluster status should be updated to ‘Running’. If the cluster status is still ‘Creating’ after that
251+
time, reach out to Redpanda support to investigate.
252+
253+
You can check the cluster status via the API or using the website.
254+
255+
Example using returned `operation_id`:
256+
257+
```shell
258+
curl -X GET "https://api.redpanda.com/v1beta2/operations/<operation_id>" \
259+
-H "accept: application/json"\
260+
-H "content-type: application/json" \
261+
-H "authorization: Bearer $PUBLIC_API_TOKEN"
262+
```
263+
264+
Example retrieving cluster:
265+
266+
```shell
267+
curl -X GET "https://api.redpanda.com/v1beta2/clusters/<cluster_id>" \
268+
-H "accept: application/json"\
269+
-H "content-type: application/json" \
270+
-H "authorization: Bearer $PUBLIC_API_TOKEN"
271+
```

‎customer-managed/aws/terraform/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
*.auto.tfvars.json
2+
**/.terraform/*
3+
*.tfstate
4+
*.tfstate.*

‎customer-managed/aws/terraform/.terraform.lock.hcl‎

Lines changed: 44 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎customer-managed/aws/terraform/iam_connectors_node_group.tf‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
data "aws_iam_policy_document" "connectors_node_group_trust" {
2+
statement {
3+
effect = "Allow"
4+
actions = ["sts:AssumeRole"]
5+
principals {
6+
type = "Service"
7+
identifiers = ["ec2.amazonaws.com"]
8+
}
9+
}
10+
}
11+
12+
resource "aws_iam_role" "connectors_node_group" {
13+
name_prefix = "${var.common_prefix}connect-"
14+
path = "/"
15+
force_detach_policies = true
16+
tags = merge(
17+
var.default_tags,
18+
{
19+
"redpanda-client" = "connectors"
20+
}
21+
)
22+
assume_role_policy = data.aws_iam_policy_document.connectors_node_group_trust.json
23+
}
24+
25+
resource "aws_iam_instance_profile" "connectors_node_group" {
26+
name_prefix = "${var.common_prefix}connect-"
27+
path = "/"
28+
role = aws_iam_role.connectors_node_group.name
29+
tags = merge(
30+
var.default_tags,
31+
{
32+
"redpanda-client" = "connectors"
33+
}
34+
)
35+
}
36+
37+
resource "aws_iam_role_policy_attachment" "connectors_node_group" {
38+
for_each = {
39+
"1" = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
40+
"2" = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
41+
"3" = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
42+
}
43+
policy_arn = each.value
44+
role = aws_iam_role.connectors_node_group.name
45+
}

0 commit comments

Comments
 (0)