As explained in the previous article "Building Dataspaces in Multi-Cloud", we utilized Rancher to manage Kubernetes clusters across multiple clouds AWS, GCP, Azure, and OTC.
By using dataspaces in a multi-cloud environment, where Kubernetes clusters can be running in any restricted or unrestricted setup, securing communication between the Rancher control plane and Rancher agents is crucial. This can be achieved by allowing managed cluster IPs in a Web Application Firewall (WAF).
To ensure connectivity between the newly created Kubernetes clusters managed by Rancher and the Rancher Load Balancer, we added the IP addresses of the NAT gateway to the Security Group associated with the Rancher Load Balancer as shown in the diagram below:
The better option is to use the Web Application Firewall (WAF), which can utilize 100 IP sets, each IP set handling 10,000 IP addresses, allowing for a total of 1 million IP addresses.
Automation WAF with Crossplane, Lambda, and DynamoDB:
The main challenge we face during cluster provisioning is bootstrapping all the default Kubernetes applications and external cloud services. Since we are provisioning the clusters dynamically and rarely manually, we decided to use crossplane as a single control plane for both Kubernetes and cloud resources
You can find more information on crossplane in their official documentation but here's an example of how we can define a dynamodb configuration as CRD in our cluster:
apiVersion: dynamodb.aws.upbound.io/v1beta1
kind: TableItem
metadata:
name: cluster1
spec:
providerConfigRef:
name: aws-provider
forProvider:
region: eu-central-1
hashKey: id
item: |
{
"id": {"S": "cluster1"},
"ips": {"S": "[\"54.123.456.78/32\"]"}
}
rangeKey: ips
tableName: cluster-bootstrapping
Web Application Firewall (WAF) It includes a Rancher Access Control List (ACL) that, by default, blocks all requests. Additionally, it maintains an allowed IP set containing all IPs originating from the Crossplane bootstrapping service. Furthermore, it requires association with AWS resources such as the Application Load Balancer ARN to function effectively.
DynamoDB: acts as a source for allowlist of cluster IPs that we manage with crossplane.
DynamoDB Streams enabled and the stream view is set to "NEW_AND_OLD_IMAGES".
Lambda function: is invoked to add or remove the IPs from the WAF set. DynamoDB Streams triggers Lambda in response to these changes.
Example of an event that triggers a Lambda function invoked by DynamoDB Streams when eventName is INSERT or REMOVE:
{'Records': [{'eventID': '34750be9350340a66221711500a1cc97',
'eventName': 'INSERT',
'eventVersion': '1.1',
'eventSource': 'aws:dynamodb',
'awsRegion': 'eu-central-1',
'dynamodb': {'ApproximateCreationDateTime': 1718874187.0,
'Keys': {'id': {'S': 'cluster1'},
'ips': {'S': '["54.123.456.78/32"]'}},
'NewImage': {'id': {'S': 'cluster1'},
'ips': {'S': '["54.123.456.78/32"]'}},
'SequenceNumber': '31142800000000062127994195',
'SizeBytes': 76,
'StreamViewType': 'NEW_AND_OLD_IMAGES'},
'eventSourceARN': 'arn:aws:dynamodb:eu-central-1:ACCOUNT_NUMBER:table/cluster-bootstrapping/stream/2024-06-13T17:43:35.560'}]
}
The Python Lambda function is invoked by events, fetches all IPs from the existing IP set, and checks the event name. If the event name is 'INSERT', it verifies if the IP in the event is already in the existing list and does nothing if it is; otherwise, it adds the IP. Similarly, if the event name is 'REMOVE', it checks if the IP is in the existing list and removes it if it is; otherwise, it does nothing.
import boto3
import json
region_name = “eu-central-1”
# Define the scope and the IPSetId
scope = “REGIONAL”
name = ip_set_name # Name of IPSet, that created in WAF
ip_set_id = ip_set_id # ID of the IPSet, that created in WAF
# Create a WAFv2 client
client = boto3.client('wafv2', region_name=region_name)
def get_ip_set():
"""Fetch the current IP set and its lock token."""
try:
response = client.get_ip_set(Name=name, Scope=scope, Id=ip_set_id)
ip_set = response['IPSet']
lock_token = response['LockToken']
return ip_set, lock_token
except Exception as e:
raise
def update_ip_set(add_ips=None, remove_ips=None):
"""
Update the IP set by adding new IPs or removing existing IPs.
Parameters:
- add_ips: List of IPs to add (e.g., ['10.0.0.1/32'])
- remove_ips: List of IPs to remove (e.g., ['10.0.0.2/32'])
"""
try:
# Get current IP set and lock token
ip_set, lock_token = get_ip_set()
current_ips = set(ip_set['Addresses'])
# Prepare sets for additions and removals
ips_to_add = set()
ips_to_remove = set()
if add_ips:
add_ips = set(add_ips)
ips_to_add = add_ips - current_ips
if remove_ips:
remove_ips = set(remove_ips)
ips_to_remove = remove_ips & current_ips
# If there are no changes to be made, return early
if not ips_to_add and not ips_to_remove:
return
# Update current IPs
updated_ips = current_ips | ips_to_add
updated_ips -= ips_to_remove
# Convert to list for update
addresses = list(updated_ips)
# Update the IP set
response = client.update_ip_set(Name=name, Scope=scope, Id=ip_set_id, Addresses=addresses, LockToken=lock_token)
return response
except Exception as e:
raise
def lambda_handler(event, context):
try:
for record in event['Records']:
event_name = record['eventName']
if event_name == 'INSERT':
ip_address = record['dynamodb']['Keys']['ips']['S']
return update_ip_set(add_ips=json.loads(ip_address))
if event_name == 'REMOVE':
ip_address = record['dynamodb']['Keys']['ips']['S']
return update_ip_set(remove_ips=json.loads(ip_address))
except Exception as e:
raise
This is just an example how we can use crossplane for infrastructure management replacing many IaC solutions.
