Automating WAF for Multi-Cloud in Dataspaces

Most Popular Insights

As explained in the previous article "Building Dataspaces in Multi-Cloud", we utilized Rancher to manage Kubernetes clusters across multiple clouds AWS, GCP, Azure, and OTC. 

 

By using dataspaces in a multi-cloud environment, where Kubernetes clusters can be running in any restricted or unrestricted setup, securing communication between the Rancher control plane and Rancher agents is crucial. This can be achieved by allowing managed cluster IPs in a Web Application Firewall (WAF).  

To ensure connectivity between the newly created Kubernetes clusters managed by Rancher and the Rancher Load Balancer, we added the IP addresses of the NAT gateway to the Security Group associated with the Rancher Load Balancer as shown in the diagram below:Multi CloudHowever, the Security Groups have a limitation of 1,000 IP addresses per Elastic Network Interface (ENI).

The better option is to use the Web Application Firewall (WAF), which can utilize 100 IP sets, each IP set handling 10,000 IP addresses, allowing for a total of 1 million IP addresses.

Automation WAF with Crossplane, Lambda, and DynamoDB:

The main challenge we face during cluster provisioning is bootstrapping all the default Kubernetes applications and external cloud services. Since we are provisioning the clusters dynamically and rarely manually, we decided to use crossplane as a single control plane for both Kubernetes and cloud resourcesScreenshot 2024 06 19 at 21.46.22Crossplane is an open-source Kubernetes tool for managing cloud infrastructure resources as code directly from Kubernetes. It supports many cloud providers but for us currently relevant are the AWS services. For example, we use TableItem to add or remove items from DynamoDB.

You can find more information on crossplane in their official documentation but here's an example of how we can define a dynamodb configuration as CRD in our cluster:

apiVersion: dynamodb.aws.upbound.io/v1beta1 kind: TableItem metadata: name: cluster1 spec: providerConfigRef: name: aws-provider forProvider: region: eu-central-1 hashKey: id item: | { "id": {"S": "cluster1"}, "ips": {"S": "[\"54.123.456.78/32\"]"} } rangeKey: ips tableName: cluster-bootstrapping

Web Application Firewall (WAF)  It includes a Rancher Access Control List (ACL) that, by default, blocks all requests. Additionally, it maintains an allowed IP set containing all IPs originating from the Crossplane bootstrapping service. Furthermore, it requires association with AWS resources such as the Application Load Balancer ARN to function effectively.

DynamoDB: acts as a source for allowlist of cluster IPs that we manage with crossplane.

DynamoDB Streams enabled and the stream view is set to "NEW_AND_OLD_IMAGES".

Lambda function: is invoked to add or remove the IPs from the WAF set. DynamoDB Streams triggers Lambda in response to these changes.

Example of an event that triggers a Lambda function invoked by DynamoDB Streams when eventName is INSERT or REMOVE:

{'Records': [{'eventID': '34750be9350340a66221711500a1cc97', 'eventName': 'INSERT', 'eventVersion': '1.1', 'eventSource': 'aws:dynamodb', 'awsRegion': 'eu-central-1', 'dynamodb': {'ApproximateCreationDateTime': 1718874187.0, 'Keys': {'id': {'S': 'cluster1'}, 'ips': {'S': '["54.123.456.78/32"]'}}, 'NewImage': {'id': {'S': 'cluster1'}, 'ips': {'S': '["54.123.456.78/32"]'}}, 'SequenceNumber': '31142800000000062127994195', 'SizeBytes': 76, 'StreamViewType': 'NEW_AND_OLD_IMAGES'}, 'eventSourceARN': 'arn:aws:dynamodb:eu-central-1:ACCOUNT_NUMBER:table/cluster-bootstrapping/stream/2024-06-13T17:43:35.560'}] }

The Python Lambda function is invoked by events, fetches all IPs from the existing IP set, and checks the event name. If the event name is 'INSERT', it verifies if the IP in the event is already in the existing list and does nothing if it is; otherwise, it adds the IP. Similarly, if the event name is 'REMOVE', it checks if the IP is in the existing list and removes it if it is; otherwise, it does nothing.

import boto3 import json region_name = “eu-central-1” # Define the scope and the IPSetId scope = “REGIONAL” name = ip_set_name # Name of IPSet, that created in WAF ip_set_id = ip_set_id # ID of the IPSet, that created in WAF # Create a WAFv2 client client = boto3.client('wafv2', region_name=region_name) def get_ip_set(): """Fetch the current IP set and its lock token.""" try: response = client.get_ip_set(Name=name, Scope=scope, Id=ip_set_id) ip_set = response['IPSet'] lock_token = response['LockToken'] return ip_set, lock_token except Exception as e: raise def update_ip_set(add_ips=None, remove_ips=None): """ Update the IP set by adding new IPs or removing existing IPs. Parameters: - add_ips: List of IPs to add (e.g., ['10.0.0.1/32']) - remove_ips: List of IPs to remove (e.g., ['10.0.0.2/32']) """ try: # Get current IP set and lock token ip_set, lock_token = get_ip_set() current_ips = set(ip_set['Addresses']) # Prepare sets for additions and removals ips_to_add = set() ips_to_remove = set() if add_ips: add_ips = set(add_ips) ips_to_add = add_ips - current_ips if remove_ips: remove_ips = set(remove_ips) ips_to_remove = remove_ips & current_ips # If there are no changes to be made, return early if not ips_to_add and not ips_to_remove: return # Update current IPs updated_ips = current_ips | ips_to_add updated_ips -= ips_to_remove # Convert to list for update addresses = list(updated_ips) # Update the IP set response = client.update_ip_set(Name=name, Scope=scope, Id=ip_set_id, Addresses=addresses, LockToken=lock_token) return response except Exception as e: raise def lambda_handler(event, context): try: for record in event['Records']: event_name = record['eventName'] if event_name == 'INSERT': ip_address = record['dynamodb']['Keys']['ips']['S'] return update_ip_set(add_ips=json.loads(ip_address)) if event_name == 'REMOVE': ip_address = record['dynamodb']['Keys']['ips']['S'] return update_ip_set(remove_ips=json.loads(ip_address)) except Exception as e: raise

This is just an example how we can use crossplane for infrastructure management replacing many IaC solutions.

Read more