CASE #005 completed Cloud Multi-region Kubernetes platform with disaster recovery

Multi-Tenant SaaS Infrastructure

A multi-tenant SaaS platform on Amazon EKS across two regions: a dual-VPC design in Virginia separating application and data, namespace-isolated tenants with restricted Pod Security Standards and NetworkPolicy, an event-driven audit pipeline, Grafana on ECS Fargate, blue/green canary releases through CodeDeploy, and an Oregon standby with Aurora Global Database and S3 cross-region replication.

Role: Cloud infrastructure engineer (module exercise). Fixing and completing Terraform modules, Kubernetes manifests, Helm values, Dockerfiles, application code, and Lambda functions that the module ships with intentional bugs; deploying with CLI tooling only.
Context: LKS Nasional 2026 Cloud Computing, Infrastructure as a Service module. Resources use the cloudtech- prefix. Console-created resources lose points, so everything goes through Terraform, Helm, kubectl, and the AWS CLI.
Period: 2026 → ongoing (Module dated LKS Nasional 2026; working dates not documented.)
Evidence: ArchitectureDocumentationTechnical breakdown

The Case

A SaaS platform has to serve several tenants from one Kubernetes cluster while keeping their workloads apart, keep its data tier away from its compute tier, survive a regional failure, and ship new versions without downtime. This module hands over a repository where every layer, from Terraform modules to Helm values to Lambda code, contains deliberate faults or missing pieces, and asks for a working platform built only with aws, terraform, helm, and kubectl. Anything created by hand in the console loses points.

Evidence

Architecture and network diagrams from the module

architecture

LKS Nasional 2026 Infrastructure as a Service module (14 pages)

document

Architecture

implemented planned external / provided Hover or focus a component to trace its connections; select it for details.

Text version of the diagram

Application VPC 10.10.0.0/16 (us-east-1)

ALB cloudtech-alb Application Load Balancer
Public subnets; /api/* to port 8080, / to port 3000.
EKS cluster Amazon EKS
EKS 1.31 in private subnets; namespaces tenant-alpha and tenant-beta; PSS restricted, ResourceQuota, NetworkPolicy, HPA.
cloudtech-api (Flask) Flask
2 replicas per tenant; port 8080 plus Prometheus metrics on 9100; publishes events to Kinesis.
cloudtech-fe (Flask) Flask
1 replica per tenant on port 3000; checks Grafana through the ECS API.
Grafana OSS on ECS Fargate Amazon ECS
Internal service in private subnets, reachable from 10.10.0.0/16 on port 3000.
NAT Gateway NAT Gateway

Data VPC 10.20.0.0/16 (us-east-1)

VPC Peering VPC Peering
cloudtech-peering-app-data with DNS resolution and routes in both directions.
Aurora PostgreSQL Multi-AZ Amazon Aurora
Isolated subnets; port 5432 from 10.10.0.0/16 only; publicly_accessible false.
ElastiCache Redis Amazon ElastiCache
Port 6379 from 10.10.0.0/16 only.

Event-driven pipeline

Kinesis cloudtech-event-stream Amazon Kinesis Data Streams
2 shards, 24 h retention.
Lambda cloudtech-event-processor AWS Lambda
Python 3.13; configuration from environment variables.
EventBridge cloudtech-saas-events Amazon EventBridge
Custom bus, source cloudtech.api.
SNS cloudtech-user-events Amazon SNS
SQS cloudtech-event-queue Amazon SQS
DLQ cloudtech-dlq, maxReceiveCount 3.
DynamoDB cloudtech-audit-log Amazon DynamoDB
PAY_PER_REQUEST with TTL expiresAt.

Delivery and monitoring

ECR (api, fe, monitoring) Amazon ECR
CodeDeploy blue/green AWS CodeDeploy
Canary 10% for 5 minutes then 100%; automatic rollback on ALB health failure.
CloudWatch alarms Amazon CloudWatch
cloudtech-api-error-rate (5xx > 10) and cloudtech-primary-health-alarm.
S3 cloudtech-assets-primary Amazon S3
Versioned primary assets bucket with Block Public Access.

DR region us-west-2 (10.30.0.0/16)

Transit Gateway peering AWS Transit Gateway
cloudtech-tgw-2026 (us-east-1) peered with cloudtech-tgw-secondary (us-west-2).
Aurora Global Database replica Aurora Global Database
S3 cross-region replication S3 Cross-Region Replication
cloudtech-assets-primary (versioned) to cloudtech-assets-dr.
DR failover Lambda AWS Lambda
Triggered by EventBridge from the health alarm; initiates Aurora failover and notifies cloudtech-dr-alerts.

Connections

ALB cloudtech-alb → cloudtech-api (Flask) · /api/*
ALB cloudtech-alb → cloudtech-fe (Flask) · /
EKS cluster → NAT Gateway · egress
cloudtech-api (Flask) → VPC Peering · 5432 / 6379
VPC Peering → Aurora PostgreSQL Multi-AZ
VPC Peering → ElastiCache Redis
cloudtech-fe (Flask) → Grafana OSS on ECS Fargate · status check
cloudtech-api (Flask) → Kinesis cloudtech-event-stream · user CRUD events
Kinesis cloudtech-event-stream → Lambda cloudtech-event-processor
Lambda cloudtech-event-processor → EventBridge cloudtech-saas-events
EventBridge cloudtech-saas-events → SNS cloudtech-user-events
EventBridge cloudtech-saas-events → SQS cloudtech-event-queue
EventBridge cloudtech-saas-events → DynamoDB cloudtech-audit-log
ECR (api, fe, monitoring) → EKS cluster · images
CodeDeploy blue/green → ALB cloudtech-alb · traffic shift
cloudtech-api (Flask) → CloudWatch alarms · 5xx metrics
CloudWatch alarms → DR failover Lambda · alarm via EventBridge
DR failover Lambda → Aurora Global Database replica · promote
Aurora PostgreSQL Multi-AZ → Aurora Global Database replica · global replication
S3 cloudtech-assets-primary → S3 cross-region replication · cross-region replication
Transit Gateway peering ↔ VPC Peering · cross-region routes

Technical Breakdown

infrastructure

EKS 1.31 cluster cloudtech-eks-cluster created by Terraform with two or more nodes, the AWS Load Balancer Controller so the ALB can target pods, three ECR repositories (cloudtech-api-app on 8080 with Prometheus metrics on 9100, cloudtech-fe-app on 3000, cloudtech-monitoring reserved for DR), and Grafana OSS as an ECS Fargate service (512 CPU, 1024 MB) with no public IP and its image pulled through the NAT Gateway.

application

Two Flask 3.1 services on Python 3.13 deployed with Helm into both tenant namespaces: cloudtech-api (2 replicas, 256m CPU, 512Mi) and cloudtech-fe (1 replica, 128m CPU, 256Mi). Database credentials come from the Kubernetes Secret cloudtech-db-secret through secretKeyRef; the ConfigMap cloudtech-config carries the Redis host and Grafana URL; the API has CORS enabled and publishes to the KINESIS_STREAM named in its environment.

security

Kubernetes Secrets rather than plaintext values, restricted Pod Security Standards, no security group open to 0.0.0.0/0 except the ALB on 80 and 443, Block Public Access on every S3 bucket, and publicly_accessible = false on Aurora. The module closes with a security audit against that checklist.

networking

Application VPC 10.10.0.0/16 with public subnets (10.10.1.0/24, 10.10.2.0/24) for the ALB and private subnets (10.10.10.0/24, 10.10.11.0/24) for EKS nodes and Grafana. Data VPC 10.20.0.0/16 with isolated subnets for Aurora and Redis plus private subnets. DR VPC 10.30.0.0/16 in us-west-2. Peering cloudtech-peering-app-data with DNS resolution; Transit Gateways cloudtech-tgw-2026 and cloudtech-tgw-secondary peered across regions with routes in both directions. Security groups: ALB open on 80 and 443, EKS pods reachable only from the ALB group on 8080 and 3000, Aurora on 5432 and Redis on 6379 only from 10.10.0.0/16.

observability

Grafana OSS on ECS Fargate as an internal service, CloudWatch alarm cloudtech-api-error-rate on 5xx responses, log group /cloudtech/ecs/grafana, and the frontend dashboard that reports API, Aurora, Redis, Kinesis, EKS, Grafana, Transit Gateway, and peering status.

deployment

CodeDeploy application cloudtech-eks-app with a canary strategy: 10% of traffic to the new version for five minutes, then 100% if no errors, with automatic rollback when the ALB health check fails. Deployments are triggered from the CLI.

Key Findings

Engineering decisions

Dual-VPC design in us-east-1: an application VPC (10.10.0.0/16) with EKS, an ALB, and Grafana on ECS Fargate, peered with a data VPC (10.20.0.0/16) holding Aurora PostgreSQL Multi-AZ and ElastiCache Redis, for compute-to-data isolation.
Cross-region connectivity through Transit Gateway peering to a DR VPC (10.30.0.0/16) in us-west-2 with an Aurora Global Database replica and S3 cross-region replication.
Tenant isolation with Kubernetes namespaces tenant-alpha and tenant-beta, restricted Pod Security Standards, ResourceQuota, NetworkPolicy (ingress from the same namespace and kube-system; egress to DNS, the data VPC, and HTTPS), and an HPA.
Event-driven audit trail: the Flask API publishes user CRUD events to Kinesis, a Lambda forwards them to a custom EventBridge bus, which fans out to SNS, SQS with a dead-letter queue, and a DynamoDB audit log with TTL.
Blue/green canary through CodeDeploy: 10% of traffic for five minutes, then 100%, with automatic rollback on ALB health-check failure.
Automated DR trigger: a CloudWatch alarm on primary health starts an EventBridge rule and a Lambda that initiates the Aurora Global Database failover and notifies through SNS.

Isolation is layered. Two VPCs separate compute from data; VPC peering with explicit routes is the only path between them. Inside the cluster, each tenant namespace carries a restricted Pod Security Standard, a ResourceQuota, and a NetworkPolicy that allows ingress only from its own namespace and kube-system and egress only to DNS, the data VPC, and HTTPS.
Fix the module, not the plan. All Terraform corrections live under modules/; main.tf stays untouched. State sits in a versioned S3 bucket so a bad apply can be rolled back.
Events are the audit log. Every user create, update, or delete is published to Kinesis by the API, forwarded by a Lambda to a custom EventBridge bus, and fanned out to SNS, an SQS queue with a dead-letter queue, and a DynamoDB table with TTL.
Disaster recovery is automated, not a runbook. A CloudWatch alarm on primary health drives an EventBridge rule and a Lambda that initiates the Aurora Global Database failover and notifies through SNS. The targets set by the module are an RTO of 15 minutes and an RPO of 1 minute.

Challenges and Solutions

Challenges

Terraform modules, Kubernetes manifests, Helm values, Dockerfiles, application code, and Lambda functions all contained intentional bugs, and some resources were missing entirely.
Pods in the application VPC must reach Aurora and Redis in another VPC while remaining isolated per tenant.
The ALB must route paths to two services in two tenant namespaces.
Database credentials must never appear in plaintext.

Solutions

Bugs fixed inside modules/ only (main.tf untouched), with a versioned S3 backend for Terraform state.
VPC peering routes in both directions with DNS resolution, security groups scoped to the application CIDR, NetworkPolicy egress to 10.20.0.0/16.
AWS Load Balancer Controller with target groups for /api/* (port 8080) and / (port 3000).
Kubernetes Secret cloudtech-db-secret consumed through secretKeyRef; ConfigMap for Redis host and Grafana URL.

Result

The module's end-to-end verification requires a CRUD lifecycle in both tenants (create with an event to Kinesis, read from Aurora across the peering, update with an audit record in DynamoDB, delete), every dashboard indicator green, and the security checklist satisfied. The RTO and RPO figures are the module's targets, not measured values. Evidence from the lab session was not retained.

Outcomes on record

The module's acceptance is CRUD in both tenants, all dashboard indicators green, cross-VPC connectivity, and a security checklist; execution evidence was not kept. Not documented

Targets from the specification

Recovery Time Objective of 15 minutes or less. Target from specification
Recovery Point Objective of 1 minute or less. Target from specification

Stack

Languages

Frameworks and libraries

Flask

Infrastructure and IaC

Terraform

Containers and orchestration

Databases and storage

Messaging and streaming

REST API

Observability

Repository, Demo, Documentation

Source code UnavailableThe working copy of the module repository was not published.

Live demo UnavailableThe Learner Lab environment is torn down after the session.

Documentation PrivateCompetition module; not redistributed.