CASE #004 completed Data End-to-end analytics platform on AWS

NusaCommerce Analytics Platform

A five-layer data platform: CSV batches land in S3 and user events stream through Kinesis; Glue, EMR, Step Functions, and EventBridge transform and orchestrate; Redshift, DynamoDB, and S3 store history, real-time metrics, and the lake; Lambda behind API Gateway serves five endpoints protected by WAF and API keys; a dashboard on Amplify shows GMV, sales, funnel, and ML segments.

Role: Cloud data engineer (module exercise). CloudFormation, Glue, Athena, Redshift, Step Functions, EventBridge, API Gateway, WAF, EMR steps, monitoring. ETL scripts, PySpark jobs, Lambda handlers, the state-machine template, and the dashboard were provided by the module.
Context: LKS Nasional 2026 Cloud Computing, Cloud AI / Data Analytics module, on AWS Academy Learner Lab. NusaCommerce is the fictional marketplace in the brief.
Period: 2026-07 (Private practice repository created 2026-07-23.)
Evidence: ArchitectureDocumentationTechnical breakdownSource code

The Case

NusaCommerce, the fictional marketplace in this competition module, has transaction, event, shipment, and seller data but no infrastructure that turns it into answers. Hourly GMV, top sellers, the conversion funnel, and regional distribution are computed by hand over days.

The module asks for a complete platform in five hours: automated ingestion, ETL, ML feature engineering, lakehouse storage, and a REST API with a monitoring dashboard, all under the nusa- prefix in us-east-1. The application code was supplied; the infrastructure, the orchestration, and every integration between the pieces were the work.

Evidence

End-to-end architecture figure from the module

architecture

CloudFormation templates in the private practice repository

code

LKS Nasional 2026 Cloud AI / Data Analytics module (14 pages)

document

Architecture

implemented planned external / provided Hover or focus a component to trace its connections; select it for details.

Text version of the diagram

Ingestion

S3 raw bucket Amazon S3
Versioned landing zone with EventBridge notifications; CSV datasets and scripts.
Kinesis stream Amazon Kinesis Data Streams
nusa-events-stream, 2 shards, 24 h retention, SSE.
Firehose delivery Amazon Kinesis Data Firehose
5 MB / 60 s buffer, dynamic year/month/day prefix, Parquet conversion.

Processing

EventBridge rule Amazon EventBridge
Object Created on transactions/, shipments/, sellers/ with input transformer.
Step Functions orchestrator AWS Step Functions
Nine states; validate, parallel ETL, crawler, parallel post-processing, ML load, notify; catch to NotifyFailure.
Glue catalog, crawlers, ETL AWS Glue
Three jobs (Glue 4.0, G.1X, 2 workers) writing partitioned Parquet. Scripts provided.
EMR cluster Amazon EMR
EMR 7.x Spark, 1 master + 2 core m5.xlarge; seller scoring and RFM segmentation steps. PySpark jobs provided.
Stream processor AWS Lambda
Batches of 100 Kinesis records aggregated hourly into DynamoDB. Handler provided.

Storage

S3 processed and curated Amazon S3
Parquet ETL output; aggregates, reports, Athena results, ML features.
DynamoDB realtime metrics Amazon DynamoDB
METRIC#date#hour keys, 48 h TTL, MetricTypeIndex GSI, streams.
Redshift warehouse Amazon Redshift
staging, analytics, reporting schemas; mv_daily_summary; ML result tables.
Athena workgroup Amazon Athena
1 GB scan cap; daily GMV, top sellers, category, province heatmap, funnel queries.

Serving

API handler and loaders AWS Lambda
api-handler routes by path; redshift-loader, view-refresher, validate-input, ml-loader. Handlers provided.
REST API nusa-analytics-api Amazon API Gateway
/health, /metrics/realtime, /analytics/sales, /analytics/funnel, /analytics/recommendations; API key and usage plan.

Security

WAF Web ACL AWS WAF
Rate limit 100 per 5 min (custom 429), SQLi managed rules (block), common rules (count).
Secrets Manager AWS Secrets Manager
nusa/redshift/credentials with a LabRole-only resource policy.
SQS dead-letter queues Amazon SQS
For api-handler and stream-processor; alarms on visible messages.

Monitoring

SNS nusa-alerts Amazon SNS
Pipeline success and failure notifications; DLQ alarm actions.
CloudWatch dashboard Amazon CloudWatch
Lambda, Kinesis, Step Functions, DynamoDB, and Glue widgets.

Frontend

Amplify dashboard AWS Amplify
Static analytics dashboard configured with the API base URL and key. Page provided.

Connections

Kinesis stream → Firehose delivery
Firehose delivery → S3 raw bucket · user_events/
Kinesis stream → Stream processor · batch 100
Stream processor → DynamoDB realtime metrics · atomic ADD
S3 raw bucket → EventBridge rule · Object Created
EventBridge rule → Step Functions orchestrator · start execution
Step Functions orchestrator → Glue catalog, crawlers, ETL · parallel ETL
Glue catalog, crawlers, ETL → S3 processed and curated · Parquet
Step Functions orchestrator → EMR cluster · steps
EMR cluster → S3 processed and curated · ml-features/
Step Functions orchestrator → API handler and loaders · load and refresh
API handler and loaders → Redshift warehouse · COPY / REFRESH
S3 processed and curated → Athena workgroup · catalog
Step Functions orchestrator → SNS nusa-alerts · notify
REST API nusa-analytics-api → API handler and loaders
API handler and loaders → DynamoDB realtime metrics
API handler and loaders → Redshift warehouse
API handler and loaders → Athena workgroup
API handler and loaders → Secrets Manager
API handler and loaders → SQS dead-letter queues · on failure
WAF Web ACL → REST API nusa-analytics-api · prod stage
Amplify dashboard → REST API nusa-analytics-api · API key (HTTPS)
SQS dead-letter queues → SNS nusa-alerts · alarm
Step Functions orchestrator → CloudWatch dashboard · metrics

Technical Breakdown

infrastructure

nusa-foundation-stack defines the raw, processed, and curated buckets (SSE-S3, Block Public Access, lifecycle to Glacier after 30 days and expiry after 120), the nusa-realtime-metrics DynamoDB table (PAY_PER_REQUEST, composite key, 48-hour TTL, MetricTypeIndex, streams), the nusa-events-stream Kinesis stream, and the nusa-firehose-delivery stream. An EMR 7.x cluster with one master and two core m5.xlarge nodes runs the Spark steps.

application

Six Lambda functions on Python 3.12: nusa-stream-processor (Kinesis batches to DynamoDB with atomic ADD), nusa-api-handler (routes /metrics/realtime to DynamoDB, /analytics/sales to Redshift, /analytics/funnel to an Athena named query, /analytics/recommendations to the analytics schema), and the pipeline utilities nusa-redshift-loader, nusa-view-refresher, nusa-validate-input, and nusa-ml-loader. The handlers were provided; deployment, environment, event source mappings, DLQs, and the runtime switch to Secrets Manager were configured here.

data

Glue database nusa-database with tables for transactions, user events, shipments, and sellers; crawler nusa-crawler-raw over the processed prefixes and nusa-crawler-curated on demand. Three Glue ETL jobs validate rows (null keys, non-positive amounts), deduplicate, standardise strings, and write year/month/day-partitioned Parquet. Athena workgroup nusa-workgroup holds five named queries. Redshift nusa-warehouse has staging, analytics, and reporting schemas; COPY loads Parquet from S3 and reporting.mv_daily_summary joins transactions and shipments per province.

Seller scoring weights four dimensions (90-day GMV 40%, completion rate 30%, average rating 20%, month-over-month growth 10%) after min-max scaling. User segmentation builds RFM features (days since last purchase, completed transactions in 90 days, spend in 90 days), standardises them, splits into quartiles, and labels CHAMPION, LOYAL, AT_RISK, and DORMANT.

security

API keys with usage plan nusa-usage-plan; WAF Web ACL nusa-api-waf with NusaRateLimitRule (block above 100 requests per five minutes with a JSON 429 body), NusaSQLInjectionRule (managed SQLi set, block), and NusaCommonRule (managed common set, count). Redshift credentials live in nusa/redshift/credentials with a resource policy limited to LabRole and are fetched at runtime by the four Redshift-facing functions. Dead-letter queues for the API handler and the stream processor receive events after three failed attempts.

observability

CloudWatch dashboard nusa-pipeline-dashboard with Lambda invocations and errors, Kinesis incoming records, Step Functions executions succeeded and failed, DynamoDB consumed capacity, and Glue job elapsed time. Alarms on each DLQ notify nusa-alerts when a message stays visible for two consecutive five-minute periods.

deployment

EventBridge rule nusa-s3-trigger matches Object Created events on the raw bucket's transactions, shipments, and sellers prefixes and starts nusa-pipeline-orchestrator with a transformed input. The state machine runs the three Glue jobs in parallel, the crawler, then two parallel branches (Redshift load and view refresh; seller scoring then user segmentation on EMR), loads the ML features, and publishes success or failure to SNS. The dashboard is a static page deployed to Amplify by manual deployment and configured with the API base URL and key.

Key Findings

Engineering decisions

Foundation in CloudFormation: three S3 buckets (versioned raw bucket with EventBridge notifications, SSE-S3, Block Public Access, 30-day Glacier and 120-day expiry lifecycle), a DynamoDB metrics table with composite key, 48-hour TTL, GSI and streams, a two-shard Kinesis stream, and a Firehose delivery stream with dynamic date partitioning and Parquet conversion.
Batch path: Glue database, four catalog tables, two crawlers, three Glue 4.0 ETL jobs writing partitioned Parquet, an Athena workgroup with a 1 GB scan cap and five named queries, and Redshift staging, analytics, and reporting schemas with a materialized daily summary.
Streaming path: Kinesis records consumed in batches of 100 by a Lambda that aggregates hourly GMV, order count, and active users into DynamoDB with atomic ADD operations.
Orchestration: a nine-state Step Functions machine (validate, parallel ETL, crawler, parallel Redshift load and EMR steps, ML load, notify) triggered by an EventBridge rule with an input transformer on S3 object creation. The provided definition contained five intentional bugs to find and fix.
Serving and security: five REST endpoints behind API keys and a usage plan (100 RPS, burst 200, 1,000 requests per day), a WAF Web ACL with a rate-based rule returning a custom 429, the SQLi managed rule group, dead-letter queues with CloudWatch alarms, and Redshift credentials in Secrets Manager with a resource policy.
ML features on EMR 7.x: a seller composite score (GMV 40%, completion rate 30%, rating 20%, growth 10%, min-max scaled, tiers TOP, GROWING, STABLE, AT_RISK) and RFM user segmentation (z-score, NTILE quartiles, labels CHAMPION, LOYAL, AT_RISK, DORMANT), loaded into Redshift.

Infrastructure as a stack, updated in place. Buckets, the DynamoDB table, the Kinesis stream, and the Firehose delivery stream were added to one CloudFormation template across successive stack updates rather than clicked together.
Two paths, one lake. Batch CSVs land in S3 and become partitioned Parquet through Glue; streamed events go through Kinesis into DynamoDB for the last 48 hours and into S3 through Firehose for history. Redshift Spectrum and Athena read the same lake.
Orchestration is where the bugs were. The Step Functions definition arrived with five intentional faults: wrong transitions, wrong resource names, missing S3 path segments, and incomplete error handling. Some failed at creation, some only when a state ran.
Protect the API in layers. API keys and a usage plan first, then a WAF Web ACL with a rate-based block, SQL-injection managed rules, and the common rule set in count mode so that legitimate traffic is logged rather than blocked.
ML as features, not claims. Seller scores and RFM segments are computed by PySpark on EMR, written as Parquet, and loaded into Redshift. No accuracy figure is reported because none was measured.

Challenges and Solutions

Challenges

The whole foundation had to be expressed as CloudFormation and updated in place as resources were added.
The Firehose Parquet conversion references a Glue table that does not exist until later, so the template cannot enable it up front.
The provided state-machine definition contained five bugs, some failing validation and some only at runtime.
Public API endpoints needed protection without breaking the dashboard.

Solutions

Incremental stack updates on one template with PascalCase logical IDs and nusa- physical names.
Format conversion enabled manually after the Glue catalog table was defined.
Fixes located through the Step Functions console errors and execution history, then a manual test execution to SUCCEEDED.
API keys with a usage plan as the baseline, WAF layered on the prod stage with rate limiting blocking and common rules in count mode.

Result

The foundation and core stacks exist as CloudFormation templates in the practice repository, and the module's verification steps (row counts in Athena and Redshift, a Step Functions execution to SUCCEEDED, a Kinesis test event loop, the five endpoints with and without an API key, a WAF rate-limit test) define what "done" meant. Logs from the lab session were not retained, so results are documented by the module rather than by kept evidence.

Outcomes on record

Three CloudFormation templates authored for the foundation and core stacks (S3, Kinesis, DynamoDB, SQS, WAF, Secrets Manager).
The module defines verification queries and endpoint checks for every phase; execution evidence was not kept. Not documented

Stack

Languages

Data and analytics

Messaging and streaming

REST API

Repository, Demo, Documentation

Source code PrivatePractice repository with the CloudFormation templates. The ETL, Lambda, EMR, Step Functions, and dashboard code were provided by the module.

Live demo UnavailableThe Learner Lab environment is torn down after the session.

Documentation PrivateCompetition module; not redistributed.