Cloud & DevOps
Multi-cloud data & platform engineering — this portfolio runs on AWS (ECS Fargate + Terraform), backed by production Databricks Lakehouse and Azure Data Factory experience across AWS and Azure.
AWS Infrastructure Architecture
Production ReadyWhat this portfolio runs on
Multi-AZ deployment with ECS Fargate, RDS PostgreSQL, ElastiCache Redis, and comprehensive monitoring. All infrastructure is defined as code using Terraform with automated deployments via GitHub Actions.
Implementation Highlights
Infrastructure as Code
Complete infrastructure defined in Terraform with 9 modular components: VPC, ECS, RDS, ElastiCache, ALB, Route53, ECR, CloudWatch, and Security Groups.
Container Orchestration
Serverless containers on ECS Fargate with auto-scaling (1-4 tasks), deployment circuit breakers, and health checks. Frontend (Next.js) and Backend (FastAPI) services with independent scaling policies.
Database & Caching
RDS PostgreSQL with automated backups, encryption at rest, and enhanced monitoring. ElastiCache Redis for session management and application caching with automatic failover capabilities.
CI/CD Pipeline
Fully automated deployments with GitHub Actions. Terraform workflow for infrastructure, separate workflows for backend and frontend with testing, building, and zero-downtime deployments to ECS.
High Availability
Multi-AZ deployment across us-east-1a and us-east-1b with Application Load Balancer, NAT Gateways in each AZ, and automatic task replacement on failure.
Security & Compliance
Private subnets for all application resources, security groups with least-privilege access, SSL/TLS encryption, image scanning, and secrets management. No hardcoded credentials.
Monitoring & Observability
CloudWatch with Container Insights, custom dashboards, log aggregation with 30-day retention, CPU/memory alarms, and enhanced RDS monitoring at 60-second intervals.
Cost Optimization
Environment-specific sizing (dev: t4g.micro, prod: t4g.small), lifecycle policies for ECR images, minimal task counts with auto-scaling, and efficient resource utilization. Dev environment ~$115/month.
Databricks Lakehouse Platform
Databricks Certified — Data Engineer ProfessionalBeyond this site's AWS footprint, my day-to-day is building production data platforms on the Databricks Lakehouse. At MojoTech I designed agentic data ingestion on Databricks and integrated and automated pipelines and APIs for a Fortune 500 company — enabling AI agents across their work streams and driving 72% growth of their analytics platform. The patterns below are how I structure those Lakehouses.
Medallion Architecture & Delta Lake
Bronze → Silver → Gold layering on Delta Lake: raw landing, cleansed/conformed tables, and curated business marts. ACID transactions, schema enforcement and evolution, time travel, and OPTIMIZE/Z-ORDER for fast reads.
Streaming & Incremental Ingestion
Structured Streaming and Auto Loader for exactly-once, incremental ingestion from object storage and message streams (Kinesis / Event Hubs), with checkpointing and backfill — the same engine powers batch and near-real-time.
Governance with Unity Catalog
Centralized governance across workspaces: three-level namespaces, fine-grained access control, column/row-level security, data lineage, and a searchable catalog so teams can trust and discover data.
Orchestration & DataOps
Delta Live Tables for declarative, testable pipelines with data-quality expectations; Databricks Workflows for job orchestration and dependencies; and MLflow for experiment tracking and model registry alongside the data.
Azure — Multi-Cloud Delivery
Multi-CloudThe same Lakehouse patterns port to Azure. I've delivered data pipelines on Azure using Data Factory for orchestration, ADLS Gen2 as the Delta storage layer, and Azure Databricks for transformation — so the platform choice follows the client's cloud rather than the other way around.
Azure Data Factory Orchestration
Pipelines with linked services, datasets, and triggers; mapping data flows for code-free transformation; and Integration Runtimes to bridge on-prem and cloud sources. Parameterized, metadata-driven pipelines for reusable ingestion.
ADLS Gen2 + Delta
Azure Data Lake Storage Gen2 as the lake layer with hierarchical namespaces, holding the same Bronze/Silver/Gold Delta tables — a consistent storage contract regardless of cloud.
Azure Databricks & Synapse
Azure Databricks for Spark transformation and ML, with Synapse Analytics for serving warehouse workloads and Event Hubs for streaming ingestion — the Azure-native equivalents of the AWS stack.
Security & Governance
Managed Identities for credential-free access between services, Key Vault for secrets, and Microsoft Purview for cataloging and lineage — least-privilege access wired in from the start.
Multi-Cloud Service Mapping
The same Lakehouse building blocks, expressed in each cloud's native services — how I keep architectures portable across AWS and Azure.
| Capability | AWS | Azure |
|---|---|---|
| Lake storage | Amazon S3 | ADLS Gen2 |
| Lakehouse compute | Databricks on AWS | Azure Databricks |
| Streaming ingestion | Kinesis Data Streams | Event Hubs |
| Orchestration / ETL | Glue + Step Functions | Data Factory |
| Analytics warehouse | Redshift | Synapse Analytics |
| Secrets management | Secrets Manager | Key Vault |
| Serverless compute | ECS Fargate / Lambda | Container Apps / Functions |
Technical Documentation
Architecture Documentation
Comprehensive documentation including Mermaid diagrams, component descriptions, traffic flows, and scalability strategies.
View on GitHub →Deployment Guide
Step-by-step guide for deploying the infrastructure, including prerequisites, AWS setup, and GitHub Actions configuration.
View on GitHub →Terraform Source Code
Browse the complete Terraform infrastructure code with 9 modular components and environment-specific configurations.
View on GitHub →