Architected and implemented a next-generation AWS data platform for real-time analytics and customer segmentation. Built scalable reusable ETL/ELT pipelines using AWS Glue and PySpark for ingestion and transformation of real-time transaction and clickstream data. Prototyped Apache Iceberg on S3 for schema evolution and performance improvements. Used serverless orchestration with AWS Lambda and Step Functions. Developed curated datasets in Snowflake and Redshift using dbt for self-service analytics and BI dashboards. Implemented governance with AWS IAM, S3 bucket policies, and AWS Glue Data Catalog. Automated infrastructure provisioning and schema migrations using Terraform and Jenkins; optimized lake partitioning and ensured data quality using Great Expectations and unit tests.

Cloud Data Engineer at MiQ

January 1, 2021 - April 1, 2023

Built an Azure unified data lake for high-volume ad impression, campaign, and bid data to support near real-time reporting and machine learning. Developed ingestion and transformation pipelines with Azure Data Factory, PySpark, and Azure Databricks. Implemented Delta Lake storage on ADLS Gen2. Integrated streaming and third-party sources via Kafka and Azure Event Hubs; migrated legacy batch to real-time using Databricks Structured Streaming. Added data lineage using Azure Purview and built REST API-based ingestion into Event Hubs. Created value-added datasets for marketing KPIs delivered to Power BI. Managed version-controlled DDL/DML in Git with Azure DevOps pipelines and deployed via GitHub Actions and ARM templates. Implemented monitoring with Azure Monitor and Log Analytics.

Data Engineer at Elanco

September 1, 2018 - December 1, 2020

Developed a centralized platform/hub for aggregating veterinary, livestock, and sales data to support marketing and regulatory compliance. Designed ETL pipelines using Apache Spark, Python, and Hive across on-prem and hybrid cloud environments. Implemented data cleansing, mapping, and transformation rules to standardize veterinary/medical records and improve data quality. Built ingestion from CSV, Excel, Oracle DB, and APIs into HDFS and Parquet. Collaborated with stakeholders for requirements and estimation; performed data quality validation throughout pipelines. Supported data science by preparing feature-rich datasets for model training. Conducted metadata and lineage tracking using Apache Atlas.