Led the design and implementation of scalable batch and streaming data pipelines integrating multiple data sources like Elasticsearch, RDS, Redshift, and Apache Spark. Reduced pipeline latency by 30% through query tuning and Python profiling. Automated workflows using Airflow with integrated monitoring and alerting. Ensured 99.9% data accuracy and compliance with validation checks. Collaborated with data scientists and product teams to deliver high-performance, secure, and fault-tolerant AWS-based architectures. Developed Kibana and QuickSight dashboards for proactive pipeline health monitoring and applied DevOps CI/CD practices for automation.

Built large-scale ETL pipelines to migrate terabytes of data to AWS Redshift, RDS, and S3. Reduced batch processing runtimes by 40% with Spark and AWS Glue optimizations. Automated data reconciliation and validation frameworks ensuring 100% accuracy. Integrated migration workflows into a unified web UI. Ensured GDPR and HIPAA compliance during cloud migrations and delivered cost optimizations using AWS tools. Conducted performance benchmarking to validate SLA adherence.

Delivered analytics solutions by designing dashboards using Tableau and AWS QuickSight. Automated ETL pipelines for fresh and reliable data input using AWS Glue and SQL. Improved reporting efficiency through query optimizations on Redshift and Hive. Implemented validation checks to guarantee data consistency. Developed secure row-level security models in dashboards and automated report scheduling, saving over 20 manual hours weekly. Collaborated with stakeholders to align analytics with business goals.

Senior Data Engineer — Eli Lilly DDR at Eli Lilly

Developed batch and real-time data pipelines processing clinical trial sensor data (ECG, accelerometer, gyroscope). Designed and deployed real-time streaming using AWS Kinesis and Spark Streaming, reducing ingestion latency to under 2 seconds. Developed Python utilities for secure S3 access from on-premise. Created data validation utilities and Kibana dashboards for monitoring. Improved querying performance with data partitioning strategies on S3 and Redshift.

Developed GDPR-compliant Information Lifecycle Management frameworks automating archival and purging of datasets on AWS S3. Implemented disaster recovery mechanisms and centralized logging for compliance and auditability. Established validation checks reducing operational risks and improved metadata management for quick recovery. Optimized S3 storage costs via lifecycle policies and Glacier archival.

Built fault-tolerant data ingestion framework across global zones using AWS DynamoDB and Redshift. Improved ETL efficiency with Python Asyncio for parallel processing. Implemented automated retry and reload mechanisms for data failure handling. Created data lineage reports for transparency and reduced ETL runtime by 20% through query optimizations.

Data Engineer — DIH-RAR Morgan Stanley at Morgan Stanley

Developed Data Integration Hub consolidating wealth management data using Hadoop-based technologies like Sqoop, Hive, and Spark. Optimized Hive/Presto queries reducing execution times by 25%. Replaced hardcoded scripts with parameterized Unix, Python, and Informatica jobs. Integrated multiple banking sources into centralized data lake. Created audit and monitoring utilities ensuring ingestion consistency. Implemented Kerberos authentication and automated performance testing scripts.

AWS Data Engineer at Freecharge

Lead DataHub Integration to migrate data from diverse sources (Lambda, Elasticsearch, RDS, Redshift, flatfiles) ensuring end-to-end data encryption. Designed architectural data pipelines, implemented triggering, orchestration, monitoring, and scheduling of pipelines with Apache Airflow. Contributed to pipeline components and architecture decisions to support high-volume data flows.

AWS Architect at Yash

Johndeer- Manufacturing Migration: migrated data from mainframe DB2 to Snowflake; built an AWS-based data lake on S3 and loaded into Snowflake. Notification layer developed using Lambda and DynamoDB with Node.js. Designed the end-to-end architecture; performed Glue-based migrations; used Airflow for scheduling and established DAGs for the team. Assessed storage integration approach for Snowflake migration.

AWS Architect at Yash

Migration Accelerator platform enabling migration of big data applications to public clouds (AWS, Azure). Supported Data, Workload, and Meta-store migrations; major technologies included Glue, EMR, Lambda, Redshift, DynamoDB, RDS. Focus areas: Security/Governance and Orchestration Migration. Architecture design for each section and one-click operation via Web UI.

AWS Architect at Yash

Solution Accelerator to deliver end-to-end DataOps and MLOps capabilities. Architecture design for Data Platform, Data and ML Ops. ETL performed with Glue. Integrated SageMaker features (Autopilot, Notebook Exp, Experiments) and enabled one-click flow for dataset upload, transform, prediction, and visualization for layperson users.

Senior Data Engineer at Yash

Eli Lilly DDR: Designed a data pipeline to ingest and process clinical trial studies in batch and streaming modes. ETL with AWS Glue and Redshift; real-time using Kinesis. Created Kinesis stream to Kibana; small files processed with Lambda; outputs stored in DynamoDB with TTL. Ensured HIPAA & GDPR compliance. Built Python utility to access S3 data from on-premises and a Data Generator to push dummy data into Kinesis and upload to S3 from on-prem.

Senior Data Engineer at Synechron

Information Lifecycle Management: designed an architecture to make the process interactive, flexible, and secure. Established a failure handling framework and disaster recovery readiness. Technologies included AWS Glue, Lambda, and DynamoDB. Implemented a single point of logging to maintain object lineage.