Poonam Kanwar

Contributed to RocheHealth ETL project, a global medical data platform analyzing product and material availability for the German market. Designed and implemented PySpark-based ETL pipelines on AWS, reducing manual processing time by 29%. Optimized data transformations using AWS Glue and Athena, improving query performance by 30%. Integrated DynamoDB and AWS Lambda for real-time metadata storage and on-demand transformations. Improved Spark job performance by 40% using efficient partitioning and caching strategies. Also worked on a Data Comparison Engine to ensure data consistency, increasing data accuracy by 98% and minimizing downtime with AWS Lambda-based recovery jobs.

Designed and implemented PySpark-based ETL pipelines on AWS, reducing manual processing time by 29%. Optimized data transformations using AWS Glue and Athena, improving query performance by 30%. Integrated DynamoDB for metadata storage and used AWS Lambda for on-demand data transformations ensuring real-time updates. Enhanced Spark job performance by 40% through efficient partitioning and caching strategies. Also developed an automated data validation framework in Python, increasing data accuracy by 98%, and implemented AWS Lambda-based data recovery jobs to minimize downtime. Technologies included PySpark, AWS Glue, Lambda, DynamoDB, Athena, Spark SQL, and SQL.

Led the design and development of scalable big data pipelines in AWS, significantly reducing data processing time by 40%. Automated ETL transformations using AWS Glue, cutting manual workloads by 35%. Developed event-driven workflows with AWS EventBridge and Step Functions to decrease operational costs by 25%. Integrated DynamoDB for metadata storage and AWS Lambda for real-time processing, improving system efficiency. Used AWS Athena for data analytics, reducing query execution times by 35%. Leveraged AWS EMR Serverless for high-volume data processing, enhancing system performance by 20%. Technologies used included AWS S3, Redshift, Glue, Lambda, DynamoDB, EventBridge, Step Functions, Athena, EMR Serverless, PySpark, Spark SQL, Databricks, and SQL.