I am Manasa Reddy, a Senior Data Engineer with about 12 years of hands-on experience designing, developing, testing, and implementing Big Data, Spark, and Hadoop projects across Azure Data Bricks and AWS-based lakehouse deployments. I specialize in building scalable data engineering pipelines in Python and Scala using Spark, Delta Lake, and Structured Streaming to support both batch and real-time data workflows. I thrive on creating production-ready data processing applications and maintaining robust, governed data platforms that empower analytics and business decision-making. I have extensive experience with cloud data services (Azure and AWS), distributed storage and compute architectures, and end-to-end CI/CD for data pipelines. My focus areas include data security, performance optimization, data governance, and observability, with a strong track record of collaborating across data engineering, data science, and business teams to deliver reliable, scalable solutions.

Manasa Reddy

I am Manasa Reddy, a Senior Data Engineer with about 12 years of hands-on experience designing, developing, testing, and implementing Big Data, Spark, and Hadoop projects across Azure Data Bricks and AWS-based lakehouse deployments. I specialize in building scalable data engineering pipelines in Python and Scala using Spark, Delta Lake, and Structured Streaming to support both batch and real-time data workflows. I thrive on creating production-ready data processing applications and maintaining robust, governed data platforms that empower analytics and business decision-making. I have extensive experience with cloud data services (Azure and AWS), distributed storage and compute architectures, and end-to-end CI/CD for data pipelines. My focus areas include data security, performance optimization, data governance, and observability, with a strong track record of collaborating across data engineering, data science, and business teams to deliver reliable, scalable solutions.

Available to hire

I am Manasa Reddy, a Senior Data Engineer with about 12 years of hands-on experience designing, developing, testing, and implementing Big Data, Spark, and Hadoop projects across Azure Data Bricks and AWS-based lakehouse deployments. I specialize in building scalable data engineering pipelines in Python and Scala using Spark, Delta Lake, and Structured Streaming to support both batch and real-time data workflows. I thrive on creating production-ready data processing applications and maintaining robust, governed data platforms that empower analytics and business decision-making.

I have extensive experience with cloud data services (Azure and AWS), distributed storage and compute architectures, and end-to-end CI/CD for data pipelines. My focus areas include data security, performance optimization, data governance, and observability, with a strong track record of collaborating across data engineering, data science, and business teams to deliver reliable, scalable solutions.

See more

Experience Level

Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
See more

Language

English
Fluent
Hindi
Advanced
Telugu
Fluent

Work Experience

Senior Data Engineer at HEB
April 1, 2024 - November 6, 2025
Designed and deployed enterprise-grade Azure Data Lake using ADLS Gen 2 to support scalable storage, processing, and analytics for structured and semi-structured data. Built end-to-end ETL pipelines with Azure Data Factory, Databricks, and Spark jobs implementing modular transformations aligned with the Medalion architecture; implemented partitioning, caching, and broadcasting for performance; integrated Unity Catalog for data governance; supported real-time streaming analytics; ensured HIPAA data handling and de-identification; established CI/CD practices and security controls.
Cloud Data Engineer at PNC Bank
March 1, 2024 - March 1, 2024
Designed and implemented an Enterprise Data Lake on ADLS Gen 2 to support storage, processing, and analytics for large and dynamic datasets. Built robust ETL pipelines using Azure Data Factory, Databricks, and Spark to ingest, transform, and govern data across diverse sources; enabled batch and real-time data movement into Azure Synapse Analytics; implemented data governance with Purview and security controls; automated CI/CD for pipelines.
Data Engineer/ Data Analyst at American Express
August 1, 2020 - August 1, 2020
Built real-time processing pipelines with Spark Streaming and Kafka; migrated legacy ETL to Azure Databricks; developed Spark SQL-based transformations for optimized analytics; implemented data governance and security; built dashboards in Power BI; enabled scalable, end-to-end data processing.
Big Data Engineer at State of New York
April 1, 2018 - April 1, 2018
Developed and maintained multi-cluster Hadoop/Spark pipelines; built Spark jobs for structured and semi-structured data; implemented HIPAA-compliant data handling for cross-agency analytics; migrated legacy queries to Spark SQL and optimized data models for governance and performance.
Hadoop Developer at Deloitte
July 1, 2013 - July 1, 2013
Developed data processing solutions using Hadoop ecosystem (MapReduce, Hive, Pig) for distributed storage and computation; designed and maintained Hive tables; built Oozie workflows for multi-stage data processing; automated job execution with shell scripts; implemented data quality checks and provided business insights through dashboards.
Data Engineer at American Express (Amex)
August 1, 2020 - August 1, 2020
Developed end-to-end data pipelines using ADF, Azure Databricks, and Spark to ingest, transform, and store data in Azure Synapse Analytics; implemented real-time streaming pipelines using Azure Event Hubs and Azure Stream Analytics; built nested JSON processing and Parquet/Avro with Delta Lake to achieve ACID compliance; built and optimized Spark SQL queries; migrated legacy batch pipelines from Teradata/Informatica to Databricks; designed and implemented data governance with Azure Purview; automated infrastructure provisioning with Terraform; built Power BI dashboards for business insight; mentored junior engineers.

Education

Master of Science in Computer Science at Wright State University
August 1, 2013 - June 1, 2015
Masters in Computer Science at Wright State University
August 1, 2013 - June 1, 2015

Qualifications

Masters in Computer Science
August 1, 2013 - June 1, 2015

Industry Experience

Software & Internet, Financial Services, Professional Services, Healthcare, Retail, Government