Looks like you have JavaScript disabled. For the full Twine experience, you will need to re-enable it.

Azure data engineering professional with 7+ years of experience designing and implementing scalable data integration and analytics solutions across Azure (ADF, Synapse, ADLS, Databricks) and AWS (EMR, Glue, S3, Redshift). Hands-on with ETL/ELT pipelines, API integrations, and performance tuning for reporting and BI. Experienced in building end-to-end data platforms and data lakes, leveraging Spark (PySpark/Scala/Spark SQL/Streaming), Delta Lake, and advanced SQL for large-scale transformations. Also applies OOP and design patterns in Python/Java, and supports CI/CD with Docker and Kubernetes while working with monitoring, governance, and secure data ingestion workflows.…Azure data engineering professional with 7+ years of experience designing and implementing scalable data integration and analytics solutions across Azure (ADF, Synapse, ADLS, Databricks) and AWS (EMR, Glue, S3, Redshift). Hands-on with ETL/ELT pipelines, API integrations, and performance tuning for reporting and BI. Experienced in building end-to-end data platforms and data lakes, leveraging Spark (PySpark/Scala/Spark SQL/Streaming), Delta Lake, and advanced SQL for large-scale transformations. Also applies OOP and design patterns in Python/Java, and supports CI/CD with Docker and Kubernetes while working with monitoring, governance, and secure data ingestion workflows.

Pavani Reddy

Data Scientist, Full Stack Developer, Cloud Developer, +6





Azure data engineering professional with 7+ years of experience designing and implementing scalable data integration and analytics solutions across Azure (ADF, Synapse, ADLS, Databricks) and AWS (EMR, Glue, S3, Redshift). Hands-on with ETL/ELT pipelines, API integrations, and performance tuning for reporting and BI. Experienced in building end-to-end data platforms and data lakes, leveraging Spark (PySpark/Scala/Spark SQL/Streaming), Delta Lake, and advanced SQL for large-scale transformations. Also applies OOP and design patterns in Python/Java, and supports CI/CD with Docker and Kubernetes while working with monitoring, governance, and secure data ingestion workflows.…Azure data engineering professional with 7+ years of experience designing and implementing scalable data integration and analytics solutions across Azure (ADF, Synapse, ADLS, Databricks) and AWS (EMR, Glue, S3, Redshift). Hands-on with ETL/ELT pipelines, API integrations, and performance tuning for reporting and BI. Experienced in building end-to-end data platforms and data lakes, leveraging Spark (PySpark/Scala/Spark SQL/Streaming), Delta Lake, and advanced SQL for large-scale transformations. Also applies OOP and design patterns in Python/Java, and supports CI/CD with Docker and Kubernetes while working with monitoring, governance, and secure data ingestion workflows.

Available to hire

Azure data engineering professional with 7+ years of experience designing and implementing scalable data integration and analytics solutions across Azure (ADF, Synapse, ADLS, Databricks) and AWS (EMR, Glue, S3, Redshift). Hands-on with ETL/ELT pipelines, API integrations, and performance tuning for reporting and BI.

Experienced in building end-to-end data platforms and data lakes, leveraging Spark (PySpark/Scala/Spark SQL/Streaming), Delta Lake, and advanced SQL for large-scale transformations. Also applies OOP and design patterns in Python/Java, and supports CI/CD with Docker and Kubernetes while working with monitoring, governance, and secure data ingestion workflows.

Skills

Experience Level

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Intermediate

Intermediate

Intermediate

Language

English

Fluent

Work Experience

Senior Data Engineer

January 1, 2024 - Present

Worked hands-on with Azure PaaS/IaaS services including Synapse Analytics, Azure SQL, Data Factory, Azure Analysis Services, HDInsight, Azure Monitoring, and Key Vault, along with Azure Data Lake. Built and monitored ADF ETL/ELT pipelines integrating Azure SQL Database, ADLS, and Blob Storage. Developed PySpark for data ingestion/transformation using DataFrames and wrote curated outputs to a Hive data warehouse layer. Created tabular models in Azure Analysis Services for business reporting. Performed performance tuning of production Spark jobs. Developed SQL queries/views for analytics in Synapse Analytics and collaborated with healthcare payor teams to securely move claims/policy data for reporting. Debugged and validated API integrations (JSON/XML/SOAP/REST) using Postman and ensured seamless ingestion into Azure environments. Built Spark jobs in Scala/Spark SQL/Streaming, designed warehouse data models for multiple real-time sources, and used Spark Streaming to micro-batch streamin

Senior Data Engineer at Intact

January 1, 2024 - November 21, 2025

Senior Data Engineer focusing on Azure cloud services, Spark/PySpark processing, Azure Synapse Analytics, and building/monitoring ETL/ELT pipelines across Azure SQL Database, Data Lake, and Blob Storage. Implemented data models and tabular models in Azure Analysis Services; debugged API integrations and supported secure data ingestion.

Senior Data Engineer at Intact

January 1, 2024 - November 7, 2025

Designed, developed, and monitored ETL/ELT pipelines using Azure Data Factory across Azure SQL Database, Data Lake, and Blob Storage; implemented PySpark/Scala data processing, Spark SQL/Streaming, and Hive-based data warehousing; built tabular models in Azure Analysis Services and optimized Azure Synapse queries; integrated healthcare data sources (claims, policies) for reporting; debugged API integrations (JSON, XML, SOAP, REST) with Postman; loaded data into Azure Synapse; Spark Streaming batch processing; partitioning and bucketing in HiveQL; worked with Hadoop, HDFS, Hive, Sqoop, Airflow, Jira, Kafka, AWS EC2/S3, EMR, Cloudera.

Senior Data Engineer at Intact

January 1, 2024 - November 3, 2025

Senior Data Engineer responsible for designing and operating Azure-based data platforms. Hands-on with Azure Data Factory, Azure Synapse Analytics, Data Lake, and Data Factory pipelines; implemented Spark processing with PySpark, creating dataframes from service-layer tables and loading into Hive data warehouse. Built tabular models in Azure Analysis Services to support BI reporting and optimized query performance in Azure Synapse. Led end-to-end ETL/ELT pipelines, including batch and streaming components using Spark Streaming. Collaborated with healthcare payor teams to securely move claims and policy data for reporting. Tested API integrations and validated data ingestion with Postman. Used Azure Blob/Data Lake storage to store large datasets and reduce costs. Implemented data governance practices, managed metadata, and monitored pipelines via Azure Monitor. Worked with Databricks/Delta Lake in Python/Scala for scalable data transformations. Technology stack included Hadoop/HDFS, Hiv

Data Engineer at Wells Fargo

October 31, 2023 - October 31, 2023

Data Engineer responsible for migrating Ab Initio jobs to Python/PySpark and building Spark applications with Spark SQL; worked with AWS cloud services (Glue, Lambda, Redshift) and data streaming with Kinesis. Implemented data governance and operational structures for governable data environments. Ingested, transformed, and moved data across systems using Glue, Spark, and SQL; performed migrations from RDBMS to NoSQL where appropriate. Designed and maintained end-to-end data solutions on AWS (storage, processing, visualization) and migrated SAS reports to Python-based reporting in Snowflake, scheduling via Airflow. Automated data ingestion from S3 to Snowflake and managed source control in GitHub. Built Pandas-based data wrangling/visualization tools and maintained AWS Data Pipeline for data movement among S3, EMR, and RDS. Implemented ELK stack for centralized logging and monitoring; collaborated with stakeholders to deliver BI-ready datasets.

Data Engineer at Wells Fargo

October 1, 2023 - October 1, 2023

Designed and migrated ETL pipelines from Ab Initio to Python using PySpark; built Spark applications (RDD transformations, Spark SQL) with Hadoop HDFS and YARN; established end-to-end data solutions in AWS (S3, Glue, Redshift, EMR) and implemented data governance; performed data ingestion/transformation for RDBMS-to-NoSQL migrations; automated Snowflake reporting via Airflow; implemented data quality checks in SQL and Snowflake; migrated SAS reports to Python; scheduled Snowflake reports via Airflow; automated data movement from S3 to Snowflake; version-controlled code in GitHub; built PySpark data frames and visualization tooling; set up ELK stack for centralized logging.

Data Engineer at Wells Fargo

March 1, 2022 - October 1, 2023

Migrated Abinitio jobs to Python/PySpark and built ETL pipelines using Spark (RDD transformations, Spark SQL) on Hadoop ecosystems (HDFS, YARN). Designed and implemented AWS infrastructure and maintained services in a managed environment. Contributed to data governance for previously ungoverned environments. Built ingestion and transformation pipelines using Kinesis, AWS Glue, and Spark. Performed data migration from RDBMS to NoSQL to integrate datasets across multiple systems. Developed end-to-end AWS solutions covering storage, integration, processing, and visualization. Built PySpark pipelines to load data to EDL via EMR and stored datasets in S3. Wrote SQL for required data analysis and implemented data quality checks/validations in AWS Snowflake versus legacy data; also migrated SAS reports to Python reports. Automated Snowflake report generation and scheduling through Python Airflow, visualized outputs in S3, and loaded generated files into Snowflake. Implemented services and w

Data Engineer at HTC Global Services

February 28, 2022 - February 28, 2022

Data Engineer focusing on migrating Hadoop jobs to EMR/PySpark; developed Spark-based ETL pipelines and moved data between HDFS and S3. Created partitioning strategy and DDL on partitioned data; wrote HiveQL to compute metrics and data quality checks. Built ETL processes in AWS Glue to migrate campaign data into Redshift, automated workflows with Airflow. Implemented Databricks Spark jobs for table-to-table transformations; loaded CSV/JSON/Parquet from S3 and managed bucket structures. Worked in Apache Hadoop ecosystem (HDFS, Hive, Pig) and Cloudera Manager for cluster operations; used Lambda to load CSV into S3 and Python scripts to parse web server outputs to HDFS. Collaborative with project teams; used Jira for issue tracking and monitoring; implemented data visualization with Python/Tableau.

Data Engineer at HTC Global Services

February 1, 2022 - February 1, 2022

Moved data between HDFS and S3; automated EMR cluster provisioning with boto3; partitioned large datasets and migrated Hadoop jobs to EMR; implemented real-time processing with Spark Streaming and Kafka; wrote HiveQL queries for metrics; developed AWS Glue ETL pipelines to migrate campaign data into Redshift; automated workflows with Airflow; worked with Azure Kubernetes Service, Databricks, and Azure Key Vault; ingested data from S3 to Snowflake; handled Avro/Parquet/XML/JSON data; built PySpark jobs in Databricks for table-to-table transformations; created Python scripts to load CSVs into S3 and manage buckets; performed data validation and visualization using Python/Tableau/Power BI.

Data Engineer at HTC Global Services

June 1, 2018 - February 1, 2022

Worked on file movements between HDFS and AWS S3 and performed extensive S3 bucket operations. Automated EMR cluster launch and Hadoop application configuration using boto3. Implemented S3 partitioning and defined DDL for partitioned data; migrated Hadoop jobs to run on EMR by tuning cluster configuration based on data size. Built real-time processing pipelines using Spark Streaming with Kafka and used HiveQL for metrics on partitioned/bucketed datasets. Created data quality scripts in SQL and Hive for validation of successful data loads. Developed ETL processes in AWS Glue to migrate campaign data from S3 (ORC/Parquet/Text) into Redshift. Scheduled daily production workflows using Airflow and shell scripting. Used Azure Kubernetes Service for development collaboration (VMs, connectors) and worked on data movement from S3 to Snowflake. Parsed semi-structured data (JSON) with PySpark and converted to Parquet; created Databricks clusters/jobs for table operations. Built Python scripts