Looks like you have JavaScript disabled. For the full Twine experience, you will need to re-enable it.

I am a Senior Data Engineer with over 9 years of experience managing the full software development life cycle, specializing in high-performing multi-tiered web applications and big data technologies. I have extensive hands-on expertise in Hadoop frameworks, Spark, Scala, Python, and cloud platforms including AWS and Azure. Passionate about solving complex data challenges, I thrive in agile environments and continually strive to optimize data processing and analytics solutions. Throughout my career, I have successfully implemented large-scale data engineering projects, including migrating traditional systems to modern cloud architectures and integrating streaming data platforms like Kafka. I am adept at developing scalable, fault-tolerant distributed systems and automating data pipelines, with strong problem-solving skills and a commitment to delivering high-quality, efficient solutions.…I am a Senior Data Engineer with over 9 years of experience managing the full software development life cycle, specializing in high-performing multi-tiered web applications and big data technologies. I have extensive hands-on expertise in Hadoop frameworks, Spark, Scala, Python, and cloud platforms including AWS and Azure. Passionate about solving complex data challenges, I thrive in agile environments and continually strive to optimize data processing and analytics solutions. Throughout my career, I have successfully implemented large-scale data engineering projects, including migrating traditional systems to modern cloud architectures and integrating streaming data platforms like Kafka. I am adept at developing scalable, fault-tolerant distributed systems and automating data pipelines, with strong problem-solving skills and a commitment to delivering high-quality, efficient solutions.

Pavan Ponugti

Developer, Data Scientist, AI Engineer, +2





I am a Senior Data Engineer with over 9 years of experience managing the full software development life cycle, specializing in high-performing multi-tiered web applications and big data technologies. I have extensive hands-on expertise in Hadoop frameworks, Spark, Scala, Python, and cloud platforms including AWS and Azure. Passionate about solving complex data challenges, I thrive in agile environments and continually strive to optimize data processing and analytics solutions. Throughout my career, I have successfully implemented large-scale data engineering projects, including migrating traditional systems to modern cloud architectures and integrating streaming data platforms like Kafka. I am adept at developing scalable, fault-tolerant distributed systems and automating data pipelines, with strong problem-solving skills and a commitment to delivering high-quality, efficient solutions.…I am a Senior Data Engineer with over 9 years of experience managing the full software development life cycle, specializing in high-performing multi-tiered web applications and big data technologies. I have extensive hands-on expertise in Hadoop frameworks, Spark, Scala, Python, and cloud platforms including AWS and Azure. Passionate about solving complex data challenges, I thrive in agile environments and continually strive to optimize data processing and analytics solutions. Throughout my career, I have successfully implemented large-scale data engineering projects, including migrating traditional systems to modern cloud architectures and integrating streaming data platforms like Kafka. I am adept at developing scalable, fault-tolerant distributed systems and automating data pipelines, with strong problem-solving skills and a commitment to delivering high-quality, efficient solutions.

Available to hire

I am a Senior Data Engineer with over 9 years of experience managing the full software development life cycle, specializing in high-performing multi-tiered web applications and big data technologies. I have extensive hands-on expertise in Hadoop frameworks, Spark, Scala, Python, and cloud platforms including AWS and Azure. Passionate about solving complex data challenges, I thrive in agile environments and continually strive to optimize data processing and analytics solutions.

Throughout my career, I have successfully implemented large-scale data engineering projects, including migrating traditional systems to modern cloud architectures and integrating streaming data platforms like Kafka. I am adept at developing scalable, fault-tolerant distributed systems and automating data pipelines, with strong problem-solving skills and a commitment to delivering high-quality, efficient solutions.

Skills

Experience Level

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Intermediate

Intermediate

Intermediate

Intermediate

Intermediate

Work Experience

Sr. Data Engineer at CareFirst SBP, MD

November 1, 2023 - Present

I have been involved in all phases of the software development life cycle to develop and deploy Spark applications using PySpark and Spark-SQL for extracting, transforming, and aggregating data from multiple file formats. I developed Hive UDFs and managed Hive tables with partitioning and bucketing to optimize query performance. I worked on migrating client data warehouse architectures to Microsoft Azure, integrating Spark for interactive and streaming data processing, and utilized NoSQL databases like HBase. I created data pipelines using Azure Data Factory and implemented cluster solutions for NoSQL tools. Further, I designed end-to-end data solutions in Azure, developed Spark applications for various streaming sources, and automated workflows using Oozie and Apache NiFi. My role also included working with Kafka for real-time data streaming and implementing autoscaling for cost-effective, fault-tolerant systems.

Data Engineer/ Spark at State of Michigan, MI

October 31, 2023 - August 26, 2025

In this role, I developed and optimized Spark jobs using Scala and Python on YARN for both batch and interactive analyses. My responsibilities included migrating on-premises data to Azure cloud services, developing NiFi workflows for data ingestion to Kafka, and using Python libraries for clinical data ETL processes and NLP analysis. I designed and managed Spark frameworks for data processing, developed Oozie workflows, worked with Cassandra for NoSQL data storage, and migrated data pipeline jobs from Oozie to Airflow. I also implemented real-time streaming ingestion with Kafka and Spark Streaming and created data visualizations and dashboards using Power BI.

Data Engineer at Global Atlantic Financial Group, Indianapolis, IN

March 31, 2021 - August 26, 2025

I managed and maintained Cloudera Hadoop clusters, developed MapReduce programs for ETL processes, and utilized Pig, Hive, and HBase for data analysis. I worked extensively in AWS environments, creating Oozie workflows and moving data between HDFS and AWS S3. I performed data validation, developed custom models to cleanse data, and handled importing data from various sources into HDFS. I also transferred data into AWS Redshift using Informatica and performed ETL using SSIS with report generation via SSRS. My role involved Hive table creation, query optimization, and cluster coordination using Zookeeper.

Data Engineer at Avon Technologies Pvt Ltd, Hyderabad, India

August 1, 2018 - August 26, 2025

I collaborated with BI teams to gather report requirements and used Sqoop to export data into HDFS and Hive. My duties included data cleaning and pre-processing using Java MapReduce, managing Flume infrastructure, and administering Pig, Hive, and HBase. I analyzed patterns in fraudulent claims using text mining in R and Hive, and exported data for claims processing teams. I developed Hive queries to support marketing analysts and automated data loading using Oozie. Additionally, I tested raw data and executed performance scripts while managing Hadoop log files.