Looks like you have JavaScript disabled. For the full Twine experience, you will need to re-enable it.

Hi, I’m Manichandana Gopireddy, a data engineer based in Atlanta, GA. I have 5 years of hands-on experience building scalable big data pipelines and real-time analytics across AWS, GCP, and hybrid cloud environments. I specialize in ETL/ELT workflows using PySpark, Scala, and Spark SQL to process structured and semi-structured data from Snowflake, Hive, Kafka, and Web APIs. I design end-to-end data platforms and orchestrate data workflows with Apache Airflow and AWS Step Functions, while integrating cloud-native services (EMR, S3, Databricks, GCS) to support analytics, reporting, and data science initiatives. I emphasize data quality, schema evolution, ACID-like production architectures, and end-to-end testing for production-grade pipelines, delivering reusable components for clients like Sirius XM, Deloitte, and Hewlett Packard.…Hi, I’m Manichandana Gopireddy, a data engineer based in Atlanta, GA. I have 5 years of hands-on experience building scalable big data pipelines and real-time analytics across AWS, GCP, and hybrid cloud environments. I specialize in ETL/ELT workflows using PySpark, Scala, and Spark SQL to process structured and semi-structured data from Snowflake, Hive, Kafka, and Web APIs. I design end-to-end data platforms and orchestrate data workflows with Apache Airflow and AWS Step Functions, while integrating cloud-native services (EMR, S3, Databricks, GCS) to support analytics, reporting, and data science initiatives. I emphasize data quality, schema evolution, ACID-like production architectures, and end-to-end testing for production-grade pipelines, delivering reusable components for clients like Sirius XM, Deloitte, and Hewlett Packard.

Manichandana Gopireddy

Data Scientist, Full Stack Developer, Back-End Developer, +2





Hi, I’m Manichandana Gopireddy, a data engineer based in Atlanta, GA. I have 5 years of hands-on experience building scalable big data pipelines and real-time analytics across AWS, GCP, and hybrid cloud environments. I specialize in ETL/ELT workflows using PySpark, Scala, and Spark SQL to process structured and semi-structured data from Snowflake, Hive, Kafka, and Web APIs. I design end-to-end data platforms and orchestrate data workflows with Apache Airflow and AWS Step Functions, while integrating cloud-native services (EMR, S3, Databricks, GCS) to support analytics, reporting, and data science initiatives. I emphasize data quality, schema evolution, ACID-like production architectures, and end-to-end testing for production-grade pipelines, delivering reusable components for clients like Sirius XM, Deloitte, and Hewlett Packard.…Hi, I’m Manichandana Gopireddy, a data engineer based in Atlanta, GA. I have 5 years of hands-on experience building scalable big data pipelines and real-time analytics across AWS, GCP, and hybrid cloud environments. I specialize in ETL/ELT workflows using PySpark, Scala, and Spark SQL to process structured and semi-structured data from Snowflake, Hive, Kafka, and Web APIs. I design end-to-end data platforms and orchestrate data workflows with Apache Airflow and AWS Step Functions, while integrating cloud-native services (EMR, S3, Databricks, GCS) to support analytics, reporting, and data science initiatives. I emphasize data quality, schema evolution, ACID-like production architectures, and end-to-end testing for production-grade pipelines, delivering reusable components for clients like Sirius XM, Deloitte, and Hewlett Packard.

Available to hire

Hi, I’m Manichandana Gopireddy, a data engineer based in Atlanta, GA. I have 5 years of hands-on experience building scalable big data pipelines and real-time analytics across AWS, GCP, and hybrid cloud environments. I specialize in ETL/ELT workflows using PySpark, Scala, and Spark SQL to process structured and semi-structured data from Snowflake, Hive, Kafka, and Web APIs.

I design end-to-end data platforms and orchestrate data workflows with Apache Airflow and AWS Step Functions, while integrating cloud-native services (EMR, S3, Databricks, GCS) to support analytics, reporting, and data science initiatives. I emphasize data quality, schema evolution, ACID-like production architectures, and end-to-end testing for production-grade pipelines, delivering reusable components for clients like Sirius XM, Deloitte, and Hewlett Packard.

Skills

Experience Level

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Intermediate

Language

English

Fluent

Work Experience

Data Engineer Intern at Sirius XM

May 1, 2024 - November 12, 2025

Built and maintained a scalable end-to-end data pipeline in the AWS cloud (EMR, S3, EC2) with Apache Airflow. Developed and orchestrated 4 PySpark jobs for data extraction and transformation from Snowflake (9 tables), S3, and external WebAPI. Implemented a master PySpark job to aggregate intermediate outputs, apply business rules, and write final datasets to S3 and OpenSearch (in progress). Performed advanced transformations in PySpark (filters, joins, aggregations, flattening, window functions). Enabled team-level access to intermediate S3 datasets for cross-unit reporting. Designed and scheduled Airflow DAGs to dynamically provision and terminate EMR clusters, with daily executions at 9 AM EST. Integrated real-time WebAPI data to enrich customer location insights and initiated Delta Lake adoption. Contributed to OpenSearch integration to support downstream API-based access and advanced search functionalities (in development).

Data Engineer at Deloitte

December 1, 2022 - December 1, 2022

Designed and deployed distributed Spark pipelines using Scala and PySpark on Google Dataproc, handling large-scale batch and streaming data workflows. Developed modular Python scripts for orchestrating ETL jobs, API ingestion, file parsing, logging, and error handling to ensure reusable components. Built SQL-based data quality checks and complex joins for source-target reconciliation across Snowflake and Hive. Integrated data from Kafka, GCS, and Snowflake; performed advanced ETL transformations using Spark RDD, DataFrame APIs, SQL, and MLlib for real-time and batch analytics. Implemented Spark Streaming jobs for real-time log analysis, anomaly detection, and dashboard updates via Kafka-Spark integration. Automated orchestration using Step Functions and Apache Airflow, with retry logic, cluster lifecycle management, and scheduled runs at 9 AM EST. Developed Delta Lake-based architecture enabling ACID transactions and time travel for location data, improving data integrity. Built and ma

Big Data Engineer at Connectial Infosolutions Pvt Ltd

March 1, 2021 - March 1, 2021

Created and managed Hive tables (managed, external, partitioned), optimizing query performance with predicate pushdown, subquery unnesting, and vectorization. Wrote complex SQL queries and views for data transformation, aggregation, and analysis across Hive, MySQL, and Impala. Developed Python utilities to automate data ingestion, validation, and ETL status reporting, improving workflow reliability. Streamlined ETL workflows using Informatica, including real-time scheduling, transformation logic, and pipeline validation. Integrated Hive with Spark and Hadoop to accelerate big data processing, enabling parallel ETL operations for structured and semi-structured datasets. Enabled schema evolution using Avro to manage changing data structures without interrupting downstream pipelines. Developed Sqoop-based data sync and replication jobs to transfer data between MySQL and HDFS, supporting backups and data lake updates.