Sheshanth Reddy
I am a Data Engineer with over 7 years of experience designing and implementing cloud-based and on-premises data solutions.
About
I am a Data Engineer with over 7 years of experience designing and implementing cloud-based and on-premises data solutions. I specialize in Databricks, PySpark, and Kafka, and have a proven track record migrating warehouses to cloud-native platforms, building scalable batch and streaming pipelines, and delivering BI-ready datasets for analytics and ML.
I enjoy collaborating with platform and product teams to create data products, implementing governance and security, and enabling self-service analytics through dashboards and reporting. My work spans from data ingestion and transformations to data modeling (Dimensional, Data Vault, Star) and modern architectures such as Medallion, Delta Lake, and lakehouse patterns.
- 9 years' experience
- Joined October 2025
Industry experience
- Software & Internet
- Professional Services
More experts in St. Louis, United States
- Cloud Developers for hire in St. Louis, United States
- Data Analysts for hire in St. Louis, United States
- Data Scientists for hire in St. Louis, United States
- Database Developers for hire in St. Louis, United States
- Full Stack Developers for hire in St. Louis, United States
- Web Developers for hire in St. Louis, United States
Work experience· 7 roles
Worked on on-premises BI solutions using SSIS, SSRS, and Power BI to build ETL pipelines, operational reports, and dashboards. Migrated to Azure Cloud, adopting Azure Databricks to modernize pipelines and process large-scale semi-structured and unstructured data. Designed and optimized workflows as the central platform for big data processing, Spark-based transformations, and ML-ready pipelines. Developed and deployed PySpark jobs for batch ingestion from APIs, flat files, and RDBMS sources. Streamed ingestion from Kafka into Cosmos DB and ADLS, handling high-velocity JSON/NoSQL data for low-latency analytics. Implemented Azure Key Vault, RBAC, and Managed Identities to safeguard sensitive datasets across production pipelines. Built automated pipelines for schema and data deployments using GitHub Actions and GraphQL Inspector, with Git as the version control system to manage and back up code. Delivered Power BI dashboards and reports by integrating with modernized cloud data sources, e
Migrated on-premises databases and data warehouses into the Databricks Lakehouse on GCP using Google Cloud Storage (GCS) as the landing zone, establishing a scalable and cloud-native foundation. Designed and developed robust PySpark pipelines in Databricks for data ingestion, cleansing, enrichment, and transformation, producing high-quality reusable datasets. Implemented near real-time streaming pipelines using Kafka and Spark Structured Streaming, later refactored with GCP Pub/Sub and Databricks Structured Streaming for cloud-native scalability. Optimized Databricks performance and cost efficiency by right-sizing clusters, enabling autoscaling, and tuning Spark jobs for large-scale batch and streaming workloads. Automated ETL workflows with Apache Airflow DAGs, ensuring reliability and reducing manual interventions in production pipelines. Built a centralized logging and monitoring framework using ELK Stack and GCP Stackdriver, enabling proactive detection and faster resolution of fai
Migrated on-premises databases and data warehouses into the Databricks Lakehouse on GCP using Google Cloud Storage (GCS) as the landing zone. Designed and developed PySpark pipelines for data ingestion, cleansing, enrichment, and transformation. Implemented near real-time streaming pipelines using Kafka and Spark Structured Streaming, later refactored with GCP Pub/Sub for cloud-native scalability. Optimized Databricks performance and cost efficiency through right-sizing clusters and autoscaling. Automated ETL workflows with Apache Airflow, and built centralized logging/monitoring using ELK Stack and Stackdriver. Partnered with platform and product teams to deliver curated data products in Databricks Lakehouse for downstream consumers.
Education
Hire Sheshanth Reddy today
Post a job and invite Sheshanth, or contact them directly. No hidden commission.

