Shubham Chourasia

Engineered and optimized data pipelines and ETL workflows using Airflow, PySpark, and AWS (S3, Redshift, Athena, Aurora) to enable seamless data movement and processing. Developed custom Airflow operators, sensors, and real-time monitoring systems integrating DAG and EC2 status APIs for enhanced alerting and reliability. Designed SQL-based views and Tableau dashboards to transform complex datasets into actionable business insights and data-driven decisions.

Experienced Data Engineer in the retail domain, specializing in building and orchestrating scalable data pipelines using Spark, Hive, and Python for large-volume processing and analytics. Led GCP migration initiatives and delivered ad hoc SQL solutions to support business intelligence and reporting needs, including Looker dashboard for visualization. Developed microservices using Java and contributed to MVP data products, ensuring high-performance, cloud-native, and production-grade data ecosystems.

Contributed to the successful execution of a Data Warehouse Migration initiative for a US Healthcare organization, ensuring seamless transition and optimized data architecture. Designed and implemented a robust data pipeline leveraging the Snowflake ecosystem — including Snowpipe, Tasks, Streams, and Stored Procedures — to efficiently move data from Teradata and FTP sources into Snowflake. Developed advanced PySpark scripts to implement complex business logic, enabling precise data deduplication, validation, and enhanced data quality assurance.