I am Shivani, a data engineer with 5+ years of experience building scalable data pipelines, real-time streaming solutions, and data warehousing across on-premises and cloud environments. I specialize in transforming raw data into reliable, well-governed datasets using Python, SQL, Spark, and modern ETL/ELT tooling. I’ve worked across Azure and AWS stacks, implemented CI/CD for data pipelines, and delivered interactive dashboards in Power BI to enable data-driven decisions. In my roles, I’ve designed end-to-end data ingestion and transformation, improved data quality and latency, and collaborated with cross-functional teams to deploy robust data solutions. I thrive in fast-paced environments, document architectures and runbooks for operational reliability, and continually upskill to leverage the latest cloud and big data technologies.

Shivani R

I am Shivani, a data engineer with 5+ years of experience building scalable data pipelines, real-time streaming solutions, and data warehousing across on-premises and cloud environments. I specialize in transforming raw data into reliable, well-governed datasets using Python, SQL, Spark, and modern ETL/ELT tooling. I’ve worked across Azure and AWS stacks, implemented CI/CD for data pipelines, and delivered interactive dashboards in Power BI to enable data-driven decisions. In my roles, I’ve designed end-to-end data ingestion and transformation, improved data quality and latency, and collaborated with cross-functional teams to deploy robust data solutions. I thrive in fast-paced environments, document architectures and runbooks for operational reliability, and continually upskill to leverage the latest cloud and big data technologies.

Available to hire

I am Shivani, a data engineer with 5+ years of experience building scalable data pipelines, real-time streaming solutions, and data warehousing across on-premises and cloud environments. I specialize in transforming raw data into reliable, well-governed datasets using Python, SQL, Spark, and modern ETL/ELT tooling. I’ve worked across Azure and AWS stacks, implemented CI/CD for data pipelines, and delivered interactive dashboards in Power BI to enable data-driven decisions.

In my roles, I’ve designed end-to-end data ingestion and transformation, improved data quality and latency, and collaborated with cross-functional teams to deploy robust data solutions. I thrive in fast-paced environments, document architectures and runbooks for operational reliability, and continually upskill to leverage the latest cloud and big data technologies.

See more

Experience Level

Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Intermediate
Intermediate
See more

Language

English
Fluent

Work Experience

Data Engineer at AT&T
September 1, 2023 - November 21, 2025
Orchestrated Azure Data Factory pipelines for data ingestion and migrated on-prem SQL Server databases to Azure Data Lake Storage Gen2. Built Delta tables and Delta Live Tables in Databricks for real-time processing, and integrated Power BI with Azure SQL for interactive dashboards. Implemented real-time monitoring with Azure Logic Apps and Azure Functions, resulting in faster issue resolution. Scheduled and managed Apache Airflow workflows and designed Snowflake modeling to improve data cleansing and optimize slowly changing dimensions and data validation.
Data Engineer at NCR (India)
December 31, 2022 - December 31, 2022
Developed SQL Server stored procedures, functions, views, and triggers. Built ETL pipelines using Azure Data Factory and SSIS packages for cloud and on-prem data integration. Created Spark applications with PySpark and Spark SQL for multi-format data processing. Enhanced pipeline deployment with CI/CD using Azure DevOps, Kubernetes, and Docker. Migrated data from SQL Server, DB2, Oracle, and MySQL to Azure Synapse Analytics and Azure Data Lake, achieving reduced data latency (≈30%) and improved production release efficiency with Agile practices.
Program Analyst at Aktrix (India)
July 31, 2021 - July 31, 2021
Built Spark streaming applications integrating AWS Kinesis and loading data into Redshift. Leveraged Snowflake features like Snow Pipes and stages for real-time ingestion. Scheduled serverless workflows with AWS Step Functions and EventBridge Scheduler. Orchestrated ETL pipelines in Glue Studio, ingesting data into S3, Redshift, and RDS, reducing completion time by ~40%. Worked with Hadoop, Spark, Hive, MapReduce, and HBase on AWS EMR clusters to optimize performance. Architected RESTful APIs with Django and AWS services (APIGateway, Lambda, Route53, IAM, S3, CloudWatch, Step Functions, ElasticSearch, DynamoDB) to reduce latency.

Education

Master of Science in Information Systems at Indiana Institute of Technology, Fort Wayne, IN
January 11, 2030 - November 21, 2025
Bachelor of Technology in Electronics & Communication Engineering at Teegala Krishna Reddy Engineering College, Hyderabad, India
January 11, 2030 - November 21, 2025

Qualifications

Advanced SQL Certification
January 11, 2030 - November 21, 2025
Python for Data Engineering
January 11, 2030 - November 21, 2025

Industry Experience

Software & Internet, Professional Services