Available to hire
Senior Data Engineer with 8+ years of experience designing scalable data platforms across Azure and AWS using Databricks, PySpark, and Azure Data Factory. Expertise includes Delta Lake architectures (ACID, schema enforcement, time travel), ETL/ELT pipeline development, and SQL performance tuning for analytics workloads.
Built and operated real-time and batch data pipelines using Kafka, streaming ingestion patterns, and workflow orchestration (Databricks Workflows/Jobs, ADF, Airflow). Strong focus on governance (Unity Catalog / data catalogs), security (RBAC, Key Vault/KMS, encryption), CI/CD, monitoring/alerting, and delivering analytics-ready datasets for Power BI/Tableau and ML use cases.
Experience Level
Work Experience
Senior Data Engineer at Mark Anthony Group
October 1, 2024 - PresentDesigned and developed scalable Azure Databricks pipelines across dev/QA/prod. Implemented Delta Lake with ACID guarantees, schema enforcement, time travel, and performance optimizations (e.g., Z-ordering). Built parameterized, reusable PySpark notebooks integrated with Azure Data Factory using parameter passing and managed orchestration. Migrated legacy SQL Server/SSIS ETL logic into Databricks to improve runtimes and maintainability. Established Unity Catalog governance with fine-grained RBAC and centralized secure access. Tuned job and cluster configurations (autoscaling, high-concurrency, spot instances) to reduce compute costs, implemented monitoring/logging with Azure Monitor and Databricks job logs, and enabled BI publishing to Power BI with optimized datasets.
Senior Data Engineer at Wells Fargo
August 1, 2022 - October 1, 2024Engineered an Intelligent Data Foundation accelerator using Azure Databricks, Azure Functions, and ADLS Gen2, including workspace management, job orchestration, and cluster performance tuning. Developed high-volume ETL/ELT using Spark SQL and PySpark with query and transformation performance optimization. Built data quality validation frameworks with PySpark/Delta Lake and delivered analytics-ready outputs for Power BI and Tableau. Used Databricks Repos/Workflows for version control and repeatable job execution across environments. Integrated MLflow for model tracking and lifecycle management. Migrated SSIS pipelines into Databricks, built parameterized notebooks, orchestrated external loads (Kafka, flat files, APIs) via Airflow, and enforced governance with Unity Catalog. Implemented monitoring and SLA compliance using job logs, Azure Monitor, and Log Analytics.
Data Engineer at State Farm Insurance
May 1, 2020 - July 31, 2022Designed conceptual/logical/physical data models for banking operations with a focus on scalability and performance. Built OLAP/OLTP dimensional models using star/snowflake design patterns, including fact/dimension tables and SCD Type 1/Type 2 handling. Developed lake and warehouse integrations using Azure Synapse Analytics, Azure Data Factory, and Azure Databricks. Established data lake architecture with ADLS Gen2 and Synapse for machine learning and advanced analytics use cases. Optimized Azure Cosmos DB configurations and tuned Synapse queries to reduce query costs. Implemented robust ETL/ELT pipelines using ADF and PySpark, and supported reporting/BI model design with Power BI. Participated in agile delivery and maintained metadata/documentation for governance and traceability.
Data Engineer at Change Healthcare
January 1, 2018 - April 30, 2020Migrated legacy healthcare systems to AWS using AWS DMS for transitioning ACA claims and EHR/EMR data. Built ingestion pipelines using AWS Glue and AWS Lambda with transformations and data cleansing aligned to HIPAA and HL7/FHIR standards. Consolidated multiple data streams into master-child ETL workflows in AWS Glue and stored data in S3 as a secure data lake. Loaded curated datasets into AWS Redshift using stored procedures for mart loading and performance tuning, including distribution/sort key optimization. Implemented data quality checks and governance tagging for compliance. Developed ML-ready datasets with Python (Pandas/NumPy) and supported visualization via AWS QuickSight. Managed Spark batch processing with EMR and ensured security using IAM and KMS.
Data Engineer at TCS
July 1, 2016 - September 30, 2017Supported BI operations and built ETL pipelines across Hadoop and SQL Server to deliver data for business units. Developed ingestion flows between HDFS and relational systems using Sqoop and built Hive/Spark-based warehouse solutions. Designed SSIS packages for automated extraction/transformation from APIs, flat files, and relational sources. Created Tableau dashboards for Media Insights and Marketing teams, and coordinated onshore-offshore reporting operations including issue resolution and stakeholder updates.
Education
Bachelor of Technology in Computer Science at JNTU (H)
June 1, 2012 - May 31, 2016Bachelor of Technology in Computer Science at JNTU (H)
June 1, 2012 - May 31, 2016Qualifications
Industry Experience
Financial Services, Healthcare
Experience Level
Hire a Data Engineer
We have the best data engineer experts on Twine. Hire a data engineer in Washington today.