Available to hire
Senior Data Engineer / AI Platform Engineer with 7+ years of experience building scalable lakehouse data platforms, governed analytics pipelines, and AI serving layers on Azure. Strong hands-on expertise in Azure Databricks, Delta Lake, PySpark, Spark SQL, Delta Live Tables, and enterprise data orchestration for batch and streaming workloads.
Built production-grade low-latency REST APIs and GenAI/RAG solutions using FastAPI, Docker, Kubernetes/AKS, Redis caching, GPT/OpenAI APIs, LangChain, vector databases, and search/embedding pipelines. Deep experience in Delta Lake internals, performance tuning, CI/CD, MLOps/DataOps, and enterprise governance using Unity Catalog and Azure Purview.
Experience Level
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Intermediate
Work Experience
Sr Data Engineer – Gen AI at T-Mobile
October 1, 2024 - PresentBuilt production-grade AI and data serving pipelines using Azure Databricks, PySpark, Spark SQL, Delta Lake, and ADLS Gen2 to support scalable lakehouse processing for enterprise GenAI workloads. Developed low-latency REST APIs using Python, FastAPI, Docker, AKS, and Azure Functions to expose RAG/semantic search and customer data lookup capabilities. Designed reusable sync pipelines using Kafka, Azure Event Hubs, Spark Structured Streaming, ADF, and Delta Lake for continuous refresh of serving, analytics, and downstream business layers. Implemented Redis-backed caching patterns for high-frequency lookup APIs to reduce latency and load on Databricks and downstream systems. Engineered RAG pipelines using LangChain, GPT-4, Pinecone, ChromaDB, Azure Cognitive Search, embeddings, chunking, and metadata filters, along with prompt validation, response evaluation, and hallucination/consistency checks for production readiness. Orchestrated resilient AI workflows with fallback routing and tool-c
AI/ML Data Engineer at BNY Mellon
May 1, 2023 - August 31, 2024Designed governed lakehouse pipelines on Azure Databricks, PySpark, Spark SQL, Delta Lake, and Azure Data Factory for large-scale financial datasets using reliable batch and incremental patterns. Built scalable AI/ML serving microservices exposing model outputs and data products via API-based deployments using Python, FastAPI, Docker, Kubernetes on AKS/EKS, and Azure services. Developed real-time streaming pipelines using Kafka, Azure Event Hubs, Spark Structured Streaming, and Delta Lake to support low-latency financial data processing and continuous data availability. Implemented Delta Live Tables pipelines for incremental ingestion, data quality checks, and controlled financial data movement. Managed ML lifecycle workflows with MLflow and Databricks ML Runtime, including experiment tracking and controlled promotion via Azure DevOps. Built NLP pipelines using BERT, Scikit-learn, and TensorFlow for document classification and entity extraction. Enhanced enterprise search using Elastic
Data ML Engineer at Berkadia Services India Private Limited
July 1, 2019 - December 31, 2022Built scalable data and ML pipelines using Azure Databricks, PySpark, Spark SQL, Delta Lake, Scikit-learn, XGBoost, and MLflow for risk analytics, reporting, and predictive modeling. Designed end-to-end batch and streaming pipelines using Azure Databricks/Spark 3.x with ADLS Gen2 storage and Delta Lake. Developed metadata-driven ETL frameworks with ADF, SQL Server, Oracle, REST APIs, and ADLS Gen2 for reusable ingestion and transformation patterns. Implemented near real-time ingestion pipelines using Kafka, Azure Event Hubs, Spark Structured Streaming, Apache NiFi, checkpointing, and watermarking. Built ML workflows with Spark MLlib, Scikit-learn, XGBoost, and MLflow for model tracking/versioning/validation. Deployed scalable inference services using FastAPI and Docker on Azure Kubernetes Service with autoscaling. Created reusable transformation frameworks and improved maintainability using modular Databricks notebooks and PySpark/Spark SQL/Scala components. Tuned Spark performance usi
Big Data Developer at AppZon Technologies Private Limited
June 1, 2018 - June 30, 2019Built scalable ETL pipelines using Azure Data Factory, Azure Databricks, PySpark, Spark SQL, and Parquet to process CRM, POS, and eCommerce datasets for analytics. Designed batch processing frameworks using Apache Spark and SQL with data warehouse technologies for high-volume transformations and reporting. Developed ingestion workflows with ADF, SQL Server, Cosmos DB, REST APIs, and Snowflake/cloud storage. Engineered near real-time pipelines using Kafka, Azure Event Hubs, Spark Structured Streaming, and Stream Analytics. Created metadata-driven and reusable ADF pipelines using parameterization and loop constructs, including incremental ingestion patterns with watermarking. Built reusable transformation logic with PySpark/Spark SQL/SQL and window functions. Tuned Spark workloads using broadcast joins, caching, checkpointing, partitioning, and shuffle optimization for stability and performance. Modeled warehouse datasets using T-SQL, star schema, and SCD Type 2 patterns for BI reporting
Education
Master's in Information Technology at Southern New Hampshire University
January 11, 2030 - January 1, 2024Bachelor's in Computer Science Engineering at Geethanjali College of Engineering & Technology
January 1, 2015 - December 31, 2019Qualifications
Azure Data Engineer Associate (DP-203)
January 11, 2030 - July 10, 2026Azure Fundamentals (AZ-900)
January 11, 2030 - July 10, 2026Industry Experience
Telecommunications, Financial Services, Healthcare, Retail, Software & Internet
Experience Level
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Intermediate
Hire a Data Scientist
We have the best data scientist experts on Twine. Hire a data scientist today.