Looks like you have JavaScript disabled. For the full Twine experience, you will need to re-enable it.

Data Engineer with 8 years in IT and 5 years building production-grade, cloud-native data pipelines on Azure Databricks—turning raw, messy, multi-source data into reliable pipelines the business can run. I focus on scalable Spark/Delta Lake pipelines, orchestrated ingestion and transformation, and automated reporting to improve runtime and reliability. I also extend the standard data stack into agentic AI tooling by building Databricks LLM agents that query governed Unity Catalog data and generate actionable business insights with explicit data completeness signals. I translate ambiguous cross-functional requirements into well-documented, end-to-end solutions and comfortably own delivery from ingestion through reporting.…Data Engineer with 8 years in IT and 5 years building production-grade, cloud-native data pipelines on Azure Databricks—turning raw, messy, multi-source data into reliable pipelines the business can run. I focus on scalable Spark/Delta Lake pipelines, orchestrated ingestion and transformation, and automated reporting to improve runtime and reliability. I also extend the standard data stack into agentic AI tooling by building Databricks LLM agents that query governed Unity Catalog data and generate actionable business insights with explicit data completeness signals. I translate ambiguous cross-functional requirements into well-documented, end-to-end solutions and comfortably own delivery from ingestion through reporting.

Sangeetha Pandiyan

Data Engineer, Software Engineer, Data Analyst





Data Engineer with 8 years in IT and 5 years building production-grade, cloud-native data pipelines on Azure Databricks—turning raw, messy, multi-source data into reliable pipelines the business can run. I focus on scalable Spark/Delta Lake pipelines, orchestrated ingestion and transformation, and automated reporting to improve runtime and reliability. I also extend the standard data stack into agentic AI tooling by building Databricks LLM agents that query governed Unity Catalog data and generate actionable business insights with explicit data completeness signals. I translate ambiguous cross-functional requirements into well-documented, end-to-end solutions and comfortably own delivery from ingestion through reporting.…Data Engineer with 8 years in IT and 5 years building production-grade, cloud-native data pipelines on Azure Databricks—turning raw, messy, multi-source data into reliable pipelines the business can run. I focus on scalable Spark/Delta Lake pipelines, orchestrated ingestion and transformation, and automated reporting to improve runtime and reliability. I also extend the standard data stack into agentic AI tooling by building Databricks LLM agents that query governed Unity Catalog data and generate actionable business insights with explicit data completeness signals. I translate ambiguous cross-functional requirements into well-documented, end-to-end solutions and comfortably own delivery from ingestion through reporting.

Available to hire

Data Engineer with 8 years in IT and 5 years building production-grade, cloud-native data pipelines on Azure Databricks—turning raw, messy, multi-source data into reliable pipelines the business can run. I focus on scalable Spark/Delta Lake pipelines, orchestrated ingestion and transformation, and automated reporting to improve runtime and reliability.

I also extend the standard data stack into agentic AI tooling by building Databricks LLM agents that query governed Unity Catalog data and generate actionable business insights with explicit data completeness signals. I translate ambiguous cross-functional requirements into well-documented, end-to-end solutions and comfortably own delivery from ingestion through reporting.

Skills

Experience Level

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Expert

Intermediate

Intermediate

Work Experience

Advanced Data Engineer at Tech Mahindra

November 1, 2025 - Present

Built an Azure Databricks retail analytics platform for Kroger business units with end-to-end data engineering delivery, agentic AI tooling, and automated reporting workflows. Developed Databricks ingestion from Excel (.xlsb) into Delta Lake; implemented geospatial-aware ingestion from REST APIs into Delta Lake with coordinate deduplication across 2,700+ stores and batched writes for efficient processing. Implemented Microsoft Fabric workflows using OneLake/Lakehouse/Data Pipelines with PySpark transformations. Added data quality validation (schema consistency, null checks, duplicates, referential integrity, business rules) and source-to-target reconciliation (counts and aggregates). Designed incremental SCD-2 loading with Delta Lake MERGE. Built LangChain/LangGraph agents that invoke Unity Catalog SQL functions as structured tools to generate retail insights (YoY ranking, mode-selected descriptors) with explicit completeness signals. Implemented 2-level HITL approvals via Streamlit, a

Technology Analyst at Infosys Ltd

April 1, 2021 - January 1, 2025

Modernized data platforms for BNSF railways with core focus on Azure Data Factory pipeline development, Databricks transformations, and Delta Lake optimization. Designed ADF pipelines to ingest multi-source data from DB2, Oracle, Teradata, REST APIs, and S3, with transformations in Databricks using SQL and PySpark. Built real-time ingestion using Azure IoT Hub and Event Hubs for GPS sensor tracking. Applied Medallion architecture (Bronze/Silver/Gold) for refined, structured data. Tuned Delta Lake for improved query performance and collaborated with data scientists/analysts to enhance data quality and reliability. Implemented RBAC, data masking, and encryption to meet compliance requirements. Built CI/CD using GitHub/Jenkins for automated ADF and Databricks deployments, and created optimized Power BI dashboards using DAX, KPIs, slicers, and visualizations adopted by business users.

Systems Engineer at Infosys Ltd

January 2, 2017 - January 31, 2017

Provided incident management and post-deployment support for PepsiCo–Frito-Lay North America. Developed and maintained mainframe-based applications, and supported scheduled mainframe batch jobs for data processing and Oracle DB integration to ensure system stability.

Senior Systems Engineer at Infosys Ltd

May 1, 2013 - January 1, 2017

Created Power BI dashboards and reports for Hallmark Cards (USA) to visualize KPIs for multiple business units, including building measures, calculated columns, and relationships using DAX. Used Power BI features such as Spotlight and bookmarks in desktop. Developed and maintained mainframe-based applications, and created PDS libraries to manage Endevor elements with coding standards validation and job/event checks before production migration. Worked with IBM DB2 SQL, Oracle, and DataStage ETL processes for enterprise reporting. Performed unit/regression testing, resolved production defects, and provided knowledge transfer/training to a team of five.