MASTHAN VALIDATA ENGINEER

Partnered with Sunderland City Council stakeholders to define data requirements and KPIs, producing source-to-target mappings and metadata documentation. Designed and implemented a Microsoft Fabric lakehouse architecture using OneLake for unified storage and scalable ingestion/analytics. Built Gold-layer data models for council operational datasets (housing, revenues & benefits, environmental services), and developed PySpark + SQL notebooks orchestrated with pipelines for batch and incremental loads. Implemented OneLake shortcuts for trusted dataset sharing across workspaces without duplication, optimized performance via partitioning/compaction/tuning (improving join/aggregation times ~20–25%), and managed capacity/concurrency to improve pipeline stability. Applied governance using workspace roles, permissions, and secure sharing patterns. Built Power BI semantic models using DirectLake on Gold datasets for low-latency analytics. Used Fabric monitoring/lineage and variable libraries

Supported an insurance data platform processing policy, claims, customer, and pricing data used by actuarial, fraud, and reporting teams. Engineered PySpark-based transformation frameworks for multi-terabyte datasets with modular, reusable, parallel execution across domains. Developed Delta Live Tables pipelines with schema enforcement and automated data quality checks for consistent curated outputs. Tuned Spark workloads using partitioning, broadcast joins, caching, and shuffle optimization (improving critical pipelines ~25–30%). Managed Databricks compute with separation of dev and prod workloads and right-sized clusters to control costs. Used Databricks SQL endpoints for ad-hoc querying on Gold Delta tables, and configured event-driven ingestion via Azure Event Hub with Maven libraries. Integrated dbt with Databricks for modular SQL transformations and dimensional models on Delta Lake. Applied Unity Catalog governance (access control and lineage) and built feature engineering pipe

Supported an Allstate insurance data platform for policy, claims, customer, and pricing use cases for actuarial, fraud, and reporting teams. Developed PySpark transformation frameworks for multi-terabyte datasets with modular, reusable, parallel processing across domains. Built Delta Live Tables pipelines with schema enforcement and automated data quality checks for curated layers. Tuned Spark workloads using partitioning, broadcast joins, caching, and shuffle optimization to reduce execution times ~25–30%. Managed Databricks compute separation for dev vs production to control costs, and used Databricks SQL serverless querying for low-latency exploration on Gold tables. Implemented event-driven ingestion via Azure Event Hub and Maven libraries. Integrated dbt with Databricks for modular SQL transformations and dimensional models. Applied Unity Catalog governance for access control and lineage. Built feature engineering pipelines for pricing/fraud ML-ready datasets, and monitored jobs