Varun Sharm a

Designed and developed an end-to-end data processing pipeline in Microsoft Fabric to ingest and process incoming ZIP files, move raw files to the Bronze layer, extract embedded text files, and load data into Lakehouse tables in the Collect layer. Implemented transformation logic from Collect to Staging including name normalization, trimming, character cleaning, and prefix/suffix extraction using lookup tables. Integrated an external Party Resolution API to map normalized person names and organization fields to Party Perm IDs and Person-in-Role IDs. Applied data governance principles using RBAC, least-privilege access, and controlled data access across lakehouse environments. Collaborated with DevOps to create CICD pipelines on GitLab for deployment across Fabric workspaces. Evaluated workload performance and resource/cost considerations using Fabric capacity guidance. Also built an ESG Data Mart on a star-schema approach in Databricks, and worked on ADF pipelines and reconciliations fo

Designed and developed an end-to-end data processing pipeline in Microsoft Fabric for ingesting and processing private company office and director data. Implemented PySpark-based pipelines and Fabric notebooks to handle incoming ZIP files, move raw files into a Bronze layer, extract embedded text files, and ingest into lakehouse tables in the Collected layer. Built transformation logic to move data from Collected to Staging, including name normalization, trimming, character cleaning, and prefix/suffix extraction using lookup tables. Integrated an external Party Resolution API to identify Party Perm IDs and Person-in-Role IDs using normalized person names and organization details. Collaborated with DevOps to create a full CI/CD pipeline on GitLab for deployment across Fabric workspaces. Applied governance principles such as RBAC, least-privilege access, and controlled data access across lakehouse environments. Evaluated workload concurrency, resource utilization, and performance/cost co

Worked with data scientists and engineering teams to build member feature generation using PySpark and DataBricks for model training, then optimized the feature store workflow to reduce feature generation time from ~35 minutes to ~10 minutes using partitioning, caching, and avoiding shuffle operations. Built reusable feature restoration pipelines and implemented unit testing in line with coding standards using Pytest. Developed an end-to-end data marketplace for Bayer using Python FastAPI with Publisher/Admin/Consumer APIs to manage dataset and artifact publishing, review, approval, and access workflows. Implemented approval-based publishing, where admins review submissions while consumers discover approved datasets and access them through workbenches. Created a Snowflake proof of concept to ingest CSVs into Snowflake tables and validate using SQL. Built supporting schemas, tables, and virtual warehouses for loading and transforming data. Additionally led multiple data platform efforts