Insight Advisors Private Limited - Data Engineer (Microsoft Fabric)

Data Scientist is needed in Pune, India.

Client: Insight Advisors Private Limited

Location: Pune, MH, in

Contract: Contract

Job Description

Data Engineer (Microsoft Fabric – ETL & Migration) located in Pune Aundh is needed to support enterprise data platform modernization by migrating legacy SQL-based systems to Microsoft Fabric. This role focuses on ETL/ELT pipeline development, converting SQL stored procedures into Fabric Spark notebooks, and delivering production-grade data layers for analytics and reporting.

Requirements

Experience: 3–6 years of experience in data engineering, with strong exposure to cloud data platforms and migration projects (AWS to Azure/Fabric preferred).

Qualification: B.E./B.Tech/MCA/Relevant Degree in Computer Science, IT, Data Engineering, or related field.

Contract: 6 Month On Contract.

Key Responsibilities

• Convert SQL stored procedures and legacy transformation logic into PySpark / Spark SQL notebooks in Microsoft Fabric.
• Design and build end-to-end ETL/ELT pipelines using Fabric Data Factory, Dataflows Gen2, and Fabric pipelines.
• Develop Gold-layer data marts optimized for Power BI dashboards and enterprise analytics.
• Implement data ingestion and transformation pipelines across Bronze, Silver, and Gold layers.
• Ensure data quality through validation, reconciliation, and comparison with legacy systems.
• Optimize data processing performance, query execution, and pipeline efficiency.
• Monitor and troubleshoot ETL pipelines, notebook runs, and data refresh failures.
• Implement CI/CD pipelines using Azure DevOps and Git for deployment across Dev, UAT, and Production environments.
• Collaborate with BI developers, business stakeholders, and ERP teams to translate business requirements into scalable data solutions.

Required Skills & Qualifications

• Strong experience in ETL/ELT pipeline development and data engineering.
• Hands-on experience with Microsoft Fabric (Data Factory, Dataflows Gen2, Notebooks, OneLake).
• Strong expertise in PySpark and Spark SQL.
• Advanced SQL skills including stored procedures and query optimization.
• Strong understanding of data modeling (star schema, dimensional modeling).
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Understanding of medallion architecture (Bronze, Silver, Gold layers).

Key Success Metrics

• Successful conversion of SQL workloads into Fabric notebooks.
• Stable and scalable ETL pipelines with minimal failures.
• High-quality, business-ready datasets for reporting.
• Improved pipeline performance and reduced processing time.
• Reliable production data platform supporting enterprise analytics.