Bitful Consulting - Senior Data Engineer

Data Engineer is needed in Pittsburgh, United States.

Client: Bitful Consulting

Location: Pittsburgh, PA

Contract: Contract

Job Description

Build and maintain scalable data pipelines and reusable datasets in a Databricks lakehouse environment to support reporting, visualization, machine learning and data-driven decision-making across finance, sales, supply chain, marketing and other business functions.

Key responsibilities include:

• Develop and maintain scalable data pipelines and data products.
• Ingest, validate, clean and transform structured and unstructured data using Python, PySpark and SQL.
• Build data models and reporting datasets for Tableau and other analytics tools.
• Integrate existing data warehouses and reporting systems with modern cloud analytics platforms.
• Improve pipeline reliability, data quality and processing performance.
• Implement automated testing, monitoring, documentation and CI/CD practices.
• Collaborate with analysts, architects, developers and business stakeholders to deliver effective data solutions.
• Provide technical guidance, conduct code reviews and mentor other data engineers.

Requirements

• At least 7 years of data engineering experience, including developing and supporting production data pipelines.
• Strong SQL skills and knowledge of data modelling, database design and data warehousing.
• Hands-on experience with Databricks, Apache Spark and cloud data platforms.
• Strong proficiency in Python and PySpark.
• Experience developing ETL/ELT pipelines and integrating data from multiple source systems.
• Experience creating reliable datasets for Tableau or comparable business intelligence tools.
• Familiarity with source control, automated testing and CI/CD practices.
• Ability to troubleshoot complex data issues and optimise pipeline performance.
• Strong communication and collaboration skills, with experience working with technical teams and business stakeholders.

Preferred Experience and Qualifications

• Experience with Spotfire, Oracle, Informatica, R or data virtualisation.
• Databricks certification.