Senior Data Platform Engineer with 10+ years of experience building data pipelines, ML infrastructure, and production systems. Proficient in Python, SQL, Spark, Airflow, Kubernetes, and cloud platforms (AWS, GCP). I bring a strong background in data modeling, workflow orchestration, and platform observability.

Abdullah Abid

Senior Data Platform Engineer with 10+ years of experience building data pipelines, ML infrastructure, and production systems. Proficient in Python, SQL, Spark, Airflow, Kubernetes, and cloud platforms (AWS, GCP). I bring a strong background in data modeling, workflow orchestration, and platform observability.

Available to hire

Senior Data Platform Engineer with 10+ years of experience building data pipelines, ML infrastructure, and production systems. Proficient in Python, SQL, Spark, Airflow, Kubernetes, and cloud platforms (AWS, GCP). I bring a strong background in data modeling, workflow orchestration, and platform observability.

See more

Experience Level

Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Expert
Intermediate
See more

Work Experience

Senior Data / AI Platform Engineer at Humanloop
January 1, 2022 - Present
Architected and owned backend data services for AI prompts, evaluation runs, feedback records, and model traces; redesigned async processing for outputs and production usage events to reduce latency; implemented vector search workflows with embeddings, Postgres, Redis; hardened workspace controls with RBAC, audit logging, encrypted secrets, tenant isolation, and GDPR‑aware data handling; partnered with product and research teams to turn evaluation workflows into stable APIs, dashboards, and data models; mentored engineers on Python services, data modeling, testing strategy, and production debugging.
Senior Data / ML Platform Engineer at Quantexa
April 1, 2019 - December 1, 2021
Architected and owned backend data services for AI prompt version evaluation runs, feedback records, and model traces; built Spark pipelines for entity resolution, transaction networks, customer relationships, and investigation datasets; developed Scala and Python jobs for ingestion, matching, enrichment, and analytic feature generation; improved fraud detection accuracy by 15%; tuned Spark jobs, partitions, and cluster resources; introduced Kafka-backed ingestion for near real-time data; standardized data quality checks for schema drift, duplicates, missing identifiers, and inconsistent attributes; supported ML feature pipelines for risk scoring, anomaly detection, and prioritized investigations; reduced incident response time by 25% and mentored data engineers on Python services, data modeling, testing strategy, and production debugging.
Data Platform Engineer at Sky
July 1, 2016 - March 1, 2019
Scaled analytics pipelines for viewing behavior, customer events, subscriptions, and content engagement (100TB+ daily); built PySpark and Airflow workflows for batch and streaming data from product and customer systems, ensuring data freshness and accuracy; produced curated datasets for analytics, marketing, retention, and product performance reporting; migrated legacy ETL jobs to Airflow DAGs with clear ownership, retries, and failure alerts, improving operational reliability.
Data Platform Engineer at Experian Nottingham
August 1, 2013 - June 1, 2016
Maintained regulated ETL workflows for credit, customer, fraud, and reference data used by analytics teams, ensuring compliance with data governance standards; wrote Python and SQL scripts for extraction, validation, transformation, and warehouse loading; supported Hadoop, Hive, and relational database workflows for large-scale data processing; added data quality checks for missing records, duplicate keys, invalid fields, and inconsistent identifiers, improving data quality by 20%; improved daily batch reliability with retry handling, structured logging, and clearer failure notifications; prepared production incident dashboards and operational handover notes to improve knowledge sharing; built a foundation in production data engineering, SQL tuning, and regulated data environments.

Education

BSc (Hons) Computer Science at University of Manchester
January 1, 2010 - January 1, 2013

Qualifications

BSc (Hons) Computer Science
January 11, 2030 - June 30, 2026

Industry Experience

Software & Internet, Professional Services