I am a results-driven software engineer specializing in Python, Django, Scrapy, React, PostgreSQL, Databricks, and PySpark. I design scalable scraping frameworks, build robust data pipelines, implement SCD Type 2 for historical tracking, and optimize workflows for large-scale data ingestion across 2000+ scrapers. I thrive on turning complex scraping challenges into reliable production systems with efficient Git-driven ticket workflows and proactive proxy management. I also bring a strong background in machine learning, dashboards, and real-time data processing. I have led small engineering teams, improved ticket velocity, and delivered end-to-end solutions—from crawling architectures to data visualization dashboards, including cricket data analytics. I am passionate about data-driven decision making and mentoring teammates to ship high-quality, production-ready software.

Habib ur Rehman

I am a results-driven software engineer specializing in Python, Django, Scrapy, React, PostgreSQL, Databricks, and PySpark. I design scalable scraping frameworks, build robust data pipelines, implement SCD Type 2 for historical tracking, and optimize workflows for large-scale data ingestion across 2000+ scrapers. I thrive on turning complex scraping challenges into reliable production systems with efficient Git-driven ticket workflows and proactive proxy management. I also bring a strong background in machine learning, dashboards, and real-time data processing. I have led small engineering teams, improved ticket velocity, and delivered end-to-end solutions—from crawling architectures to data visualization dashboards, including cricket data analytics. I am passionate about data-driven decision making and mentoring teammates to ship high-quality, production-ready software.

Available to hire

I am a results-driven software engineer specializing in Python, Django, Scrapy, React, PostgreSQL, Databricks, and PySpark. I design scalable scraping frameworks, build robust data pipelines, implement SCD Type 2 for historical tracking, and optimize workflows for large-scale data ingestion across 2000+ scrapers. I thrive on turning complex scraping challenges into reliable production systems with efficient Git-driven ticket workflows and proactive proxy management.

I also bring a strong background in machine learning, dashboards, and real-time data processing. I have led small engineering teams, improved ticket velocity, and delivered end-to-end solutions—from crawling architectures to data visualization dashboards, including cricket data analytics. I am passionate about data-driven decision making and mentoring teammates to ship high-quality, production-ready software.

See more

Experience Level

Expert
Expert
Expert
Expert
Expert
Intermediate
Intermediate
Intermediate

Language

English
Advanced
Urdu
Fluent

Work Experience

Software Engineer at Arbisoft
August 8, 2022 - Present
Managed and developed 2000 Scrapy spiders, using efficient Git branching and ticket workflows. Built highly optimized category + listing crawling flows using spider_idle logic. Designed mini-item category processing and improved mapping performance. Worked on proxy rotation, DLR management, gap time reduction, and large-scale scraping reliability. Improved team productivity handling 100–120 bug tickets while simultaneously delivering new spiders for new clients. Contributed to a crawling framework with request prioritization, callbacks, errbacks, and size request logic.

Education

Add your educational history here.

Qualifications

Bachelor's degree in Computer Science
January 11, 2030 - December 3, 2025
Databricks Certified Data Engineer Associate
January 11, 2030 - December 3, 2025
Databricks Certified Data Engineer Professional
January 11, 2030 - December 3, 2025
dbt Fundamentals Certification
January 11, 2030 - December 3, 2025

Industry Experience

Software & Internet, Professional Services, Media & Entertainment

Experience Level

Expert
Expert
Expert
Expert
Expert
Intermediate
Intermediate
Intermediate