INSPYR Solutions - Lead Data Engineer

Data Scientist is needed in Doraville, United States.

Client: INSPYR Solutions

Location: Doraville, GA

Contract: Temp-to-hire

Job Description

The Lead Data Engineer will architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment. This position involves designing and maintaining enterprise Lakehouse architecture with Delta Lake, enabling machine learning workflows, and leading data quality and access control frameworks. The role also includes collaborating with various teams to create scalable data solutions and ensuring production reliability of data and ML pipelines.

Requirements

Experience:

• 7+ years of experience in advanced data engineering with distributed compute technologies.
• Expert level Spark engineering (performance tuning, cluster configuration, optimization of large datasets).
• Hands-on experience with Lakehouse architectures including ACID transactions and governance frameworks.
• Deep proficiency in Python and SQL for large-scale data transformation.
• Experience supporting machine learning pipelines or model operationalization.
• Proven experience architecting cloud native data platforms (Azure, AWS, or GCP).
• Strong background integrating diverse, complex data sources at enterprise scale.
• Demonstrated ability to manage mission-critical production systems.

Core Tools:

• Databricks (Spark, Delta Lake, MLflow, Notebooks)
• Python & SQL
• Apache Spark (via Databricks)
• Delta Lake (for lakehouse architecture)
• Cloud Platforms: Azure, AWS, or GCP
• Cloud Storage: ADLS, S3, GCS
• Data Integration: Kafka or Event Hubs (streaming), Auto Loader (Databricks file ingestion), REST APIs
• AI/ML: MLflow, Hugging Face Transformers, LangChain / LlamaIndex
• LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
• DevOps: Git (GitHub, GitLab, Azure Repos), Databricks Repos, CI/CD (GitHub Actions, Azure DevOps)
• Security & Governance: Unity Catalog, RBAC

Additional Qualifications:

• Experience with distributed streaming frameworks (Kafka, Event Hubs, etc.).
• Experience building or supporting ML platforms or feature stores.
• Background in data security, compliance controls, or audit-ready governance.
• Experience automating data operations with CI/CD and infrastructure as code.

Additional Information

Title: Lead Data Engineer

Location: GA, Remote

Duration: Temp to perm

Compensation: $60/hr to $70/hr

Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.

Benefits Package: (EXCLUDE on perm placements)

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan
• …and much more!

About INSPYR Solutions:
INSPYR Solutions specializes in delivering technology and talent solutions aligned with business objectives. We offer a variety of professional services and strive to match the right solutions to the right opportunities while adhering to a commitment to quality and excellence.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. INSPYR Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.

Information collected through your application is subject to INSPYR Solutions’ Privacy Policy and AI and Automated Employment Decision Tool Policy. By submitting an application, you consent to being contacted by INSPYR Solutions through phone, email, or text.