Software Resources, Inc. - Senior Data Engineer

Client: Software Resources, Inc.

Location: Glendale, CA

Contract: Contract

Job Description

Software Resources is seeking a Senior Data Engineer for a major corporation in Glendale, California. This role supports enterprise data engineering and data platform modernization initiatives, with responsibility for designing, building, optimizing, and operating scalable data solutions.

The Senior Data Engineer will own and support the Core Data platform on Databricks, developing reliable batch and streaming Spark pipelines, implementing platform governance, and providing solution architecture across AWS, Kubernetes, and Airflow. The role involves close collaboration with engineers, architects, product managers, and business stakeholders.

• Manage Databricks governance, including Unity Catalog, access controls, lineage, data discovery, and privacy tools.
• Design, develop, test, deploy, and maintain batch and streaming data pipelines using PySpark, Scala, SQL, Python, and/or Kotlin.
• Gather stakeholder requirements and translate them into scalable data platform solutions.
• Use Databricks APIs, SDKs, and CLI tools to automate updates, audit platform activity, diagnose errors, and troubleshoot integrations.
• Explain Spark architecture and pipeline behavior, identify root causes, and recommend solutions.
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow/MWAA.
• Build and maintain Kubernetes containers and containerized utilities supporting data platform services.
• Troubleshoot networking, connectivity, and cross-platform integration issues.
• Administer the data platform by managing access, monitoring resource utilization and platform health, controlling costs, and evaluating stakeholder requests.
• Implement infrastructure automation, CI/CD practices, and deployment standards using Terraform and Git-based workflows.
• Develop data quality checks, monitoring, logging, and documentation to support pipeline reliability and governance.
• Collaborate with cross-functional engineering and product teams and participate in Agile/Scrum ceremonies.
• Optimize data processing environments and scalable ETL/ELT pipelines integrating structured and unstructured data sources.

Requirements

• At least 5 years of experience developing and operating large-scale data pipelines and data engineering solutions.
• Strong hands-on experience with Databricks and Apache Spark, including batch and streaming pipelines.
• Advanced Python development skills and proficiency in at least one additional language or tool, such as Scala, Kotlin, or SQL-based pipeline tooling.
• Experience with Terraform and Infrastructure as Code.
• Strong understanding of Spark architecture, including executors, stages, partitioning, shuffles, and performance tuning.
• Proficiency with Databricks APIs, SDKs, and CLI tools for automation, governance, auditing, and troubleshooting.
• Advanced SQL skills, including query and performance tuning.
• Production experience with Airflow, preferably AWS Managed Workflows for Apache Airflow (MWAA).
• Experience managing Databricks governance, including Unity Catalog, ACLs, lineage, and access provisioning.
• Experience designing and optimizing scalable ETL/ELT pipelines.
• Experience with AWS services, including compute, storage, networking, and IAM.
• Proficiency with Docker and Kubernetes for containerized data platform services.
• Working knowledge of networking concepts and cross-platform connectivity troubleshooting.
• Familiarity with Snowflake or comparable data platform technologies.
• Experience implementing CI/CD, DevOps, data quality, monitoring, and logging practices.
• Ability to participate in architecture and system design discussions.
• Strong analytical, problem-solving, communication, and stakeholder-management skills.
• Familiarity with Agile and Scrum methodologies.
• Bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent work experience. A master’s degree is a plus.

Additional Information

• Location: Glendale, CA
• Work Arrangement: Hybrid; two days onsite per week, with candidates able to select their onsite days.
• Duration: 12+ months
• Pay Rate: $90–$93 per hour, depending on experience.
• Required Technologies: Databricks, Python, and Terraform.
• Preferred Technology: AWS.

About Software Resources

Software Resources is a staffing partner with more than 30 years of experience specializing in technology, creative and marketing, finance, and accounting placements. The company connects skilled professionals with organizations nationwide.

Available benefits may include medical, dental, and vision coverage; a 401(k) with company match; short-term disability; and life insurance with AD&D coverage.