Hudson Australia - Senior Data Engineer

Data Scientist is needed in Melbourne, Australia.

Client: Hudson Australia

Location: Melbourne VIC, Australia

Contract: Contractor

Job Description

The project will deliver a foundational data and analytics platform, including a Microsoft Azure based data platform with Databricks as a key component. This involves establishing the platform, ingesting SQL CDC and REST API data feeds into a data lake, and performing analysis and design tasks. The candidate will be responsible for overall planning, analysis, preparation, execution, and reporting of test activities, ensuring the quality of data products.

Requirements & Skills

• Demonstrable hands-on experience with Azure Data services such as Azure Data Factory, Azure Storage Account, Azure Data Lake Storage, Blob Storage, Cosmos DB, and Azure Functions.
• Demonstrable hands-on experience with Azure Databricks, including Medallion Architecture, Unity Catalog, Compute Clusters, etc.
• Deep technical skills across Apache Spark, Spark SQL, and ETL, with experience with REST APIs and SQL CDC.
• Hands-on experience with data warehousing technologies such as Oracle, Teradata, SAP BW, etc.
• Expert SQL skills.
• Experience in developing data ingestion, data processing, and analytical pipelines for various data solutions.
• Experience in Data Engineering and Architecture supporting complex integration projects.
• Excellent communication skills.

Additional Information

The project will enable the delivery of business use cases as resources and future investment allows. The candidate will also be involved in data quality checking, ad hoc testing, functionality testing of Power BI reports, designing and developing data pipelines and ETL processes using Databricks, optimizing data models and database performance, and extending the existing Databricks implementation per the Medallion Framework.