Senior Data Engineer
Context
A contract role in Sydney for a senior data engineer to design and deliver production-grade data platforms across Azure, Databricks and on-premises environments. The work covers batch and near-real-time pipelines and works alongside architecture, business, governance and testing teams.
What you'll do
- Design and develop complex, production-grade batch and near-real-time data pipelines on Azure, Databricks and on-premises platforms
- Design end-to-end data solutions covering ingestion, transformation, data modelling, storage and consumption
- Establish reusable data engineering patterns, standards and best practices, and implement CI/CD and automated testing
- Monitor pipeline performance, troubleshoot production issues and provide mentoring and code reviews for data engineers
Profile
- 7+ years of experience in data engineering, data warehousing or related technologies
- Strong hands-on experience with Azure Databricks (minimum 2 years)
- Advanced Python and SQL skills, with Apache Spark and PySpark for distributed processing
- Experience with ETL/ELT, data modelling, orchestration tools such as Airflow or Control-M, and JSON, XML and Avro formats