AOttawa, ONSenior Data Engineer, Databricks and AWS
Context
An enterprise client is building scalable data platforms to support analytics, AI and business intelligence. The role is hybrid, based in Ottawa, for a contract of 6 months or more with possible extensions.
What you'll do
- Build and maintain scalable enterprise data platforms on Databricks and AWS
- Develop ETL and ELT pipelines with PySpark, Spark SQL and Python, and optimize their performance
- Implement Medallion architecture (Bronze, Silver, Gold) using Delta Lake
- Set up Unity Catalog governance and data lineage, along with data quality, orchestration, monitoring and CI/CD
- Design enterprise data models and reusable data products
Profile
- Strong hands-on experience with Databricks, PySpark, Spark SQL, Python and SQL
- Expertise in Delta Lake and Medallion architecture
- Strong AWS experience with S3, Glue, Lambda and Redshift
- Experience with Unity Catalog, governance and data lineage; exposure to Databricks Genie is an advantage