DRemoteDatabricks Data Engineer
Context
A retail-focused data initiative on Microsoft Azure works with a wide range of retail datasets. The work centres on scalable data solutions built on Databricks, PySpark and Kafka streaming.
What you'll do
- Develop new end-to-end data engineering pipelines with change data capture on Databricks
- Enhance and optimize existing data workflows for performance and scalability
- Build streaming pipelines using Kafka and implement automation for pipeline operations
- Integrate data from multiple structured and unstructured sources into Delta Lake
Profile
- Extensive hands-on experience with Databricks and the ability to contribute from day one
- Strong proficiency in PySpark and SQL, Python is a plus
- Experience in Microsoft Azure cloud environments
- Knowledge of Delta Lake, distributed processing, Kafka and SCD Type 1 and Type 2 data modeling
- Familiarity with GitHub and CI/CD pipelines