ARemoteDatabricks Engineer, Test Data Management
Context
A US client needs a Databricks data engineer for a 6+ month remote contract focused on test data management. Day 1 onsite in Chicago, IL is required for equipment pickup.
What you'll do
- Design and build scalable data ingestion and transformation pipelines on Databricks
- Implement Medallion Architecture (Bronze, Silver, Gold) layers with Delta Lake
- Manage data governance, access control (RBAC), metadata and lineage with Unity Catalog
- Support test data management, including data masking, de-identification and synthetic data generation
Profile
- 8+ years of experience in data engineering or data architecture
- 3+ years of hands-on Databricks experience
- Strong skills in PySpark, Python, Delta Lake and Unity Catalog
- Experience with Azure and/or AWS cloud platforms
- Preferred: PII, GDPR, CCPA or HIPAA compliance experience, Terraform, Azure DevOps or GitHub Actions