KRemoteDatabricks Engineer with Test Data Management
Context
Remote contract role for a W2-only engagement lasting 6 to 12+ months, with a required day one onsite visit in Chicago, Illinois, to pick up equipment. The work centers on data engineering, data governance and test data management on Databricks.
What you'll do
- Build and maintain PySpark and Python data pipelines on Databricks using Delta Lake and a medallion architecture
- Implement Unity Catalog governance, including role-based access control and metadata management
- Set up test data management with data masking, synthetic data generation and PII deidentification
- Track data lineage and support data classification and observability across pipelines
Profile
- Experienced Databricks engineer with strong data engineering skills in PySpark, Python and Delta Lake
- Solid knowledge of data governance and test data management, including data masking and deidentification
- Hands-on experience with Unity Catalog and lakehouse architecture
- Cloud experience with Azure Data Factory, Azure DevOps, Azure Synapse or AWS; knowledge of GDPR or HIPAA is a plus