ASan Francisco, CASenior Databricks Solution Architect
Context
A contract role for a senior architect to design and deliver a Databricks lakehouse data platform for a client. The role is based in San Francisco, CA with remote work considered, on Pacific Time.
What you'll do
- Design the Databricks lakehouse architecture and Delta Lake data layers
- Build and optimize ETL/ELT pipelines and Structured Streaming workloads with Spark and PySpark
- Set up governance with Unity Catalog and manage data workflows with Lakeflow and MLflow
- Tune Spark performance and implement CI/CD and MLOps practices
- Advise the client on enterprise data architecture
Profile
- 12 to 15+ years of experience in data and solution architecture
- Strong hands-on experience with Databricks, Apache Spark, PySpark and Spark SQL
- Experience with AWS, Azure or GCP cloud platforms
- Experience with ETL/ELT pipelines, DevOps, CI/CD and MLOps
- Databricks Certified Data Engineer Professional certification is a plus