ARemoteDatabricks Engineer, Big Data
Context
A US state government agency that relies on large-scale data processing for its data engineering, machine learning and analytics initiatives. The role is remote and open to Texas residents only.
What you'll do
- Design and develop scalable data pipelines with Apache Spark on Databricks
- Implement ETL and ELT workflows for structured and unstructured data
- Optimize Spark jobs for performance and cost efficiency
- Integrate Databricks with Azure Data Factory and Azure Data Lake Storage
- Automate deployments with CI/CD pipelines and ensure data quality and security with Unity Catalog and Delta Lake
Profile
- Deep understanding of Apache Spark architecture, RDDs, DataFrames and Spark SQL
- Hands-on experience with Databricks notebooks, clusters, jobs and Delta Lake
- Proficiency in Python and R, and strong SQL skills
- Experience with the Azure cloud platform, DevOps and version control systems
- Knowledge of ML libraries such as MLflow, Scikit-learn and TensorFlow