MPlano, TXSenior Data Engineer (Python, PySpark, AWS)
Context
A US employer is hiring a senior data engineer on a W2 contract with a right-to-hire option. The role is onsite five days a week in Plano, Texas, and focuses on large-scale data processing and production pipelines.
What you'll do
- Build and maintain ETL and ELT pipelines and production data pipelines using Python and PySpark
- Work with AWS services such as S3, EMR and Lambda to process big data
- Provision and manage infrastructure with Terraform
- Integrate data flows through Kafka, MQ messaging and REST APIs and orchestrate jobs with Airflow / Control-M
- Optimize big data processing and pipeline performance
Profile
- Senior-level experience in Python and PySpark
- Hands-on AWS experience with S3, EMR and Lambda
- Experience with ETL/ELT, Terraform, Kafka, Cassandra and Airflow or Control-M
- Preferred: Databricks and machine learning workflow experience
- Authorized to work in the US (USC, GC or GC EAD)