CAtlanta, GASenior Data Engineer (Azure / PySpark)
Context
The role sits within the alternative investments team of a global fund services technology group. The work moves legacy SSIS-based data integrations to scalable, cloud-native PySpark ETL and ELT pipelines.
What you'll do
- Design and develop PySpark-based ETL and ELT pipelines for cloud data platforms
- Lead the migration of legacy SSIS packages and on-premises workloads to Azure
- Build data-processing workflows on Azure Synapse Analytics, Azure Data Lake Storage, Azure Data Factory, Databricks and Kubernetes
- Define and execute migration strategies while keeping reliability, data quality and performance
- Apply CI/CD, automated testing, observability and performance tuning practices, and work with product, security and cloud architecture teams on highly available solutions
Profile
- Strong hands-on experience building complex financial data-processing frameworks in large enterprise environments
- Solid operational and development experience with SSIS and third-party data-processing tools
- Strong knowledge of Apache Spark and PySpark, and expertise with Microsoft Azure data services including Synapse, Data Lake Storage, Data Factory, Azure SQL Database, Purview and Machine Learning
- Familiarity with Databricks, Kubernetes, Kafka, MQ and AWS SQS, plus knowledge of IAM, RBAC, OAuth, TLS and JWT security concepts
- Experience with cloud migration, data governance, lineage and compliance; experience with financial accounting platforms is a plus