Data Scientist

We are working with a crucial state government agency that provides vital protective and regulatory services, safeguarding children, adults, and the elderly. They are seeking experienced professionals to support data engineering, machine learning, and analytics initiatives involving large-scale data processing.

This agency is committed to leveraging cutting-edge technology to enhance their public service mission.

The Role

• Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks
• Implement ETL/ELT workflows for structured and unstructured data, optimizing Spark jobs for performance
• Build and maintain data models, schemas, and database structures supporting analytical and operational use cases
• Integrate Databricks solutions with Azure Data Factory and other Azure cloud services, including Azure Data Lake Storage
• Contribute to data governance, metadata management, data lineage, and data cataloging
• Implement data security measures, including encryption, access controls, auditing, and compliance with regulations

What You'll Need

• 4+ years of experience in ETL/ELT workflows, data modeling, data storage solutions (Azure Data Lake, data warehouses), data validation, and data governance
• 4+ years of experience with Python, R, and strong SQL querying and data manipulation
• 4+ years of experience with Microsoft Azure cloud platform, DevOps, CI/CD pipelines, and version control systems
• 3+ years designing and developing scalable data pipelines using Apache Spark on Databricks
• 3+ years optimizing Spark jobs and integrating Databricks with Azure Data Factory
• Strong understanding of Apache Spark architecture, RDDs, DataFrames, Spark SQL, and hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake

What's On Offer

• Opportunity to work on critical data initiatives for a significant state agency
• Engage with large-scale data processing, machine learning, and analytics
• Utilize advanced technologies like Databricks, Apache Spark, and Microsoft Azure
• Long-term contract with potential for extension

Apply via Haystack today!