Skip to main content

Senior Data Engineer

Technology
Haystack
1 month ago
Full timeOn-site

Job description

We are working with a pioneering technology company that empowers businesses with cutting-edge data solutions and advanced analytics, driving innovation and efficiency across various industries, including healthcare.

The Role

  • Design and build scalable, secure data pipelines and lakehouse solutions using Databricks, Delta Lake, Spark, Unity Catalog, and Workflows.
  • Develop both batch and real-time data pipelines leveraging AWS services such as S3, Glue, Redshift, Athena, Lambda, Step Functions, and Kinesis.
  • Optimize lakehouse architecture, data models, and ETL/ELT processes to support analytics and AI/ML initiatives.
  • Implement robust data security, governance, and HIPAA-compliant data handling practices, including access controls and auditing.
  • Collaborate closely with Data Science, Engineering, Product, and business teams to deliver impactful data solutions.
  • Mentor junior engineers, review code, and establish engineering best practices.

What You'll Need

  • 8+ years of hands-on experience in Data Engineering, with a strong focus on cloud-based data lake/lakehouse architectures.
  • Deep expertise in Databricks, Apache Spark/PySpark, Delta Lake, Unity Catalog, and Workflows.
  • Strong proficiency with AWS data and analytics services, including S3, Glue, Redshift, Athena, Lambda, Step Functions, and Kinesis.
  • Excellent Python and SQL skills, with experience in tools like Airflow/dbt and Kafka/Kinesis.
  • Proven experience with CI/CD, Docker/Kubernetes, and implementing data quality practices.
  • Required experience in healthcare or regulated-data environments, ensuring HIPAA compliance.

What's On Offer

  • Opportunity to work on challenging projects supporting advanced analytics and AI/ML initiatives.
  • A collaborative environment focused on innovation and engineering best practices.
  • Significant impact on the development of scalable and secure data platforms.
  • Contribute to mentoring and technical leadership within the engineering team.

Apply via Haystack today!

Keywords
planning-and-designvisual-art-designproduct-development-and-designdata-pipelinedata-lakehouseazure-databricksdatabricksdelta-lakesparkunity-3dunity-and-unreal-software-and-trainingtime-and-attendancereal-time-dataamazon-web-servicesamazon-s3aws-glueamazon-redshiftamazon-athenaaws-lambdaaws-step-functions

Interested in this role?