Skip to main content

Senior Data Engineer

Technology
Haystack
1 month ago
Full timeOn-site

Job description

A fast-growing tech company that's disrupting the data and analytics space is seeking a skilled professional to build scalable, secure data pipelines and lakehouse solutions. This role is crucial for supporting advanced analytics, AI/ML initiatives, and handling sensitive healthcare data within a cutting-edge environment.

The Role

  • Design and build robust data pipelines using Databricks, Delta Lake, Spark, Unity Catalog, and Workflows.
  • Develop both batch and real-time data pipelines leveraging AWS services such as S3, Glue, Redshift, Athena, Lambda, Step Functions, and Kinesis.
  • Optimize lakehouse architecture, create efficient data models, and manage ETL/ELT processes.
  • Support machine learning and artificial intelligence initiatives, including the development of feature stores and LLM/RAG data pipelines.
  • Implement stringent data security, governance, and HIPAA-compliant data handling practices with access controls and auditing.
  • Collaborate closely with Data Science, Engineering, Product, and business teams to drive data-driven solutions.

What You'll Need

  • 8+ years of hands-on experience in Data Engineering.
  • 4+ years of experience with cloud-based data lake/lakehouse architectures.
  • Deep expertise with Databricks, Apache Spark/PySpark, Delta Lake, Unity Catalog, and Workflows.
  • Strong command of AWS, specifically S3 and various data/analytics services.
  • Proficiency in Python and SQL.
  • Experience with Airflow/dbt, Kafka/Kinesis, CI/CD, Docker/Kubernetes, and data quality practices.

What's On Offer

  • The opportunity to work with cutting-edge technologies like Databricks and AWS.
  • A role central to supporting advanced analytics, AI/ML, and healthcare data initiatives.
  • A collaborative environment with opportunities to mentor junior engineers and establish engineering best practices.
  • Involvement in projects requiring HIPAA-compliant data handling, enhancing your expertise in regulated data environments.

Apply via Haystack today!

Keywords
analyticsdata-analyticsdata-pipelinedata-lakehouseadvanced-analyticsartificial-intelligencemachine-learninghealth-careenvironment-health-and-safety-hsseecology-environmentplanning-and-designvisual-art-designproduct-development-and-designazure-databricksdatabricksdelta-lakesparkunity-3dunity-and-unreal-software-and-trainingtime-and-attendance

Interested in this role?