Senior Data Engineer
Technology
Haystack1 month ago
Full timeOn-site
Job description
A fast-growing tech company that's disrupting the data and analytics space is seeking a skilled professional to build scalable, secure data pipelines and lakehouse solutions. This role is crucial for supporting advanced analytics, AI/ML initiatives, and handling sensitive healthcare data within a cutting-edge environment.
The Role
- Design and build robust data pipelines using Databricks, Delta Lake, Spark, Unity Catalog, and Workflows.
- Develop both batch and real-time data pipelines leveraging AWS services such as S3, Glue, Redshift, Athena, Lambda, Step Functions, and Kinesis.
- Optimize lakehouse architecture, create efficient data models, and manage ETL/ELT processes.
- Support machine learning and artificial intelligence initiatives, including the development of feature stores and LLM/RAG data pipelines.
- Implement stringent data security, governance, and HIPAA-compliant data handling practices with access controls and auditing.
- Collaborate closely with Data Science, Engineering, Product, and business teams to drive data-driven solutions.
What You'll Need
- 8+ years of hands-on experience in Data Engineering.
- 4+ years of experience with cloud-based data lake/lakehouse architectures.
- Deep expertise with Databricks, Apache Spark/PySpark, Delta Lake, Unity Catalog, and Workflows.
- Strong command of AWS, specifically S3 and various data/analytics services.
- Proficiency in Python and SQL.
- Experience with Airflow/dbt, Kafka/Kinesis, CI/CD, Docker/Kubernetes, and data quality practices.
What's On Offer
- The opportunity to work with cutting-edge technologies like Databricks and AWS.
- A role central to supporting advanced analytics, AI/ML, and healthcare data initiatives.
- A collaborative environment with opportunities to mentor junior engineers and establish engineering best practices.
- Involvement in projects requiring HIPAA-compliant data handling, enhancing your expertise in regulated data environments.
Apply via Haystack today!
Keywords
analyticsdata-analyticsdata-pipelinedata-lakehouseadvanced-analyticsartificial-intelligencemachine-learninghealth-careenvironment-health-and-safety-hsseecology-environmentplanning-and-designvisual-art-designproduct-development-and-designazure-databricksdatabricksdelta-lakesparkunity-3dunity-and-unreal-software-and-trainingtime-and-attendance
Interested in this role?