Lead Data Engineer

We're working with a leading global specialty insurance firm and wholesale brokerage disruptor on this exciting opportunity. We are seeking a powerhouse technical leader to architect and scale sophisticated data environments within the Azure ecosystem, transforming raw data into high-value business intelligence.

As the Lead Data Engineer, you will own the Databricks strategy and lead a high-performing team to build resilient, real-time data pipelines. This is a high-impact role where you will bridge the gap between complex engineering—utilizing Delta Lake, Azure Data Factory, and PySpark—and strategic business objectives in a fast-paced, remote-friendly environment.

• *The Role
• Lead the design and implementation of scalable data pipelines using Azure Data Factory and Databricks notebooks for both batch and real-time big data processing.
• Architect and manage advanced data storage solutions using Delta Parquet files in Azure Data Lake Storage (ADLS), optimizing for maximum query performance and cost-efficiency.
• Drive data excellence by integrating Profisee Master Data Management (MDM) and implementing Change Data Capture (CDC) for real-time data synchronization.
• Establish rigorous data governance frameworks using Azure Purview for automated cataloging, classification, and lineage tracking.
• Mentor and coach junior engineers while leading the charge on CI/CD pipeline automation via Azure DevOps and optimizing complex SQL Server environments.
• *What You'll Need
• 10+ years of deep Data Engineering experience (or 8 years with a Master’s degree) with a proven track record of technical leadership.
• 6+ years of hands-on expertise specifically within Databricks and Big Data analytics environments.
• Mastery of the Azure Data Stack, including Azure Data Factory (ADF), SQL Server Management, and Delta Parquet File Management.
• Advanced proficiency in SQL Server Integration Services (SSIS), SSRS, and Tableau for high-level business intelligence reporting.
• Strong background in automated deployment, process optimization, and working with Profisee MDM and Azure Purview tools.
• *What's On Offer
• Highly competitive salary range of $112,000 - $140,000 based on experience and impact.
• Remote-friendly flexibility with a focus on work-life balance and physical, financial, and emotional well-being.
• Comprehensive "Best-in-Class" benefits package including paid parental leave, mental health services, and generous PTO.
• Opportunity to work at the forefront of insurance technology with a focus on innovation and professional growth.
• *Apply via Haystack today!