Data Engineer

Databricks Data Engineer

Location: Quincy, MA (Onsite)

About the Role

We are seeking an experienced Databricks Data Engineer to support a critical initiative focused on building scalable, high-performance data solutions across a broad range of datasets. This role requires a highly skilled engineer who can contribute immediately, developing new data pipelines while enhancing and optimizing existing data workflows.

The ideal candidate has deep expertise in Databricks, Azure, PySpark, CDC, and Kafka, along with a strong track record of building enterprise-grade data engineering solutions in large-scale environments.

Responsibilities

• Design, develop, and optimize end-to-end ETL/ELT pipelines using Databricks and PySpark.
• Build and maintain scalable data solutions within the Microsoft Azure ecosystem.
• Develop robust Change Data Capture (CDC) pipelines to support incremental and near real-time data processing.
• Enhance and maintain existing data pipelines while contributing to new development initiatives.
• Integrate data from multiple structured and unstructured sources, including:
• SQL Server and JDBC databases
• JSON, XML, and flat files
• Kafka streams
• APIs and MQ platforms
• Unity Catalog volumes and enterprise systems
• Implement data quality frameworks, validation rules, and error-handling processes.
• Design and support dimensional data models, including SCD Type 1 and Type 2 implementations.
• Optimize data pipelines for scalability, reliability, and performance.
• Automate workflows and deployments using Databricks Workflows, GitHub, and CI/CD best practices.
• Ensure compliance with data governance, architecture, and security standards.
• Partner with architects, product owners, developers, and business stakeholders to deliver high-quality solutions.
• Participate in Agile/Scrum ceremonies and contribute to continuous process improvement.

Required Qualifications

• Extensive hands-on experience with Databricks in enterprise environments.
• Proven experience building end-to-end data engineering pipelines with CDC (Change Data Capture) functionality.
• Strong proficiency in PySpark and SQL.
• Experience working within Microsoft Azure cloud environments.
• Strong knowledge of:
• Delta Lake
• Apache Spark
• Kafka and streaming technologies
• Distributed data processing
• Data modeling concepts, including SCD Type 1 and Type 2
• Experience integrating data from a variety of structured and unstructured data sources.
• Experience with GitHub and CI/CD deployment processes.
• Strong understanding of data architecture, performance tuning, and scalable pipeline design.
• Excellent analytical and problem-solving skills.

Preferred Qualifications

• Python development experience.
• Retail or large enterprise data platform experience.
• Experience with real-time and event-driven architectures.
• Familiarity with enterprise data quality and governance practices.

What You'll Accomplish

Within your first 60 days, you will:

• Quickly ramp up on the Retail Spine ecosystem and architecture.
• Contribute to both new development efforts and existing pipeline enhancements.
• Deliver scalable, reliable, and efficient data engineering solutions.
• Identify opportunities to improve data quality, automation, and operational efficiency.
• Become an active contributor within Agile teams and cross-functional initiatives.

Key Competencies

• Strong ownership and accountability
• Proactive, self-driven problem solver
• Ability to work effectively in complex data environments
• Excellent collaboration and communication skills
• Commitment to quality, scalability, and continuous improvement

Keywords: Databricks, Azure, PySpark, Apache Spark, Data Engineer, CDC, Change Data Capture, Delta Lake, Kafka, SQL, ETL, ELT, Data Pipelines, Streaming Data, Azure Data Platform, Data Modeling, SCD Type 1, SCD Type 2, GitHub, CI/CD, Quincy MA.