Data Engineer - Databricks/Python

Job Description :

• Architect and implement enterprise-grade Lakehouse solutions using Databricks.
• Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL).
• Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks.
• Implement and optimize Databricks components : Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows.
• Design large-scale data warehousing solutions with 3NF and dimensional modeling.
• Establish data governance, security, and data quality frameworks, including Unity Catalog.
• Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI).
• Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow).
• Drive CI/CD, DevOps practices, and performance optimization of Spark workloads.
• Provide technical leadership, mentor teams, and ensure successful delivery.
• Collaborate with stakeholders to translate business requirements into scalable solutions.

Ideal Candidate :

• Strong Databricks Architect Profile with end-to-end Lakehouse ownership.
• Must have 10 years of software engineering experience with at least 5 years in Data Engineering with hands-on exposure to Databricks and strong ownership of end-to-end data pipeline development.
• Must have at least 5 years of expertise across the Databricks ecosystem Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Workflows, Unity Catalog.
• Must have worked at architecture level, owning end-to-end design through deployment.
• Must have strong experience with Python and SQL for data processing and Apache Spark for performance tuning & scalability.
• Must have experience in large-scale data warehousing & advanced data modeling (3NF and dimensional) across batch and real-time systems.
• Must have at least a basic working understanding of how AI services or tools work.
• Must have strong stakeholder management & requirement-gathering experience with US or UK clients.
• Must come from a B2B IT services or IT consulting background.