Architect and implement enterprise-grade Lakehouse solutions using Databricks.
Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL).
Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks.
Implement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows.
Design large-scale data warehousing solutions with 3NF and dimensional modeling.
Establish data governance, security, and data quality frameworks, including Unity Catalog.
Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI).
Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow).
Drive CI/CD, DevOps practices, and performance optimization of Spark workloads.
Provide technical leadership, mentor teams, and ensure successful delivery.
Collaborate with stakeholders to translate business requirements into scalable solutions.
Ideal Candidate :
Strong Databricks Architect Profile with end-to-end Lakehouse ownership.
Mandatory (Experience 1) : Must have 10+ years of software engineering experience with at least 5+ years in Data Engineering with hands on exposure to Databricks and strong ownership of end-to-end data pipeline development.
Mandatory (Experience 2) : Must have at least 5+ years of expertise across the Databricks ecosystem Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Workflows, Unity Catalog.
Mandatory (Tech skill 1) : Must have worked at architecture level, owning end-to-end design through deployment.
Mandatory (Tech skill 2) : Must have strong experience with Python and SQL for data processing and Apache Spark for performance tuning & scalability.
Mandatory (Tech skill 3) : Must have experience in large-scale data warehousing & advanced data modeling (3NF and dimensional) across batch and real-time systems.
Mandatory (AI Exposure) : Must have at least a basic working understanding of how AI services or tools work.
Mandatory (Communication) : Must have strong stakeholder management & requirement-gathering experience with US or UK clients.
Mandatory (Company) : Must come from a B2B IT services or IT consulting background.
Mandatory (Note 1) : CTC is inclusive of 5% variable.
Preferred (Tech skill 1) : Azure Databricks or Azure data services experience (project runs on Azure DevOps).
Preferred (Tech skill 2) : MLflow or MLOps practices and AI use cases (RAG, AI/BI).
Preferred (Tech skill 3) : CI/CD, Databricks Asset Bundles (DABs) or equivalent packaging, Terraform or IaC, reusable deployment templates.
Preferred (Integrations) : ServiceNow or enterprise system integrations.
Preferred (Certifications) : Databricks (Data Engineer Associate or Professional, ML or GenAI tracks), Azure or AWS cloud certifications.
Perks, Benefits and Work Culture :
The company provides free AWS and Azure certification training.
Group medical insurance of 5 lakhs is included in the benefits package, along with meal allowances.
AB INFOTECH