Rajesh B
Lead Azure Databricks Engineer
Hire this person
Send a job offer directly to this candidate
About
Lead Azure Databricks Engineer with 16+ years of experience building enterprise data platforms, with a strong background spanning traditional SQL Server/SSIS data warehousing, Azure data engineering, and modern Databricks Lakehouse architectures. Experienced in designing and leading end-to-end Azure data solutions, using Azure Databricks, ADLS Gen2, Azure Data Factory, and Azure Synapse Analytics to bring together data from SQL Server, Oracle, PostgreSQL, MongoDB, APIs, files, and operational systems for analytics and reporting. Strong hands-on expertise in Databricks-based ETL and Lakehouse development, using PySpark, Spark SQL, Python, Delta Lake, and Medallion Architecture to move data through Bronze, Silver, and Gold layers, with incremental processing and reusable transformation frameworks.
Experienced in building incremental ingestion patterns where ADF handles source extraction and orchestration, while Databricks Auto Loader and Structured Streaming process continuously arriving files and event data in ADLS, with Delta MERGE, CDC, and watermarking used to keep target datasets current. MMicrosoft Learn Designed both batch and near-real-time pipelines using Kafka and Azure Event Hubs, applying PySpark transformations, checkpointing, validation, and error-handling patterns to deliver reliable data for downstream analytical workloads. Strong understanding of Delta Lake optimization and Spark performance tuning, including partition strategy, efficient joins, broadcast techniques, caching, Adaptive Query Execution, cluster configuration, Photon, OPTIMIZE, and Z-Ordering for large-volume workloads.
Experienced in establishing data governance within Databricks, using Unity Catalog, RBAC, Azure Key Vault, Collibra, lineage, auditing, metadata, and access controls to provide governed and secure access to enterprise datasets. MMicrosoft Learn Hands-on experience developing Databricks Workflows and ADF orchestration, incorporating dependencies, parameterization, scheduling, retries, monitoring, validation, and recovery processes for production data pipelines. Proven experience with Azure DevOps and Git-based CI/CD, using repositories, pull requests, automated pipelines, and Databricks Asset Bundles to promote Databricks notebooks, ADF pipelines, and supporting data artifacts across development, testing, and production environments.
Strong data warehousing foundation in T-SQL, dimensional modeling, Star/Snowflake schemas, fact and dimension design, SCD Type 1/Type 2, stored procedures, and ETL frameworks, providing a solid foundation for modernizing legacy warehouse workloads into Azure Lakehouse platforms. Experienced in delivering the final analytical layer through Power BI, DAX, Power Query, SSAS, and Synapse, developing curated datasets, semantic models, KPIs, and reporting solutions while performing source-to-target reconciliation to resolve data discrepancies. Familiar with applying MLflow, Azure Machine Learning, Azure OpenAI, Copilot, LLM, and
Experience
Lead Azure Databricks Engineer - Oldcastle Building Envelope (OBE) - Texas, USA
(2024-08)
OBE is a leading manufacturer of architectural glass and aluminum systems for the commercial building industry. The project focused on building a scalable data platform to enable advanced analytics and reporting.
- Led and developed scalable data pipelines on Azure Databricks to process data from relational databases, APIs, files, and event-based sources for reporting and analytics requirements.
- Created incremental ingestion processes using Azure Data Factory, Databricks Auto Loader, Delta MERGE, CDC, and watermarking techniques to efficiently process new and changed records.
- Used Git and Azure DevOps to manage and deploy Python, PySpark, ADF, and Power BI changes across development, testing, and production environments, with Azure Key Vault used to securely manage connection details and credentials.
- Developed batch and streaming pipelines using PySpark, Kafka, and Azure Event Hubs, handling continuous data ingestion and near-real-time processing requirements.
- Used ADLS Gen2 as the primary cloud storage layer and implemented Delta tables for reliable data storage, schema management, and transactional processing.
- Implemented data governance practices using Collibra for metadata management, data cataloging, and end-to-end data lineage.
- Applied Unity Catalog to manage dataset permissions, role-based access, data lineage, auditing, and controlled access to sensitive business data.
- Created Databricks Workflows for scheduling and monitoring data processing jobs, including dependency handling, retries, alerts, and failure recovery.
- Implemented CI/CD pipelines using Git, Azure DevOps, and Databricks Asset Bundles to automate code promotion across Development, Testing, and Production environments.
- Supported production data platforms by investigating Databricks, and reporting failures, performing root-cause analysis and implementing fixes also explored Azure OpenAI, Azure Machine Learning, Copilot, LLM, and MCP-based solutions for data discovery, automation, and analytics use cases.
- Used MLflow for tracking machine learning experiments and supporting integration of analytical and machine learning workloads with the Databricks environment.
- Monitored production pipelines, investigated data and job failures, performed root-cause analysis, and implemented fixes to maintain reliable data processing.
- Collaborated with data analysts, application teams, and business users to understand data requirements and deliver reusable data processing solutions.
- Improved Spark job performance by tuning joins, partitions, caching, broadcast strategies, cluster configurations, Photon execution, and Delta table maintenance using OPTIMIZE and Z-Ordering.
- Worked with Azure SQL, Snowflake, and Microsoft Fabric to integrate data across multiple platforms and prepare curated datasets for downstream analytics.
- Developed and optimized SQL Server data models, T-SQL procedures, views, and ETL processes, then integrated the resulting datasets into Azure Data Factory and ADLS Gen2 pipelines for downstream analytics and reporting.
- Migrated on-premises data workloads to Azure by building ADF ingestion pipelines that moved data from SQL Server, Oracle, PostgreSQL, MongoDB, APIs, and file sources into ADLS, with Synapse and Databricks used for subsequent processing and analytics.
- Designed parameterized and metadata-driven ADF pipelines with incremental load logic, dependency handling, validation, and error recovery, using SQL-based control tables and Python utilities to make recurring ingestion processes easier to maintain.
- Built Databricks solutions using PySpark, Python, and Delta Lake to cleanse, join, enrich, and aggregate data from the lake, applying partitioning and incremental processing techniques to improve performance on large datasets.
- Developed curated datasets for Snowflake, Synapse, and Microsoft Fabric by combining transformed data from multiple source systems, supporting both traditional warehouse reporting and newer lakehouse-based analytical workloads.
- Built PySpark and Python-based data processing jobs to perform data cleansing, transformation, enrichment, aggregation, and validation across large datasets.
- Implemented Delta Lake-based data processing using Bronze, Silver, and Gold layers to organize raw, cleansed, and business-r