Gaurav Joshi

Data Engineer with 2 years of experience in building and optimizing enterprise data platforms using Azure Databricks, Azure Data Factory, PySpark, SQL, and Delta Lake. Experienced in developing scalable ETL pipelines, performing complex transformations, and ensuring data quality through validation and reconciliation. Strong understanding of data governance, access control, and schema management for secure and well-structured data environments.

Proficient in performance optimization, partitioning strategies, SCD handling, and multiple target write modes. Solid foundation in data warehousing and medallion architecture, with strong analytical and problem-solving skills to deliver reliable, production-ready datasets.

Associate Data Engineer at Celebal Technologies (2024-09 – Present)

• Engineered a standardized, multi-layer Databricks pipeline architecture, version-controlled in Git and promoted through a 4-environment CI/CD pipeline (Work → Dev → ACC → Prod), enabling parallel execution and validated, low-risk production releases.
• Developed an automated JSON-generation utility to drive metadata-driven ADF pipelines, generating config that defines source, target, write mode, SQL query, and test cases — reducing manual configuration effort by ~70% and minimizing setup errors.
• Applied data validation, reconciliation, partitioning, salting, optimized join strategies, and Spark performance tuning to achieve 100% reconciliation accuracy across daily production runs.
• Led a 4-member data engineering pod to on-time delivery and supported production runs, root-cause analysis, and client communication; earned a Certificate of Standout Performer from senior leadership.
• Conducted code reviews across the team to maintain code quality, enforce standards, and catch issues early in the development cycle.

Bachelor of Technology (B.Tech) in Computer Science and Engineering – Jodhpur Institute of Engineering & Technology (2020-07 – 2024-05)