Sourabh Sharma

Azure Data Engineer with 4+ years of experience designing scalable enterprise data platforms using Microsoft Fabric, Azure Databricks, PySpark, and Delta Lake. Specialized in SAP-to-Lakehouse integrations, metadata-driven ETL frameworks, Medallion architecture, and high-performance distributed data processing. Experienced in building optimized incremental ingestion pipelines, improving Spark performance, and modernizing enterprise analytics workflows for scalable and reliable reporting solutions.

Data Engineer - Polestar Analytics Pvt Ltd.

(2023-11)

• Engineered Microsoft Fabric Lakehouse solutions using Medallion (Bronze–Silver–Gold) architecture for scalable enterprise data processing.
• Developed metadata-driven ingestion frameworks and parameterized ETL pipelines for REST and SAP OData API ingestion.
• Reduced pipeline loading time by 30% for high-volume SAP tables by implementing incremental ingestion using Delta Token-based processing.
• Designed scalable transformation pipelines in Azure Databricks using PySpark for distributed enterprise data processing.
• Improved notebook runtime and Spark job performance by resolving data skewness using optimization techniques such as salting and partition tuning.
• Leveraged Databricks Auto Loader with schema evolution support for optimized incremental ingestion and scalable file processing.
• Optimized Delta Lake storage and ingestion strategies, improving processing efficiency and reducing unnecessary data reloads.
• Collaborated with business stakeholders to modernize legacy BW reporting workflows using PySpark-based transformations, improving reliability, scalability, and processing performance.

Data Engineer - Tata Power Delhi Distribution Limited

(2018-06 - 2019-08)

• Re-engineered Qlik ETL workflows into optimized T-SQL transformations, improving performance and ensuring business rule consistency during BI migration.
• Built and automated Azure Data Factory (ADF) pipelines with parameterization, scheduling, dependency management, and robust error handling for seamless data integration.
• Reduced pipeline failures by 30% by developing resilient data pipelines and implementing validation and error-handling mechanisms to ensure accurate data delivery to business users.
• Implemented performance tuning techniques, reducing overall data refresh time by 35%.
• Ensured data quality and governance through validation frameworks and structured logging mechanisms.

Bachelor of Technology - Maharaja Agrasen Institute of Technology (2019-08 - 2023-07) Delhi, India