Pavan Naga Raju

Data Engineer with over 4.7 years of experience designing and building scalable batch data pipelines on Azure. Skilled in Azure Databricks (PySpark, Delta Lake, Jobs/Workflows), ADF, ADLS, Microsoft Fabric (Data Factory, Notebook, Lakehouse, Warehouse) and metadata-driven ETL pipelines. Experienced in distributed data processing, Delta Lake optimization, data quality frameworks, incremental loads, and secure data orchestration.

Skilled in delivering reliable, production-grade data systems and collaborating with cross-functional teams to support analytics, reporting, and enterprise data products.

Azure Data Engineer at Vation Digital (2026-01 – Present)

Client: Schindler

• Migrated an enterprise data warehouse from Azure Synapse Analytics to Microsoft Fabric Warehouse, validating 40+ fact and dimension tables across Lakehouse and Warehouse layers through SIT, QA, and Production.
• Developed and optimized T-SQL stored procedures and PySpark notebooks within a metadata-driven ELT framework, enabling configuration-based onboarding of new entities.
• Enhanced Microsoft Fabric Data Factory pipelines by implementing parallel batch processing, significantly improving data load performance and scalability.
• Created and managed Microsoft Fabric Shortcuts to integrate data across OneLake, Lakehouse, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Synapse Analytics, enabling centralized and efficient data access.
• Designed and implemented Microsoft Fabric Data Factory pipelines to extract data from SAP extractors and ingest SharePoint files (CSV, Excel) into ADLS Gen2, supporting scalable data processing and analytics.
• Migrated and optimized Azure Synapse Analytics views in Microsoft Fabric Warehouse by redesigning them using Enterprise Warehouse tables, improving query performance and enabling seamless Power BI reporting.

Data Engineer at Cognizant Technology Solutions (2021-11 – 2026-01)

• Designed and operated scalable batch pipelines using Azure Databricks (PySpark, Delta Lake), ADF, and ADLS to deliver curated, analytics-ready datasets for enterprise consumption.
• Built metadata-driven ingestion and transformation frameworks with schema validation, data quality rules, audit logging, and automated exception handling to improve pipeline stability.
• Implemented Medallion Architecture (Bronze/Silver/Gold) and optimized Delta Lake workloads using Z-Order, Optimize, and Vacuum, improving performance and storage efficiency.
• Automated orchestration and monitoring using ADF and Databricks Jobs/Workflows, enabling reliable, repeatable, and production-ready pipeline executions.
• Collaborated with architects, BI developers, QA, and business stakeholders to ensure accurate data processing, seamless downstream consumption, and improved reporting SLAs.
• Worked with Unity Catalog for managing basic access control and governance across Databricks environments, improving security and standardization.
• Designed and delivered an end-to-end enterprise data ingestion and transformation pipeline using ADF, Azure Databricks, Synapse, and ADLS for large-scale marketing and member analytics.
• Built incremental Delta processing and metadata-driven orchestration patterns aligned to Medallion Architecture, enabling structured and efficient data refinement.
• Developed automated compliance workflows with routing, validation, enrichment, and Kafka-based publishing, supporting audit and regulatory requirements.
• Enhanced pipeline observability with detailed logging, exception handling, and performance checks, improving overall reliability and reducing operational overhead.

Bachelor of Technology in Electronics and Communication Engineering – SRKR Engineering College, Bhimavaram (2017 – 2021)