Nikhil Kudva

Around 5+ years of experience as an Azure Data Engineer in designing and developing end-to-end data engineering solutions on Microsoft Azure. Hands-on experience in building scalable ETL and ELT pipelines using Azure Data Factory for ingesting data from multiple heterogeneous data sources. Good experience in developing data transformation pipelines using Azure Databricks, PySpark, and Spark SQL.

Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2) for enterprise data storage following Bronze, Silver, and Gold architecture. Worked extensively on loading structured, semi-structured, and unstructured data from SQL Server, REST APIs, FTP, Blob Storage, and CSV/JSON/XML files. Good knowledge of Delta Lake, ACID Transactions, Time Travel, Schema Enforcement, and Schema Evolution.

Experience implementing incremental data loading using watermark columns, Change Data Capture (CDC), and Delta Merge operations. Hands-on experience writing optimized SQL queries, stored procedures, views, CTEs, window functions, and performance tuning techniques.

Experience in developing reusable PySpark notebooks for data cleansing, transformation, aggregation, deduplication, and validation. Worked on data quality validations and reconciliation checks before publishing data into business reporting layers.

Experience integrating Azure Data Factory with Azure Databricks notebooks for orchestration of enterprise ETL workflows. Good exposure to Azure Key Vault for secure credential management across data pipelines.

Experience working with Azure Event Hub and Apache Kafka for streaming data ingestion and real-time processing. Hands-on experience with Azure DevOps for source code management, branching strategy, pull requests, CI/CD pipelines, and release deployments.

Experience handling production support activities including monitoring failed pipelines, root cause analysis, issue resolution, and deployment validations. Worked with ServiceNow for incident management, problem tickets, service requests, and change management processes.

Experience using Datadog for application monitoring, log analysis, dashboard creation, and proactive alert monitoring. Good understanding of Spark optimization techniques including caching, partitioning, repartitioning, broadcast joins, adaptive query execution, and skew handling.

Experience working in Agile methodology involving Sprint Planning, Daily Stand-ups, Sprint Reviews, Retrospectives, and client status meetings.

AZURE DATA ENGINEER - VERITAS COMUNICATIONS PRIVATE LIMITED

(2024-07)

• Designed and developed Azure Data Factory pipelines to ingest daily sales, inventory, distributor, retailer and production data from SQL Server, REST APIs and flat files into Azure Data Lake Storage Gen2.
• Integrated Azure Data Factory with Azure Databricks notebooks through parameterized pipelines to support reusable ETL frameworks.
• Developed reusable PySpark notebooks in Azure Databricks for cleaning, standardizing and transforming raw datasets into curated business layers.
• Implemented incremental loading using watermark columns and Delta MERGE statements, reducing full data loads and improving overall pipeline performance.
• Executed ACID transactions and schema evolution using Delta Lake to maintain historical records.
• Built Bronze, Silver and Gold layer architecture to separate raw, validated and business-ready data for downstream reporting.
• Optimized Spark jobs using partitioning, caching, broadcast joins and Adaptive Query Execution (AQE) to enhance notebook execution efficiency.
• Implemented data quality validations including duplicate checks, null validations, schema validations and record count reconciliation before loading curated data.
• Monitored scheduled ADF pipelines, investigated failed activities, and resolved production issues within SLA timelines to ensure operational continuity.
• Configured Azure Key Vault to securely retrieve database credentials, API secrets and storage account access keys during pipeline execution.
• Used Datadog dashboards and alerts to monitor pipeline execution, cluster utilization and application logs.
• Worked on ServiceNow incidents, problem tickets and enhancement requests while coordinating with business users during production support.
• Performed deployment activities across Development, QA and Production environments using Azure DevOps Release Pipelines.

AZURE DATA ENGINEER - KGISL

(2024-04 - 2024-07)

• Worked on a prototype Azure Data Factory framework to extract daily sales and CRM data into Azure Blob Storage for reporting.
• Configured Azure Data Factory pipelines to automate ingestion of sales metrics from CRM applications.
• Utilized ADF Web Activities to invoke external Sales APIs and capture lead generation data for downstream processing.
• Performed data validation and preprocessing using Python before loading data into Azure storage.
• Used Azure Databricks Community Edition to test Python-based data cleansing and transformation scripts.
• Assisted in validating ingestion workflows and improving pipeline execution reliability.
• Collaborated with business stakeholders to understand reporting requirements and automate manual data collection activities.
• Participated in testing and documenting Azure-based proof-of-concept solutions.

AZURE DATA ENGINEER - SAMVITTI CAPITAL

(2024-01 - 2024-04)

• Configured Azure Data Factory Linked Services to securely connect with external financial data providers.
• Automated ingestion of daily client portfolio and transaction files into Azure Storage using Azure Data Factory.
• Loaded portfolio datasets into Azure Databricks and developed PySpark scripts for data validation, joins, and auditing.
• Scheduled automated ADF triggers for processing end-of-day transaction files.
• Performed data reconciliation between source files and processed datasets to ensure data consistency.
• Assisted in implementing transformation logic for financial reporting datasets.
• Worked closely with business and operations teams to validate processed financial data.
• Supported troubleshooting of ingestion failures and monitored scheduled pipeline executions.

AZURE DATA ENGINEER - Think & Learn Pvt Ltd (Byju's) - Bengaluru - Bengaluru

(2022-05 - 2023-12)

• Developed end-to-end ETL pipelines using Azure Data Factory to ingest student activity, course enrollment, and learning analytics data into Azure Data Lake, enhancing data accessibility for analytical insights.
• Built PySpark transformation logic in Azure Databricks to process millions of student interaction records received daily from multiple source systems.
• Consumed streaming events from Apache Kafka and Azure Event Hub to process real-time learning activities for analytical reporting.
• Developed Spark SQL scripts to aggregate user engagement metrics and publish them into reporting tables consumed by Power BI dashboards.
• Worked with Delta Lake tables to implement Slowly Changing Dimension (SCD Type-1 and Type-2) processing for dimension management.
• Created reusable parameterized ADF pipelines, reducing duplicate development effort and streamlining multiple ingestion processes.
• Performed schema validation and handled schema drift using dynamic mapping techniques in Azure Data Factory.
• Optimized Spark transformations by minimising shuffles and tuning partition sizes, improving processing efficiency for large-scale data.
• Created SQL stored procedures for post-load validations and audit logging.
• Monitored production workloads through Datadog dashboards and proactively addressed long-running jobs before SLA breaches.
• Performed root cause analysis for production failures and implemented permanent fixes after impact assessment.
• Supported monthly production deployments and coordinated release validation with QA and business teams.
• Used Azure DevOps Boards for sprint tracking and Git repositories for version control and code collaboration.
• Worked with ServiceNow Change Requests during production deployments and followed standard release management processes.

AZURE DATA ENGINEER - LIVFAST - Mangalore - Mangalore

(2019-11 - 2020-09)

• Contributed to building Azure Data Factory pipelines to extract operational data from SQL Server databases and FTP locations, enhancing data accessibility.
• Facilitated implementation of Delta Lake architecture, ensuring reliable data processing and enabling historical data tracking.
• Developed PySpark notebooks for data cleansing, filtering, joining and transformation activities.
• Crafted complex SQL queries with joins, CTEs, window functions, aggregate functions, and stored procedures to enable effective data analysis.
• Loaded processed datasets into Azure SQL Database for reporting and dashboard development.
• Worked on file-based ingestion of CSV, JSON and Excel files into Azure Data Lake Storage.
• Created reusable notebooks for common data transformation logic to improve development efficiency.
• Performed data reconciliation between source systems and target databases to ensure data accuracy.
• Assisted senior engineers in troubleshooting failed ETL jobs and resolving production issues.
• Participated in code reviews and implemented optimization suggestions received from technical leads.

Bachelor of Business Management - Business Management - St. Aloysius College - Mangalore (2018)