Shirish kumar Veggalam

Azure Data Engineer at Vertex Computer Systems (2025-07 – Present)

Worked on a manufacturing analytics project focused on transforming legacy data processes into a modern Lakehouse architecture using Microsoft Fabric.
Designed and implemented metadata-driven data ingestion pipelines to process ERP datasets, CSV extracts, and multi-region manufacturing data.
Utilized Fabric Notebooks and Delta Lake–based Lakehouse architecture to build efficient transformation layers (Bronze → Silver → Gold) supporting large-scale production workloads.
Extensively worked with PySpark, SQL, and Python to build high-performance ETL.
Re-engineered existing SQL stored procedures and migrated business logic into Fabric notebooks.
Implemented data quality checks, surrogate keys, and incremental load strategies to ensure reliability and accuracy of enterprise datasets.
Collaborated with cross-functional teams including analysts, leads, and SMEs to understand business rules and translate them into technical solutions.
Supported validation, reconciliation, and performance tuning of ETL pipelines to ensure consistent and high-quality output.
Prepared technical documentation, pipeline diagrams, and mapping sheets to support ongoing development and production readiness.

Azure Data Engineer at Ascendion Engineering Pvt Ltd. (2025-02 – 2025-07)

Worked on a healthcare analytics project aimed at streamlining and modernizing data processing pipelines using Microsoft Fabric.

Worked on a healthcare analytics project aimed at streamlining and modernizing data processing pipelines using Microsoft Fabric. Key responsibilities and contributions include: Designed and implemented data ingestion pipelines leveraging Microsoft Fabric's integration capabilities.
Utilized Notebooks and Lakehouse architecture for efficient data transformation and unified storage.
Extensively worked with PySpark, SQL, and Python to build scalable and high-performance ETL workflows.
Ensured clean, governed data pipelines that support healthcare reporting and analytics use cases.
Collaborated with data analysts and domain experts to align pipeline outputs with regulatory and business requirements.

Azure Data Engineer at HCLTech (2021-12 – 2024-11)

Developed a robust Medallion Architecture utilizing cutting-edge technologies like Azure ADF, Azure Databricks, Delta Lake, and Unity Catalog as the storage solution.
Actively involved in developing the Data ingestion and creating pipelines as per the gathered requirements using Azure Data Factory.
Created different layers of data on the Azure Data Lake and cleaned the data. Performed transformations on the data using Azure Data Bricks.
Monitoring the developed pipelines to measure the performance or the throughput.
Involved in triggering and monitoring the data ingestion and creating pipelines on Production.
Preparing documentation of the data pipelines developed for the reference purpose.
Leveraged Pyspark and a metadata-driven process to ensure efficient data processing and management within the architecture.
Proactively planned, debugged, and resolved operational challenges faced by the system, ensuring smooth project execution.
Applied optimization techniques in PySpark to enhance the performance of data processing tasks. Optimized PySpark jobs by fine-tuning configurations and leveraging techniques such as partitioning and caching for improved efficiency.
Effectively communicated technical concepts and solutions to both technical and non-technical stakeholders, fostering a collaborative and cohesive work environment.
Applied key performance indicators on the data and then the data was aggregated. Moved the data to synapse to facilitate the Power BI team to populate the reports.

Azure Data Engineer at Cognizant (2021-08 – 2021-12)

Involved in enhancing the features of the data curation logic as per the requirements and testing the developed logic in Dev environment.

Involved in enhancing the features of the data curation logic as per the requirements and Testing the developed logic in Dev environment. Also, Migrating the changes to the Quality and Production and testing on it.
Involved in triggering and monitoring the data ingestion and curation pipelines on production.
Working on the transformation of the curation business logic provided in the form of raw Code to the PySpark code to develop a meaningful data curation framework.

About