Meet Shah

Big Data Engineer with 4 years of experience designing, building, and optimizing scalable ETL/ELT data pipelines and cloud data architectures. Proficient in Azure Data Factory (ADF), Azure Synapse Analytics, PySpark, Apache Spark, Databricks, Microsoft Fabric, and SQL. Experienced in full and incremental load strategies, data modeling, data warehousing, data cleansing, and distributed computing.

Proven ability to deliver high-quality data engineering solutions in Banking and Supply Chain domains. Holds multiple Microsoft and Databricks certifications.

Data Engineer at Bizmetric (2024-07 – Present)

Design and maintain scalable ETL/ELT data pipelines and build data workflows in Microsoft Fabric for centralized data transformation, cleansing, and staging.

• Design and maintain scalable ETL/ELT data pipelines using Azure Data Factory (ADF) and Azure Synapse Analytics, supporting full load and incremental load strategies across multiple business domains.
• Build and orchestrate data workflows in Microsoft Fabric using Lakehouse, Dataflows Gen2, Pipelines, and Notebooks for centralized data transformation, cleansing, and staging.
• Write complex SQL queries using window functions, Common Table Expressions (CTEs), and nested subqueries for large-scale data analysis and business reporting.
• Implement data modeling best practices and enforce referential integrity across Azure cloud platforms, improving data quality and downstream analytics reliability.
• Work with cross-functional teams to gather data requirements and deliver reliable, optimized data solutions that reduce pipeline processing time.

Big Data Engineer at Celebal Technologies (2021-11 – 2023-11)

Built production-grade Azure Data Factory pipelines and developed high-performance PySpark workflows in Databricks for data cleansing and harmonization.

• Built production-grade Azure Data Factory (ADF) pipelines for full and incremental loads, ensuring referential integrity within Azure Synapse Analytics dedicated SQL pools.
• Developed high-performance PySpark workflows in Databricks for data cleansing and harmonization, using the DataFrame API and SparkSQL to process large volumes of records on distributed clusters.
• Applied PySpark transformations including map, flatMap, filter, groupBy, and reduce to process structured and semi-structured datasets, improving throughput and reducing job execution time.
• Created PySpark ingestion scripts to parse, clean, and convert unstructured XML and JSON data into structured Delta Lake tables using Spark DataFrames in Databricks.
• Gathered requirements from domain experts in Banking and Supply Chain, defined data contracts, and delivered reliable big data solutions within project timelines.

Master of Science (M.S.) in Computer Science – MIT World Peace University (2020 – 2022)

Bachelor of Science (B.S.) in Information Technology – GLS University (2017 – 2020)