Mohamed Arshad M.R.

Azure Data Engineer with 4+ years of experience designing and delivering cloud-native ETL/ELT pipelines, real-time and batch data workflows, and lakehouse architectures on Microsoft Azure. Expert in Azure Data Factory, Azure Databricks, ADLS Gen2, Delta Lake, and Azure Synapse Analytics. Proven ability to process terabytes of data daily using PySpark, implement Medallion Architecture, and optimise Spark workloads for production-grade performance.

Azure Data Engineer at Tata Consultancy Services (TCS) (2021-11 – Present)

• Designed and deployed end-to-end ADF pipelines ingesting raw data from 10+ sources (REST APIs, SQL, flat files) into ADLS Gen2, processing 50 GB+ daily with automated error handling and retry logic.
• Built PySpark transformation scripts on Azure Databricks following Medallion Architecture (Bronze → Silver → Gold); applied deduplication, null handling, and business rule enforcement across all layers.
• Implemented SCD Type 2 logic using PySpark Window functions to maintain full historical dimension records — enabling time-travel analysis and regulatory audit compliance.
• Stored curated datasets as Delta Lake tables with ACID transactions, MERGE-based upserts, and time-travel queries, reducing stale-data incidents by 100%.
• Authored complex SQL on Azure Synapse Analytics for aggregation, validation, and self-service BI; optimised slow queries by 60% through partitioning and statistics tuning.
• Tuned Spark jobs using broadcast joins, dynamic resource allocation, and partition pruning — cut average job runtime by 40% on production clusters.
• Automated pipeline scheduling, triggers, and alerting via ADF Monitor and Azure Monitor; reduced mean-time-to-detect (MTTD) for pipeline failures from hours to minutes.
• Maintained CI/CD workflows for pipeline code using Git and Azure DevOps, ensuring zero-downtime deployments across dev / staging / production environments.

B.E. in Instrumentation Engineering – Anna University (2021)