Abishake Singaravelu

Azure Data Engineer with 7+ years of experience designing, developing, and optimizing enterprise-scale ETL/ELT data pipelines on Azure Databricks and Azure Data Factory, processing 3+ TB of data daily across Azure Data Lake Storage, Snowflake, and Web API sources. Skilled in building Lakehouse (Bronze-Silver-Gold / Medallion) architectures, metadata-driven ETL frameworks, distributed PySpark/Spark SQL solutions, large-scale data migration, and CI/CD-driven deployment automation, reducing pipeline onboarding time by 45% and improving data accuracy to 99.8%. Experienced in cloud migration, workflow orchestration, production support, and data quality management, with a strong track record of collaborating across cross-functional teams to deliver reliable, scalable cloud data platforms for global, remote-first organizations.

Azure Data Engineer (Remote) at Nationwide Accommodation Ltd (2024-06 – Present)

• Built an end-to-end Azure Lakehouse pipeline for Clearsprings Ready Homes, consolidating housing and property management data from Azure Data Lake Storage (ADLS), 5 Snowflake databases, and 4 Web APIs, processing 3+ TB of enterprise data daily.
• Engineered scalable PySpark applications in Azure Databricks, developing 3 ingestion jobs to cleanse, validate, and transform tenancy, occupancy, maintenance, and compliance data into Silver Layer Lakehouse tables, improving batch processing efficiency by 40%.
• Built a master PySpark job consolidating Silver Layer tables into curated Lakehouse datasets used by BI, Property Management, Housing Support, Maintenance, and Compliance teams for dashboards and KPI monitoring.
• Transformed 50+ million records using PySpark joins, aggregations, and window functions across Snowflake, ADLS, and API sources to support complex business logic.
• Led a large-scale data migration initiative, migrating legacy on-premises and Snowflake datasets to Azure Data Lake Storage using PySpark and ADF Copy Activities, ensuring zero data loss and full referential integrity across 5 Snowflake source systems.
• Designed and executed source-to-target reconciliation and validation frameworks during the data migration, verifying accuracy across 50+ million migrated records and achieving 99.8% post-migration data accuracy.
• Orchestrated the pipeline using Azure Data Factory, scheduling 4 Databricks notebooks to run daily at 9:00 AM ET, ensuring downstream teams always have current operational data.
• Developed 25+ parameterized Azure Data Factory pipelines automating end-to-end ETL workflows across Dev, QA, and Production environments.
• Built metadata-driven ETL frameworks and reusable Databricks notebooks, cutting new pipeline onboarding time by 45% and development effort by 35%.
• Implemented CI/CD pipelines using Azure DevOps to automate build, test, and deployment of ADF pipelines and Databricks notebooks across Dev, QA, and Production environments, reducing manual deployment effort by 60%.
• Integrated Git-based version control with Azure DevOps CI/CD pipelines to enable automated code validation, artifact packaging, and release approvals, improving deployment consistency and reducing production errors.
• Optimized Spark workloads through partitioning, caching, and broadcast joins, reducing job execution time by 50%.
• Created Spark SQL validation scripts and automated source-to-target reconciliation, improving data accuracy to 99.8%.

Cloud Data Engineer at Marketplace Direct Limited (2022-05 – 2024-06)

• Designed and implemented 20+ enterprise Azure Data Factory pipelines supporting critical business reporting and analytics.
• Developed PySpark transformations processing 2+ TB of structured and semi-structured data with improved reliability.
• Automated Databricks notebook execution via Azure Data Factory, reducing manual scheduling effort by 80%.
• Built reusable PySpark components, cutting development time by 30% across multiple ETL projects.
• Developed dynamic, parameter-driven ADF pipelines supporting 15+ data sources and multiple deployment environments.
• Optimized Spark jobs through partitioning, caching, and broadcast joins, improving overall performance by 45%.
• Scheduled and monitored 100+ recurring Databricks jobs, processing 10,000+ CSV, JSON, and Parquet files into curated Data Lake layers.

ETL Testing Engineer at The Princess Alexandra Hospital NHS Trust (2019-03 – 2022-05)

• Executed ETL testing for 50+ healthcare data migration and integration releases, achieving 99.9% data accuracy.
• Designed and ran 800+ SQL validation scripts and performed source-to-target reconciliation across 20+ million records.
• Identified and resolved 300+ ETL defects, improving production data quality by 30% and cutting testing effort by 25%.
• Created and managed Hive tables, including managed, external, and partitioned tables, integrating with Hadoop, HBase, and Impala.

MSc in Telecommunications – London South Bank University (2017 – 2019)

BE in Electronics & Communication Engineering – St. Joseph's Institute of Technology (2012 – 2016)