Senior Data Engineer | 9+ years Experience
Send a job offer directly to this candidate
I am a results-driven Data Engineer with over 9 years of experience, including 4 years specializing in Big Data and cloud-based data engineering and 5 years in database development. My expertise lies in leveraging Azure Databricks, Azure Data Factory, and Apache Spark to design and implement scalable, secure, and high-performance data pipelines.
Designed and maintained end-to-end ETL pipelines using
ADF to orchestrate data ingestion, transformation, and loading across Azure SQL Database, Data Lake Storage.
➔ Optimized PySpark ETL jobs processing large datasets by configuring Spark parameters, implementing effective data partitioning, and using caching strategies, resulting in a 50%
improvement in job performance.
➔ Established multi-layered data lakehouse architecture using Azure Databricks and Delta Lake, streamlining ETL workflows and improving data reusability across teams.
➔ Implemented data partitioning and bucketing in PySpark for optimal query performance on large datasets, improving job execution speed and minimizing shuffle operations.
2010 – 2014 B.E(ECE)
