Abdullah Ansari

Azure Data Engineer with 3 + year of hands-on experience in designing and building end-to-end data pipelines. Skilled in Azure Data Factory, Azure Databricks, ADLS Gen2, and PySpark. Experienced in data ingestion, transformation, and implementing incremental data loading for optimized performance.

Azure Data Engineer at DREAMCARE DEVELOPERS PVT.LTD (2023-01 – Present)

Experience in creating Data Ingestion Pipelines, Data Transformations an enterprise level. Designed and implemented data solutions using Azure Services, including Azure Databricks, ADLS and Data Factory.

• Developed an end-to-end ETL pipeline using Azure Data Factory.
• Created Pipelines in Azure Data Factory using Linked Services/Datasets/Pipeline to load the data into target systems.
• Designed and developed Azure Data Factory pipelines to ingest data from Azure SQL Database into ADLS Gen2 as raw data for downstream processing.
• Designed Azure Data Factory pipelines with Incremental loading logic to ingest only new records from source systems, optimizing performance and minimizing resource consumption.
• Performed data transformation and cleansing using PySpark in Azure Databricks.
• Stored processed data in ADLS Gen2 in a structured format for downstream analytics.
• Enhanced data pipeline performance by transforming raw data into Delta Lake format in Azure Databricks, improving storage optimization, query efficiency, and data consistency.
• Developed incremental data pipelines using Databricks Auto Loader and Delta Lake UpSert and Merge operations to efficiently process only new and incremental data.
• Optimized pipeline performance by Partitioning data and collaborating with reporting teams to deliver curated datasets for business analytics.

Bachelor Of Engineering in (COMPUTER ENGINEER)