Farhan Shoukat Ali
Data Engineer
Hire this person
Send a job offer directly to this candidate
About
Data Engineer and Computer Science student specializing in building scalable, real-time data pipelines and cloud-native architectures. Expert in Azure, Microsoft Fabric, Databricks, and dbt, with a proven track record of developing metadata-driven frameworks and containerized ELT workflows. Skilled in transforming complex datasets into actionable insights through Medallion architectures and automated orchestration.
Experience
Real-Time Data Engineering (Azure & Databricks) at Uber
Engineering a high-throughput, low-latency streaming architecture to handle concurrent ride-sharing events while ensuring data integrity across a Medallion architecture. Real-time GPS and transaction streams ingested via FastAPI and Azure Event Hubs, processed into Delta Tables.
- Designed a streaming ETL pipeline in Azure Databricks utilizing Spark Structured Streaming.
- Implemented Medallion Architecture (Bronze, Silver, Gold) to provide clean, aggregated datasets.
- Utilized Azure Schema Registry to manage schema evolution across the event-driven ecosystem.
Advanced Azure Data Engineering (Synapse Analytics Integration) at Enterprise
Integrating diverse enterprise data sources into a unified analytics platform while optimizing cost and performance for massive-scale querying. Large-scale enterprise datasets moved through Synapse Pipelines and stored in Dedicated SQL Pools for high-performance BI.
- Architected an end-to-end integration using Azure Synapse Analytics, leveraging Serverless SQL pools for data exploration.
- Developed complex T-SQL transformation scripts to build dimension and fact tables optimized for analytical workloads.
- Integrated with Azure Key Vault to manage secure connection strings and credentials across the pipeline.
ShoppingMart Data Warehouse (Microsoft Fabric) at Retail
Migrating traditional retail ETL processes to a modern SaaS data lakehouse environment to simplify management and scale-out performance. Multi-source retail transaction data ingested into OneLake using Fabric Notebooks and Data Factory.
- Built a unified data repository in Microsoft Fabric Lakehouse, utilizing Delta Tables for ACID compliance.
- Orchestrated data flows using Fabric Data Pipelines, automating movement from ingestion to curated Gold layers.
- Utilized SQL Analytics Endpoints within Fabric to enable direct Power BI connectivity for real-time sales reporting.
Airflow Config-Driven Pipelines (Orchestration Scalability) at Enterprise
Overcoming "DAG sprawl" by developing a system to programmatically generate Airflow workflows based on central configuration files. Enterprise-grade orchestration capable of scaling across hundreds of independent data tasks.
- Built a dynamic DAG generator in Apache Airflow that maps metadata definitions to executable workflow patterns.
- Integrated CI/CD pipelines to automatically validate and deploy configuration changes.
- Standardized error handling and alerting mechanisms across the entire pipeline ecosystem.
Real-Time Data Lakehouse on Azure (FastAPI & Medallion) at Azure
Building a unified platform that supports both real-time ingestion and historical analytics without data duplication. Streaming event data processed through a Medallion Architecture with sub-second ingestion latency.
- Deployed a FastAPI application to serve as a high-performance entry point for data producers.
- Leveraged Azure Event Hubs and Databricks to maintain the Silver and Gold layers of the lakehouse.
- Automated infrastructure deployment using specialized Python scripts for consistency.
Car Sales Data Warehouse (AWS Native Stack) at AWS
Building a cost-effective, serverless data warehousing solution that can handle high volumes of semi-structured automotive sales data while ensuring the final dataset is optimized for complex analytical queries and business intelligence reporting. Massive-scale automotive sales records, inventory logs, and customer interaction data stored as raw files in Amazon S3, refined through multiple architectural layers to support a production-grade dimensional model.
- Architected a serverless ETL pipeline utilizing AWS Glue for heavy data processing and AWS Lambda for event-driven orchestration.
- Implemented a Medallion Architecture (Bronze, Silver, Gold) to incrementally clean, validate, and enrich data stored in Amazon S3.
- Designed and executed a Star Schema dimensional model to build final Gold tables, delivering a high-quality, analytics-ready dataset for executive BI consumption.
Education
Bachelor Of Computer Science in Computer Science – Superior University, Lahore, Pakistan


