Farhan Shoukat Ali

Data Engineer and Computer Science student specializing in building scalable, real-time data pipelines and cloud-native architectures. Expert in Azure, Microsoft Fabric, Databricks, and dbt, with a proven track record of developing metadata-driven frameworks and containerized ELT workflows. Skilled in transforming complex datasets into actionable insights through Medallion architectures and automated orchestration.

Real-Time Data Engineering (Azure & Databricks) at Uber

Engineering a high-throughput, low-latency streaming architecture to handle concurrent ride-sharing events while ensuring data integrity across a Medallion architecture. Real-time GPS and transaction streams ingested via FastAPI and Azure Event Hubs, processed into Delta Tables.

• Designed a streaming ETL pipeline in Azure Databricks utilizing Spark Structured Streaming.
• Implemented Medallion Architecture (Bronze, Silver, Gold) to provide clean, aggregated datasets.
• Utilized Azure Schema Registry to manage schema evolution across the event-driven ecosystem.

Advanced Azure Data Engineering (Synapse Analytics Integration) at Enterprise

Integrating diverse enterprise data sources into a unified analytics platform while optimizing cost and performance for massive-scale querying. Large-scale enterprise datasets moved through Synapse Pipelines and stored in Dedicated SQL Pools for high-performance BI.

• Architected an end-to-end integration using Azure Synapse Analytics, leveraging Serverless SQL pools for data exploration.
• Developed complex T-SQL transformation scripts to build dimension and fact tables optimized for analytical workloads.
• Integrated with Azure Key Vault to manage secure connection strings and credentials across the pipeline.

ShoppingMart Data Warehouse (Microsoft Fabric) at Retail

Migrating traditional retail ETL processes to a modern SaaS data lakehouse environment to simplify management and scale-out performance. Multi-source retail transaction data ingested into OneLake using Fabric Notebooks and Data Factory.

• Built a unified data repository in Microsoft Fabric Lakehouse, utilizing Delta Tables for ACID compliance.
• Orchestrated data flows using Fabric Data Pipelines, automating movement from ingestion to curated Gold layers.
• Utilized SQL Analytics Endpoints within Fabric to enable direct Power BI connectivity for real-time sales reporting.

Airflow Config-Driven Pipelines (Orchestration Scalability) at Enterprise

Overcoming "DAG sprawl" by developing a system to programmatically generate Airflow workflows based on central configuration files. Enterprise-grade orchestration capable of scaling across hundreds of independent data tasks.

• Built a dynamic DAG generator in Apache Airflow that maps metadata definitions to executable workflow patterns.
• Integrated CI/CD pipelines to automatically validate and deploy configuration changes.
• Standardized error handling and alerting mechanisms across the entire pipeline ecosystem.

Real-Time Data Lakehouse on Azure (FastAPI & Medallion) at Azure

Building a unified platform that supports both real-time ingestion and historical analytics without data duplication. Streaming event data processed through a Medallion Architecture with sub-second ingestion latency.

• Deployed a FastAPI application to serve as a high-performance entry point for data producers.
• Leveraged Azure Event Hubs and Databricks to maintain the Silver and Gold layers of the lakehouse.
• Automated infrastructure deployment using specialized Python scripts for consistency.

Car Sales Data Warehouse (AWS Native Stack) at AWS

Building a cost-effective, serverless data warehousing solution that can handle high volumes of semi-structured automotive sales data while ensuring the final dataset is optimized for complex analytical queries and business intelligence reporting. Massive-scale automotive sales records, inventory logs, and customer interaction data stored as raw files in Amazon S3, refined through multiple architectural layers to support a production-grade dimensional model.

• Architected a serverless ETL pipeline utilizing AWS Glue for heavy data processing and AWS Lambda for event-driven orchestration.
• Implemented a Medallion Architecture (Bronze, Silver, Gold) to incrementally clean, validate, and enrich data stored in Amazon S3.
• Designed and executed a Star Schema dimensional model to build final Gold tables, delivering a high-quality, analytics-ready dataset for executive BI consumption.

Bachelor Of Computer Science in Computer Science – Superior University, Lahore, Pakistan