Ali Khan

Senior Data Engineer with 7+ years of experience designing, developing, and optimizing cloud-native data platforms, enterprise data warehouses, and real-time streaming solutions across Azure, AWS, and GCP. Specialized in building high-performance ETL/ELT pipelines, modern Lakehouse architectures, and distributed data processing systems using Databricks, Apache Spark, Kafka, Snowflake, Delta Lake, Airflow, dbt, and Python. Proven expertise in cloud migrations, infrastructure automation, performance optimization, data governance, and scalable analytics platforms that enable reliable business intelligence and machine learning workloads.

Strong collaborator with cross-functional engineering, analytics, and data science teams, delivering secure, resilient, and production-ready data solutions

Senior Data Engineer - Nexla

(2024-06)

• Designed and deployed cloud-native data platforms using Databricks, Apache Spark, Snowflake, and Python, enabling scalable analytics and supporting multiple enterprise business domains.
• Built 40+ enterprise ETL/ELT pipelines using Apache Airflow, dbt, Spark, and cloud-native services, processing 100M+ records monthly while improving pipeline reliability, maintainability, and overall data quality.
• Developed real-time streaming solutions with Apache Kafka and Spark Structured Streaming, processing millions of events daily and enabling low-latency analytics for business-critical applications.
• Implemented modern Lakehouse architectures using Delta Lake and Apache Iceberg, improving data consistency, schema evolution, governance, and long-term platform scalability.
• Optimized Spark workloads, SQL queries, storage layouts, and partitioning strategies, reducing data processing time by 50%+ while improving compute efficiency and reporting performance.
• Automated infrastructure provisioning, CI/CD deployments, monitoring, and governance using Terraform, GitHub Actions, Unity Catalog, and RBAC, significantly reducing manual operations and improving production reliability.

Data Engineering Team Lead - Redpanda Data

(2021-04 - 2024-05)

• Led the design and development of distributed data platforms using Apache Kafka, Apache Spark, Python, and cloud-native technologies, supporting enterprise-scale batch and streaming workloads.
• Architected scalable streaming pipelines capable of processing millions of events daily, improving throughput, fault tolerance, operational visibility, and real-time analytics capabilities.
• Modernized 30+ production ETL pipelines by migrating legacy workloads to Databricks, Delta Lake, and Snowflake, improving scalability, simplifying maintenance, and accelerating cloud adoption.
• Designed reusable Apache Airflow workflows with automated scheduling, dependency management, monitoring, alerting, and recovery, improving production stability and reducing operational overhead.
• Collaborated with engineering, analytics, and platform teams to optimize SQL workloads, strengthen data governance, implement monitoring frameworks, and deliver highly available enterprise data platforms.

Data Engineer - dbt Labs

(2019-04 - 2021-03)

• Developed 30+ scalable ETL/ELT pipelines using Python, SQL, dbt, Apache Airflow, and Snowflake, delivering trusted analytics-ready datasets for enterprise reporting and business intelligence.
• Designed dimensional data models using Star Schema and Snowflake Schema, supporting scalable reporting, self-service analytics, and enterprise data warehousing initiatives.
• Built automated testing, validation, and data quality frameworks that improved data consistency, pipeline reliability, and production stability across analytical environments.
• Optimized Snowflake warehouses, PostgreSQL databases, and SQL workloads through query tuning, indexing, and performance optimization, improving query execution and overall platform efficiency.
• Partnered with analytics engineers, business stakeholders, and cross-functional teams to translate complex business requirements into scalable, production-grade data pipelines and analytical data models.

Master of Science - Computer Science - Binghamton University (2017-03 - 2019-02)

BSc - Computer Science - National University of Sciences and Technology (NUST) (2013-03 - 2017-02)