Abdullah Solaiman

Results-driven Lead Data Engineer with 10+ years of experience designing and modernizing cloud-native data platforms and production-grade ETL/ELT pipelines across AWS and Azure. Strong expertise in Databricks, Azure Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, Snowflake, and Medallion Architecture, with hands-on experience administering Databricks Unity Catalog, managing Databricks Jobs and Workflows, and implementing governed data access and Delta Sharing. Proven ability to build scalable batch and streaming pipelines, optimize large-scale SQL and Spark workloads, implement data quality, observability, lineage, and security frameworks, and deliver reliable analytics-ready data platforms across financial services, healthcare, reporting, and compliance environments.

Lead Data Engineer at Wells Fargo (2024-12 – Present)

• Led the modernization of enterprise data platforms by migrating legacy ETL workloads to cloud-native Lakehouse architecture using AWS Glue, Databricks, Snowflake, and Apache Spark, improving scalability, governance, and operational efficiency across business-critical reporting systems.
• Architected and implemented scalable Medallion Architecture (Bronze, Silver, Gold) pipelines for ingesting and transforming structured and semi-structured datasets, enabling trusted and analytics-ready data products for risk, compliance, and finance teams.
• Administered Databricks Unity Catalog structures including catalogs, schemas, permissions, and role-based access controls, supporting governed access to bronze, silver, and gold data assets across business domains.
• Designed and implemented production-grade ETL/ELT pipelines using Python, SQL, AWS Glue, Databricks, and Snowflake, supporting multi-step data ingestion, transformation, validation, and delivery workflows.
• Implemented Python unit, integration, and system testing frameworks using PyTest, improving code quality, deployment confidence, and production stability.
• Managed Databricks Jobs and Workflows for scheduled batch pipelines, implementing dependency management, validation, failure recovery, monitoring, and alerting to improve pipeline reliability and reduce manual intervention.
• Built high-performance distributed processing frameworks using PySpark and Spark SQL to process multi-terabyte datasets, reducing pipeline execution times and improving resource utilization across cloud environments.
• Implemented Databricks Unity Catalog and enterprise governance controls to provide centralized metadata management, lineage tracking, fine-grained security policies, and regulatory compliance across multiple data domains.
• Optimized large-scale SQL and Spark workloads through query tuning, partitioning, clustering, workload optimization, and resource management to improve pipeline performance and reliability.
• Supported governed data distribution through Delta Sharing, enabling secure and controlled delivery of curated datasets to internal or external data consumers while maintaining appropriate access policies and data governance.
• Built scalable data pipelines and transformation frameworks from the ground up for structured and semi-structured datasets, incorporating incremental processing, data validation, monitoring, and failure recovery patterns.
• Developed near real-time streaming solutions using Kafka, Kinesis, and Spark Structured Streaming to support fraud monitoring, operational intelligence, and event-driven analytics with low-latency data processing.
• Established enterprise-wide data observability and quality frameworks using Great Expectations, OpenLineage, and CloudWatch, enabling proactive monitoring, anomaly detection, and improved SLA compliance.
• Optimized Snowflake and Redshift workloads through advanced partitioning, clustering, workload isolation, and query tuning techniques, significantly improving performance for enterprise reporting and analytical workloads.
• Automated infrastructure deployment and platform provisioning using Terraform and Infrastructure-as-Code principles, ensuring consistent, repeatable, and secure environments across Development, QA, and Production.

Senior Data Engineer at CDW Corporation (2022-01 – 2024-11)

• Architected and built cloud-native Azure data platforms from the ground up using Azure Data Factory, Azure Databricks, Synapse Analytics, and ADLS Gen2 to support enterprise reporting, analytics, and large-scale data integration.
• Designed and implemented multi-step ETL/ELT pipelines using Azure Data Factory, Python, PySpark, and SQL for structured and semi-structured data across multiple source systems.
• Designed and developed modern data integration pipelines using Microsoft Fabric Data Factory and Azure Data Factory (ADF) to orchestrate ingestion, transformation, validation, and delivery of enterprise data across analytics workloads.
• Developed reusable PySpark and Python transformation frameworks and standardized pipeline patterns to improve code quality, maintainability, and delivery consistency across enterprise data workloads.
• Built enterprise Master Data Management (MDM) solutions for customer, product, and vendor domains, ensuring data consistency, governance, and trusted reporting across business functions.
• Supported Azure Databricks platform standards by establishing consistent workspace, cluster, storage, and pipeline configuration practices across development and production environments.
• Implemented event-driven architectures using Azure Event Hubs and Azure Functions to support near real-time analytics, operational monitoring, and streaming data integration requirements.
• Designed and implemented healthcare data ingestion frameworks supporting HL7, FHIR, and EMR integrations, enabling compliant and scalable healthcare analytics platforms.
• Automated CI/CD deployment pipelines using Azure DevOps and Git-based workflows, improving release reliability across development and production environments.
• Established metadata management, lineage tracking, and governance standards across enterprise data assets, improving regulatory compliance, transparency, and data stewardship capabilities.
• Developed dimensional data models and curated analytical datasets in Synapse Analytics supporting executive dashboards, self-service reporting, and operational decision-making.
• Built automated reconciliation, validation, and data quality frameworks that improved reporting accuracy and reduced manual intervention in critical business processes.
• Optimized Azure Databricks compute and Spark workloads through cluster sizing, partitioning, caching, and adaptive query execution, balancing pipeline performance, reliability, and cloud compute costs.
• Collaborated with business stakeholders, architects, and analytics teams to deliver scalable data solutions supporting strategic initiatives and enterprise-wide digital transformation programs.

Senior Data Engineer at Pinterest (2018-09 – 2021-12)

• Led enterprise data warehouse modernization initiatives by migrating legacy reporting platforms to Snowflake, improving scalability, performance, and operational efficiency for analytics workloads.
• Designed and implemented scalable ingestion frameworks utilizing Snowpipe, Streams, and Tasks to automate batch and incremental data processing from diverse enterprise data sources.
• Developed cloud-native ELT pipelines for ingesting and transforming structured and semi-structured datasets, enabling domain-driven analytics and improved business intelligence capabilities.
• Implemented enterprise governance and security controls including RBAC, auditing, and data-sharing policies to ensure secure and compliant access to business-critical information.
• Designed high-performance dimensional data models supporting product analytics, marketing insights, financial reporting, and executive dashboards across multiple business functions.
• Strengthened governed data-access practices through RBAC, auditing, data-sharing policies, and controlled permissions for business-critical datasets, supporting secure access for internal analytics and downstream data consumers.
• Built reusable data engineering frameworks and automation processes that accelerated onboarding of new datasets while improving consistency and maintainability across the platform.
• Integrated Kafka event streams with Spark processing pipelines to support near real-time analytics, customer behavior analysis, and operational reporting requirements.
• Optimized Snowflake analytical