Abdulahi Mubara Aidid

Senior AI and Data Engineer experienced in designing cloud data platforms, production machine learning systems, and practical generative AI applications. Builds reliable batch and streaming pipelines, lakehouses, and analytics layers using Python, SQL, PySpark, Spark, and Databricks, with ingestion from databases, APIs, files, and event streams. Develops maintainable solutions from data ingestion and feature engineering through deployment, monitoring, governance, and collaboration with product, analytics, and engineering teams.

Senior AI & Data Platform Engineer - Mikata Health - Calgary, Alberta, Canada

(2021-08 - 2026-09)

Healthcare technology

• Tools / Environment: Python, SQL, PySpark, Databricks, Delta Lake, AWS S3, Azure Data Lake, Kafka, Airflow, MLflow, PyTorch, FastAPI, LangGraph, RAG, pgvector, Docker, Kubernetes, Terraform, GitHub Actions, Prometheus
• Architected cloud-native data platforms for clinical, pharmacy, claims, and demographic datasets. Created ingestion, standardization, and curated analytics layers in SQL that supported reporting, risk scoring, and machine learning.
• Developed scalable batch and near-real-time pipelines with Python, PySpark, Kafka, and Airflow. Added schema checks, retry handling, alerting, and replay procedures to improve reliability across multiple source systems.
• Built governed lakehouse datasets with Delta Lake and Databricks on Azure Data Lake. Designed partition strategies, incremental processing, and reusable transformations for structured and semi-structured healthcare records.
• Implemented feature engineering and model-training workflows for patient risk prediction, operational forecasting, and clinical resource planning. Standardized training data, validation, and experiment tracking with MLflow.
• Designed generative AI applications using retrieval-augmented generation (RAG), embedding search, and LLM APIs. Connected approved internal documents to secure services that return source-grounded responses.
• Created document ingestion, chunking, retrieval, and evaluation pipelines for AI knowledge assistants. Added access-aware filtering, prompt tests, and human review for sensitive responses.
• Built model-serving APIs with FastAPI and containerized inference workloads using Docker and Kubernetes. Improved service observability through request metrics, latency tracking, and error reporting.
• Established model monitoring practices for prediction quality, data drift, feature freshness, and operational health. Documented response procedures for degraded data feeds and failed deployments.
• Integrated data governance controls including role-based access, audit logging, lineage, sensitive-data handling, and documented retention policies for regulated information.
• Automated infrastructure provisioning and release workflows using Terraform and CI/CD pipelines. Added automated tests for data transformations, model interfaces, and deployment configurations.
• Partnered with data scientists, clinicians, product managers, and security teams to convert complex business questions into measurable AI and data engineering deliverables.
• Mentored engineers through design reviews, code reviews, production troubleshooting, and technical documentation. Encouraged reusable components and clear ownership of data and model services

Senior Data Engineer / ML Platform Engineer - Optimus Information - Vancouver, British Columbia, Canada

(2018-07 - 2021-07)

Enterprise cloud consulting

• Tools / Environment: Python, Scala, SQL, Apache Spark, Hadoop, Hive, Kafka, Apache NiFi, Airflow, Databricks, Delta Lake, Snowflake, Azure, AWS, GCP, MLflow, Terraform, Jenkins, Docker, Power BI
• Designed and implemented cloud data lakehouse architectures for enterprise clients migrating from legacy data warehouses. Used Databricks, Delta Lake, Azure Data Lake, cloud object storage, and curated SQL layers to support analytics.
• Developed large-scale batch and streaming data pipelines with Apache Spark, Kafka, Hadoop, and Airflow. Standardized ingestion from transactional databases, APIs, file feeds, and event streams using Python and SQL.
• Automated ETL and ELT workflows using Apache NiFi, Airflow, and Python. Introduced dependency management, validation checkpoints, and operational dashboards for daily processing.
• Built dimensional models, fact and dimension tables, and reusable data marts. Worked with reporting teams to define trusted business measures and consistent transformation rules.
• Optimized Spark processing with partition tuning, caching, efficient joins, SQL query optimization, and file compaction. Investigated bottlenecks in distributed workloads and improved execution stability.
• Led data warehouse modernization activities including source inventory, mapping, migration planning, historical backfills, reconciliation, and cutover support.
• Integrated Snowflake and Databricks datasets for business intelligence, self-service reporting, and advanced analytics. Tuned SQL queries and storage layouts for high-volume reporting.
• Collaborated with machine learning teams to build feature pipelines and deploy predictive models. Used MLflow for experiment tracking, model packaging, and repeatable release procedures.
• Developed data quality frameworks for completeness, uniqueness, schema consistency, and reconciliation. Added alerts and documented exception-handling processes for failed checks.
• Implemented data governance practices covering metadata, lineage, role-based access, and documentation of sensitive datasets across cloud platforms.
• Created CI/CD pipelines and infrastructure automation for data jobs, notebooks, and supporting services using Jenkins, Terraform, and containerized development environments.
• Presented technical designs and migration trade-offs to architects and business stakeholders, and supported junior engineers with troubleshooting and coding standards

Data Engineer - Massive Insights Inc. - North York, Toronto, Ontario, Canada

(2017-08 - 2018-05)

Retail and business analytics

• Tools / Environment: Python, SQL, Apache Beam, Google Cloud Dataflow, BigQuery, AWS S3, PostgreSQL, Airflow, Talend, REST APIs, Great Expectations, Tableau, Power BI, Git
• Built and maintained data ingestion and transformation pipelines for sales, inventory, customer, and operational datasets from multiple business systems.
• Developed ETL workflows using Python, SQL, Apache Beam, and managed cloud processing services to prepare analytics-ready data at scale.
• Designed a cloud data lake and organized raw, standardized, and curated data zones to make reporting datasets easier to discover and reuse.
• Integrated data from REST APIs, relational databases, CSV exports, and third-party platforms. Implemented incremental loading and repeatable source-to-target mappings.
• Created dimensional warehouse models with staging tables, facts, dimensions, and business-friendly reporting views for sales and customer analytics.
• Implemented automated data validation and reconciliation with Python and Great Expectations. Investigated mismatched records and improved source-data quality.
• Optimized SQL queries and data processing jobs through indexing, partitioning, and transformation refactoring to support faster reporting refreshes.
• Prepared clean feature datasets for forecasting, customer segmentation, and churn analysis in partnership with analysts and data scientists.
• Developed operational dashboards and reporting datasets for Tableau and Power BI, with consistent definitions for key performance indicators.
• Monitored scheduled jobs, resolved pipeline failures, and documented recovery procedures for production support and business continuity.
• Worked with product and operations teams to gather requirements, explain data limitations, and prioritize reliable delivery of new reporting capabilities

Bachelor of Science - Computer Science - University of Toronto (2013 - 2017)