Skip to main content
Kartik Patel

Kartik Patel

Senior Data Engineer

Talent
VancouverMember since November 19, 2026
KafkaCassandraApache SparkSpark StreamingHBaseImpalaHDFSHiveSqoopFlumeOozieZookeeperCloudera CDHApacheAWSHortonworks HDPSQLPL/SQLPythonUNIXPysparkHiveQLScalaShell ScriptingAWS KinesisApache FlinkSystem Development Life Cycle (SDLC)AgileDatabricksSynapse Analytics
·

Hire this person

Send a job offer directly to this candidate

KafkaCassandraApache SparkSpark StreamingHBase

About

Senior Data Engineer with 6+ years of experience designing and scaling cloud-native data platforms, data lakehouse architectures, and distributed data pipelines for large-scale enterprise analytics. Strong expertise in AWS and Azure ecosystems, building high-performance data solutions using Snowflake, Databricks, Redshift, and Synapse Analytics. Designed and implemented scalable GCP cloud architectures using Compute Engine, GKE, Cloud Storage, BigQuery, Cloud SQL, and VPC.

Proven ability to develop scalable ETL/ELT pipelines using PySpark, SQL, and Spark-based frameworks, enabling efficient processing of structured and semi-structured data. Hands-on experience in real-time data streaming and event-driven architectures using Apache Kafka, AWS Kinesis, and Spark Streaming for low-latency analytics. Built CI/CD pipelines for GCP workloads using Cloud Build, GitHub Actions, Jenkins, Docker, and Kubernetes.

Strong expertise in building high-performance distributed data systems using Snowflake, Databricks, Azure Synapse, and Amazon Redshift, with deep focus on data modeling, performance optimization, and scalable analytics architecture. Developed and optimized ETL/ELT pipelines using Cloud Dataflow, Dataproc, BigQuery, and Cloud Storage. Migrated on-premises and cloud data platforms to GCP BigQuery and Cloud Storage, improving scalability and reducing infrastructure overhead.

Skilled in data modeling and architecture, including dimensional modeling, medallion architecture, and data warehousing for optimized analytics performance. Strong background in machine learning pipelines and MLOps, leveraging Databricks, MLflow, and AWS SageMaker for scalable model training and deployment. Managed relational databases using Azure SQL Database.

Expertise in workflow orchestration and DataOps, using Apache Airflow, CI/CD pipelines, and Infrastructure as Code for automated, reliable deployments. Implemented GCP Pub/Sub for event-driven and real-time data ingestion architectures. Experienced in implementing data governance, security, and compliance frameworks (HIPAA, PCI DSS, SOX) using IAM, encryption, and audit controls.

Experienced in cloud-native optimization using partitioning, bucketing, and Parquet/ORC file formats for cost-efficient storage and high-speed analytics. Strong expertise in PySpark, Apache Flink, and Scala programming, writing distributed code optimized to handle growing data volumes while maintaining SLAs. In-depth knowledge of relational databases, including Oracle, MySQL, PostgreSQL, and SQL Server, designing normalized schemas and writing optimized SQL, PL/SQL, HiveQL, and SnowSQL queries.

Skilled in Azure data services like Synapse, Data Lake, Data Factory, and SQL Database, creating hybrid cloud data solutions integrated with AWS via Direct Connect. Built ACID-compliant data pipelines using Delta Lake, improving data reliability and consistency across ingestion and transformation layers. Automated GCP infrastructure provisioning using Terraf

Experience

Senior Data Engineer - Deloitte - Toronto

(2024-10)

  • Implemented CDC-based data pipelines using AWS DMS and Snowflake Streams, ensuring sub-minute data synchronization across order, warehouse, and logistics systems.
  • Designed dimensional data models in Snowflake, improving historical tracking and enhancing demand forecasting accuracy by 25%.
  • Built and deployed end-to-end ML pipelines using Databricks, MLflow, and AWS SageMaker, reducing model deployment time by 40% and enabling scalable batch inference.
  • Designed secure and scalable GCP VPC networking, including subnets, firewall rules, Cloud NAT, private connectivity, and load balancing.
  • Implemented secure data access using Azure Key Vault, Managed Identity, and Azure RBAC.
  • Developed data movement pipelines using Sqoop for RDBMS-to-Hadoop transfers, Flume for streaming ingestion, and Oozie for dependable execution of large-scale data processes.
  • Designed and implemented Hadoop-based Big Data pipelines on AWS EMR for large-scale data processing and analytics.
  • Built distributed data processing workflows using HDFS, MapReduce, Hive, and Spark on Amazon EMR clusters.
  • Managed production workloads on GCP with a focus on reliability, scalability, performance, security, and cost optimization.
  • Designed and implemented scalable data pipelines using Azure Data Factory and Azure Databricks, processing multi-terabyte datasets with optimized performance.
  • Experienced in working with NoSQL technologies such as MongoDB and DynamoDB, implementing flexible schema models to support semi-structured data.
  • Experience building data lakehouse solutions with Delta Lake, ensuring ACID compliance, data reliability, and unified batch and streaming processing.
  • Worked extensively with GCP BigQuery for enterprise data warehousing, analytics, partitioning, clustering, and performance optimization.
  • Experienced in core AWS services, including S3, EC2, Lambda, Redshift, Glue, DynamoDB, RDS, Aurora, IAM, SNS, and SQS for building scalable, secure solutions.
  • Built MLOps workflows for continuous training, validation, and monitoring of machine learning models using Jenkins, GitLab CI, and Kubernetes.
  • Configured Azure Key Vault for centralized management of secrets, keys, and credentials.
  • Integrated Generative AI and LLM-based workflows with cloud-native data platforms to support intelligent analytics and automated data enrichment.
  • Developed AI-driven data quality frameworks using Python and Spark to identify anomalies, improve data reliability, and reduce manual validation efforts.
  • Developed scalable ETL pipelines using GCP Dataflow, Apache Beam, Python, and PySpark.
  • Developed CI/CD pipelines using Azure DevOps, enabling automated deployment, versioning, and promotion of Databricks and data engineering workloads.
  • Integrated Amazon S3 with Hadoop as a scalable data lake storage layer replacing traditional HDFS storage.
  • Designed and implemented end-to-end Machine Learning pipelines with feature engineering, MLflow-based experiment tracking, and scalable model deployment.
  • Implemented role-based access control (RBAC) and managed data security using Azure Key Vault and managed identities.
  • Implemented large-scale data processing using GCP Dataproc, Apache Spark, PySpark, and Scala.
  • Integrated data from multiple sources including REST APIs, SQL databases, and streaming sources using Azure Event Hubs.
  • Integrated GCP services with Snowflake, Databricks, AWS, and Azure to support multi-cloud data platforms.
  • Designed and developed dashboards and reports using Tableau, Power BI, and Amazon QuickSight, turning complex datasets into actionable analytics.
  • Designed and implemented a Data Lakehouse architecture using Delta Lake on AWS and Databricks, enabling unified batch and streaming analytics across enterprise datasets.
  • Built medallion architecture to standardize data ingestion, transformation, and serving layers for scalable analytics workloads.
  • Implemented disaster

Education

BTech - Information Tech - Charotar University of Science and Technology, India (2020-04)

Skills

Senior Data Engineer

Reviews

Related