Skip to main content

Embark Group - Data Engineer - Google Cloud Platform

Technology
3 weeks ago

Job description

Role Overview :

We are looking for an experienced GCP Data Engineer to design, develop, and optimize scalable cloud-based data solutions. The role involves building and maintaining data pipelines, data models, and analytics platforms using GCP, dbt, and Apache Airflow.

Responsibilities :

  • Design, develop, and maintain scalable data pipelines and workflows on Google Cloud Platform (GCP).
  • Build and optimize data models using dbt Cloud, including models, macros, tests, and Slowly Changing Dimensions (SCDs).
  • Develop and manage Apache Airflow workflows, optimizing orchestration, scheduling, and pipeline performance.
  • Work with BigQuery to develop complex queries, optimize performance, and manage data processing costs.
  • Implement data modelling practices using fact and dimension tables, SCDs, and appropriate schema design.
  • Develop data quality checks, validation frameworks, and monitoring mechanisms to ensure data accuracy and reliability.
  • Manage version control and CI/CD workflows using GitHub, Bitbucket, and relevant deployment tools.
  • Support infrastructure automation using Terraform or CloudFormation.
  • Apply cloud security and IAM best practices to protect data and manage access.
  • Build and maintain cross-cloud data pipelines where required.
  • Collaborate with cross-functional teams to deliver reliable and scalable data engineering solutions.

Requirements :

  • 4 - 9 years of experience in data engineering, with strong hands-on expertise in GCP.
  • Mandatory experience with GCP, dbt, and Apache Airflow.
  • Strong proficiency in BigQuery, including query tuning, performance optimization, and cost management.
  • Hands-on experience with dbt Cloud, including data models, SCDs, tests, macros, and governance.
  • Strong SQL skills and proficiency in Python for data processing and pipeline development.
  • Experience designing and optimizing Airflow workflows and data orchestration processes.
  • Knowledge of data modelling concepts, including fact and dimension tables and SCDs.
  • Experience with GitHub or Bitbucket, CI/CD tools, and infrastructure-as-code tools such as Terraform or CloudFormation.
  • Understanding of data quality frameworks, cloud security, and IAM best practices.
  • Exposure to cross-cloud data pipelines is an advantage.
  • B.E./B.Tech in Computer Science, Information Technology, or a related discipline is preferred.
Keywords
OrchestrationMacroApache AirflowAirflowPythonSqlApache Http ServerCI/CDBigQuery

Interested in this role?