Data Engineer - GCP/BigQuery

Data Engineer

Experience: 6+ years in data engineering, ETL/ELT, analytics engineering, data warehouse development, or big data pipelines; minimum 4 years on GCP cloud data platforms preferred.

What we're looking for:

• Strong SQL and Python.
• Data modelling, warehousing concepts, partitions, clustering, indexing, optimization.
• ETL/ELT pipeline design and production support.
• Batch and streaming pipeline fundamentals.
• Orchestration using Airflow, Composer, Control-M, ADF, dbt, or equivalent.
• Big data processing using Spark, PySpark, Beam, Dataflow, Dataproc, Databricks, EMR, or equivalent.
• Cloud storage and data lake concepts.
• Data quality, reconciliation, schema validation, lineage, monitoring, and alerting.
• CI/CD for data pipelines and code versioning.
• Ability to explain scale scenarios, not just tool definitions.

Technical Skills:

• Python and advanced SQL.
• Data quality checks, reconciliation, alerting, retries, and backfills.
• Production pipeline monitoring and troubleshooting.

GCP Skills:

• Mandatory: BigQuery: datasets, tables, views, partitioning, clustering, SQL optimisation, cost awareness.
• Option 1: Dataflow / Apache Beam or equivalent streaming/batch framework.
• Option 2: Dataproc / PySpark or strong Spark experience.
• Pub/Sub or equivalent event streaming.
• Cloud Composer / Airflow DAG development.

Nice to have:

• Dataform, dbt, CI/CD for data.
• BigQuery ML.
• Dataflow templates.
• Looker / LookML basics.
• Terraform for data infrastructure.