GCP Data Engineer - Python / PySpark

Experience : 5.5 - 9 Years

Openings : 8

Role Overview :

We are looking for experienced Data Engineers with strong expertise in Python or PySpark, SQL, GCP cloud services, data processing, data warehousing, and data pipelines. The ideal candidate should have hands-on experience with BigQuery, Dataflow or Dataproc, Pub/Sub, Apache Kafka, Airflow/Cloud Composer, and Terraform.

Key Responsibilities :

Design, develop, and maintain scalable data pipelines.
Develop data processing solutions using Python and/or PySpark.
Develop complex SQL queries and optimize data processing workloads.
Work with GCP data services including BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Functions.
Build and maintain data ingestion and transformation pipelines.
Develop and manage workflows using Airflow, Astronomer, or Cloud Composer.
Work with Apache Kafka for real-time data processing and streaming.
Implement infrastructure automation using Terraform.
Monitor data pipelines and troubleshoot production issues.
Develop monitoring dashboards and ensure pipeline reliability.
Collaborate with Data Scientists, Software Engineers, business stakeholders, and other technical teams.

Mandatory Technical Skills :

SQL - minimum 4 years
Python or PySpark - minimum 4 years
GCP - minimum 3 years
BigQuery
Dataflow or Dataproc
Pub/Sub
Scheduled Queries
Cloud Functions
Monitoring Tools / Dashboards
Apache Kafka
Terraform scripting - minimum 2 years
Airflow / Astronomer / Cloud Composer - minimum 2 years

Good to Have :

Data Fusion, Tekton Pipeline, CI/CD Pipeline, GCP Secret Manager
Data/ML solutions and advanced analytics exposure
Prompt Engineering, Generative AI, LLM, Agentic AI, LangGraph, CrewAI, OpenAI SDK, Google ADK

Soft Skills :

Strong written and verbal communication skills
Strong presentation and stakeholder management skills
Excellent analytical and problem-solving abilities
Strong collaboration and interpersonal skills

GCP Data Engineer - Python

Job description