Data Engineer - Python/SQL

Description :

Job Title : Data engineer GCP

Experience needed : 5 to 8 years

Job Purpose :

We are looking for a skilled and proactive Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The ideal candidate will have strong experience in Google Cloud Platform (GCP) technologies, infrastructure as code, and modern data engineering practices, with a focus on delivering reliable and high-quality data systems.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines and data processing systems.
• Work with GCP data services such as BigQuery, Dataflow, Pub/Sub, and Cloud Functions.
• Implement and manage cloud infrastructure using Terraform (Infrastructure as Code).
• Build and maintain CI/CD pipelines using Azure DevOps for automated deployment and testing.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions.
• Ensure data quality, reliability, and performance optimization across pipelines.
• Develop and optimize data transformations using ANSI SQL and Python.
• Work with GKE/Kubernetes for containerized deployments.
• Leverage GCP Vertex AI where applicable for advanced data and ML workflows.
• Follow Agile/Scrum methodologies and actively participate in sprint planning, stand-ups, and retrospectives.
• Use tools like Jira and Confluence for tracking tasks and documentation.
• Monitor, troubleshoot, and resolve data pipeline issues proactively.
• Maintain version control and collaborate using Git.
• Continuously improve data engineering practices, tools, and processes.

Required Skills & Qualifications :

• Strong experience in cloud-based data architecture and platforms (preferably GCP).
• Hands-on experience with BigQuery, Dataflow, Pub/Sub, and Cloud Functions.
• Proficiency in Python for data engineering and pipeline development.
• Strong knowledge of ANSI SQL.
• Experience with Terraform for infrastructure management.
• Experience with Azure DevOps CI/CD pipelines.
• Working knowledge of GKE/Kubernetes.
• Solid understanding of data pipeline design, development, and testing.
• Familiarity with Git for version control.
• Experience working in Agile/Scrum environments.
• Familiarity with Jira and Confluence.
• Strong problem-solving skills with a proactive and ownership-driven mindset.
• Excellent communication and collaboration abilities.