Data Engineer

Job Title : Data Engineer

Experience : 4 - 6 Years

Location : Pune

Job Type : Full-Time

Shift Timing : 2:00 PM - 11:00 PM

Job Overview :

We are looking for an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data integration workflows that support analytics, operational reporting, and AI/ML applications. The ideal candidate should have strong expertise in Python, SQL, cloud-based data engineering, and distributed data processing systems.

Key Responsibilities :

• Design, build, and maintain scalable batch and streaming data pipelines for structured and unstructured data.
• Develop data ingestion, transformation, and integration workflows using Python, SQL, and modern data engineering frameworks.
• Build and maintain streaming pipelines using Kafka or equivalent messaging platforms.
• Work with cloud data services, preferably GCP, including BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Cloud Spanner.
• Develop and maintain REST and GraphQL APIs for data access and integration across microservices.
• Integrate data pipelines with AI/ML models and collaborate with Data Scientists and MLOps teams.
• Build data processing workflows for NLP applications and healthcare datasets, including claims and clinical documents.
• Implement data quality checks, pipeline monitoring, logging, and observability frameworks.
• Automate data pipeline deployment using CI/CD tools such as GitHub Actions, Jenkins, or Argo CD.
• Optimize pipeline performance, data reliability, scalability, and processing costs.

Required Qualifications :

• 4 - 6 years of experience in data engineering, data pipelines, or backend data workflows.
• Strong proficiency in Python and SQL.
• Experience with relational databases such as PostgreSQL.
• Hands-on experience building batch and streaming data pipelines.
• Experience with Kafka or similar streaming technologies.
• Experience with REST APIs and GraphQL for data integration and service development.
• Experience with cloud platforms, preferably Google Cloud Platform (GCP).
• Familiarity with BigQuery, Cloud Storage, Dataflow, Pub/Sub, or equivalent cloud data services.
• Experience with CI/CD practices and version control tools.
• Understanding of distributed systems, data modeling, data quality, and pipeline monitoring.

Preferred Qualifications :

• Experience processing healthcare data, claims data, or clinical documents.
• Exposure to AI/ML data pipelines and NLP workflows.
• Familiarity with GCP Composer, Cloud Spanner, or Cloud Functions.
• Knowledge of Docker, Kubernetes, and Infrastructure as Code.
• Experience with large-scale data processing and cloud cost optimization.

Education :

• Bachelors degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent practical experience.