Senior Data Engineer - Python/AWS

Role: Senior Data Engineer

Experience: 7+ Years

Location: Pune

Joining: Immediate to 30 Days' Notice Period

Role Summary:

We are looking for a skilled Sr. Data Engineer with strong expertise in Python and cloud-native technologies to design, develop, and maintain scalable data pipelines and distributed systems. The ideal candidate should have hands-on experience with ETL/ELT pipelines, workflow orchestration, AWS cloud services, data visualization, distributed architectures, and modern data engineering practices.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build and manage distributed and event-driven architectures for large-scale data processing.
• Develop backend services and data infrastructure using Python.
• Implement DAG-based workflow orchestration using Airflow, Prefect, or Dagster.
• Work with AWS services including Lambda, S3, ECS/EKS, SQS, and CloudWatch.
• Design and optimize data models and schema structures.
• Develop analytics dashboards and reporting solutions using Amazon QuickSight.
• Implement CI/CD pipelines and Infrastructure as Code (IaC).
• Monitor and maintain data platforms using observability and monitoring tools.
• Collaborate with Data Scientists, ML Engineers, and Backend Developers.
• Support AWS cloud migration and modernization initiatives.

Mandatory Skills:

• Strong proficiency in Python.
• Experience with Docker and Kubernetes.
• Strong understanding of distributed and event-driven architectures.
• Hands-on experience building ETL/ELT pipelines.
• Strong knowledge of data modeling and schema design.
• Experience with Airflow, Prefect, or Dagster.
• Hands-on experience with AWS Lambda, S3, ECS/EKS, SQS, and CloudWatch.
• Experience with CI/CD and Infrastructure as Code (IaC).
• Hands-on experience with Amazon QuickSight for dashboards, reporting, and analytics.

Good to Have:

• AWS Application Migration Service (MGN/AMC) or similar migration technologies.
• Logstash or other data ingestion frameworks.
• OpenSearch / Elasticsearch.
• Kafka / Kinesis and streaming technologies.
• Prometheus, Grafana, or ELK Stack.
• MLOps lifecycle knowledge.
• Vector databases and embedding pipelines.
• Large-scale indexing and search optimization.
• RAG, Natural Language Search, GenAI, knowledge graphs, prompt engineering, and embedding models.
• Knowledge of Golang, Rust, Java, C/C++, or other compiled languages.