Senior Data Engineer - Python/SQL/ETL

Job Summary

We are seeking an experienced Senior Data Engineer to design, build, and optimize scalable data pipelines and modern cloud-based data platforms. The ideal candidate will possess strong expertise in Python, PySpark, Azure Databricks, ETL/ELT frameworks, SQL optimization, and DevOps practices.

You will be responsible for developing robust data solutions, ensuring data quality and performance, and enabling efficient analytics and reporting across enterprise systems.

Key Responsibilities :

• Design, develop, and maintain scalable ETL/ELT pipelines for processing large volumes of structured and unstructured data.
• Build and optimize data workflows using Python, PySpark, and Azure Databricks.
• Develop high-performance data models, schemas, and transformation logic for enterprise data platforms.
• Create and optimize complex SQL queries, stored procedures, and database operations.
• Implement data quality checks, validation frameworks, and monitoring solutions.
• Collaborate with Data Scientists, Analysts, Architects, and Business stakeholders to deliver data-driven solutions.
• Automate deployment, testing, and release processes using CI/CD pipelines and DevOps best practices.
• Deploy and manage applications and data services in containerized environments using OpenShift, Kubernetes, and HELM.
• Configure dashboards and observability tools to monitor data pipeline health, system performance, and operational metrics.
• Ensure security, scalability, reliability, and governance across data engineering solutions.

Required Skills &

Qualifications :

Programming &

• Data Processing :
• Strong proficiency in :

a. Python b. PySpark

• Experience handling large-scale distributed data processing and transformation workloads.
• Strong understanding of data structures, performance optimization, and parallel processing concepts.

Data Warehousing &

• SQL :
• Expertise in SQL for :

a. Complex querying b. Data manipulation c. Schema design d. Data modeling

• Strong understanding of relational and dimensional data warehousing concepts.
• Experience in SQL optimization and performance tuning is highly preferred.

ETL / ELT Development :

• Proven experience in designing, developing, and maintaining enterprise-grade ETL/ELT pipelines.
• Hands-on experience with :

a. Data ingestion b. Data transformation c. Workflow orchestration d. Batch and near real-time processing

• Experience building reusable and scalable data frameworks.

Cloud Data Platform (Azure Focus) :

• Hands-on experience with :

a.

Azure

Databricks b.

Azure Data

Factory c. Azure SQL Server d.

Azure Key

Vault

• Strong understanding of Azure-based analytics and data engineering ecosystems.
• Experience with cloud-native architecture and secure data handling practices.

DevOps &

• CI/CD :
• Experience implementing CI/CD pipelines using :

a. GitHub b. GitHub Actions

• Knowledge of :

a. Automated testing b. Unit testing frameworks c. Build and deployment automation

• Familiarity with version control and release management processes.

Containerization &

• Orchestration :
• Practical experience with :

a. OpenShift b. Kubernetes c. HELM

• Experience deploying, configuring, and managing containerized applications and services.

Monitoring &

• Observability :
• Experience setting up and configuring :

a. Grafana dashboards b. Monitoring and alerting systems

• Ability to monitor :

a. Data quality b. Pipeline health c. System performance d. Operational metrics

Preferred Qualifications :

• Experience with Delta Lake, Spark optimization, and lakehouse architecture.
• Familiarity with orchestration tools such as Apache Airflow.
• Exposure to streaming technologies like Kafka or Event Hub.
• Knowledge of data governance, security, and compliance standards.
• Experience working in Agile/Scrum delivery environments.

Key Performance Indicators (KPIs) :

• Data Pipeline Reliability and Uptime
• ETL/ELT Processing Performance
• Data Quality and Accuracy Metrics
• Query Optimization and Processing Efficiency
• Deployment Automation Success Rate
• Monitoring and Incident Resolution Efficiency