Senior Python Developer - Data Engineering

Description

:

Job Summary

:

We are looking for an experienced Senior Python Developer with strong expertise in designing, developing, and deploying scalable applications. The ideal candidate should have hands-on experience in Python-based frameworks, backend development, APIs, and cloud technologies, along with a solid understanding of software engineering best practices.

Key Responsibilities

:

• Design, develop, and optimize scalable data pipelines and ETL processes using Azure Data Services and Databricks.
• Implement data transformations and analytics using PySpark and Python.
• Configure and manage Databricks Unity Catalog for secure data governance and access control.
• Orchestrate workflows using Apache Airflow for scheduling and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions in an Agile environment.
• Set up and maintain CI/CD pipelines for data workflows and deployments.
• Manage version control and branching strategies using Git.
• Write and execute unit tests to ensure code quality and reliability.
• Monitor, troubleshoot, and optimize data processes for performance and cost efficiency.

Role Requirements and Qualification

:

• Azure Knowledge

:

Strong experience with Azure Data Factory, Azure Data Lake, Azure Synapse, or related services.

• Programming

:

Proficiency in Python and PySpark for data engineering tasks.

• Databricks

:

Hands-on experience in building and managing workflows in Databricks, including Unity Catalog.

• Orchestration

:

Experience with Apache Airflow for workflow scheduling and automation.

• CI/CD

:

Familiarity with tools like Azure DevOps or Jenkins for automated deployments.

• Version Control

:

Expertise in Git for code management.

• Testing

:

Strong understanding of unit testing frameworks for data pipelines.

• Agile Methodology

:

Experience working in Agile teams and using tools like Jira or Azure Boards.

• 5+ years of relevant experience in Python development.

Mandatory skills

:

• Python, Databricks, git

Desired skills

:

• Deep expertise in python
• Experience with Databricks, pyspark and Git.
• Experience with Azure cloud platform