Role: Python Developer – Data Engineering

Location: Toronto, ON (4 days hybrid)

Develop and optimize data manipulation and transformation workflows using pandas and Polars.
Design, build, and maintain efficient data processing pipelines for large-scale datasets.
Optimize Python code and data workflows for performance, scalability, and memory efficiency.
Develop containerized applications using Docker and deploy scalable services using Kubernetes.
Build and maintain data pipelines integrated with ClickHouse or similar columnar databases for analytical workloads.
Design and implement event-driven architectures using NATS for asynchronous data processing and messaging.
Develop distributed computing solutions using Dask to process datasets that exceed single-machine memory limitations.
Write and maintain comprehensive unit and integration tests using pytest.
Maintain high code quality, test coverage, reliability, and performance standards.
Use Git for version control and follow established branching, code review, and collaboration practices.
Collaborate with data engineers, software developers, DevOps teams, and other stakeholders to deliver reliable data solutions.
Troubleshoot production issues and support the continuous improvement of data engineering processes.

Requirements:

Bachelor’s degree in computer science or Diploma in IT with equivalent experience.

Python and Data Processing

Advanced proficiency in Python.
Strong experience with pandas and Polars for data manipulation, transformation, and analysis.
Experience working with large datasets and optimizing code for performance and memory usage.
Strong understanding of Python development best practices, object-oriented programming, and clean code principles.

Containerization and Orchestration

Hands-on experience building and managing container images using Docker.
Experience creating and maintaining multi-container applications.
Working knowledge of Kubernetes for container orchestration, deployment, scaling, and service management.

Data Infrastructure

Experience with ClickHouse or a similar columnar database.
Understanding of OLAP workloads, analytical queries, data modeling, and query optimization.
Experience designing data pipelines for analytical and reporting use cases.

Messaging and Streaming

Familiarity with NATS.io and message-driven architectures.
Understanding of asynchronous processing, event-driven systems, messaging patterns, and distributed workflows.

Testing and Quality Assurance

Proficiency with pytest.
Experience writing unit tests, integration tests, and regression tests.
Understanding of test coverage, mocking, test automation, and quality assurance practices.

Distributed Computing

Experience with Dask for parallel processing and out-of-core computation.
Understanding of distributed data processing and techniques for handling datasets that exceed available system memory.

Version Control

Strong command of Git, including branching strategies, pull requests, code reviews, merging, and conflict resolution.
Experience collaborating through shared code repositories and following software development best practices.

Python Developer

Explore more