Senior Data Engineer (Spark & Python Specialist)

Recruiter

WatersEdge Solutions

Job Ref

1000039007

Date posted

Monday, March 16, 2026

Location

Cape Town, South Africa

SUMMARY

POSITION INFO

Location: Remote (South Africa)

Employment Type: Full-Time

Industry: Data Engineering | Cloud Platforms | Financial Services Technology

WatersEdge Solutions is partnering with a client to recruit a highly skilled Senior Data Engineer (Spark & Python Specialist). This is a strong opportunity for a technically advanced engineer who enjoys building scalable, high-performance data solutions in a modern cloud environment. The role is ideal for someone who thrives on optimisation, code-first engineering, and modernising legacy data logic into clean, portable, Python-centric solutions.

About The Role

As a Senior Data Engineer, you’ll serve as a key technical contributor within the engineering team, focused on building, maintaining, and optimising large-scale data processing engines. You’ll work extensively with Spark, PySpark, Delta Lake, and cloud-based lakehouse environments, helping shape a provider-agnostic platform with strong engineering standards, portability, and performance at its core.

Key Responsibilities

Optimise Spark-based processing through best practices in memory management, shuffle tuning, and partitioning

Build and maintain data pipelines using Python, PySpark, Delta Lake, and Parquet

Refactor legacy SQL-based ETL logic into modular, testable, maintainable Python libraries

Build and optimise medallion architecture layers across Bronze, Silver, and Gold

Support code-first orchestration approaches using tools such as Airflow, Dagster, or Python-based wrappers

Participate in code reviews and mentor junior engineers in PySpark best practices

Contribute to automated testing frameworks using Pytest

Work closely with data scientists, analysts, and business stakeholders to deliver fit-for-purpose solutions

Lead initiatives that strengthen data engineering capability, tooling, and technical best practice

Ensure strong security, compliance, and governance across engineering processes

What You’ll Bring

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field

6 years of Spark / PySpark experience

Strong production-grade Python capability

Solid T-SQL skills with experience interpreting and migrating existing SQL logic

Experience with Azure Synapse Analytics, Dedicated SQL Pools, and Data Factory

Hands-on expertise with Delta Lake and Parquet in high-volume environments

Experience with Docker and open-source, cloud-agnostic engineering standards

Strong collaboration skills and a proven track record modernising large-scale data workloads

Experience mentoring engineers and contributing to technical excellence across a team

Strong understanding of security, compliance, and data governance principles

Nice to Have

Exposure to Microsoft Fabric

Experience with code-first orchestration tooling such as Airflow or Dagster

Experience building reusable internal Python libraries and automated testing patterns

Background in high-scale cloud data platform modernisation

What’s On Offer

Fully remote role based in South Africa

Flexible working arrangements

Wellness support and home office reimbursement

Continuous learning opportunities

Competitive salary, ESOP, and recognition for performance

Supportive and inclusive team culture focused on accountability and work-life balance

Company Culture

This is a team that values transparency, accountability, inclusion, and technical excellence. You’ll join an environment that supports strong engineering standards, continuous learning, and collaborative problem-solving while giving people the flexibility to do their best work in a sustainable way.

If you have not been contacted within 10 working days, please consider your application unsuccessful.