Python + AWS - Developer

Job Description

Role description The Senior Data Engineer will be responsible for designing developing and maintaining largescale data ingestion and processing pipelines The role focuses on Oracle to Big Data ingestion Sparkbased transformations and production support for critical data pipelines

Roles Responsibilities

Data Engineering Development

Design and build endtoend data pipelines from OracleRDBMS to Big Data platforms

Develop scalable transformations using Apache Spark PySpark Scala Spark SQL

Implement staging curated and consumption layers

Create and manage Hive external tables ORC Iceberg

Handle snapshot incremental and CDCbased ingestion

Scheduling Operations

Develop and maintain Autosys jobs JILs box jobs dependencies

Monitor batch pipelines and ensure SLA adherence

Provide production support issue analysis and RCA

DevOps Collaboration

Use Git Bitbucket for version control and PR reviews

Support CICD deployments RLM Jenkins Lightspeed

Prepare technical documentation runbooks and KT documents

Mentor junior engineers

Required Skills

Apache Spark PySpark Scala Spark SQL

Strong SQL Oracle SQL

Iceberg

Unix Shell scripting

Autosys scheduling

Git Bitbucket

Production support experience

Good to Have

Iceberg Starburst Trino

CDP Public Cloud

Databricks

Data quality frameworks

Mandatory Certification Cloudera CCA Spark and Hadoop Developer CCA175