Job description
Role: Senior Data Engineer
Skill
- Fluent in Python programming language
- Proficient in SQL
- Understanding of OLTP concepts including normalised schema design
- Understanding of OLAP concepts and data model design
- Experience leading the designing and building data pipelines, ETL processes, and managing data infrastructure
- Experience leading workflows automation, managing task dependencies and transformations using Apache Airflow and DBT
- Experience in real time streaming pipelines like Kafka/pubsub and, Pyspark/dataflows
- A strong analytical mindset, a passion for resolving complex technical challenges, and the ability to find creative solutions to issues
- Excellent communication and interpersonal skills, with the ability to lead technical discussions, build consensus, and work effectively with cross-functional teams
- Deep understanding of cloud migration strategies, best practices, and potential pitfalls
- Experience with Google Cloud Platform, AWS or Azure including services for data storage, processing, and analytics
- Experience with cloud deployment and test automation using a CI/CD solution such as Cloud Build or Concourse
- Familiarity with consumer behaviour data (Google Analytics, Adobe Analytics or Snowplow Analytics) and/or subscribers data
- Proficient usage of AI to gain working efficiency to focus on higher impact architectural tasks
Experience and Role Responsibilities
- 5 + Senior Data Engineer
- Design, develop and maintain data pipelines that extract, load and transform data from various sources into the data warehouse
- Design and implement data models that align with business requirements and support efficient querying and analysis
- Ensure the team's orchestration pipelines are reliable, scalable and efficient whilst considering factors including data volume, velocity and variety
- Work across the data engineering teams, vendors and internal stakeholders to contribute to the continuous improvement of data products, processes and systems.
- Follow data engineering best practices including coding, DevOps, documentation and metadata standardisations
- Experience with the management and nature of audience tracking and subscribers datasets
- Take ownership and accountability for assigned work, including development and end-to-end testing
- Ability to conduct comprehensive feasibility assessments of architectural and data modelling options, systematically evaluating the trade-offs, advantages, and limitations of each proposed solution
- Ensure high standard and ongoing quality of Engineering output whilst meeting project timelines
- Document existing and future solution designs to enhance the team's collective knowledge
Keywords
OrchestrationCodingApache KafkaMetadataApache AirflowDevOpsAirflowPythonSqlApache Http ServerCI/CDBig data
Interested in this role?