Data Analytics Engineer

We are looking for an experienced Data Analytics Engineer with strong experience in Python in the financial sector, and good communication skills, ability to refine requirements together with the business.

Tasks & Responsibilities:

• Develop required code for end-to-end data pipelines with optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, SQL and other technologies. Incorporate business rules and functional capabilities, security, data retention requirements etc
• Assist business analysts and interact with the business to identify, capture and analyse business requirements
• Translate functional and technical requirements into detailed design
• Participate and contribute to overall data architecture and design
• Provide responsive support, both during and outside business hours, per team rotation. The existing data pipelines are implemented in C# however will be migrated to Python.
• Work with stakeholders including the business area sponsors, product owners, data architect, data engineers and business analysts to assist them with their data-related technical issues and support their data needs.
• Ensure effective design and development of system architectures through frequent product deliveries; employing effective governance methods for transparency and communication.
• Remain up-to-date with industry standards and technological advancements that will improve the quality of your output
• Assist and support proof of concepts as data technology evolves

Must haves

• Minimum 5 years work experience in modern data technologies
• Experience in building data ingestion pipelines for data warehouse and/ or data lake/lakehouse architecture
• Hands-on development in using open-source data technologies such as Hadoop, Hive, Spark, HBase, Kafka, Impala, dbt, ELK etc. preferably with Cloudera
• Strong experience in data modelling, design patterns and building highly scalable applications
• Strong experience with Python

Should have:

• Experience in a Banking/Financial institution, e.g. knowledge of financial instruments, trade lifecycle, risk management etc. – analysing, implementing and testing data products
• Experience with lakehouse technologies, in particular Apache Iceberg
• Experience with relational SQL and NoSQL databases: SQL Server, Sybase IQ, Postgres, Cassandra, etc.
• Experience with data pipeline and workflow management tools: Airflow, RunDeck, Nifi etc
• Experience with stream-processing systems: Kafka, Spark-Streaming, etc.
• Experience with CICD pipelines
• Experience in Agile methodologies like Scrum, Kanban
• Experience in Automated Testing, Test-driven Development, debugging, troubleshooting, and optimizing code

• Experience with .NET/C#

Nice to have:

• Experience with cloud-based technologies such as Databricks, Snowflake, Azure Synapse
• Knowledge of micro-services architecture and experience in API creation and management technologies (REST, SOAP etc)