Big Data Developer

Requirements

• Between 3 and 6 years of experience in a similar field;
• Strong experience in Apache Spark 3, including application optimization techniques and large-scale data processing.
• Proficiency in Scala 2.12, particularly for Spark application development.
• Solid knowledge of coding best practices, clean code principles and software development standards.
• Very good knowledge of the Hadoop ecosystem, including YARN, HDFS, Hive, HBase and Kafka.
• Knowledge of Big Data security mechanisms, particularly Kerberos and Apache Ranger.
• Experience implementing and maintaining CI/CD pipelines in a DevOps environment.
• Knowledge of deployment and automation tools, namely Jenkins, XL Deploy and XL Release.
• Experience with version control systems, particularly Bitbucket/Git.
• Knowledge of job scheduling and orchestration tools, particularly Control-M.
• Experience working with on-premises Big Data environments, preferably Cloudera.
• Knowledge of functional programming frameworks such as ZIO or Cats is considered an advantage.

Responsibilities

• Develop new features and components within the Big Data platform.
• Optimize Spark applications to ensure performance, efficiency and scalability when processing large volumes of data.
• Rationalize, optimize and reuse existing components to build a consistent, scalable and non-redundant architecture.
• Participate in code reviews, ensuring code quality, readability, maintainability and consistency.
• Participate in deployments to testing and production environments within a DevOps context.
• Manage component upgrades, including the associated testing and validation processes.
• Implement and maintain CI/CD pipelines.
• Write and maintain technical documentation whenever required.
• Plan and estimate development tasks, identifying technical requirements and dependencies.
• Participate in Agile/Scrum ceremonies, including daily meetings, sprint planning, sprint reviews and retrospectives.
• Use collaboration and project management tools such as Jira and Confluence.
• Collaborate with Infrastructure teams, Developers, Business Analysts and end users, adapting communication to different technical audiences.
• Contribute to continuous improvement initiatives, enhancing development processes, architecture and overall code quality.

2x week in Porto or Lisbon