Opis stanowiska
We are seeking a Data Engineer (Databricks / Spark) to join a lean, engineering-focused team building a new data platform for a global payments technology company. In this role, you will work closely with our Tech Lead and the client’s engineers to build Databricks on AWS pipelines across a medallion architecture, develop ClickHouse structures for real-time analytics, and own the quality of your code end to end, from automated tests to data quality checks, while giving pragmatic support to a legacy Hadoop platform.
Responsibilities
- New Platform Development: Build pipelines and data transformations on Databricks on AWS, developing Bronze, Silver and Gold layers as part of a platform that unifies the engineering approach across many consumers and use cases.
- Spark Performance: Develop and optimize Spark / PySpark workloads, including those behind monthly reports for around 40,000 merchants.
- Real-Time Analytics: Build ClickHouse ingestion and analytical structures for low-latency use cases.
- Engineering Quality Ownership: Implement automated tests, integration tests and data quality checks yourself, with no dedicated QA on the team, and take part in code reviews.
- Legacy Platform Support: Troubleshoot production issues and deliver targeted fixes in a legacy Hadoop platform with file-based sources and Kafka streams (around 10% of your time).
- Reusable Patterns: Contribute to shared engineering patterns that become the standard for the new platform, with potential growth into AI/ML and MLOps.
- Commercial experience in data engineering, building production data pipelines and transformations.
- Databricks: Hands-on experience building pipelines and workloads on Databricks.
- Spark / PySpark: Strong practical skills in developing and optimizing Spark workloads.
- Python &
- SQL: Strong programming skills in Python and SQL.
Nice to have
- Data Platform Test Automation: Strong experience building automated testing for data platforms.
- Hadoop &
- Streaming: Knowledge of the Hadoop ecosystem, Kafka and streaming data.
- MLOps: Fundamentals of AI/ML and MLOps, including Databricks MLOps.
- Delivery: Experience with AWS, Delta Lake, CI/CD and performance optimization.
We offer
- Opportunity to work on bleeding-edge projects
- Work with a highly motivated and dedicated team
- Competitive salary
- Flexible schedule
- Benefits package - medical insurance, sports
- Corporate social events
- Professional development opportunities
- Well-equipped office
- DevOps, application modernization and customer experience. Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.
Keywords
ClickHouseApache HadoopApache KafkaApache SparkDevOpsPythonSqlHadoopCI/CDBig data
Interesuje Cię ta oferta?