Data Annotator

Lemon.io helps companies in the US and Europe hire experienced engineers, from funded startups to teams like Klarna, Retool, and Pinterest. We’ve been doing this since 2015, and more than 1,500 companies have hired engineers through us.

We are looking for a Data Annotation + cybersecurity specialist for our client - a leading European AI company that builds and improves advanced AI models used for complex real-world applications. The company recently raised €3 billion at a valuation of over €21 billion, works with major global enterprises, and focuses on making powerful AI more open, customizable, and secure.

About the role:

• Create high-quality coding prompts, environments, and reference solutions (benchmark-style problems similar to Terminal Bench and SWE Bench) with close attention to detail
• Evaluate LLM outputs across code generation, refactoring, debugging, and end-to-end implementation tasks
• Identify, reproduce, and document model failures, edge cases, and reasoning or logic gaps
• Conduct head-to-head model evaluations between internal AI-based LLMs and leading external models
• Build, configure, or adapt coding environments to support systematic evaluation and reinforcement learning workflows
• Apply detailed annotation and evaluation guidelines with a high degree of consistency and rigor
• Collaborate cross-functionally with engineering, product, and research teams to improve model performance

Requirements:

• Coders with 3-5 years of experience and strong in Python 3 (critical)
• Strong in 1–2 languages, comfortable reviewing across many
• Full-stack / cross-functional / multi-job polyglot background preferred
• Must be detail-oriented and comfortable with operational work
• Need to commit to throughput (20–30 evaluations/day). Hours must be spread evenly across the working week (not condensed into 2–3 days) to allow for daily iteration
• Candidates must be located in GMT or similar time zones (+/- 2-3 hours)

Nice-to-have:

• Cybersecurity experience; especially familiarity with exploits and verifiers in the cybersecurity domain
• Experience evaluating LLM outputs across code generation, refactoring, debugging, and end-to-end implementation tasks
• Model benchmarking
• Code-related annotation tasks

Duration

• 1-2 month trial, then potential long-term collaboration (renewal every three months)