Engineering the "Ground Truth" : Curate and maintain "Golden Datasets" definitive sets of prompts and ideal responsesthat serve as the benchmark for every model update we ship.
Building the Eval Pipeline : Implement automated evaluation frameworks using tools like Ragas or DeepEval to quantify performance metrics such as Faithfulness, Relevancy, and Context Precision.
Red Teaming & Jailbreaking : Proactively act as an adversary to bypass guardrails, inject malicious prompts, and force hallucinations to ensure our safety layers are bulletproof.
RAG Debugging : Diagnose the "Retrieval" side of our systems. You will determine if a bad answer was caused by the vector database fetching the wrong data or the LLM failing to interpret the right data.
LLM-as-a-Judge : Design and tune "Judge" models (using high-reasoning models like GPT-4o or Claude 3.5) to programmatically grade the outputs of our production models.
Guardrail Governance : Implement and test real-time guardrails to intercept PII leakage, toxic content, or off-topic conversations before they reach the user.
Technical Skills &
Qualifications :
Python Native : Deep proficiency in Python. You should be comfortable building test suites that interact with asynchronous AI APIs and handle complex data structures.
AI Orchestration : Working knowledge of LangChain, LangGraph, or Haystack. You understand how "chains" and "agents" behave and, more importantly, how they fail.
The Eval Stack : Hands-on experience with (or a strong desire to master) LangSmith for tracing, debugging, and evaluating LLM workflows.
Statistical Thinking : You understand that "accuracy" in GenAI is a moving target. You are comfortable using semantic similarity, cosine distance, and BLEU/ROUGE scores, while understanding their specific limitations.
Data Driven : Experience working with vector databases and an understanding of how embeddings work.
Carbynetech