Lead Quality Engineer ETL
CGI GroupJob description
Position Description:
CGI is looking to hire a Lead Quality Engineer with ETL testing expertise, focused on ensuring reliability, performance, and data integrity across applications. The person will work on projects for one of the largest financial institutions in the US in a challenging technical environment and gain exposure to advanced tools and technology suites. This person will be skilled in validating ETL pipelines and workflows through automated and exploratory testing.
Location:
Pittsburgh, PA preferred but open to Dallas, TX and Cleveland, OH. Onsite five days a week.
The AI QE - LLM Evaluation Engineer will be responsible for validating and benchmarking Large Language Model (LLM)-based solutions across a wide range of enterprise use cases. This includes evaluating RAG pipelines (mandatory), texttoSQL, document intelligence, and other generative AI workflows. The role requires strong analytical skills, a deep understanding of LLM behavior, and handson experience designing evaluation frameworks for complex AI systems.
Key Responsibilities
- LLM Evaluation: Design and execute evaluation strategies for diverse LLM use cases, ensuring accuracy, consistency, robustness, and hallucination control.
- RAG Pipeline Testing: Validate retrieval quality, grounding accuracy, context relevance, and endtoend RAG performance.
- TexttoSQL QE: Test SQL generation for correctness, schema alignment, and performance.
- Document Intelligence QE: Validate extraction, classification, summarization, and semantic understanding across structured/unstructured documents.
- Prompt & Model Behavior Testing: Evaluate prompt templates, variations, and tuning strategies to improve reliability.
- Pipeline Validation: Perform endtoend testing of LLM workflows including ingestion, preprocessing, inference, and postprocessing.
- Benchmarking: Build benchmark datasets and scoring rubrics for regression testing and model comparison.
- Error Analysis: Identify hallucinations, failure patterns, and edge cases; provide actionable insights to engineering teams.
- Automation: Develop automated test suites using Python, PyTest, or similar frameworks.
- CrossFunctional Collaboration: Work with Data Scientists, ML Engineers, and Product teams to refine model behavior and improve quality.
- Strong experience in AI QE, ML testing, or LLM evaluation.
- Handson experience with RAG pipelines (mandatory).
- Proficiency in Python and automated testing frameworks.
- Solid understanding of LLMs, prompt engineering, and GenAI architectures.
- SQL proficiency for validating texttoSQL use cases.
- Experience with Document Intelligence tools (Azure Form Recognizer, AWS Textract, Google Document AI, etc.).
- Familiarity with evaluation metrics (accuracy, semantic similarity, BLEU, ROUGE, hallucination scoring).
- Ability to design test datasets, evaluation frameworks, and benchmarking suites.
- Bachelor's degree in computer science or related field.
Compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range for this role in the U.S. is $70,800.00 - $156,700.00.
CGI's benefits are offered to eligible professionals on their first day of employment to include:
- Competitive compensation
- Comprehensive insurance options
- Matching contributions through the 401(k) plan and the share purchase plan
- Paid time off for vacation, holidays, and sick time
- Paid parental leave
- Learning opportunities and tuition assistance
- Wellness and well-being programs
Category: Testing/Quality Assurance
Main location: United States, Various
Position ID: J0926-0437
Employment Type: Full Time
Interested in this role?