QA Analyst (AI Systems)

About CodeRoad

CodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our nearshore technology services empower businesses to thrive in an ever-evolving digital landscape.

About the Role

We’re looking for a QA Analyst specialized in modern AI systems. You’ll be responsible for validating not only APIs and performance, but also the behavior and reliability of AI-driven outputs. This role goes beyond traditional QA — you’ll test intelligent systems, not just deterministic ones.

Key Responsibilities

Design and execute test strategies for AI-powered APIs and workflows

Validate agent inputs/outputs and ensure consistency and reliability

Perform API testing, exploratory testing, and automated testing

Run performance, load, and bulk testing scenarios

Collaborate with engineers to identify edge cases in AI behavior

Contribute to testing frameworks for LLM-based systems

Requirements

3+ years of experience in QA (Automation + Manual).

Strong API Testing: Expert level with Postman, Insomnia, or RestAssured.

Automation Specialist: Proficient in Playwright or Selenium for end-to-end flows.

Performance & Scale: Experience with Load/Bulk testing using Locust or JMeter.

AI/LLM Validation: Familiarity with testing Agentic workflows, validating AI payloads, and using tools like LangSmith.

Observability: Exposure to OpenTelemetry for tracing and system monitoring.

Analytical Rigor: Ability to design test cases for non-linear, agentic AI tasks.

Advanced English: Fluent for documentation and global collaboration.

Nice to Have

Experience testing AI/ML systems

Familiarity with LangSmith or similar tools

Experience with MLFlow or model evaluation frameworks

What You’ll Love

100% Remote

Holidays off

Paid Time Off

Health insurance assistance program

Competitive USD compensation

Strong team culture and collaborative environment

Ongoing training and growth opportunities