Job Summary

AI QE

Role Summary

The AI Quality Engineer (AI QE) is responsible for validating the quality, reliability, safety, and user experience of AI and Generative AI solutions. The role combines software quality engineering with AI evaluation to test Large Language Model (LLM) applications, copilots, chatbots, Retrieval-Augmented Generation (RAG) systems, and agentic workflows across real-world user scenarios. The AI QE establishes repeatable evaluation methods, ground-truth datasets, scoring rubrics, regression suites, and quality reporting to ensure AI-powered features are accurate, grounded, relevant, complete, and production-ready.

Key Responsibilities

Key Responsibilities

· Validate AI-powered features, LLM applications, copilots, chatbots, agents, and conversational experiences across functional and non-functional requirements.

· Design test strategies for deterministic application behavior and probabilistic AI outputs, covering positive, negative, adversarial, boundary, and edge-case scenarios.

· Test agent behavior across user personas, multi-step workflows, tool usage, function calling, memory, routing, fallback, and error-recovery paths.

· Evaluate AI-generated responses across single-turn and multi-turn conversations for factual correctness, grounding, relevance, completeness, coherence, and adherence to instructions.

· Create and maintain ground-truth, golden, and benchmark datasets for evaluation and regression testing.

· Define evaluation rubrics, metrics, scoring scales, acceptance thresholds, and detailed rationale for qualitative and quantitative assessments.

· Perform Human-in-the-Loop reviews and analyze agreement, variance, and recurring quality gaps across evaluators.

· Validate RAG pipelines, including retrieval relevance, context utilization, citation accuracy, faithfulness, and hallucination behavior.

· Test prompt templates and prompt changes for robustness, consistency, injection resistance, and unintended regressions.

· Build or execute automated AI evaluation and regression suites, integrating quality gates into CI/CD pipelines where applicable.

· Log defects with reproducible evidence, expected behavior, evaluation scores, rationale, and business impact.

· Produce quality dashboards, evaluation reports, trend analysis, release recommendations, and improvement feedback for engineering and product teams.

· Collaborate with AI Engineers, Fullstack Engineers, LLMOps, Observability, Product, Security, and Responsible AI teams.

AI Evaluation & Quality Scope

· LLM and conversational evaluation: response correctness, relevance, completeness, consistency, tone, instruction following, context retention, and multi-turn coherence.

· Grounding and RAG validation: retrieval precision, context relevance, faithfulness, citation validation, unsupported claims, and hallucination detection.

· Agentic workflow testing: planning, routing, tool selection, tool-call arguments, execution sequence, memory, guardrails, retries, and termination behavior.

· Safety and Responsible AI testing: prompt injection, jailbreak attempts, harmful or restricted outputs, sensitive-data leakage, bias checks, and policy compliance.

· Operational quality validation: latency, availability, throughput, cost, failure handling, observability signals, and production-readiness criteria.

Skill Requirements

Required Skills & Experience

· Hands-on experience in software quality engineering, test design, defect management, and test reporting.

· Practical exposure to Generative AI, LLMs, copilots, chatbots, RAG systems, and AI agents.

· Experience defining evaluation rubrics, metrics, scoring frameworks, acceptance thresholds, and evaluation rationale.

· Experience creating ground-truth or golden datasets and conducting Human-in-the-Loop reviews.

· Understanding of prompt engineering and testing prompt behavior across varied user inputs and scenarios.

· Ability to assess grounding, factual correctness, relevance, completeness, hallucinations, and conversational quality.

· Experience with API testing and automation using Python and tools such as PyTest, REST clients, or equivalent frameworks.

· Familiarity with AI evaluation or observability frameworks such as RAGAS, DeepEval, promptfoo, LangSmith, Azure AI evaluation capabilities, or equivalent.

· Working knowledge of CI/CD, test automation, version control, and cloud-based application environments.

· Strong analytical, communication, documentation, and stakeholder-reporting skills.

Preferred / Nice-to-Have

· Experience with LLM-as-a-judge approaches, pairwise comparison, model-based evaluation, and correlation with human evaluation.

· Knowledge of statistical sampling, inter-rater agreement, confusion matrices, precision/recall, and quality trend analysis.

· Exposure to performance, security, accessibility, localization, or usability testing for AI-enabled applications.

· Familiarity with Responsible AI controls, privacy requirements, auditability, and regulated-domain quality practices.

· Experience testing enterprise AI solutions deployed on Azure, AWS, or GCP.

Experience Level

· 4–8 years of overall software testing, quality engineering, automation, or AI/ML validation experience.

· 1–3+ years of hands-on exposure to AI, ML, Generative AI, conversational AI, or data/model evaluation is preferred.

· Proven experience validating complex digital products or production-grade AI-enabled applications.

Success Measures

· Repeatable and auditable AI evaluation frameworks established for key use cases.

· Higher accuracy, grounding, relevance, completeness, and consistency of AI-generated outputs.

· Reduced hallucinations, unsafe responses, escaped defects, and regressions across releases.

· Effective automated and Human-in-the-Loop quality gates supporting confident production releases.

· Clear quality reports and trend insights that drive measurable product and model improvements

Other Requirements

Good Communication Skils