Senior Test Lead
India
Job Description
Senior Test Lead
Pune, Maharashtra
Job Summary
Design and execute testing strategies for AI/LLM systems with strong grounding in LLM testing concepts.
Validate and evaluate LLM outputs using frameworks such as DeepEval, RAGAS, TruLens, and LangSmith.
Perform prompt validation, optimization, and robustness testing across diverse use cases.
Detect, analyze, and mitigate hallucinations to ensure response accuracy and reliability.
Assess quality, bias, and performance of LLM-driven applications in production-like scenarios.
Support validation of multi-agent AI systems and contribute to continuous quality improvements.
Key Responsibilities
Define and execute test strategies for AI/LLM‑based applications.
Validate LLM responses using evaluation frameworks (DeepEval, RAGAS, TruLens, LangSmith).
Perform prompt validation, robustness testing, and regression testing for LLM workflows.
Identify, analyze, and mitigate hallucinations, bias, and response inconsistencies.
Ensure quality, accuracy, and reliability of LLM outputs across business scenarios.
Support testing and validation of multi‑agent AI systems where applicable.
Skill Requirements
|
Skill |
Mandatory |
Proficiency Level (1-5) |
|
AI/LLM Testing concepts |
Must Have |
4–5 |
|
LLM evaluation frameworks (DeepEval, RAGAS, TruLens, LangSmith), |
Must Have |
4 |
|
Prompt validation & testing |
Must Have |
4–5 |
|
Hallucination detection |
Must Have |
4 |
|
Multi-agent system validation |
Good to Have |
3–4 |
Other Requirements
null
Information at a Glance