We are looking for an experienced AI/GenAI Test Engineer to design and execute testing strategies for LLM-powered applications, AI agents, and conversational AI solutions. The role requires strong expertise in validating AI-generated responses, prompt behavior, agent workflows, and end-to-end application functionality using Playwright and JavaScript.

The candidate should combine traditional automation testing skills with emerging GenAI/LLM testing techniques, ensuring AI solutions are accurate, reliable, safe, and consistent.

Design and execute test strategies for LLM-based applications and AI agents.Perform Agent Testing covering multi-step workflows, tool/function interactions, context handling, memory behavior, error recovery, and response consistency.Validate LLM responses for relevance, correctness, groundedness, hallucination, consistency, and completeness.Design and maintain structured test datasets and evaluation scenarios for GenAI applications.Perform Prompt Engineering and Prompt Testing, including:

• System and user prompt validation
• Prompt variations and edge cases
• Context-window testing
• Prompt regression testing
• Adversarial and negative scenarios

Establish evaluation criteria and measurable acceptance thresholds for LLM outputs.Develop automated functional and regression test suites using Playwright and JavaScript.Automate testing of web applications integrating conversational AI/LLM capabilities.Validate API integrations, agent tool calls, structured outputs, and downstream system interactions.Perform end-to-end testing covering UI → Agent → LLM → Tools/APIs → Response flows.Identify AI-specific failure scenarios such as hallucinations, inconsistent responses, incorrect tool selection, malformed output, and loss of context.Integrate automated test suites with CI/CD pipelines and support continuous regression testing.Analyze defects, perform root-cause analysis, and collaborate with developers, AI engineers, architects, and product teams.Define and maintain AI/LLM quality metrics, test reports, defect dashboards, and release-quality criteria.

Good understanding of Prompt Engineering and Prompt Testing.Knowledge of LLM validation techniques covering:

• Groundedness and factual correctness
• Hallucination detection
• Relevance
• Response consistency
• Context retention
• Instruction adherence

Strong automation experience with Playwright.Strong programming/scripting skills in JavaScript.Experience with API testing and JSON-based request/response validation.Good understanding of functional, regression, integration, negative, and exploratory testing.Experience working in Agile/Scrum and CI/CD environments.

Experience with RAG-based application testing, including retrieval relevance and grounded response validation.Exposure to LLM evaluation frameworks or observability tools such as Ragas, DeepEval, LangSmith, or promptfoo.Understanding of agentic AI concepts such as tool calling, orchestration, memory, planning, and multi-agent workflows.Experience with LLM APIs and platforms such as Azure OpenAI/OpenAI.Familiarity with Git, Azure DevOps, Jenkins, or GitHub Actions.Knowledge of AI security testing, including prompt injection and sensitive-data handling.