Job Summary
Job Description: QA Engineer – AI Powered Quality Engineer
Job Description
- We are looking for a QA Engineer with min 8 yr of experience and at least 2+ years in QA/Test Automation with exposure to AI-enabled applications. This role is primarily for testing an AI Agent-driven workflow application that automates business processes. The role covers end-to-end workflow validation, AI Agent behaviour testing, integration testing across system components, and business rule correctness.
Key Responsibilities
Key Responsibilities
- Design and execute test cases covering end-to-end automated business workflows
- Validate AI Agent behavior: data classification, input validation, decision-making, and output generation
- Test decision logic at each workflow gateway and branching point
- Validate AI Agent outputs against expected business rules (black-box testing of agent decisions)
- Perform integration testing across system and service boundaries
- Experience creating AI evaluation frameworks and automated regression testing for LLM/RAG systems.
- Exposure to Responsible AI, model performance evaluation, and AI observability practices.
Skill Requirements
Required Skill Set
Category Skills
AI Agent Testing Black-box testing of AI agent behaviour — validating classification accuracy, decision outputs, input/output consistency, edge-case handling, failure/fallback scenarios, and non-deterministic output handling
Application testing LLM, RAG and AI Agents
Testing Types Functional, integration, end-to-end, negative/boundary, regression
Automation Scripting skills (Python/JavaScript) for test automation of repeatable workflow and agent scenarios
Soft Skills Attention to detail, clear communication for cross-team defect reporting, analytical thinking for complex workflow and agent behaviour scenarios
4. Tools
Tool / Framework
Postman / REST Assured Must
LangSmith/ DeepEval Must
Ragas Must
Python (pytest) Must
Selenium / Playwright Must
Grafana / Datadog Good to have
Jupyter Notebooks Must
Git + CI/CD (Git Runner, GitHub Actions, Jenkins) Good to have