Job Summary
AI Evaluation & Observability Architect(CoE)-
The AI Evaluation & Observability Architect makes AI systems measurable, reliable, and safe to run in production. The role designs the full evaluation lifecycle (offline and online evaluations, ground truth, golden datasets, deterministic and LLM-as-judge scoring) and turns it into enforceable engineering controls: regression gates, observability, SLOs, and resilience patterns. It also provides Responsible AI assurance and packages everything into reusable evaluation assets and production-readiness scorecards that Solutions and Advisory teams apply to client engagements.
The role is a hands-on, Al-native engineer-architect who uses agent harnesses in daily delivery and sets the quality bar for how the organization ships AI.
Design offline/online evaluations, ground truth, golden datasets, deterministic and LLM-as-judge scoring, regression gates, observability, SLOs, resilience and Responsible AI assurance; create reusable evaluation assets and production-readiness scorecards. Mandatory across all roles: practical AI-native engineering; strong modern application development; cloud-native, API, microservices and DevSecOps grounding; hands-on experience with at least one agent harness such as Claude Code, GitHub Copilot, Codex, Kiro or Cursor; enterprise security, governance, evaluation, observability and cost awareness.
Key Responsibilities
A. Evaluation Strategy & Design • Define evaluation strategy per use case: what "good" means, which dimensions matter (accuracy, groundedness, relevance, safety, tone, latency, cost), and how each is measured. Design offline evaluations (pre-release benchmarks, experiment comparison, model/prompt/retrieval variants) and online evaluations (shadow testing, A/B tests, canary releases, user feedback loops, production sampling). • Cover the full AI system stack: LLM calls, RAG/GraphRAGretrieva1, tool use, multi-step agents, and end-to-end workflows.
B. Ground Truth & Golden Datasets Build and govern ground truth and golden datasets: sourcing, labeling guidelines, subject-matter-expert review, inter-annotator agreement, versioning, and coverage Of edge cases and adversarial inputs. • Generate and curate synthetic test data responsibly, with human validation and bias checks. Maintain dataset lifecycle: drift detection, refresh cadence, and protection against test- set leakage and contamination.
C. Scoring Methods Design deterministic scoring (exact/fuzzy match, schema and format validation, citation checks, retrieval metrics such as recall@k and MRR, tool-call correctness, latency and cost metrics). Design LLM-as-judge scoring: rubric design, judge prompt engineering, calibration against human labels, bias and consistency checks (position, verbosity, self-preference), and judge-model selection and cost control. Define when to use each method and how to combine them into composite quality scores with confidence intervals.
D. Regression Gates & Release Governance • Implementregressiongatesin CI/CD: threshold-based, statistical, and trend-based gates that block or flag releases for model, prompt, retrieval, or tool changes. • Define release criteria, rollback triggers, and approval workflows with clear ownership. • Integrate evaluation into pull requests, pre-production, and post-deployment verification.
E. Observability, SLOs & Resilience Design observability for AI systems: distributed tracing across agent steps and tool calls, prompt/response logging with privacy controls, quality and drift dashboards, anomaly alerting, and usage and cost telemetry. Define SLIs/SLOs and error budgets for quality, latency, availability, cost, and safety. Design resilience patterns: fallbacks and model failover, timeouts, retries and circuit breakers, graceful degradation, rate-limit handling, and chaos/fault-injection testing for AI components.
F. Responsible AI Assurance Design assurance for fairness and bias, safety and harmful content, privacy and PII leakage, security (prompt injection, jailbreaks, data exfiltration, tool misuse), transparency, and human oversight. Run red-teaming and adversarial testing; maintain risk registers and evidence packs. Map controls to relevant frameworks and regulations (e.g„ NIST AI RMF, ISO/IEC 42001, EU AI Act, OWASP Top IO for LLM Applications) and to client policies.
G. Reusable Assets & Production -Readiness Scorecards Create reusable evaluation assets: evaluation harnesses and libraries, golden dataset templates, rubric and judge-prompt libraries, CI/CD gate templates, dashboard templates, and red-team suites. Define production-readiness scorecards covering quality, safety, security, resilience, observability, cost, and operability, with clear pass/conditional/fail criteria. Publish playbooks and reference architectures; enable and mentor Solutions and Advisory teams.
Skill Requirements
Bachelor's or Master's degree in Computer Science, Data Science, Software Engineering, or an equivalent field, or equivalent practical experience. 10+ years in software engineering, quality engineering, ML/data engineering, or architecture, with at least 3 years evaluating or operating LLM-based, ML, or search/retrieval systems in production. Track record of establishmg an evaluation or quality framework that gated real releases. Technical Skills (Must Have) 1. Evaluation design: offline and online evaluation methods for LLM, RAG, and agentic systems; experiment design and NB testing. 2. Ground truth&datasets: golden dataset creation, labeling guidelines, SMEreview workflows, versioning, and leakage prevention. 3. Scoring: deterministic metrics and LLM-as-judge techniques, including calibration against human judgment. 4. Statisticsforevaluation: sampling, confidenceintervals, significance testing, variance and non-determinism handling. 5. Regressiongating:CI/CD integration ofevaluation suites; threshold and statistical gates. 6. Observability: tracing, logging, metrics, and dashboardsforAIworkloads (e.g., OpenTelemetry-based tooling). 7. Reliability engineering: SLI/SLO/error budget design; resilience patterns and fault- injection testing. 8. Responsible AI: bias, safety, privacy, and security testing (including prompt injection and jailbreak testing); red-teaming; framework mapping. 9. Al-native engineering: practical use 0fAIin the development lifecycle; hands-on With at least one agent harness (Claude Code, GitHub Copilot, Codex, Kiro, or Cursor). IO. Modern application development: strongproficiencyin at leastone languagesuch as Python or TypeScript; testing frameworks, clean code, code review. 11. AWS/Azure/GCP, containers, service design, API design and testing. 12. DevSecOps: CI/CDpipelines,infrastructureas code, secrets management, secure SDLC. 13. Enterprise readiness: security, governance, and costawarenessforAIsystems,including cost-per-evaluation and judge-model spend management. Behavioral & Professional Skills Rigorous, evidence-driven thinking; comfortable saying "the data does not support this." • Abilityto translate business risk into measurable criteria and communicate results to executives and engineers. Influences without authority across delivery, security, legal, and product teams. • Ownership, mentoring, and a collaborative approach in a COE setting
Other Requirements
Other Skills (Good to Have) Role-specific Evaluation and observability tooling (e.g., RAGAS, DeepEval, promptfoo, TruLens, LangSmith, Langfuse, Arize Phoenix, MLflow, OpenAI Evals, Inspect). Red-team and security tooling (e.g., garak, PYRIT, Giskard) and guardrail frameworks. Retrieval/search evaluation depth (IR metrics, relevance judgments, GraphRAG evaluation). Agent evaluation: trajectory scoring, tool-use accuracy, multi-turn and long-horizon task evaluation. • Human-in-the-loop operations: annotation platforms, reviewer calibration, labeling quality management. Data quality and drift tooling; feature and model monitoring. Performance and load testing (k6, Locust) for LLM-backed services. FinOps for AI: token budgeting, caching, model routing, cost-quality trade-off analysis. Domain exposure in regulated industries (BFSI, healthcare/life sciences, public sector). Common competencies (mandatory across all roles) Practical Al-native engineering and strong modern application development. Cloud-native, API, microservices, and DevSecOps grounding. Hands-on experience with at least one agent harness (Claude Code, GitHub Copilot, Codex, Kiro, Cursor). • Enterprise security, governance, evaluation, observability, and cost awareness. Certifications (preferred, not mandatory) Cloud architect or DevOps certifications (AWS, Azure, or GCP). • Responsible AI and governance credentials (e.g., ISO/IEC 42001 lead implementer/auditor, AIGP). • SREor quality engineering certifications (e.g., ISTQB)