Job Summary

AI Quality & Performance Analyst 

Your goal is to maintain and improve AI agent health, performance sustainability, and operational reliability across the Global Technical Support Organization. By conducting systematic analysis of AI agent performance trends, you will identify root causes of degradation, measure gaps between forecasted and realized business value, and drive rapid corrective action. As an AI Quality & Performance Analyst, you will provide recurring health monitoring, execute diagnostic deep dives, and operationalize data-driven interventions to sustain and improve AI-driven case resolution outcomes. 

You are a driver, passionate about operational excellence and data-driven decision-making. You thrive on rapid problem diagnosis, root cause clarity, and translating performance intelligence into measurable improvements. 

Key Responsibilities

What You Get To Do In This Role: 

·                Master AI agent performance frameworks, monitoring dashboards, and quality governance processes within our Support organization. 

·                Establish baseline forecasts for AI agent business value realization (CSAT, resolution rates, cost per resolution, case velocity) and systematically track actuals against targets. 

·                Conduct systematic root cause analysis of AI agent performance degradation—identifying input drift, model decay, process breakdowns, and escalation patterns. 

·                Monitor key health indicators and KPI trajectories in real-time; flag and rapidly escalate material drops in performance with immediate diagnostic context. 

·                Execute deep-dive investigations into underperformance drivers using case-level data, agent logs, and customer signals; document findings with actionable remediation paths. 

·                Perform gap analysis between forecasted and realized value outcomes; quantify impact and recommend corrective actions (retraining, process adjustment, tool configuration, escalation to specialist review). 

·                Build dashboards and reports that translate large data sets into clear performance narratives; own recurring health scorecards and trend communications. 

·                Identify and escalate systematic risks to AI sustainability—data quality issues, emerging blind spots, performance ceiling risks, or process gaps that undermine AI reliability. 

·                Partner with operations teams to prioritize and execute corrective actions; track remediation impact against baseline. 

·                Participate in AI quality calibration forums to ensure consistency in performance assessment and governance decision-making. 

Skill Requirements

Qualifications: 

·                3-5 years working as an Analyst in a technical or operational field (support, quality, AI/ML operations preferred). 

·                Deep proficiency with Excel; ability to build and maintain complex analytical models and dashboards. 

·                Demonstrated commitment to data integrity and systematic root cause methodology. 

·                Ability to work independently on diagnostic projects and drive multi-stakeholder initiatives. 

·                Strong communication skills; ability to package complex performance findings into executive-ready reports and presentations (PowerPoint, narrative briefs). 

·                Ability to manage concurrent analytical projects and deliver insights under tight timelines in a high-velocity environment. 

·                Excellent problem-solving and attention to statistical detail; comfort with ambiguity in emerging problem spaces. 

Other Requirements

Bonus Points: 

·                ServiceNow product knowledge (Now Assist, Agentic AI, or case management modules). 

·                Experience with SQL or relational database queries for large-scale data analysis. 

·                Familiarity with performance monitoring tools, observability platforms, or AI model health frameworks. 

·                Prior experience in AI/ML operations or AI quality assurance.