Job Summary

Requirement: AI Agent Runtime Observability & Performance

Core theme: Runtime behavior of AI agents — explicitly not development-time skills.

Key Responsibilities

• Observe agents in production. Instrument deployed agents to trace what they do and how they do it (reasoning steps, tool calls, retrieval, handoffs), so behavior can be explained after the fact.
• Benchmark across LLMs. Run the same agent against different models and compare quality, cost, latency, and failure modes.
• Measure behavior. Define and track accuracy, reliability, and repeatability metrics. Build evaluation harnesses that run on live or production-like traffic, not just offline test sets.
• Own runtime performance and scalability. Find bottlenecks under real load, such as model latency, tool-call chains, and token usage, and tune for throughput and concurrency.
• Protect chat UX. Monitor and reduce response latency (his benchmark: 5 minutes is unacceptable). Track time-to-first-token, end-to-end response time, and session abandonment.
• Validate that the agent solves real user needs. Check that it handles genuine customer problems and not just general-purpose questions, using real usage data and feedback.
• Give pre-release confidence. Assess whether an agent will "behave nicely" in a customer's actual environment before DISW ships it.
• Evaluate and recommend tooling. Since this is a new area, help select or build the observability stack.

Skill Requirements

• Hands-on experience observing or evaluating an AI agent after deployment, in production or a production-like setup. Dev-only agent building doesn't qualify.
• Agent observability and tracing tools (for example LangSmith, Langfuse, Arize Phoenix, or OpenTelemetry-based tracing; these are examples, not names he gave).
• LLM and agent evaluation: metrics, test harnesses, regression testing, and handling non-determinism.
• Experience working with multiple LLMs and comparing them.
• Performance engineering for LLM applications: latency profiling, load testing, caching, and scaling.
• Familiarity with chat-based or conversational interfaces and their latency and UX constraints.

Other Requirements

• Python, plus general monitoring and logging stacks (Prometheus, Grafana, ELK or similar).
• Agent frameworks and architectures (RAG, tool use, multi-agent orchestration), enough to understand what's being observed.
• Production and DevOps or SRE exposure, such as deployments, CI/CD, and incident analysis.
• Data analysis, to turn traces and logs into insight about behavior.