Job Summary
Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails
Key Responsibilities
Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails
Skill Requirements
Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails
Other Requirements
Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails