Job Summary
Key Responsibilities:Pipeline Engineering & Automation: Design, automate, and maintain robust ML evaluation pipelines (utilizing concepts similar to Apache Airflow, Kubeflow, or MLflow) for continuous, post-production model evaluation.Agentic Evaluation Infrastructure: Configure isolated/hermetic agent environments (using container orchestration principles similar to Kubernetes or Docker), integrating tools for synthetic data generation, and setting up simulated execution traces.Continuous Monitoring & Observability: Develop and maintain quality dashboards and leaderboards (similar to Grafana, Datadog, or ELK stack). Set up configurable alerting based on specific metric thresholds (Accuracy, AUC, Trajectory Exact Match) to proactively identify model degradation.Privacy & Security Implementation: Enforce strict Data Isolation Models for sensitive data. Implement strict Identity and Access Management (IAM), Role-Based Access Control (RBAC), and adhere to rigorous data retention and deletion policies (aligned with Zero Trust and strict compliance standards).Production Support: Participate in operational support to triage incidents, mitigate production issues, manage traffic rollouts, and ensure the high reliability of distributed evaluation services.Cross-Functional Collaboration: Partner with downstream client teams to onboard new AI agents (built on frameworks similar to LangChain or LlamaIndex), ingest datasets from distributed SQL or messaging systems (like Apache Kafka), and standardize model evaluation processes.Minimum Qualifications:Bachelor's degree in Computer Science, Software Engineering, or a related technical field (or equivalent practical experience).Experience in MLOps, DevOps, or Data Engineering, focusing on the ML lifecycle (continuous training, model deployment, and continuous monitoring).Proficiency in programming languages such as Python or Go.Experience working with large-scale production deployment, container orchestration (e.g., Kubernetes, Docker), and CI/CD pipelines (e.g., Jenkins, GitLab CI).Preferred Qualifications:Familiarity with Large Language Models (LLMs) and open-source Agentic AI frameworks (e.g., LangChain, AutoGen, LlamaIndex).Knowledge of standard ML evaluation metrics for Generative and Classification models (e.g., BLEURT, ROUGE, Precision/Recall, AUC).Experience managing highly sensitive data (e.g., PII/PHI) and implementing strict access control and encryption mechanisms.Strong analytical and debugging skills, with the ability to troubleshoot complex, distributed ML systems and infrastructure dependencies.
Key Responsibilities
Key Responsibilities:Pipeline Engineering & Automation: Design, automate, and maintain robust ML evaluation pipelines (utilizing concepts similar to Apache Airflow, Kubeflow, or MLflow) for continuous, post-production model evaluation.Agentic Evaluation Infrastructure: Configure isolated/hermetic agent environments (using container orchestration principles similar to Kubernetes or Docker), integrating tools for synthetic data generation, and setting up simulated execution traces.Continuous Monitoring & Observability: Develop and maintain quality dashboards and leaderboards (similar to Grafana, Datadog, or ELK stack). Set up configurable alerting based on specific metric thresholds (Accuracy, AUC, Trajectory Exact Match) to proactively identify model degradation.Privacy & Security Implementation: Enforce strict Data Isolation Models for sensitive data. Implement strict Identity and Access Management (IAM), Role-Based Access Control (RBAC), and adhere to rigorous data retention and deletion policies (aligned with Zero Trust and strict compliance standards).Production Support: Participate in operational support to triage incidents, mitigate production issues, manage traffic rollouts, and ensure the high reliability of distributed evaluation services.Cross-Functional Collaboration: Partner with downstream client teams to onboard new AI agents (built on frameworks similar to LangChain or LlamaIndex), ingest datasets from distributed SQL or messaging systems (like Apache Kafka), and standardize model evaluation processes.Minimum Qualifications:Bachelor's degree in Computer Science, Software Engineering, or a related technical field (or equivalent practical experience).Experience in MLOps, DevOps, or Data Engineering, focusing on the ML lifecycle (continuous training, model deployment, and continuous monitoring).Proficiency in programming languages such as Python or Go.Experience working with large-scale production deployment, container orchestration (e.g., Kubernetes, Docker), and CI/CD pipelines (e.g., Jenkins, GitLab CI).Preferred Qualifications:Familiarity with Large Language Models (LLMs) and open-source Agentic AI frameworks (e.g., LangChain, AutoGen, LlamaIndex).Knowledge of standard ML evaluation metrics for Generative and Classification models (e.g., BLEURT, ROUGE, Precision/Recall, AUC).Experience managing highly sensitive data (e.g., PII/PHI) and implementing strict access control and encryption mechanisms.Strong analytical and debugging skills, with the ability to troubleshoot complex, distributed ML systems and infrastructure dependencies.