Data Scientist — Agent Evaluations & Quality

Clera · remoteRemotosenior

En corto

▸Diseñar y mantener sistemas automatizados de evaluación para asistentes de IA en entornos reales.
▸Traducir capacidades complejas de agentes en criterios claros de éxito, fallo y parcialidad.
▸Trabajar directamente con ingenieros para convertir datos de producción en decisiones de producto concretas.

En ~1 minuto te damos: quién te entrevista, las preguntas probables con respuestas desde tu CV, y tu CV adaptado a esta vacante. Gratis, sin tarjeta.

¿Qué piden?

✓4+ años en ciencia de datos aplicada o ML con experiencia en sistemas de evaluación.
✓Experiencia en frameworks de evaluación automatizados para sistemas de IA o agentes.
✓Dominio de Python y SQL para pipelines analíticos en producción.
✓Habilidades estadísticas aplicadas: pruebas de significancia, análisis de varianza, muestreo.
✓Capacidad para crear datasets de referencia y guías de anotación en entornos dinámicos.
✓Conocimiento práctico del comportamiento de agentes LLM: uso de herramientas, ejecución multi-paso, fallas comunes.

¿No cumplís todo? Es lo normal — tu dossier gratis te dice qué gaps tenés y cómo cubrirlos en la entrevista.

PythonSQLLLM-as-a-judgemodel-based gradingAI benchmarking platformsproduction telemetryobservability datadashboardsregression suitesevaluation pipelines

🎯 ¿A quién escribirle en Clera?

Tu dossier gratis identifica a las personas que te entrevistarían — con su background, qué valoran y cómo escribirles para destacar antes de aplicar.

About the Role This company is building an AI executive assistant that operates across email, calendars, meetings, and business software. As a Data Scientist — Agent Evaluations & Quality , you will own the measurement system that determines whether the assistant is genuinely improving in ambiguous, real-world environments. You'll partner directly with AI Agent Capabilities engineers to generate the evidence that shapes product decisions, model choices, and release quality. This is a high-ownership, deeply technical role at the intersection of applied data science, LLM evaluation, and product quality — ideal for someone who thrives on turning hard, open-ended quality questions into rigorous, actionable answers. What You'll Do Architect and maintain automated evaluation pipelines that measure agent quality across product surfaces. Translate agent capabilities into explicit pass, partial-pass, and failure criteria for complex multi-step tasks. Build representative gold datasets and regression suites covering real workflows, edge cases, and adversarial scenarios. Define meaningful metrics — task success, tool-selection accuracy, instruction adherence, factual consistency, latency, cost, and reliability. Design deterministic and model-based graders, calibrate LLM-as-a-judge systems, and track grader agreement. Compare models, prompts, and implementations using rigorous offline experiments and production evidence. Analyze traces and production outcomes to identify root causes and build a practical failure taxonomy. Turn production failures into regression cases and continuously close gaps in evaluation coverage. Build dashboards and release-quality signals that make results actionable for engineering, product, and leadership. Recommend improvements to capability engineers and verify that fixes raise quality without unacceptable regressions. What We're Looking For Required 4+ years in Applied Data Science or Machine Learning roles, with a track record of building and delivering evaluation systems, automated data pipelines, or production ML infrastructure. Experience designing and implementing automated evaluation frameworks, success criteria, and regression suites for complex AI/ML or agentic systems. Production-grade proficiency in Python and SQL , with experience building and maintaining automated analytical pipelines on large datasets. Applied statistical and experimental skills: significance testing, variance analysis, and sampling to evaluate non-deterministic AI/ML systems. Experience developing labeled datasets, annotation guidelines, and quality-control processes for ground-truth data in dynamic product environments. Solid understanding of LLM agent behaviors: tool use, multi-step execution, retrieval, and practical failure modes. Demonstrated ability to analyze model traces, tool calls, and outputs to identify root causes across model, prompt, tool, and data layers. Experience using production telemetry and observability data to monitor system quality, build dashboards, and analyze real-world user outcomes. Nice to Have Hands-on experience with LLM-as-a-judge systems, model-based grading, or AI benchmarking platforms. Experience shipping or operating production ML products, agentic systems, or customer-facing consumer software. Experience reviewing and adapting public research benchmarks or academic evaluation methodologies to real-world product problems. What makes you a great fit You're product-oriented — you prioritize metrics tied to real user outcomes, not just convenient measurements. You drive ambiguous quality questions from evaluation design all the way into product decisions. You write maintainable, production-quality code — not just ad-hoc notebooks. You collaborate naturally with engineers and are comfortable digging into traces and system internals. Location This role is on-site . Visa sponsorship is not available for this position. Benefits Compensation details were not provided for this listing

No apliques sin prepararte

Investigamos quién te entrevista, adaptamos tu CV y te ensayamos en vivo — gratis la primera.