Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote

Role: AI Engineer – Agent Evaluation Specialist

Location: Los Angeles, CA (Hybrid)

Duration: 3+ Months (Possible Extension)

Key Requirements: (Must-Have):LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability. Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation. Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures. Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality. Critical Screening Criteria: Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development. Agent evaluation is the core requirement and will be the primary selection criterion.

Additional Information: All your information will be kept confidential according to EEO guidelines.