AI Evaluation & Reliability Engineer

Job Title 6+ years building evaluation or testing infrastructure for ML/LLM systems or large-scale distributed systemsProven ability to translate product requirements into measurable metrics and test plansStrong Python skillsStrong experience with modern data toolingHands-on experience running A/B tests, canaries, or experiment frameworksExperience defining and maintaining operational reliability metrics (SLIs/SLOs) for AI-driven systemsFamiliarity with large-scale distributed or streaming systems serving AI/agent workflows (millions of events or alerts/day) Excellent communication skills -- able to clearly convey technical results and trade-offs to engineer, PMs, and analysts