Skip to main content

About the role

What this role does Design realistic, difficult shopping-related tasks (e.g. meal planning, restocking, gifting, travel prep) that stress-test AI models, then build the rubrics used to grade how well a model performs them. Who we're looking for Subject-matter expertise in retail, e-commerce, consumer shopping behavior, or a related domain (nutrition, logistics, home/office management, etc.) Strong analytical and writing skills — able to define precise, low-variance grading criteria Comfortable working with AI outputs and evaluating them critically and objectively Prior experience in eval design, QA, tutoring, or curriculum/rubric design is a plus
Core responsibilities: Author realistic shopping tasks with a clear, concrete end goal Build a grading rubric/verifier for each task that two independent reviewers would score consistently Review an AI agent's attempt at the task and grade it against the rubric Document where the AI succeeded or failed, with clear reasoning Work style High attention to detail and consistency Comfortable with ambiguity — tasks are self-directed with light guidelines

Matching similar jobs

JOB OVERVIEW

Experience level

Mid

Location

New York, NY

Occupation

Market Research Analysts and Marketing Specialists

Industry

Other Scientific and Technical Consulting Services

Posted

9 days ago

Tired of running searches?

Rank the roles you'd take once, and matches like these arrive on their own.

CREATE PROFILE (opens in a new tab)