Software Engineer - AI Code Evaluation
braintrustSeattle, WA
$75–$105APPLY NOW (opens in a new tab)
About the role
About the Opportunity:
Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion. You'll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent's code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.
What You'll Do:
Read unfamiliar code and trace an AI coding agent's changes through the task requirements, test suite, and run logs.
Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
Determine whether the issue changed the task's pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.
A typical investigation takes 30–60 minutes .What We're Looking For~2+ years of professional experience writing, reviewing, and debugging software, or a completed STEM master's degree or PhD with intensive programming and computational research experience.~ Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.~ Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .~ Practical experience with Git, Docker, and software build or test environments .~ Experience with a test framework such as pytest, unittest, Jest, or Mocha. You can read test assertions and assess whether they match the documented behavior.~ Clear written and spoken English for explaining technical findings and the evidence behind them.
Nice to Have:
Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces. Familiarity with coding benchmarks such as SWE-bench or Code Contests. Experience with AI reliability evaluation or collaborating with ML researchers. Engagement Details
Location: Remote, United States only
Expected commitment — 35 hours per week Start — Immediate
Compensation: $75–$105 USD per hour, depending on relevant skills and experience
Interview Process: Shortlisted candidates will need to complete a 15-minute AI interview.
Matching similar jobs
JOB OVERVIEW
Salary
$75–$105
Experience level
Senior
Location
Seattle, WA
Occupation
Software Developers
Industry
Custom Computer Programming Services
Posted
today
Tired of running searches?
Rank the roles you'd take once, and matches like these arrive on their own.
CREATE PROFILE (opens in a new tab)