Skip to main content

About the role

Overview: In this role you will define and ship AI-powered product experiences, combining deep ML expertise with product judgment. You’ll shape evaluation frameworks, guide end-to-end AI product development, and work with cross-functional teams to deliver measurable business impact. You help establish quality and safety standards while advancing production-ready AI systems. This is a hands-on, impact-focused opportunity at scale within Microsoft’s AI efforts.
Responsibilities: Lead applied science strategy for AI-powered product areas across discovery, launch, and improvement Define success metrics for AI products including customer value, quality, reliability, and business impact Design and operationalize LLM evaluation frameworks across the product lifecycle Develop evaluation methodologies for LLM-based, agentic, retrieval-augmented, and multimodal AI systems Build offline experimentation and evaluation pipelines to compare models, prompts, systems, and product behavior Partner with engineering and PM leadership to align science priorities with product strategy Collaborate with software teams to build production-quality AI systems with reliability, scalability, and MLOps best practices Drive rapid iteration through hypotheses, metrics, experiments, and actionable recommendations Establish quality bars and decision frameworks for launches, including regression testing and responsible AI considerations Mentor scientists and engineers and share best practices across teams Communicate findings, tradeoffs, risks, and recommendations to stakeholders
Key requirements: Bachelor's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering or related field with 4+ years of related experience OR Master's degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering or related field with 3+ years of related experience OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering or related field with 1+ year of related experience Equivalent experience strong communication with technical and non-technical stakeholders mentoring and leadership of scientists and engineers cross-functional collaboration large language models (LLMs) prompt engineering RAG and agents

Matching similar jobs

JOB OVERVIEW

Experience level

Lead

Location

Redmond, WA

Occupation

Data Scientists

Industry

Other Scientific and Technical Consulting Services

Posted

today

Tired of running searches?

Rank the roles you'd take once, and matches like these arrive on their own.

CREATE PROFILE (opens in a new tab)