26-0098-003 | LLM / RAG and Agent Evaluation Engineer — ICE AI / Data Advisory (Contingent)

Responsibilities: Evaluate retrieval quality, grounding, model routing, tool access and agent behavior against defined baselines. Test prompt injection, prohibited access, model/provider failure, retries, budget exhaustion, escalation, human override and recovery in authorized environments. Capture versioned test inputs, configurations, results and traceable evidence; quantify quality, latency and cost tradeoffs. Explain findings and limitations to architecture, assurance and mission leads; recommend and verify targeted improvements. Hands‑on implementation and evaluation of LLM/RAG systems, with attributable work on agent/tool behavior or model routing. Ability to build repeatable tests and inspect logs, retrieval behavior, permissions and failure handling. Experience translating test results into operational decisions and recognizing limitations in test data and metrics. Evidence we will discuss An existing candidate‑owned, public or authorized demonstration or redacted test report with rerunnable methodology. Evidence of safe failure, denied actions, recovery and cost/retry controls, with your specific contribution identified. Technical context The RFI describes an existing STELLA environment spanning AWS, Microsoft Azure and Google Cloud, with Kubernetes and LLM/RAG capabilities, and a data environment integrating Databricks, Collibra, Mule Soft and Traceable. Its enterprise control plane, agentic orchestration/runtime and independent assurance layers are future‑state concepts to be evaluated. Direct STELLA experience is preferred when lawfully demonstrable; comparable federal or regulated production experience is welcome. No applicant is expected to master every listed product or all three clouds. Engagement conditions Contingent on government contract award and funding. Work location, work model, travel and access requirements will be determined by the scope and agency requirements. Start date, engagement structure, compensation and any paid work terms will be agreed before work begins. This posting represents a coverage function, not a promise of a separate full‑time position. Candidates may cover multiple functions where experience, availability and independent‑review requirements permit.