Senior Site Reliability Engineer Consultant (AI)

ABOUT IWU TECHNOLOGYABOUT THE ROLE: IWU is re-platforming its enterprise on a dual-site, active/active, and on-premises data center foundation and an AI-first application stack. Reliability is a product requirement – not an afterthought bolted on after launch. We are seeking a Senior Site Reliability Engineer Consultant with Sovereign/On-Prem AI Stack Experience – a principal-level individual contributor to define service-level objectives, build the observability platform, and lead incident response for the applications and platform services that serve our 15,000 learners and internal operations. This is not a cloud SRE role. IWU runs production workloads on infrastructure we own and operate in commercial colocation. You will be responsible for application and platform reliability – SLOs, monitoring, alerting, on-call, and post-incident learning – while partnering with Infrastructure on hardware and network availability signals. Application-level performance and error budgets are yours; the data center floor is theirs – and the partnership between the two is what makes uptime real. You will be a hands-on principal: you will translate industry reliability standards into measurable SLOs and deployed tooling, mentor engineers through runbooks and postmortems, and hold the bar for operational excellence across dual sites. Reliability Strategy and SLOsDefine and maintain service-level objectives (SLOs), SLIs, and error budgets for critical platform and application services – availability, latency, throughput, and data freshness – aligned to business impact. Partner with product, application, and platform teams to negotiate realistic targets, document dependencies, and prioritize reliability work against feature delivery. Contribute to the enterprise observability reference architecture – metrics, logs, traces, and synthetic checks – per organizational standards.

Observability Platform: Build AI platform observability in partnership with the Enterprise AI Architect – GPU health (DCGM), inference latency and throughput, model endpoint SLOs, queue depth, and cost/utilization dashboards. Design, deploy, and operate the observability stack – metrics platform, log aggregation, distributed tracing, and dashboards. Implement alerting and escalation – on-call rotations, alert hygiene (actionable alerts only), and runbook linkage. Ensure observability services themselves meet availability and retention targets – dual-site redundancy, capacity planning, and lifecycle management.

Incident Response and Operations: Lead incident response for application and platform outages – triage, communication, mitigation, and resolution – including AI-specific incidents (model degradation, inference failures, data pipeline stalls) where standards apply. Drive post-incident reviews – root-cause analysis, corrective actions, and tracking to completion; feed learnings back into SLOs, runbooks, and architecture. Maintain and improve runbooks, playbooks, and escalation paths – clear ownership, severity definitions, and executive communication templates.

Capacity, Performance, and Resilience: Partner with Infrastructure, Database, and Network teams on capacity signals – trend analysis, saturation forecasting, and proactive scaling before user impact. Design and exercise failure-mode testing – game days, failover drills, and chaos experiments appropriate for dual-site active/active topologies. Support release and deployment reliability – canary patterns, rollback criteria, deployment health checks, and integration with CI/CD pipelines. Analyze performance regressions – APM, tracing, and profiling data to isolate bottlenecks across application, database, and infrastructure layers.

WHAT WE ARE LOOKING FOR: Required

Strongly Preferred: Experience with on-premises or colocation production environments – dual-site active/active topologies, and partnering with infrastructure teams on shared incident boundaries. Experience observing AI/ML platform workloads – GPU monitoring, inference SLOs, batch pipeline latency, or model-serving health checks. Familiarity with APM and synthetic monitoring. Experience with Infrastructure as Code – for observability-as-code deployments. Background in a regulated or compliance-sensitive environment (education/FERPA-Title-IV, financial services/GLBA, healthcare/HIPAA, insurance, public sector).Familiarity with ITIL service management and on-call best practices (Google SRE book principles applied pragmatically).Relevant certifications or demonstrated depth equivalent to CKA/CKAD, or similar. How You WorkYou measure before you optimize – SLOs and data drive priorities, not loudest stakeholder or latest outage memory. You design for operability – if on-call cannot diagnose it from the dashboard, the service is not done. You reduce toil relentlessly – automation and self-service beat heroics every time. You mentor by example – your runbooks, SLO docs, and calm incident leadership become the standard others follow.