On-Call for AI Agent Systems: Alert Routing, Escalation, and Response Procedures

On-Call Challenges Unique to AI Agents

Traditional on-call rotations handle server outages, database failures, and deployment rollbacks. AI agent systems add a new class of issues: behavioral problems. The agent is technically running, latency is normal, no errors in the logs — but it is giving users wrong answers, calling tools with fabricated parameters, or responding in an inappropriate tone.

These behavioral alerts require on-call engineers who understand not just infrastructure, but also prompt engineering, model behavior, and the agent's domain context.

Designing Alert Routing for Agents

Route alerts to the right team based on the failure type, not just severity.

from dataclasses import dataclass
from enum import Enum
from typing import List

class AlertCategory(Enum):
    INFRASTRUCTURE = "infrastructure"  # pods, networking, database
    LLM_PROVIDER = "llm_provider"      # API errors, rate limits, latency
    AGENT_BEHAVIOR = "agent_behavior"  # wrong answers, safety issues
    BUSINESS_LOGIC = "business_logic"  # tool failures, workflow errors

@dataclass
class AlertRoute:
    category: AlertCategory
    severity: str
    pagerduty_service: str
    escalation_policy: str
    notification_channels: List[str]

ALERT_ROUTES = [
    AlertRoute(
        category=AlertCategory.INFRASTRUCTURE,
        severity="critical",
        pagerduty_service="ai-platform-infra",
        escalation_policy="infra-escalation",
        notification_channels=["#agent-ops", "#infra-alerts"],
    ),
    AlertRoute(
        category=AlertCategory.AGENT_BEHAVIOR,
        severity="critical",
        pagerduty_service="ai-agent-safety",
        escalation_policy="safety-escalation",
        notification_channels=["#agent-safety", "#agent-ops"],
    ),
    AlertRoute(
        category=AlertCategory.LLM_PROVIDER,
        severity="warning",
        pagerduty_service="ai-platform-infra",
        escalation_policy="provider-escalation",
        notification_channels=["#agent-ops"],
    ),
    AlertRoute(
        category=AlertCategory.BUSINESS_LOGIC,
        severity="warning",
        pagerduty_service="ai-agent-product",
        escalation_policy="product-escalation",
        notification_channels=["#agent-product"],
    ),
]

class AlertRouter:
    def __init__(self, routes: List[AlertRoute], pagerduty_client):
        self.routes = {(r.category, r.severity): r for r in routes}
        self.pd = pagerduty_client

    async def route_alert(self, category: AlertCategory,
                          severity: str, title: str, details: dict):
        route = self.routes.get((category, severity))
        if not route:
            # Default: page infra team for unknown alerts
            route = self.routes[(AlertCategory.INFRASTRUCTURE, "critical")]

        await self.pd.create_incident(
            service=route.pagerduty_service,
            escalation_policy=route.escalation_policy,
            title=title,
            severity=severity,
            details=details,
        )

        for channel in route.notification_channels:
            await self.notify_channel(channel, title, severity)

The key insight is separating infrastructure alerts from behavioral alerts. An infra engineer can restart pods, but investigating why the agent recommended a dangerous medication dosage requires someone who understands the agent's guardrails and prompt architecture.

Rotation Design

# on-call-rotation.yaml
rotations:
  - name: "agent-infra-primary"
    type: weekly
    handoff_day: monday
    handoff_time: "09:00"
    timezone: "America/New_York"
    members:
      - "engineer-a"
      - "engineer-b"
      - "engineer-c"
      - "engineer-d"
    restrictions:
      max_consecutive_weeks: 2
      min_gap_between_shifts: 2  # weeks

  - name: "agent-behavior-primary"
    type: weekly
    handoff_day: monday
    handoff_time: "09:00"
    timezone: "America/New_York"
    members:
      - "ai-engineer-a"
      - "ai-engineer-b"
      - "ai-engineer-c"
    restrictions:
      max_consecutive_weeks: 1
      min_gap_between_shifts: 3

escalation_policies:
  infra-escalation:
    - level: 1
      target: "agent-infra-primary"
      timeout_minutes: 10
    - level: 2
      target: "infra-team-lead"
      timeout_minutes: 15
    - level: 3
      target: "vp-engineering"
      timeout_minutes: 30

  safety-escalation:
    - level: 1
      target: "agent-behavior-primary"
      timeout_minutes: 5
    - level: 2
      target: "ai-safety-lead"
      timeout_minutes: 10
    - level: 3
      target: "cto"
      timeout_minutes: 15

Notice the safety escalation has shorter timeouts at every level. A safety issue that is not acknowledged within 5 minutes automatically escalates to the AI safety lead.

Alert Quality Management

Alert fatigue is the number one cause of missed critical incidents. Manage alert quality aggressively.

from datetime import datetime, timedelta
from collections import defaultdict

class AlertQualityTracker:
    def __init__(self):
        self.alerts = []

    def record_alert(self, alert_name: str, was_actionable: bool,
                     time_to_acknowledge: float, time_to_resolve: float):
        self.alerts.append({
            "name": alert_name,
            "timestamp": datetime.utcnow(),
            "actionable": was_actionable,
            "tta_minutes": time_to_acknowledge,
            "ttr_minutes": time_to_resolve,
        })

    def weekly_report(self) -> dict:
        week_ago = datetime.utcnow() - timedelta(days=7)
        recent = [a for a in self.alerts if a["timestamp"] > week_ago]

        if not recent:
            return {"total_alerts": 0}

        by_name = defaultdict(list)
        for a in recent:
            by_name[a["name"]].append(a)

        actionable_rate = sum(1 for a in recent if a["actionable"]) / len(recent)

        noisy_alerts = [
            name for name, alerts in by_name.items()
            if len(alerts) > 10 and
            sum(1 for a in alerts if a["actionable"]) / len(alerts) < 0.3
        ]

        return {
            "total_alerts": len(recent),
            "actionable_rate": round(actionable_rate, 2),
            "avg_tta_minutes": round(
                sum(a["tta_minutes"] for a in recent) / len(recent), 1
            ),
            "noisy_alerts_to_tune": noisy_alerts,
            "recommendation": (
                "TUNE ALERTS" if actionable_rate < 0.7
                else "OK" if actionable_rate >= 0.85
                else "REVIEW needed"
            ),
        }

If fewer than 70% of your alerts are actionable, engineers will start ignoring pages. Review and tune or remove noisy alerts weekly.

Post-Incident Review Integration

Every page should feed back into the system improvement cycle.

class OnCallHandoffReport:
    def generate(self, shift_start: datetime, shift_end: datetime,
                 incidents: list, alerts: list) -> dict:
        return {
            "shift_period": f"{shift_start.isoformat()} to {shift_end.isoformat()}",
            "total_pages": len(alerts),
            "incidents_opened": len([i for i in incidents if i["opened_during_shift"]]),
            "incidents_resolved": len([i for i in incidents if i["resolved_during_shift"]]),
            "sleep_interruptions": len([
                a for a in alerts
                if a["timestamp"].hour >= 22 or a["timestamp"].hour <= 6
            ]),
            "action_items": [
                i.get("follow_up") for i in incidents if i.get("follow_up")
            ],
            "alerts_to_tune": [
                a["name"] for a in alerts if not a.get("actionable", True)
            ],
        }

FAQ

Should AI engineers or infrastructure engineers be on-call for agent systems?

Both, with separate rotations. Infrastructure engineers handle pod failures, database issues, and networking problems. AI engineers handle behavioral issues — hallucinations, safety violations, and prompt regressions. Route alerts to the right rotation based on the alert category, not a single combined rotation.

How do I reduce alert fatigue for AI agent systems?

Track your actionable alert rate and target above 85%. Remove alerts that fire frequently but never require action. Consolidate related alerts into a single notification with context. Use alert grouping to batch multiple instances of the same issue. Review the noisiest alerts weekly and either tune thresholds, add suppression rules, or delete them.

What should an on-call handoff include for AI agent systems?

Include: active incidents and their status, alerts that fired and whether they were actionable, any ongoing behavioral issues being monitored, recent deployments that might cause problems, and LLM provider status. The handoff should take less than 15 minutes. Write it as a structured document, not a verbal conversation.

---

#OnCall #AIAgents #Alerting #PagerDuty #IncidentResponse #AgenticAI #LearnAI #AIEngineering