Predictika · Incident runbooks for operations agents

An AI operations agent's incident plan, checked before any step runs

During an incident, an AI operations agent proposes a remediation plan of up to five steps. Before it runs anything, the plan goes to Predictika, which checks it against a declarative model of capacity floors, database snapshots, change freezes, approvals and verification, then returns a verdict and every reason.

Scope: Incident remediation plans of up to five steps over six example services (checkout, orders database, search, recommendations, authentication and users database) and sixteen action types. See everything this validator covers

This validator models incident remediation plans proposed by an AI or human responder: up to five steps, each an action (such as restart pods, roll back a deploy, fail over a database or take a snapshot) on one of the services in its catalog, for an incident of severity 1, 2 or 3 and a responding team.

  • Capacity — no step may leave a service with fewer healthy replicas than its minimum (restart, rolling restart, restart all, scale down).
  • Safety — destructive actions are blocked; a failover or migration needs a snapshot of the same database in an earlier step; during a change freeze only mitigations are allowed.
  • Approvals — irreversible and high-risk tier-1 actions need incident-commander approval; changing another team's service needs that team's sign-off.
  • Discipline — diagnose before acting (except at severity 1), and finish with a verification step.

Any plan built from these services, actions and rules is in scope, not only the scenarios pictured here. Not covered: services and actions outside the catalog, executing the plan, or any real company's runbook or policy. The services are fictional examples; the rules follow common incident and change-management practice and are not any company's policy.

The AI operations agentDrafts the remediation plan for the incident and asks to run it.
Predictika validatorChecks every rule at once and returns the verdict, each finding, and the computed figures.

7 scenarios in 3 groups · each group runs from the agent’s first attempt to the outcome · click one to switch

What the agent proposes

The AI operations agent says

Illustrative agent statement. The validator receives the structured request below.

Proposing agent: AI operations agent (scripted)

Before sending, the agent must call Recorded run

Predictika validator


      

        
        
        
      

      
Waiting for the call.

      
    

  

  
  

    
Where an ordinary runbook or approval workflow falls short
Click to expand
    
Our assessment of what a typical runbook tool or change-approval workflow does with the same cases. It is a judgment about effort and reliability, not a claim that nothing can be built. This page produces no price or quote, so only the policy checks are compared.

    
What the validator caughtTypical runbook tool or approval workflowPredictika
Capacity floors: a restart that would drop a service below its minimum healthy replicasOnly with workarounds
A runbook step does not know the current replica count and minimum of the service; it takes a custom pre-check for each action.		The replicas, the minimum and the number of pods a step touches are in one request and every step is checked.
A snapshot of the same database before a failover or migrationOnly with care
An approval workflow can ask for a backup, but it rarely checks that the snapshot is of the same database and comes earlier in the plan.		The plan is checked as a whole: the snapshot must be of the same database and in an earlier step.
Change freezes that allow mitigations but not changesOnly with care
A freeze flag can block deploys; telling a rollback (a mitigation) from a hotfix (a change) takes per-action classification kept in step with the runbooks.		Every action carries its kind, so a rollback passes and a hotfix is blocked during a freeze.
Every problem returned in one answer before anything runsOnly with workarounds
Checks tend to run step by step as the plan executes, so a responder meets the problems in the middle of an incident.		All findings come back together, whatever order the steps were listed in.

  

  

    
What this validator catches: findings from its test cases
Click to expand
    
Every rule is checked on every call, in any order the choices arrive; one problem never hides another. Blocking findings stop the action; approval findings need a person to sign off; everything else is computed.

    
1A step leaves a service with fewer healthy replicas than its minimum (restart, rolling restart, restart all, scale down).
2A destructive action such as dropping a table or deleting a volume.
3A change (as opposed to a mitigation) during a change freeze.
4A failover or migration with no snapshot of the same database in an earlier step.
5An irreversible action needs incident-commander approval.
6A high-risk action on a tier-1 service needs incident-commander approval.
7A change to another team's service needs that team's sign-off.
8The plan acts before diagnosing anything, outside a severity 1 incident.
9The plan ends on a change with no verification step.

  

  

    
Scenarios and results
Click to expand
    
The recorded result of the validator for each scenario on this page: the request the AI agent sent, the verdict, and every finding, in the order they appear above.

    
1. Capacity: Restarting too many Checkout pods at once

    
The agent's request. Incident severity SEV2, Payments team responding. Please run this plan: step 1, check the logs of the Checkout API; step 2, restart 3 pods of the Checkout API; step 3, check the dashboards and metrics for the Checkout API.

    
Verdict: Blocked.

    
    
      
  • Capacity below the minimum. Step 2 (Restart pods on Checkout API) leaves 3 healthy replicas; the minimum is 4.
    • 
      
  • Plan summary. 1 change step; capacity falls 1 replica below the minimum.
    • 
    

    
2. Capacity: The agent resubmits after fixing the errors

    
The agent's request. Incident severity SEV2, Payments team responding. Please run this plan: step 1, check the logs of the Checkout API; step 2, restart 2 pods of the Checkout API; step 3, check the dashboards and metrics for the Checkout API.

    
Verdict: Allowed.

    
    
      
  • Plan summary. 1 change step; tightest capacity margin is exactly at the minimum.
    • 
    

    
3. Change freeze: A hotfix deploy during a change freeze

    
The agent's request. Incident severity SEV2, Payments team responding. A change freeze is in effect. Please run this plan: step 1, check the logs of the Checkout API; step 2, deploy a hotfix to the Checkout API; step 3, check the dashboards and metrics for the Checkout API.

    
Verdict: Blocked.

    
    
      
  • Change during a change freeze. Step 2 (Deploy a hotfix on Checkout API) is a change during a change freeze; only mitigations are allowed.
    • 
      
  • High-risk action on a tier-1 service. Step 2 (Deploy a hotfix on Checkout API) is high risk on a tier-1 service and needs incident-commander approval.
    • 
      
  • Plan summary. 1 change step; tightest capacity margin is 1 replica above the minimum.
    • 
    

    
4. Change freeze: The agent resubmits after fixing the errors

    
The agent's request. Incident severity SEV2, Payments team responding. A change freeze is in effect. Please run this plan: step 1, check the logs of the Checkout API; step 2, roll back the last deploy of the Checkout API; step 3, check the dashboards and metrics for the Checkout API.

    
Verdict: Allowed.

    
    
      
  • Plan summary. 1 change step; tightest capacity margin is 1 replica above the minimum.
    • 
    

    
5. Database failover: A database failover with no snapshot before it

    
The agent's request. Incident severity SEV2, Payments team responding. Please run this plan: step 1, check the dashboards and metrics for the Orders database; step 2, fail over the Orders database; step 3, check the dashboards and metrics for the Orders database.

    
Verdict: Blocked.

    
    
      
  • No snapshot of the same database first. Step 2 (Fail over the database on Orders database) needs a snapshot of the same database in an earlier step.
    • 
      
  • Irreversible action. Step 2 (Fail over the database on Orders database) cannot be undone and needs incident-commander approval.
    • 
      
  • High-risk action on a tier-1 service. Step 2 (Fail over the database on Orders database) is high risk on a tier-1 service and needs incident-commander approval.
    • 
      
  • Plan summary. 1 change step; tightest capacity margin is 1 replica above the minimum.
    • 
    

    
6. Database failover: The agent resubmits after fixing the errors

    
The agent's request. Incident severity SEV2, Payments team responding. Please run this plan: step 1, check the dashboards and metrics for the Orders database; step 2, take a snapshot of the Users database; step 3, fail over the Orders database; step 4, check the dashboards and metrics for the Orders database.

    
Verdict: Blocked.

    
    
      
  • No snapshot of the same database first. Step 3 (Fail over the database on Orders database) needs a snapshot of the same database in an earlier step.
    • 
      
  • Irreversible action. Step 3 (Fail over the database on Orders database) cannot be undone and needs incident-commander approval.
    • 
      
  • High-risk action on a tier-1 service. Step 3 (Fail over the database on Orders database) is high risk on a tier-1 service and needs incident-commander approval.
    • 
      
  • Plan summary. 1 change step; tightest capacity margin is 1 replica above the minimum.
    • 
    

    
7. Database failover: The agent resubmits after fixing the errors

    
The agent's request. Incident severity SEV2, Payments team responding. Please run this plan: step 1, check the dashboards and metrics for the Orders database; step 2, take a snapshot of the Orders database; step 3, fail over the Orders database; step 4, check the dashboards and metrics for the Orders database.

    
Verdict: Needs approval.

    
    
      
  • Irreversible action. Step 3 (Fail over the database on Orders database) cannot be undone and needs incident-commander approval.
    • 
      
  • High-risk action on a tier-1 service. Step 3 (Fail over the database on Orders database) is high risk on a tier-1 service and needs incident-commander approval.
    • 
      
  • Plan summary. 1 change step; tightest capacity margin is 1 replica above the minimum.
    • 
    

  

  

    
Narration transcript
Click to expand
    
The text of the narrated walkthrough, step by step.

    
    
      
  1. Welcome to a demo of Predictika's Validator. During an incident, an AI operations agent proposes a remediation plan of up to five steps. Before it runs anything, the plan goes to Predictika, which checks it against a declarative model of capacity floors, database snapshots, change freezes, approvals and verification, then returns a verdict and every reason.
    2. 
      
  2. I will start with the first request by the agent.
    3. 
      
  3. The agent says. Incident severity SEV2, Payments team responding. Please run this plan: step 1, check the dashboards and metrics for the Orders database; step 2, fail over the Orders database; step 3. The rest of the request is on screen.
    4. 
      
  4. Before it goes out, the agent calls the validator.
    5. 
      
  5. Blocked, with one finding and two approvals, all in one answer. Step 2 (Fail over the database on Orders database) needs a snapshot of the same database in an earlier step.
    6. 
      
  6. The validator returned one blocking finding and two approvals in one answer. A check that reads one field at a time would pass most of this; it shows only when the choices in the runbook plan are checked against each other.
    7. 
      
  7. What this shows: for a snapshot of the same database before a failover or migration, a runbook tool or approval workflow is error-prone for this.
    8. 
      
  8. Now the agent resubmits.
    9. 
      
  9. The corrected request. It changes the step 2 to Take a database snapshot on Users database and the step 3 to Fail over the database on Orders database. It sets the step 4 to Check dashboards and metrics on Orders database.
    10. 
      
  10. It calls the validator again.
    11. 
      
  11. Still blocked, with one finding and two approvals, all in one answer. Step 3 (Fail over the database on Orders database) needs a snapshot of the same database in an earlier step.
    12. 
      
  12. 0 findings cleared. Still blocking: no snapshot of the same database first. Two approvals also remain for a person: irreversible action and high-risk action on a tier-1 service.
    13. 
      
  13. The agent resubmits once more.
    14. 
      
  14. The revised request. It changes the step 2 to Take a database snapshot on Orders database.
    15. 
      
  15. It calls the validator once more.
    16. 
      
  16. Not blocked, but held for a person.
    17. 
      
  17. The one blocking finding is cleared; two approvals remain for a person: irreversible action and high-risk action on a tier-1 service.
    18. 
      
  18. Three problems caught in one call, fixed in two passes, and the corrected change now goes to a person for approval. That is the value of Predictika's validator: catching mistakes before they reach production. Now you can try the other scenarios yourself.
    19.