Predictika · Cisco Nexus fabric changes

An AI agent's fabric change, checked before it is pushed

An AI or AIOps agent proposes a batch of changes to a Cisco Nexus NX-OS leaf-spine data center fabric: configuration changes, NX-OS software upgrades, route policy changes, optic and Spectrum-X enablement and link shutdowns. Before anything is pushed, the batch goes to the Predictika validator, which checks it against the fabric graph (N9364E-SG2-O and N9332D-GX2B leaves, N9364E-SG2-Q border leaves, N9364E-SG2-O spines, their vPC pairs and the NX-OS release each runs), the NX-OS release Spectrum-X needs, the optics each switch's ports take, and the customer's own change policy, and returns every finding in one answer. The validator runs on the graph a fabric management tool exports, with the customer's own policy. The fabric, the tickets and the change policy on this page belong to a fictional customer, Sable Ridge AI Cloud; the switch models, optics and the Spectrum-X release requirement come from Cisco's public documents. Cisco is not involved with it and has not endorsed it.

Scope: change batches for leaf-spine data center fabrics with leaves, spines, border leaves and vPC pairs. See everything this validator covers

This validator checks one kind of request: a batch of proposed changes to a leaf-spine fabric, from an AI agent, an AIOps tool or a person, against the fabric graph and a change policy. It handles any batch of such changes on the devices in the graph, not only the scenarios shown on this page. It does not generate configurations and does not push anything.

  • Covered: redundancy — both leaves (or both border leaves) of an vPC pair in one batch are blocked; the pair comes from the graph, not from the batch. An NX-OS upgrade that would leave the two peers more than one minor release apart is blocked, in either direction.
  • Covered: Spectrum-X, optics and NX-OS releases — enabling Spectrum-X on a switch that does not yet run NX-OS 10.6(1)F is blocked, and an upgrade in the same batch does not count; enabling an optic that does not fit the switch's ports, such as a QSFP-DD optic in an OSFP switch, is blocked.
  • Covered: scope, rollback and window — a fabric-wide change with no rollback reference, or outside the approved change window, is blocked; a narrower change outside the window needs approval.
  • Covered: agent actions on live state — an agent action to shut a link whose optic is degraded while the redundant path is already down is blocked, because it would take the endpoint off the fabric; with the redundant path up, the same action is allowed.
  • Covered: approvals and velocity — border leaf changes need a minimum approval rank on the ticket, and a limit on spine and border leaf changes per window sends the batch to a person.
  • Supplied by the customer's own systems, not by the validator — the fabric graph, the ticket approvals, the change policy values, whether each change is inside the approved window, the rollback reference, and the optic and redundant-path state from monitoring.
  • Not covered — configuration syntax, traffic or routing simulation, performance modeling, capacity, and any rule the customer has not written into its policy. Other rules can be added.
The AI agentProposes the batch of fabric changes and says it is ready to push.
Predictika validatorChecks every rule at once and returns the verdict, each finding, and the computed figures.

8 scenarios in 4 groups · each group runs from the agent’s first attempt to the outcome · click one to switch

What the agent proposes

The AI agent says

Illustrative agent statement. The validator receives the structured request below.

Proposing agent: AI agent (scripted)

Before sending, the agent must call Recorded run

Predictika validator


      
Waiting for the call.

Where rules-based change approval gets it wrong

Click to expand

Our assessment of what a typical rules-based change approval workflow (ticket fields, approval chains and validation rules) does with the same cases. It is a judgment about effort and reliability, not a claim that nothing can be built.

What the validator caughtTypical rules-based change approvalPredictika
A feature that needs a newer NX-OS release than the switch runsOnly with workarounds
Whether Spectrum-X can be enabled depends on the release the switch runs today, not the one a batch will leave it on. Ticket rules do not know either, so the check is done by an engineer reading release notes, or not at all.
The running release comes from the fabric graph, the feature's minimum release from the catalog of rules, and an upgrade in the same batch is correctly not counted.
vPC peers on NX-OS releases too far apartOnly with care
An upgrade ticket names one switch. Whether it leaves its vPC peer too far behind depends on the peer's release and a policy limit, which approval rules rarely hold side by side.
The target release is compared with the peer's running release from the graph, in either direction, against the customer's limit.
Both halves of a redundant pair in one batchOnly with workarounds
A change-approval workflow looks at one change at a time. Seeing that two changes in the same batch hit the two halves of a pair needs a view across the whole batch and the pairing from the topology, which is usually done in a script outside the workflow.
The whole batch is evaluated together against the graph; both changes are reported, each naming its partner.
Fabric-wide scope that depends on a rollback and a windowOnly with care
Approval rules can require a rollback or a window for a change type, but scope has to be set correctly on each ticket, and a rule per combination grows quickly.
Scope, rollback reference and window are checked together for each change in the batch; partial fixes show exactly what is still missing.
An agent action that depends on live stateOnly with workarounds
Whether it is safe to shut a degraded link depends on the state of the redundant path at that moment. Ticket rules do not see telemetry, so the check ends up inside the agent or not at all.
The monitoring state travels with the change as a field, so the same shutdown is blocked with the redundant path down and allowed with it up.
Approval rank by device role, plus a rate limitOnly with care
A rank requirement per device role and a limit on core changes per window are separate thresholds that approval chains handle one at a time.
Both are checked on every batch and reported together, with the numbers behind them.

What this validator catches: findings from its test cases

Click to expand

Every rule is checked on every call, in any order the choices arrive; one problem never hides another. Blocking findings stop the batch; approval findings need a person to sign off. The wording shown is an example from the test runs; device names and numbers change with the batch. The policy values come from a fictional customer.

1Both halves of a redundant pair in one batch (blocks): change item-02 on leaf-301b and its vPC peer leaf-301a are both in this batch. Change policy: the two halves of a redundant pair are changed in separate batches.
2Degraded link shut while the redundant path is down (blocks): change item-01 would shut a link on leaf-301a whose optic is degraded, but the redundant path is already down. Shutting it would take the endpoint off the fabric; restore the redundant path first.
3Device not in the fabric graph (blocks): change item-01 names leaf-999z, which is not a device in fabric sableridge-ai1.
4Fabric-wide change without a rollback (blocks): change item-03 is fabric-wide and has no rollback reference. Change policy: stage a rollback before a fabric-wide change.
5Optic does not fit the ports of the switch (blocks): change item-04 enables QDD-400G-DR4 on spine-2, a N9364E-SG2-O. The optic does not fit the ports of this switch: OSFP-800G-DR8 is for the OSFP ports of the N9364E-SG2-O and QDD-400G-DR4 for the QSFP-DD ports of the N9332D-GX2B.
6Spectrum-X needs a newer NX-OS release than the device runs (blocks): change item-02 enables Spectrum-X on leaf-301a, which runs NX-OS 10.5(3)F; Spectrum-X on Nexus 9000 needs NX-OS 10.6(1)F or later. An upgrade in the same batch does not count: the feature can be enabled only once the device is running the release.
7vPC peers would run NX-OS releases too far apart (blocks): change item-01 upgrades leaf-302a to NX-OS 10.6(1)F, but its vPC peer leaf-302b runs NX-OS 10.4(3)F. Change policy: the two peers of an vPC pair may not run NX-OS releases more than one minor release apart.
8Border leaf change needs higher approval (needs approval): change item-01 is on border leaf border-leaf-1a and ticket chg-8102 carries rank 1 approval; change policy requires rank 3 for border leaf changes.
9Change outside the change window (needs approval): change item-01 on border-leaf-1a is outside the approved change window. Change policy: a person must approve it.

Scenarios and results

Click to expand

The recorded result of the validator for each scenario on this page: the request the AI agent sent, the verdict, and every finding, in the order they appear above.

1. Software releases and optics: A two-release upgrade, Spectrum-X on 10.5(3)F, a fabric-wide change, a QSFP-DD optic in an OSFP spine

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8103: an upgrade of leaf-302a to NX-OS 10.6(1)F (rollback rb-9101), the enabling of Spectrum-X on leaf-301a (rollback rb-9102), a fabric-wide route policy change through spine-1, and the enabling of a QDD-400G-DR4 optic on spine-2 (rollback rb-9104). All of it is in tonight's approved change window. No spine or border leaf change has been made in this window yet. It is ready to push.

Verdict: Blocked.

  • vPC peers would run NX-OS releases too far apart. Change item-01 upgrades leaf-302a to NX-OS 10.6(1)F, but its vPC peer leaf-302b runs NX-OS 10.4(3)F. Change policy: the two peers of an vPC pair may not run NX-OS releases more than one minor release apart.
  • Spectrum-X needs a newer NX-OS release than the device runs. Change item-02 enables Spectrum-X on leaf-301a, which runs NX-OS 10.5(3)F; Spectrum-X on Nexus 9000 needs NX-OS 10.6(1)F or later. An upgrade in the same batch does not count: the feature can be enabled only once the device is running the release.
  • Fabric-wide change without a rollback. Change item-03 is fabric-wide and has no rollback reference. Change policy: stage a rollback before a fabric-wide change.
  • Optic does not fit the ports of the switch. Change item-04 enables QDD-400G-DR4 on spine-2, a N9364E-SG2-O. The optic does not fit the ports of this switch: OSFP-800G-DR8 is for the OSFP ports of the N9364E-SG2-O and QDD-400G-DR4 for the QSFP-DD ports of the N9332D-GX2B.

2. Software releases and optics: Upgrade lowered and rollback added; the Spectrum-X fix and the optic still fail

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8103: an upgrade of leaf-302a to NX-OS 10.5(3)F (rollback rb-9101), an upgrade of leaf-301a to NX-OS 10.6(1)F (rollback rb-9105), the enabling of Spectrum-X on leaf-301a (rollback rb-9102), a fabric-wide route policy change through spine-1 (rollback rb-9103), and the enabling of a QDD-400G-DR4 optic on spine-2 (rollback rb-9104). All of it is in tonight's approved change window. No spine or border leaf change has been made in this window yet. It is ready to push.

Verdict: Blocked.

  • Spectrum-X needs a newer NX-OS release than the device runs. Change item-03 enables Spectrum-X on leaf-301a, which runs NX-OS 10.5(3)F; Spectrum-X on Nexus 9000 needs NX-OS 10.6(1)F or later. An upgrade in the same batch does not count: the feature can be enabled only once the device is running the release.
  • Optic does not fit the ports of the switch. Change item-05 enables QDD-400G-DR4 on spine-2, a N9364E-SG2-O. The optic does not fit the ports of this switch: OSFP-800G-DR8 is for the OSFP ports of the N9364E-SG2-O and QDD-400G-DR4 for the QSFP-DD ports of the N9332D-GX2B.

3. Software releases and optics: Spectrum-X on a spine already on 10.6(1)F, and the right optic

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8103: an upgrade of leaf-302a to NX-OS 10.5(3)F (rollback rb-9101), the enabling of Spectrum-X on spine-3 (rollback rb-9102), a fabric-wide route policy change through spine-1 (rollback rb-9103), and the enabling of an OSFP-800G-DR8 optic on spine-2 (rollback rb-9104). All of it is in tonight's approved change window. No spine or border leaf change has been made in this window yet. It is ready to push.

Verdict: Allowed.

4. Redundant pairs: Upgrade of both vPC peers

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8101: an upgrade of leaf-301a to NX-OS 10.6(1)F (rollback rb-9201) and an upgrade of leaf-301b to NX-OS 10.6(1)F (rollback rb-9202). Both are single-device changes in tonight's approved change window. No spine or border leaf change has been made in this window yet. It is ready to push.

Verdict: Blocked.

  • Both halves of a redundant pair in one batch. Change item-01 on leaf-301a and its vPC peer leaf-301b are both in this batch. Change policy: the two halves of a redundant pair are changed in separate batches.
  • Both halves of a redundant pair in one batch. Change item-02 on leaf-301b and its vPC peer leaf-301a are both in this batch. Change policy: the two halves of a redundant pair are changed in separate batches.

5. Redundant pairs: Upgrade one peer, change another pair

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8101: an upgrade of leaf-301a to NX-OS 10.6(1)F (rollback rb-9201) and a configuration change on leaf-303a (rollback rb-9202). Both are single-device changes in tonight's approved change window. No spine or border leaf change has been made in this window yet. It is ready to push.

Verdict: Allowed.

6. AIOps link shutdown: Shut a degraded link, redundant path down

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8102: shut the interface on leaf-301a, whose optic monitoring has flagged as degraded (rollback rb-9301), and make a configuration change on leaf-303a (rollback rb-9302). Both are in tonight's approved change window. One spine or border leaf change has already been made in this window. It is ready to push.

Verdict: Blocked.

  • Degraded link shut while the redundant path is down. Change item-01 would shut a link on leaf-301a whose optic is degraded, but the redundant path is already down. Shutting it would take the endpoint off the fabric; restore the redundant path first.

7. AIOps link shutdown: Redundant path restored

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8102: shut the interface on leaf-301a, whose optic monitoring has flagged as degraded (rollback rb-9301), now that the redundant path for that link has been restored, and make a configuration change on leaf-303a (rollback rb-9302). Both are in tonight's approved change window. One spine or border leaf change has already been made in this window. It is ready to push.

Verdict: Allowed.

  • Redundant path is up. Change item-01 shuts a link on leaf-301a while its redundant path is up; traffic moves to the redundant path.

8. Approvals: Border leaf change outside the window on a lower-rank ticket

The agent's request. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8102: a configuration change on border-leaf-1a (rollback rb-9401), scheduled outside tonight's approved change window, and a configuration change on leaf-303a (rollback rb-9402), inside it. Both are single-device changes. No spine or border leaf change has been made in this window yet. It is ready to push.

Verdict: Needs approval.

  • Change outside the change window. Change item-01 on border-leaf-1a is outside the approved change window. Change policy: a person must approve it.
  • Border leaf change needs higher approval. Change item-01 is on border leaf border-leaf-1a and ticket chg-8102 carries rank 1 approval; change policy requires rank 3 for border leaf changes.

Narration transcript

Click to expand

The text of the narrated walkthrough, step by step.

  1. Welcome to a demo of Predictika's Validator. An AI or AIOps agent proposes a batch of changes to a Cisco Nexus NX-OS leaf-spine data center fabric: configuration changes, NX-OS software upgrades, route policy changes, and more. Before anything is pushed, the batch goes to the Predictika validator, which checks it against the fabric graph (N9364E-SG2-O and N9332D-GX2B leaves, N9364E-SG2-Q border leaves, N9364E-SG2-O spines, their vPC pairs and the NX-OS release each runs), the NX-OS release Spectrum-X needs, the optics each switch's ports take, and the customer's own change policy, and returns every finding in one answer. The validator runs on the graph a fabric management tool exports, with the customer's own policy. The fabric, the tickets and the change policy on this page belong to a fictional customer, Sable Ridge AI Cloud; the switch models, optics and the Spectrum-X release requirement come from Cisco's public documents. Cisco is not involved with it and has not endorsed it.
  2. I will start with the first request by the agent.
  3. The agent says. Please push this batch on the Sable Ridge AI Cloud fabric under ticket chg-8103: an upgrade of leaf-302a to NX-OS 10.6(1)F (rollback rb-9101). All of it is in tonight's approved change window. The rest of the request is on screen.
  4. Before it goes out, the agent calls the validator.
  5. Blocked, with four findings, all in one answer. Change item-01 upgrades leaf-302a to NX-OS 10.6(1)F, but its vPC peer leaf-302b runs NX-OS 10.4(3)F. Change policy: the two peers of an vPC pair may not run NX-OS releases more than one minor release apart. Change item-02 enables Spectrum-X on leaf-301a, which runs NX-OS 10.5(3)F; Spectrum-X on Nexus 9000 needs NX-OS 10.6(1)F or later. An upgrade in the same batch does not count: the feature can be enabled only once the device is running the release. Change item-03 is fabric-wide and has no rollback reference. Change policy: stage a rollback before a fabric-wide change. The other one is on screen.
  6. The validator returned four blocking findings in one answer. A check that reads one field at a time would pass most of this; it shows only when the choices in the batch are checked against each other.
  7. What this shows: for vPC peers on NX-OS releases too far apart, a rules-based change approval is error-prone for this.
  8. Now the agent resubmits.
  9. The corrected request. It changes the item Software upgrade on leaf-302a, one device, rollback rb-9101, inside the change window; target NX-OS release 10.6(1)F to Software upgrade on leaf-302a, one device, rollback rb-9101, inside the change window; target NX-OS release 10.5(3)F and the item Route policy change on spine-1, fabric-wide, rollback none, inside the change window to Route policy change on spine-1, fabric-wide, rollback rb-9103, inside the change window. It adds the item Software upgrade on leaf-301a, one device, rollback rb-9105, inside the change window; target NX-OS release 10.6(1)F.
  10. It calls the validator again.
  11. Still blocked, with two findings, all in one answer. Change item-03 enables Spectrum-X on leaf-301a, which runs NX-OS 10.5(3)F; Spectrum-X on Nexus 9000 needs NX-OS 10.6(1)F or later. An upgrade in the same batch does not count: the feature can be enabled only once the device is running the release. Change item-05 enables QDD-400G-DR4 on spine-2, a N9364E-SG2-O. The optic does not fit the ports of this switch: OSFP-800G-DR8 is for the OSFP ports of the N9364E-SG2-O and QDD-400G-DR4 for the QSFP-DD ports of the N9332D-GX2B.
  12. Two findings cleared. Still blocking: Spectrum-X needs a newer NX-OS release than the device runs and optic does not fit the ports of the switch.
  13. The agent resubmits once more.
  14. The revised request. It changes the item Enable Spectrum-X on leaf-301a, one device, rollback rb-9102, inside the change window to Enable Spectrum-X on spine-3, one device, rollback rb-9102, inside the change window and the item Enable an optic on spine-2, one device, rollback rb-9104, inside the change window; optic QDD-400G-DR4 to Enable an optic on spine-2, one device, rollback rb-9104, inside the change window; optic OSFP-800G-DR8. It removes the item Software upgrade on leaf-301a, one device, rollback rb-9105, inside the change window; target NX-OS release 10.6(1)F.
  15. It calls the validator once more.
  16. This time the validator clears it.
  17. The validator also returns the line-item quote. It starts with Software upgrade on leaf-302a and Enable Spectrum-X on spine-3; the other two lines are on screen.
  18. Both earlier findings are cleared.
  19. Four problems caught in one call, fixed in two passes, and a clean batch. That is the value of Predictika's validator: catching mistakes before the change is made. Now you can try the other scenarios yourself.