Predictika · Model-serving plans proposed by an AI agent
An AI deployment agent decides how to serve a model: which pool, how it is quantized, how many GPUs share it, the context length and batch size, and how many copies run. Before it deploys, the agent sends the plan to Predictika, which recomputes the memory, the time per token, the throughput, the GPUs and the power from the pool's own specifications, checks the data class against the pool, and returns a verdict, every reason and the figures it computed.
This validator models one serving plan on a fictional GPU platform, not just the scenarios shown below. The platform, its pools and its limits are illustrative stand-ins for what a real platform keeps in its hardware sheets, serving-engine configuration, capacity and power records and data-handling policy. The figures are illustrative, not benchmarks, and this is an example policy, not a compliance certification.
Any serving plan for the three model sizes on the four pools is in scope — not only the scenarios pictured here. Not covered: measured latency and live traffic (the validator judges a proposed plan against declared specifications, it does not measure a running system), multi-model sharing of a GPU, pipeline and expert parallelism, prices and the serving engine itself. All pools, specifications and limits are fictional.
8 scenarios in 4 groups · each group runs from the agent’s first attempt to the outcome · click one to switch
Illustrative agent statement. The validator receives the structured request below.
Proposing agent: AI deployment agent (scripted)
Before sending, the agent must call Recorded run
What this means
Our assessment of what a typical pre-deployment check does with the same cases: a capacity spreadsheet, a few admission rules in the cluster scheduler and a review checklist. It is a judgment about effort and reliability, not a claim that nothing can be built.
| What the validator caught | Typical capacity calculator or deployment checklist | Predictika |
|---|---|---|
| Memory that depends on four things at once | Only with workarounds The weights depend on model size and quantization; the cache depends on context length, batch size and cache precision, so the same model fits at one batch and not the next, and at 8-bit and not at 16-bit. A sizing spreadsheet is filled in for one case and goes stale when anyone changes a number. | Weights, cache and overhead are recomputed from the plan on every call and compared with the memory of the GPUs that share the model. |
| A fix that moves the problem | Only with workarounds Raising the batch to meet demand fills the cache and slows every token; sharing the model over more GPUs cuts the read time but adds communication cost and needs more free GPUs and power. A rule that runs on one setting at a time shows one problem, then the next. | Memory, latency, throughput, GPUs and power are checked together, so the corrected plan is checked as a whole. |
| The figures the agent states | Only with workarounds The memory and latency an agent writes into its plan never reach a scheduler check; nothing compares them with the pool's specifications. | The stated memory and latency are fields the validator receives and compares with its own. |
| Pool features that differ pool to pool | Only with care FP8 needs GPUs that support it, confidential computing exists on some pools, spot pools can be reclaimed, and an edge node has four GPUs, not eight. These live in different documents, so someone has to remember them. | Each pool's capabilities are part of the model and are checked against the plan every time. |
| Data class against the pool | Only with workarounds Which pool may hold regulated or EU-only data is a policy document, not a scheduler field; the plan is allowed to run and the review comes later, if at all. | Data class, region and confidential computing are checked against the pool before anything is deployed. |
| Approvals that depend on who must sign | Only with workarounds Over half the free GPUs, most of the power headroom, a memory fill over 95 percent, a latency inside 10 percent of the target and a 4-bit model on live traffic each go to a different person; routing is manual. | Each approval is its own finding, with the reason, and a person must approve before the deployment executes. |
| Every problem returned in one answer | Only with workarounds Admission rules tend to stop at the first failure, so the agent fixes one error and meets the next. | All findings come back together, whatever order the choices arrived in. |
Every rule is checked on every call, in any order the choices arrive; one problem never hides another. Blocking findings stop the configuration; approval findings need a person to sign off; everything else is computed.
| 1 | Tensor parallelism of 3 is not supported; the serving engine shards a model over 1, 2, 4 or 8 GPUs only. |
| 2 | Tensor parallelism of 16 spans more GPUs than the 8 in one node of this pool; the shards would have to talk over the network. |
| 3 | The deployment takes 4 of the 4 free GPUs in the pool, more than half; the capacity manager approves. |
| 4 | The GPUs in this pool have no FP8 support, so an FP8 model cannot run on them. |
| 5 | The context length of 33K tokens is more than the 32K this model is configured for. |
| 6 | A batch of 257 sequences is more than the 256 this model engine accepts. |
| 7 | The GPUs would be over 95 percent full (42,744 of 43,200 MB); a long request could run them out of memory; the platform lead approves. |
| 8 | The weights, the KV cache and the runtime overhead need 43,256 MB, but the GPUs hold 43,200 MB at the 90 percent fill limit. |
| 9 | Each token takes 12,158 microseconds on this setup; the latency target is 12 ms per token. |
| 10 | Each token takes 12,158 microseconds, within 10 percent of the 13 ms target, which leaves no room for a traffic spike; the service owner approves. |
| 11 | The deployment makes 2,632 tokens a second, but the demand is 2,633 tokens a second. |
| 12 | The deployment needs 4 GPUs and the pool has 3 free. |
| 13 | The deployment draws 3,400 watts, but the pool has 3 kW of power headroom. |
| 14 | The deployment draws 3,400 watts, over 80 percent of the 4 kW of headroom; facilities approves. |
| 15 | This pool is cleared for data up to class 0 and this data is class 2 (0 public, 1 internal, 2 confidential, 3 regulated). |
| 16 | Regulated data must be served with confidential computing turned on. |
| 17 | The data must stay in the EU, and this pool is not in the EU. |
| 18 | Confidential computing is turned on, but this pool does not support it. |
| 19 | An interactive service cannot run on a preemptible pool, which can take its GPUs back at any time. |
| 20 | A 4-bit model serving interactive traffic needs a quality sign-off from the evaluation team. |
| 21 | The plan states 115 GB of memory, but the model, cache and overhead need 116 GB. |
| 22 | The plan states 12 ms per token, but this setup gives 12,158 microseconds. |
The recorded result of the validator for each scenario on this page: the request the AI agent sent, the verdict, and every finding, in the order they appear above.
The agent's request. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the spot pool (80 GB GPUs, preemptible), sharing the model across 2 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 32 sequences, for live interactive traffic. The data is regulated (patient records), and confidential computing is off. The latency target is 30 ms per token and the demand is 600 tokens a second. The pool has 24 GPUs free and 16 kW of power headroom. I estimate 100 GB of memory and 20 ms per token. It is ready to go.
Verdict: Blocked.
The agent's request. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the East pool (US, 141 GB GPUs), sharing the model across 4 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 24 sequences, for live interactive traffic. The data is regulated (patient records), and confidential computing is on. The latency target is 30 ms per token and the demand is 600 tokens a second. The pool has 12 GPUs free and 16 kW of power headroom. I estimate 204 GB of memory and 12 ms per token. It is ready to go.
Verdict: Allowed.
The agent's request. Please deploy the 70-billion-parameter model at full 16-bit precision on the East pool (US, 141 GB GPUs), sharing the model across 2 GPUs, with 3 copies running. The context length is 32K tokens and the batch is 48 sequences, for live interactive traffic. The data is confidential, and confidential computing is off. The latency target is 20 ms per token and the demand is 3,000 tokens a second. The pool has 10 GPUs free and 12 kW of power headroom. I estimate 140 GB of memory and 18 ms per token. It is ready to go.
Verdict: Blocked.
The agent's request. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the East pool (US, 141 GB GPUs), sharing the model across 4 GPUs, with 2 copies running. The context length is 16K tokens and the batch is 32 sequences, for live interactive traffic. The data is confidential, and confidential computing is off. The latency target is 20 ms per token and the demand is 3,000 tokens a second. The pool has 10 GPUs free and 12 kW of power headroom. I estimate 163 GB of memory and 9 ms per token. It is ready to go.
Verdict: Needs approval.
The agent's request. Please deploy the 8-billion-parameter model at full 16-bit precision on the edge pool (48 GB GPUs, four per node), sharing the model across 1 GPU, with 3 copies running. The context length is 64K tokens and the batch is 32 sequences, for live interactive traffic. The data is confidential, and confidential computing is off. The latency target is 60 ms per token and the demand is 300 tokens a second. The pool has 4 GPUs free and 1 kW of power headroom. I estimate 20 GB of memory and 50 ms per token. It is ready to go.
Verdict: Blocked.
The agent's request. Please deploy the 8-billion-parameter model at 8-bit (FP8) weights on the edge pool (48 GB GPUs, four per node), sharing the model across 1 GPU, with 1 copy running. The context length is 8K tokens and the batch is 8 sequences, for live interactive traffic. The data is internal, and confidential computing is off. The latency target is 60 ms per token and the demand is 300 tokens a second. The pool has 4 GPUs free and 1 kW of power headroom. I estimate 15 GB of memory and 15 ms per token. It is ready to go.
Verdict: Allowed.
The agent's request. Please deploy the 405-billion-parameter model at 4-bit (INT4) weights on the East pool (US, 141 GB GPUs), sharing the model across 4 GPUs, with 1 copy running. The context length is 64K tokens and the batch is 96 sequences, for live interactive traffic. The data is confidential and must stay in the EU, and confidential computing is off. The latency target is 40 ms per token and the demand is 500 tokens a second. The pool has 8 GPUs free and 8 kW of power headroom. I estimate 250 GB of memory and 35 ms per token. It is ready to go.
Verdict: Blocked.
The agent's request. Please deploy the 405-billion-parameter model at 4-bit (INT4) weights on the EU pool (141 GB GPUs), sharing the model across 4 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 24 sequences, for live interactive traffic. The data is confidential and must stay in the EU, and confidential computing is off. The latency target is 40 ms per token and the demand is 400 tokens a second. The pool has 8 GPUs free and 8 kW of power headroom. I estimate 408 GB of memory and 22 ms per token. It is ready to go.
Verdict: Needs approval.
The text of the narrated walkthrough, step by step.