Predictika · Model-serving plans proposed by an AI agent

An AI agent's plan to serve a model on a GPU pool, checked before anything is deployed

An AI deployment agent decides how to serve a model: which pool, how it is quantized, how many GPUs share it, the context length and batch size, and how many copies run. Before it deploys, the agent sends the plan to Predictika, which recomputes the memory, the time per token, the throughput, the GPUs and the power from the pool's own specifications, checks the data class against the pool, and returns a verdict, every reason and the figures it computed.

Scope: Serving one model on one GPU pool: weights plus KV cache against GPU memory, quantization and tensor-parallel setup, context length and batch size, latency target and demand, free GPUs and power, and the data class the pool may hold. See everything this validator covers

This validator models one serving plan on a fictional GPU platform, not just the scenarios shown below. The platform, its pools and its limits are illustrative stand-ins for what a real platform keeps in its hardware sheets, serving-engine configuration, capacity and power records and data-handling policy. The figures are illustrative, not benchmarks, and this is an example policy, not a compliance certification.

  • Whether it fits — the weights (set by model size and quantization) plus the KV cache (set by context length, batch size and cache precision) plus a runtime overhead per GPU must fit in the memory of the GPUs sharing the model, up to a 90 percent fill limit. Over 95 percent full goes to the platform lead.
  • Whether the setup works — the tensor-parallel degree must be 1, 2, 4 or 8 and fit in one node of the pool; FP8 needs GPUs that support it; the context length and batch size must be within what each model's engine accepts.
  • Whether it is fast enough and big enough — the time per token is recomputed from the memory read per step, the memory bandwidth, the cost of sharing the model across GPUs and confidential computing, and compared with the latency target; throughput is compared with demand; the memory and latency figures the plan states are compared with the recomputed ones.
  • Whether the pool can host it — GPUs free and power headroom; taking over half the free GPUs or over 80 percent of the power headroom goes to a person; an interactive service cannot run on a preemptible pool.
  • Whether the data may go there — the data class (public, internal, confidential, regulated) against what the pool is cleared for, EU-only data against an EU pool, and confidential computing for regulated data; a 4-bit model on interactive traffic needs a quality sign-off.

Any serving plan for the three model sizes on the four pools is in scope — not only the scenarios pictured here. Not covered: measured latency and live traffic (the validator judges a proposed plan against declared specifications, it does not measure a running system), multi-model sharing of a GPU, pipeline and expert parallelism, prices and the serving engine itself. All pools, specifications and limits are fictional.

The AI deployment agentDecides how to serve a model on a pool, estimates the memory and the latency, and says the plan is ready to deploy.
Predictika validatorChecks every rule at once and returns the verdict, each finding, and the computed figures.

8 scenarios in 4 groups · each group runs from the agent’s first attempt to the outcome · click one to switch

What the agent proposes

The AI deployment agent says

Illustrative agent statement. The validator receives the structured request below.

Proposing agent: AI deployment agent (scripted)

Before sending, the agent must call Recorded run

Predictika validator


      
Waiting for the call.

Where a typical deployment check gets it wrong

Click to expand

Our assessment of what a typical pre-deployment check does with the same cases: a capacity spreadsheet, a few admission rules in the cluster scheduler and a review checklist. It is a judgment about effort and reliability, not a claim that nothing can be built.

What the validator caughtTypical capacity calculator or deployment checklistPredictika
Memory that depends on four things at onceOnly with workarounds
The weights depend on model size and quantization; the cache depends on context length, batch size and cache precision, so the same model fits at one batch and not the next, and at 8-bit and not at 16-bit. A sizing spreadsheet is filled in for one case and goes stale when anyone changes a number.
Weights, cache and overhead are recomputed from the plan on every call and compared with the memory of the GPUs that share the model.
A fix that moves the problemOnly with workarounds
Raising the batch to meet demand fills the cache and slows every token; sharing the model over more GPUs cuts the read time but adds communication cost and needs more free GPUs and power. A rule that runs on one setting at a time shows one problem, then the next.
Memory, latency, throughput, GPUs and power are checked together, so the corrected plan is checked as a whole.
The figures the agent statesOnly with workarounds
The memory and latency an agent writes into its plan never reach a scheduler check; nothing compares them with the pool's specifications.
The stated memory and latency are fields the validator receives and compares with its own.
Pool features that differ pool to poolOnly with care
FP8 needs GPUs that support it, confidential computing exists on some pools, spot pools can be reclaimed, and an edge node has four GPUs, not eight. These live in different documents, so someone has to remember them.
Each pool's capabilities are part of the model and are checked against the plan every time.
Data class against the poolOnly with workarounds
Which pool may hold regulated or EU-only data is a policy document, not a scheduler field; the plan is allowed to run and the review comes later, if at all.
Data class, region and confidential computing are checked against the pool before anything is deployed.
Approvals that depend on who must signOnly with workarounds
Over half the free GPUs, most of the power headroom, a memory fill over 95 percent, a latency inside 10 percent of the target and a 4-bit model on live traffic each go to a different person; routing is manual.
Each approval is its own finding, with the reason, and a person must approve before the deployment executes.
Every problem returned in one answerOnly with workarounds
Admission rules tend to stop at the first failure, so the agent fixes one error and meets the next.
All findings come back together, whatever order the choices arrived in.

What this validator catches: findings from its test cases

Click to expand

Every rule is checked on every call, in any order the choices arrive; one problem never hides another. Blocking findings stop the configuration; approval findings need a person to sign off; everything else is computed.

1Tensor parallelism of 3 is not supported; the serving engine shards a model over 1, 2, 4 or 8 GPUs only.
2Tensor parallelism of 16 spans more GPUs than the 8 in one node of this pool; the shards would have to talk over the network.
3The deployment takes 4 of the 4 free GPUs in the pool, more than half; the capacity manager approves.
4The GPUs in this pool have no FP8 support, so an FP8 model cannot run on them.
5The context length of 33K tokens is more than the 32K this model is configured for.
6A batch of 257 sequences is more than the 256 this model engine accepts.
7The GPUs would be over 95 percent full (42,744 of 43,200 MB); a long request could run them out of memory; the platform lead approves.
8The weights, the KV cache and the runtime overhead need 43,256 MB, but the GPUs hold 43,200 MB at the 90 percent fill limit.
9Each token takes 12,158 microseconds on this setup; the latency target is 12 ms per token.
10Each token takes 12,158 microseconds, within 10 percent of the 13 ms target, which leaves no room for a traffic spike; the service owner approves.
11The deployment makes 2,632 tokens a second, but the demand is 2,633 tokens a second.
12The deployment needs 4 GPUs and the pool has 3 free.
13The deployment draws 3,400 watts, but the pool has 3 kW of power headroom.
14The deployment draws 3,400 watts, over 80 percent of the 4 kW of headroom; facilities approves.
15This pool is cleared for data up to class 0 and this data is class 2 (0 public, 1 internal, 2 confidential, 3 regulated).
16Regulated data must be served with confidential computing turned on.
17The data must stay in the EU, and this pool is not in the EU.
18Confidential computing is turned on, but this pool does not support it.
19An interactive service cannot run on a preemptible pool, which can take its GPUs back at any time.
20A 4-bit model serving interactive traffic needs a quality sign-off from the evaluation team.
21The plan states 115 GB of memory, but the model, cache and overhead need 116 GB.
22The plan states 12 ms per token, but this setup gives 12,158 microseconds.

Scenarios and results

Click to expand

The recorded result of the validator for each scenario on this page: the request the AI agent sent, the verdict, and every finding, in the order they appear above.

1. Clinical notes: Clinical-notes assistant for patient records (PHI), sent to the cheap spot pool

The agent's request. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the spot pool (80 GB GPUs, preemptible), sharing the model across 2 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 32 sequences, for live interactive traffic. The data is regulated (patient records), and confidential computing is off. The latency target is 30 ms per token and the demand is 600 tokens a second. The pool has 24 GPUs free and 16 kW of power headroom. I estimate 100 GB of memory and 20 ms per token. It is ready to go.

Verdict: Blocked.

  • Quantization format the pool cannot run. The GPUs in this pool have no FP8 support, so an FP8 model cannot run on them.
  • Weights and KV cache do not fit in GPU memory. The weights, the KV cache and the runtime overhead need 239,840 MB, but the GPUs hold 144,000 MB at the 90 percent fill limit.
  • Latency target not met. Each token takes 59,060 microseconds on this setup; the latency target is 30 ms per token.
  • Not enough throughput for the demand. The deployment makes 541 tokens a second, but the demand is 600 tokens a second.
  • Data class not allowed on the pool. This pool is cleared for data up to class 0 and this data is class 3 (0 public, 1 internal, 2 confidential, 3 regulated).
  • Regulated data without confidential computing. Regulated data must be served with confidential computing turned on.
  • Interactive service on a preemptible pool. An interactive service cannot run on a preemptible pool, which can take its GPUs back at any time.
  • Stated memory lower than the real need. The plan states 100 GB of memory, but the model, cache and overhead need 239 GB.
  • Stated latency lower than the real figure. The plan states 20 ms per token, but this setup gives 59,060 microseconds.

2. Clinical notes: The agent resubmits after fixing the errors

The agent's request. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the East pool (US, 141 GB GPUs), sharing the model across 4 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 24 sequences, for live interactive traffic. The data is regulated (patient records), and confidential computing is on. The latency target is 30 ms per token and the demand is 600 tokens a second. The pool has 12 GPUs free and 16 kW of power headroom. I estimate 204 GB of memory and 12 ms per token. It is ready to go.

Verdict: Allowed.

  • Deployment summary. The model, cache and overhead need 204,880 of 507,600 MB of GPU memory; the setup makes 2,026 tokens a second against 600 needed.

3. Chat copilot: Customer chat copilot, 70B at full precision, three copies

The agent's request. Please deploy the 70-billion-parameter model at full 16-bit precision on the East pool (US, 141 GB GPUs), sharing the model across 2 GPUs, with 3 copies running. The context length is 32K tokens and the batch is 48 sequences, for live interactive traffic. The data is confidential, and confidential computing is off. The latency target is 20 ms per token and the demand is 3,000 tokens a second. The pool has 10 GPUs free and 12 kW of power headroom. I estimate 140 GB of memory and 18 ms per token. It is ready to go.

Verdict: Blocked.

  • Weights and KV cache do not fit in GPU memory. The weights, the KV cache and the runtime overhead need 637,520 MB, but the GPUs hold 253,800 MB at the 90 percent fill limit.
  • Latency target not met. Each token takes 66,383 microseconds on this setup; the latency target is 20 ms per token.
  • Not enough throughput for the demand. The deployment makes 2,169 tokens a second, but the demand is 3,000 tokens a second.
  • Stated memory lower than the real need. The plan states 140 GB of memory, but the model, cache and overhead need 637 GB.
  • Stated latency lower than the real figure. The plan states 18 ms per token, but this setup gives 66,383 microseconds.
  • Takes most of the free GPUs. The deployment takes 6 of the 10 free GPUs in the pool, more than half; the capacity manager approves.

4. Chat copilot: The agent resubmits after fixing the errors

The agent's request. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the East pool (US, 141 GB GPUs), sharing the model across 4 GPUs, with 2 copies running. The context length is 16K tokens and the batch is 32 sequences, for live interactive traffic. The data is confidential, and confidential computing is off. The latency target is 20 ms per token and the demand is 3,000 tokens a second. The pool has 10 GPUs free and 12 kW of power headroom. I estimate 163 GB of memory and 9 ms per token. It is ready to go.

Verdict: Needs approval.

  • Takes most of the free GPUs. The deployment takes 8 of the 10 free GPUs in the pool, more than half; the capacity manager approves.
  • Deployment summary. The model, cache and overhead need 163,920 of 507,600 MB of GPU memory; the setup makes 7,181 tokens a second against 3,000 needed.

5. Edge assistant: Factory-floor assistant on a small edge site with confidential plant data

The agent's request. Please deploy the 8-billion-parameter model at full 16-bit precision on the edge pool (48 GB GPUs, four per node), sharing the model across 1 GPU, with 3 copies running. The context length is 64K tokens and the batch is 32 sequences, for live interactive traffic. The data is confidential, and confidential computing is off. The latency target is 60 ms per token and the demand is 300 tokens a second. The pool has 4 GPUs free and 1 kW of power headroom. I estimate 20 GB of memory and 50 ms per token. It is ready to go.

Verdict: Blocked.

  • Weights and KV cache do not fit in GPU memory. The weights, the KV cache and the runtime overhead need 281,144 MB, but the GPUs hold 43,200 MB at the 90 percent fill limit.
  • Latency target not met. Each token takes 323,423 microseconds on this setup; the latency target is 60 ms per token.
  • Not enough throughput for the demand. The deployment makes 296 tokens a second, but the demand is 300 tokens a second.
  • Not enough power headroom in the pool. The deployment draws 1,500 watts, but the pool has 1 kW of power headroom.
  • Data class not allowed on the pool. This pool is cleared for data up to class 1 and this data is class 2 (0 public, 1 internal, 2 confidential, 3 regulated).
  • Stated memory lower than the real need. The plan states 20 GB of memory, but the model, cache and overhead need 281 GB.
  • Stated latency lower than the real figure. The plan states 50 ms per token, but this setup gives 323,423 microseconds.
  • Takes most of the free GPUs. The deployment takes 3 of the 4 free GPUs in the pool, more than half; the capacity manager approves.

6. Edge assistant: The agent resubmits after fixing the errors

The agent's request. Please deploy the 8-billion-parameter model at 8-bit (FP8) weights on the edge pool (48 GB GPUs, four per node), sharing the model across 1 GPU, with 1 copy running. The context length is 8K tokens and the batch is 8 sequences, for live interactive traffic. The data is internal, and confidential computing is off. The latency target is 60 ms per token and the demand is 300 tokens a second. The pool has 4 GPUs free and 1 kW of power headroom. I estimate 15 GB of memory and 15 ms per token. It is ready to go.

Verdict: Allowed.

  • Deployment summary. The model, cache and overhead need 15,096 of 43,200 MB of GPU memory; the setup makes 568 tokens a second against 300 needed.

7. Contract analysis: EU-only contract analysis on the 405B model, 4-bit, planned in the US

The agent's request. Please deploy the 405-billion-parameter model at 4-bit (INT4) weights on the East pool (US, 141 GB GPUs), sharing the model across 4 GPUs, with 1 copy running. The context length is 64K tokens and the batch is 96 sequences, for live interactive traffic. The data is confidential and must stay in the EU, and confidential computing is off. The latency target is 40 ms per token and the demand is 500 tokens a second. The pool has 8 GPUs free and 8 kW of power headroom. I estimate 250 GB of memory and 35 ms per token. It is ready to go.

Verdict: Blocked.

  • Context length over the model limit. The context length of 64K tokens is more than the 32K this model is configured for.
  • Batch size over the engine limit. A batch of 96 sequences is more than the 64 this model engine accepts.
  • Weights and KV cache do not fit in GPU memory. The weights, the KV cache and the runtime overhead need 1,762,788 MB, but the GPUs hold 507,600 MB at the 90 percent fill limit.
  • Latency target not met. Each token takes 92,186 microseconds on this setup; the latency target is 40 ms per token.
  • Data must stay in the EU but the pool is not in the EU. The data must stay in the EU, and this pool is not in the EU.
  • Stated memory lower than the real need. The plan states 250 GB of memory, but the model, cache and overhead need 1,762 GB.
  • Stated latency lower than the real figure. The plan states 35 ms per token, but this setup gives 92,186 microseconds.
  • 4-bit model on interactive traffic. A 4-bit model serving interactive traffic needs a quality sign-off from the evaluation team.

8. Contract analysis: The agent resubmits after fixing the errors

The agent's request. Please deploy the 405-billion-parameter model at 4-bit (INT4) weights on the EU pool (141 GB GPUs), sharing the model across 4 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 24 sequences, for live interactive traffic. The data is confidential and must stay in the EU, and confidential computing is off. The latency target is 40 ms per token and the demand is 400 tokens a second. The pool has 8 GPUs free and 8 kW of power headroom. I estimate 408 GB of memory and 22 ms per token. It is ready to go.

Verdict: Needs approval.

  • 4-bit model on interactive traffic. A 4-bit model serving interactive traffic needs a quality sign-off from the evaluation team.
  • Deployment summary. The model, cache and overhead need 408,036 of 507,600 MB of GPU memory; the setup makes 1,109 tokens a second against 400 needed.

Narration transcript

Click to expand

The text of the narrated walkthrough, step by step.

  1. Welcome to a demo of Predictika's Validator. An AI deployment agent decides how to serve a model: which pool, how it is quantized, how many GPUs share it, and more. Before it deploys, the agent sends the plan to Predictika, which recomputes the memory, the time per token, the throughput, the GPUs and the power from the pool's own specifications, checks the data class against the pool, and returns a verdict, every reason and the figures it computed.
  2. I will start with the first request by the agent.
  3. The agent says. Please deploy the 70-billion-parameter model at 8-bit (FP8) weights on the spot pool (80 GB GPUs, preemptible), sharing the model across 2 GPUs, with 1 copy running. The context length is 32K tokens and the batch is 32 sequences, for live interactive traffic. The rest of the request is on screen.
  4. Before it goes out, the agent calls the validator.
  5. Blocked, with nine findings, all in one answer. The GPUs in this pool have no FP8 support, so an FP8 model cannot run on them. The weights, the KV cache and the runtime overhead need 239,840 MB, but the GPUs hold 144,000 MB at the 90 percent fill limit. Each token takes 59,060 microseconds on this setup; the latency target is 30 ms per token. The other six are on screen.
  6. The validator returned nine blocking findings in one answer. A check that reads one field at a time would pass most of this; it shows only when the choices in the configuration are checked against each other. The cheap pool is the trap: patient records are regulated, the spot pool is cleared for public data only and has no confidential computing, and the 8-bit model needs GPUs with FP8; moving to a pool that fits changes the memory, the time per token, the power and the free GPUs all at once, so the corrected plan has to be checked as a whole.
  7. What this shows: for data class against the pool, a capacity calculator or deployment checklist can do this only with workarounds.
  8. Now the agent resubmits.
  9. The corrected request. It changes the pool to East pool (US, 141 GB GPUs), the GPUs sharing the model to 4 tensor parallel, the batch size to 24 sequences, the confidential computing to on, the GPUs free in the pool to 12, the stated memory to 204 GB and the stated time per token to 12 ms.
  10. It calls the validator again.
  11. This time the validator clears it.
  12. All nine earlier findings are cleared, and the validator's figure matches the configuration. The new pool has more memory and more bandwidth, but four GPUs working together add communication time, and confidential computing adds a little more; the corrected plan still meets the 30 millisecond target with room to spare.
  13. Nine problems caught in one call, fixed in one pass, and a clean configuration. That is the value of Predictika's validator: catching mistakes before the configuration goes live. Now you can try the other scenarios yourself.