Why Enterprises Still Need Cloud Models After Deploying LLMs On-Prem
Why Enterprises Still Need Cloud Models After Deploying LLMs On-Prem
This is the third post in the AI Gateway series.
Self-Hosting Looks Like the Finish Line
Once enterprise LLM usage reaches a certain scale, a lot of teams make the same call: deploy an open model — DeepSeek, Qwen, GLM — on their own GPU cluster. The reasoning is straightforward. Data never leaves the internal network, which satisfies compliance. Cost shifts from per-token billing to GPU depreciation, which pencils out better at scale. And you stop being at the mercy of a vendor's rate limits, price hikes, or regional outages.
Once the self-hosted cluster is live, an obvious question follows:
Does this mean we can cut cloud API usage way down now?
A few months in, most teams find that internal users actively avoid the self-hosted model. The usual culprit is that the self-hosted deployment itself isn't stable under load — request volume ticks up even slightly and it starts rate-limiting, or response times noticeably degrade. Once internal users hit that a couple of times, they instinctively route around the self-hosted model and go back to the cloud API, and the self-hosted deployment never delivers what it was supposed to.
What Self-Hosting Can't Fix
1. Capacity is fixed, and traffic spikes overrun it
A GPU cluster is a fixed asset you bought — it doesn't scale in seconds the way public cloud does. Flash sales, campaign launches, a feature that suddenly goes viral: peak traffic can be several times baseline, and a local cluster's concurrency has a hard ceiling. Once it's hit, requests either queue or get dropped.
Cloud APIs sit on a pay-as-you-go compute pool that's effectively unbounded — exactly what you want for short, unpredictable spikes. The right model isn't "on-prem or cloud" — it's on-prem carrying the everyday, predictable load, with cloud absorbing whatever overflows past capacity.
2. Model capability has a ceiling, and frontier tasks still need the strongest model available
Self-hosted open models typically lag the most capable closed models in the cloud — in parameter count, training data, and the depth of post-training work that goes into them. That's an industry-wide fact, not a sign of a poorly run deployment: the open community iterates fast, but what cloud vendors pour into their top-tier models is a different order of investment, and no amount of ops discipline closes that gap.
Complex reasoning, long-context understanding, multimodal tasks, hard code generation — a mid-sized self-hosted model may simply not clear the accuracy bar these need. A small local model is plenty for customer support Q&A, but an agent doing contract review or complex data analysis often still needs to route to a stronger cloud model to hit the required accuracy.
3. Hardware maintenance and failures are a cost self-hosting can't avoid
GPU failures, driver and inference-framework upgrades, model version bumps — all of these need maintenance windows. Even when the self-hosted model is otherwise sufficient, the business needs a backup backend that can take over traffic during a maintenance window or an unplanned outage, or the business itself goes down with the hardware.
Cloud APIs are a natural fit for this "always-available fallback" role — they don't depend on the box you're currently working on.
4. Model iteration moves fast, and a self-hosted stack never quite catches up
Even enterprises that fine-tune their own models and track new open releases are still up against cloud vendors shipping a capability jump every few months. Wanting the business to always run on the most capable model available is a race the self-hosted path structurally loses — catching up means downloading new weights, evaluating them, and rolling out gradually, a cycle measured in weeks or months. A cloud API just needs a parameter change to start using the newest model immediately.
5. Some capabilities simply have no self-hosted equivalent
Text-to-image, text-to-video, large-scale embedding retrieval at scale — most enterprises won't and shouldn't try to run these themselves. A cloud-hosted specialized model service is the better call on both cost and quality.
The Real Answer: Not Either/Or, but Hybrid Scheduling
Self-hosting handles the load that's everyday, predictable, and data-sensitive. Cloud models fill in the gaps self-hosting structurally can't cover: traffic peaks, frontier capability, availability fallback, and specialized use cases. The two aren't substitutes for each other — they're layers that work together.
But if the decision of "when to use on-prem, when to use cloud" is left to each business system to figure out on its own, the org lands right back at the original problem — every team integrating its own vendor SDK — except now it's "on-prem plus several cloud vendors" instead of just several cloud vendors. Integration cost goes up, not down.
A Gateway Is What Makes On-Prem and Cloud Work Together
A better architecture puts both the on-prem cluster and cloud APIs behind a gateway. Business systems call a single model alias, and the gateway decides which backend actually handles the request:
What this gateway layer needs to do:
- Primary/backup tiered routing: requests go to the on-prem cluster first; when it hits a rate limit or fails a health check, the gateway automatically falls back to a cloud API — the business system keeps calling the same model alias and never notices the switch;
- Complexity-aware routing: simple, high-frequency tasks stay on a small local model, while complex tasks or those with a hard accuracy requirement route straight to a stronger cloud model;
- Circuit breaking and automatic recovery: a GPU node under maintenance or in a failed state gets pulled out of rotation automatically, with traffic shifting to cloud as a fallback and shifting back once the node is healthy again;
- Unified metering and cost attribution: whether a request lands on-prem or in the cloud, the gateway records usage and token consumption in one place, so the real combined cost of on-prem vs. cloud can be compared with data instead of guesswork;
- Data residency policy enforced upstream: which requests must stay on-prem and which can leave the network for a cloud API gets enforced once at the gateway, rather than relying on every developer remembering the rule.
How ModelPointer Solves This
ModelPointer is built around scheduling across multiple backends — an on-prem cluster and a cloud API are just different nodes in the same routing setup as far as it's concerned:
- Primary/backup tiered routing with circuit breaking: the on-prem cluster is the primary tier and cloud APIs are the backup tier; when capacity is exceeded or a node is unhealthy, the gateway switches automatically instead of returning a 429, and switches back once things recover;
- Protocol-compatible, backend-transparent: compatible with OpenAI and Anthropic protocols — business systems integrate with one gateway interface, and whether the backend is a self-hosted vLLM/SGLang deployment or a cloud API is the gateway's call;
- Fine-grained rate limiting by key and model: on-prem capacity can be rate-limited separately per business line, so one team's traffic spike doesn't saturate the whole cluster;
- Full access logging: structured JSON logs and Prometheus metrics record which backend actually served each call, giving you the data to compare on-prem and cloud cost and quality directly;
- Two configuration modes: hot-reloaded YAML or database-backed config — adjusting primary/backup policy or routing rules never requires redeploying a business system.
Conclusion
Self-hosting handles the deterministic load — sensitive data, steady call volume, predictable scale. Cloud models cover what self-hosting structurally can't: unpredictable load, capability ceilings, and availability fallback.
On-prem and cloud aren't a question of one replacing the other — it's a question of which one owns which layer of load
A gateway is what lets the on-prem cluster and cloud APIs work as one system, instead of forcing business systems to pick a side between two separate vendor stacks.
Website: https://modelpointer.com · GitHub: https://github.com/modelpointer/modelpointer
About This Series
This is the third post in the "AI Gateway and Enterprise AI Infrastructure" series. The first two posts covered why this layer of infrastructure is emerging, and why API keys shouldn't be handed directly to business systems:
👉 Why Enterprises Are Starting to Need an AI Gateway 👉 Why Enterprises Shouldn't Hand LLM API Keys Directly to Business Systems
More posts in this series are on the way — covering complexity-aware model routing and on-prem vs. cloud cost comparisons in practice.