New Models Keep Shipping — Why Is Upgrading So Hard for Enterprises?

This is the fifth post in the AI Gateway series.

A Scene That Keeps Repeating

One day, a tech lead needs to handle something company-wide — maybe a vendor is deprecating a model, maybe security flagged a model as non-compliant, maybe finance just wants to consolidate spend on whichever models give the best value. The first thing they need is a simple fact: which systems are calling which models, from which vendors, at which versions.

Asking around, nobody can answer it in full. Customer support was wired up to OpenAI years ago. Knowledge-base Q&A later moved to Alibaba Cloud. The data analytics agent was set up by an engineer who just requested an Anthropic key directly. Nobody's sure whether some other system is quietly calling yet another model somewhere. Before "upgrade everything" can even start, it's already stuck on "we don't know what's actually running."

This isn't an isolated case. Once an enterprise reaches any real scale, model usage almost inevitably fragments — not because any one team lacks the skill, but because "which model to use" was never treated as a decision the enterprise makes once, centrally. It was left to each business team to decide on its own.

Why Model Governance Is So Hard

1. The decision of which model to use has always been scattered across teams

There's no central plan — each team picks based on whatever resources, vendor relationships, or budget it happened to have at the time. Customer support adopted OpenAI first; knowledge-base Q&A later connected to Alibaba Cloud; the analytics team went with Anthropic because it was convenient. Nobody at the company level has ever decided which models the company should run and who owns them — selection authority is scattered by default.

2. Nobody holds an accurate, company-wide inventory of what's running

Ask "how many business systems call an LLM, which models, which versions, through which vendors," and nobody can answer completely — each team only knows its own corner. Trying to run a unified upgrade fails at step one: you can't even take inventory, let alone plan a rollout.

3. Every team upgrades on its own timeline, so old and new versions coexist indefinitely

Even when one team upgrades, others stay on older versions — sometimes even calling models a completely different way, some through a shared wrapper, others with a vendor key wired in directly. Model versions, vendors, and calling conventions stay fragmented across the company with no tendency to converge — not because nobody wants unification, but because there's simply nowhere for "unification" to actually happen.

4. Governance decisions have no single place to execute

Security says "this vendor's model has a compliance issue, stop using it." Finance says "consolidate spend on the models with the best value." These are company-level governance calls, but because models are scattered across each business system's own code, there's no single place to enforce them — someone has to go negotiate and change things team by team. A governance decision becomes a prolonged back-and-forth instead of a config change that takes effect immediately.

The Root Cause: Models Were Never Treated as an Asset That Needs Unified Governance

All of the above comes down to one thing: every business system is independently choosing, calling, and maintaining its own models, with no place that sees the whole picture or makes a unified decision. Selection and access authority are scattered across teams, so upgrading, replacing, or retiring any model becomes a long project of persuading and coordinating a pile of independent teams — instead of a single config change.

The Better Approach: Centralize Model Access at One Layer, So Upgrades Have Somewhere to Land

The gateway already sits in the one place every model call is guaranteed to pass through — every request from every business system ends up routed through it to a specific backend. That means "which models are running across the company, and what they actually point to" can be fully visible at the gateway layer — and upgrading, replacing, or retiring a model can be executed there too, with business code never needing to know and no team-by-team negotiation required.

Concretely, this layer needs to provide at least three things:

• Decouple the client-facing model name from the actual backend it routes to: business systems always call a stable name (say, company-chat); the gateway's config decides which vendor and which model that name currently forwards to, so business code never has to change;
• Keep old names working: even after a model's official name changes, the old name can be set as an alias for the new config — existing callers get the upgrade automatically, with no migration required on their end;
• Shift traffic gradually by weight: instead of "switch everything" or "switch nothing," route a small weighted slice of traffic to the new model first, confirm quality, cost, and stability, then gradually raise the weight to 100% — and dial it back instantly if something's wrong, without any business system redeploying.

Whether an enterprise can govern its models has nothing to do with how many coordination meetings it holds — it depends on whether there's one place that shows exactly which models are running

How ModelPointer Solves This

In ModelPointer's design, the client-facing model name and the backend it actually routes to are two separate things from the start, and every model an enterprise uses is declared in one shared routes.yaml — for the first time, there's an actual inventory to look at:

• One config, the whole routing picture: every model, vendor, and version the company calls is declared in the same routes.yaml — no more asking each team "what model are you using";
• Model name decoupled from the upstream model: routes.yaml uses upstream_model to translate the name a client sends into whatever name the upstream actually expects — the name a business system calls never has to change, even after the upstream model has been through several version bumps;
• Aliases: a model can be configured with multiple aliases, so both the old name and the new name route to the same config — upgrading a model doesn't require business systems to update their call parameters in lockstep;
• Weighted routing for canary rollouts: a single model name can have multiple upstreams configured under it, with weights (SWRR) controlling the traffic split — old and new models can coexist proportionally while weight gradually shifts from old to new;
• Hot-reload, zero-downtime switching: whether config lives in YAML files or a database, weight changes take effect at the next sync interval — no gateway restart, and no coordination required with business systems;
• Protocol independence: OpenAI and Anthropic protocols can have their own routing and weights configured independently, so upgrading a model on one protocol never affects the other.

Conclusion

New models keep shipping, but enterprises stay stuck on old ones — on the surface it looks like slow decision-making. In practice, it's that models were never treated as an asset that needs unified governance: selection authority stayed scattered across teams, the enterprise couldn't even say how many models were running, and upgrading was never really on the table.

How quickly a new model gets adopted has nothing to do with how impressive its benchmark scores are — it depends on whether the enterprise has one place that can see, and change, every model call at once

Centralize model selection and routing at the gateway layer, and a model upgrade stops being a governance problem that requires persuading a dozen teams one by one — it becomes a config edit and a weight adjustment.

Website: https://modelpointer.com · GitHub: https://github.com/modelpointer/modelpointer

---

About This Series

This is the fifth post in the "AI Gateway and Enterprise AI Infrastructure" series. The first four posts covered why this layer of infrastructure is emerging, why API keys shouldn't be handed directly to business systems, why cloud models are still needed after self-hosting, and why enterprises can't say where their monthly LLM bill went:

👉 Why Enterprises Are Starting to Need an AI Gateway 👉 Why Enterprises Shouldn't Hand LLM API Keys Directly to Business Systems 👉 Why Enterprises Still Need Cloud Models After Deploying LLMs On-Prem 👉 Why Enterprises Can't Say Where Their Monthly LLM Bill Went

More posts in this series are on the way — covering complexity-aware model routing and on-prem vs. cloud cost comparisons in practice.