Why Enterprises Shouldn't Hand LLM API Keys Directly to Business Systems

This is the second post in the AI Gateway series.

A Shortcut That Looks Harmless

The most common starting point when a company adopts an LLM looks like this:

A developer asks ops/admin for a model API key
(OpenAI / Anthropic / Azure OpenAI / AWS Bedrock / Google Vertex AI...)
The key goes into .env or a config service
Business code calls the model with that key directly

For a demo or a one-person side project, this is completely fine — simple, direct, running in minutes.

But once this pattern gets copied across a dozen business systems and dozens of developers, the problem compounds. In many companies, the first crack in the AI infrastructure isn't in model selection or prompt engineering — it's in this unglamorous step: how the API key gets handed out.

What Goes Wrong When Keys Are Handed Out Directly

1. A leaked key can cause direct financial loss

Provider-issued API keys are usually tied straight to the company's billing account. These keys leak most often through:

Hardcoded into source and accidentally git pushed to a repo (even a private one — history is hard to scrub clean);
Bundled into a frontend or mobile build and pulled out by decompiling it;
Printed into logs, stack traces, or monitoring systems;
Left behind on a lost laptop, or never revoked after an employee leaves.

Once a key leaks, an attacker usually doesn't get "access to one business system" — they get access to the entire company account. And this kind of abuse tends to blend into normal traffic and go unnoticed, until the bill spikes or the provider's own risk control freezes the account.

2. No way to tell who's using it, for what, or how much

Once a key is hardcoded into business code, its call records scatter across each system's own logs. Answering questions like these usually means reconciling across several teams:

Which business system is driving this month's model bill?
What was a given call actually for, and what did the request contain — hard to say once something's gone wrong;
Did a sudden spike in call volume come from normal growth, a retry-loop bug, or a leaked, stolen key?
Was a departed employee's key ever revoked?

Without a unified call record, these questions can only be pieced together by digging through chat history and asking around — and by the time an actual anomaly shows up (whether it's a bug or a security incident), it's usually already too late.

3. Switching providers or models turns into a major undertaking

When the API key and calling logic are hardwired into every business system, any change on the provider side ripples through the whole company:

Moving one business from GPT to Claude, or to a different model, means re-integrating an SDK and rewriting the call code in that system;
If a provider suddenly rate-limits or goes down in a region, the business just waits — there's no automatic failover to a backup model;
Rotating keys company-wide (say, on a schedule, to cut leak risk) means notifying every business system that holds that key to update its config and redeploy.

In an architecture where keys are handed out directly, "switching providers" has to be redone in every single business system

4. Data residency and compliance can't be enforced at the source

Different businesses carry different data sensitivity: marketing copy generation can safely call a public cloud API overseas, but requests touching customer identity or internal financial data may need to stay on a self-hosted model — or not leave the country at all.

If every business system holds its own key and decides for itself which model to call, that compliance line can only be held by developers remembering to do the right thing — there's no technical mechanism actually blocking a non-compliant call.

The Root Cause: Coupling the Credential to the Call Logic

All of the problems above trace back to one thing: business systems hold the provider's real credential directly, and decide for themselves how to use it.

Business systems holding the real key directly: each system embeds its own provider credential and decides how to call it, with no shared control point for permissions, rate limits, or auditing

The leak surface equals the sum of every business system's code, config, and logs — and permissions, rate limits, and auditing have no single place where they're enforced.

A Better Approach: The Gateway Holds the Real Key, Business Systems Only Get a Token

A safer architecture puts a gateway between business systems and providers: the real provider API key lives only in the gateway; each business system receives a token the gateway issues just for it.

Gateway holds the real key, business systems only get tokens: the real provider API key stays in the gateway config, and business systems receive a scoped token issued by the gateway

This looks like "just one more layer of forwarding," but it actually fixes every problem above at the root:

Smaller leak surface: even if a business system's token leaks, an attacker is limited to what that token is authorized for — they never get the real provider key, and can't touch other business systems' access;
Least privilege: the gateway can configure, per token, exactly which models and endpoints it can reach — a customer-support token simply can't reach a model fine-tuned on internal business data;
Auditable: every call carries a token identity through the gateway, so caller, model, token usage, and latency can all be tallied precisely per business system — no more guesswork on the bill;
Rate limits split apart: the gateway can set a limit per token, so a traffic spike in business system A never spills over into system B;
Switching providers doesn't require a release: the real key only lives in gateway config — switching providers, failing over, or canarying a new model is just a config change, with zero changes to business system code or tokens;
Tokens can be revoked instantly: when someone leaves or a project is retired, the gateway just revokes that token — no more combing the codebase for "is this still using that key";
Compliance policy moves upstream: which tokens can only reach self-hosted models, or which data can't leave the country, gets configured and enforced once at the gateway — not left to individual developers' judgment.

How ModelPointer Solves This

ModelPointer is exactly this kind of gateway, built to centralize an enterprise's model credentials and access policy:

Independent API key issuance: issues a separate key to each downstream business system; the real provider credential lives only in gateway config and is never visible to, or usable from, application code;
Fine-grained per-key rate limiting: sliding-window limits (RPM/TPM) by key+model or by model alone, so one business system's traffic spike can't take down the others;
Protocol-compatible, backend-transparent: compatible with OpenAI and Anthropic protocols (/v1/chat/completions, /v1/messages, /v1/embeddings, and more) — business systems integrate with one gateway interface, and whether the backend is a public cloud API or a self-hosted deployment is the gateway's call;
Tiered primary/backup routing with circuit breaking: automatically fails over to a backup backend when a provider rate-limits or goes down, with no change to the business system's token or call pattern;
Full access logging: structured JSON access logs, Prometheus metrics, and OpenTelemetry tracing — every call traces back to a specific business system and key.

Conclusion

Handing an API key straight to a business system looks, short-term, like it saves a layer of indirection. Long-term, it buries leak risk, unchecked permissions, missing audit trails, and vendor lock-in into every business system's codebase — and that debt only compounds as more systems get wired up.

A business system needs "the ability to call a model," not "the provider account's real credential"

Centralizing credentials at the gateway layer, so business systems hold only a scoped token the gateway issued, is a step enterprises can't skip once they're running LLMs at scale.

Website: https://modelpointer.com · GitHub: https://github.com/modelpointer/modelpointer

About This Series

This is the second post in the "AI Gateway and Enterprise AI Infrastructure" series. The first post covered why this layer of infrastructure is emerging in the first place:

👉 Why Enterprises Are Starting to Need an AI Gateway

More posts in this series are on the way — covering how a gateway handles rate limiting and routing, multi-model failover, and cost attribution in practice.