On-Premise Architecture, Gateway Routing, and Locally Controlled Infrastructure for Local-First Agents

Local-first agentic tooling means running the models your agents call on infrastructure you control.

If you’re currently renting inference from someone else's API, the obvious question is: is the switch worth it?

It depends on three things. Sustained token volume high enough to beat metered pricing, data that legally or contractually cannot leave your boundary, and a level of operational control you have to prove. 

Is Renting Inference a Good Default?

It generally is, but as always, it depends on the specifics of the use case. 

Token-metered pricing scales linearly and without a ceiling. A workload may be cheap during the pilot phase, but then cost 10x more when engineers run agentic workflows against it all day.

Running a model inside a virtual private cloud still means sharing physical silicon. Sovereign cloud packages sold by US-headquartered providers remain reachable under the CLOUD Act regardless of where data sits. 

Data location is no longer sufficient on its own, because the operational control plane, meaning software updates, identity management and orchestration decisions, can sit outside your boundary even when the data does not. 

A May 2026 Futurum and IBM study called this ‘control-plane dependency’ and named it alongside vendor lock-in and missing AI governance as the three gaps that geography-only sovereign offerings leave open.

And what about availability? Research from Cohere and IDC opens on model access restrictions and security incidents over the previous year, which showed enterprises how quickly a dependency they do not control can be interrupted by a decision they were not part of.

Is Full Sovereign AI Independence Possible?

None of that argues for maximalism. The Economist has described full sovereign AI independence from US and Chinese firms as effectively a pipe dream as of 2026, while allowing that a degree of protection from coercion is achievable. Decide which parts of the stack you want to recreate at home, and which dependencies are worth living with.

IBM's Institute for Business Value arrived at the same place with a model it called selective AI sovereignty, where vendor lock-in is acceptable for peripheral capabilities like transcription and unacceptable for the ones your business runs on. 

In that research, 72% of executives said they would accept a 20% cost increase to keep multiple AI vendors in play if it bought strategic freedom. The goal is not to bring everything home. It is to know which workloads you would be unable to move, and to stop adding to that list by accident.

How Much Volume Do You Need To Justify Owning the Metal?

Language model inference is memory-bound. Generating a single token means pulling every model parameter out of memory, so capacity and bandwidth set your ceiling long before FLOPS do.

Baseline VRAM is the parameter count times bytes per parameter, plus roughly 20% for runtime overhead and activations. 

Parameter count is the trap. 

Most current open-weight flagships are mixture-of-experts models. The number that matters is the total parameter count, not the active one. A model that activates 32 billion parameters per token out of a trillion still needs all trillion resident in VRAM, because the router can reach for any expert on any token. The active figure governs generation speed, and your memory budget answers to the total.

Quantization is therefore necessary. For a trillion-parameter model, 16-bit weights need roughly 2,400 GB and exceed any single node on the market. Eight-bit weights need roughly 1,200 GB and fit an eight-way B200 node. Four-bit weights need roughly 600 GB and fit several single-node builds. One precision choice separates a server from a cluster.

Fitting inside one chassis saves more than rack space. Training needs thousands of GPUs synchronizing gradients, which is what justifies InfiniBand fabrics, while inference runs forward passes against static weights and rides the internal system bus. Specifying an inference node like a training cluster is a common and expensive mistake.

The KV cache is the other half, and its size depends on the attention mechanism. On a grouped-query attention model, the cache grows with layers, heads and total context across every in-flight request, and it can outgrow the weights themselves. 

On a multi-head latent attention model, which caches a compressed representation in place of expanded per-head tensors, 32 users at 16,000 tokens each cost about 36 GB, roughly 6% of a 600 GB footprint. Check the attention architecture before sizing anything, because the two designs produce hardware bills an order of magnitude apart.

The Serving Stack Is Where Your Capacity Comes From

Buying hardware is the easy half. Running open-weight models under a naive Python runner wastes a large fraction of the GPU you just paid for, and the gap between a naive deployment and a tuned one is measured in multiples.

PagedAttention, introduced by vLLM, treats KV cache memory like operating system virtual memory pages, removing the fragmentation a contiguous allocator leaves behind so one card holds far more concurrent sessions. FlashAttention cuts traffic between high-bandwidth memory and on-chip SRAM, which is what makes long context windows affordable, and compilers like TensorRT-LLM fuse operation sequences into single optimized kernels.

Continuous batching is the property that makes the economics work at all. Each weight read is amortized across every request in the batch, so adding users to a GPU node degrades per-user throughput slowly rather than linearly. That is why one well-configured node serves a department, and why utilization is the number to watch. An idle GPU you own costs the same as a saturated one.

NVIDIA's inference reference architecture shows where this goes at the top end, with prefill and decode in separate pools, cache-aware routing, and gang scheduling so a service never comes up half-placed with GPUs idling. Most enterprises will never need that tier. 

Routing, prefill, decode, KV cache and model movement each scale against a different bottleneck, and collapsing them into one deployment hides which one is actually binding.

Can You Change Your Mind Later?

An AI gateway is a proxy between your applications and every model backend you call, giving all of them one API with shared authentication, routing, failover and spend tracking. Its strategic value is reversibility. 

When applications talk to a gateway instead of a hardcoded provider SDK, switching backends becomes a configuration change, and the OpenAI-compatible interface that both self-hosted servers and commercial APIs expose means the swap is often just a base URL.

That makes the gateway the piece to adopt first, before any hardware decision. It turns running your own inference into a reversible experiment, and it lets frontier APIs and local models run side by side while you learn which workloads belong where.

Self-hosted options split on runtime and on how much sits inside the open-source core. Vercel's mid-2026 comparison maps the field well, with the caveat that Vercel sells a competing managed product.

LiteLLM is the fastest route to a working proxy and reaches the widest provider list. Envoy AI Gateway suits teams already running a service mesh who want model routing inside the mesh instead of beside it. MLflow argues on Linux Foundation governance and built-in tracing, which matters more than it sounds. 

Without it, client-side agent traces and proxy-side request logs stay disconnected, and debugging an agent across that seam is genuinely unpleasant.

How Much Does It Cost To Self-Host a Gateway

Spend tracking usually puts a database in the path of every model call, so a slow query slows every request your agents make. Single sign-on, audit logging and data-retention guarantees frequently sit outside the open-source core, so a team that needs them on day one can end up running a commercial edition anyway.

A self-hosted gateway is only as safe as its release pipeline. In March 2026, two LiteLLM releases briefly shipped compromised packages to PyPI that targeted cloud credentials, and BerriAI engaged Mandiant in response. That incident is closed, the general exposure is not, and it applies to anything you self-host. 

Pin verified releases and treat the gateway as production infrastructure, because it holds every provider key you own, which puts it squarely inside your agentic AI security model.

How Do You Increase Operational Control of Self-Hosted Agents?

IBM defines digital sovereignty as an organization's ability to retain control over its systems, data, operations and AI, plus the ability to prove that control when it matters. The second half is what fails audits. It breaks into four properties, each of them testable.

Operational sovereignty asks who runs the control plane, i.e. do you hold full authority over configuration, upgrades and lifecycle management? 

Data sovereignty asks whether access, secrets, keys, logs and audit evidence stay under your control, a stricter question than where the bytes live. 

Technology sovereignty asks whether you could exit the stack without major disruption, which is why open standards count as a sovereignty property and not merely an engineering preference. 

AI sovereignty, the newest of the four, asks whether you can enforce where inference runs and how agent decisions are logged.

How Europe Scores Sovereignty in Practice

The European Commission has turned this into a scorecard. Its Cloud Sovereignty Framework rates providers across eight objectives on five SEAL tiers, from SEAL-0 (no sovereignty) through SEAL-2 (data sovereignty) and SEAL-3 (digital resilience) to SEAL-4 (full digital sovereignty). 

When the Commission awarded its first sovereign cloud tender in April 2026, worth up to 180 million euros, it set SEAL-2 as the floor. Three of the four winners reached SEAL-3, one reached SEAL-2, and none reached SEAL-4. Full sovereignty remains an aspiration even for providers building specifically to win a sovereignty tender.

Regulation is moving the same way. The EU's 2023 Data Act already requires cloud contracts to include the right to switch providers and port data to on-premise infrastructure. Under the AI Act, the Commission's enforcement powers over general-purpose AI providers and the Article 50 transparency obligations both took effect on 2 August 2026. 

The proposed Cloud and AI Development Act, published 3 June 2026, would add an EU-wide sovereignty assessment built on four assurance levels, though agreement is not targeted until Q4 2027. EU financial institutions have been managing ICT supplier risk under DORA since January 2025.

What "Local-First" Meant Before It Meant This

The term is borrowed, and readers deserve to know from where. In 2019, Martin Kleppmann, Adam Wiggins, Peter van Hardenberg and Mark McGranaghan published "Local-First Software: You Own Your Data, in spite of the Cloud" at Ink & Switch. It was about collaborative documents, CRDTs and keeping a user's data on their own device, and it set out seven ideals for software that prioritizes local storage and networks over remote servers.

Six of the seven transfer to enterprise agent infrastructure almost without modification, which is more than a borrowed metaphor usually earns.

"No spinners: your work at your fingertips" becomes latency you own, with queue depth, batch policy and admission control yours to tune instead of a shared tenant's to compete for. "Your work is not trapped on one device" becomes model portability, delivered by the gateway and the OpenAI-compatible interface. "The network is optional" becomes air-gap capability, the hardest of the seven to retrofit, because an agent that must reach a hosted model to think cannot run in an isolated environment at all. "The Long Now" becomes freedom from remote shutdown and forced deprecation, the exposure the Cohere and IDC research found enterprises reacting to. "Security and privacy by default" becomes data that never crosses the boundary, and "You retain ultimate ownership and control" becomes the difference between controlling your AI and renting it.

The seventh has a seam in it. "Seamless collaboration with your colleagues" was about merging edits between individuals' devices without a server. Its enterprise analogue, many users sharing a serving node and reusing cached prefixes across a team, is a throughput property rather than a data-ownership one. Better to know that now than halfway through a design review.

Three Ways to Run the Same Workload

What Must Be True Before You Commit

Sustained volume has to be real, not projected. The vendor whose cost model is quoted most often in favor of on-premise inference is candid that at one tenth of the volume it assumes, the arithmetic tilts back toward the API. Model your actual throughput over a quarter, not your best month.

Someone has to own the serving stack. CUDA versions, driver compatibility, quantization pipelines and inference engine upgrades become your operational responsibility, and this is the cost most consistently underestimated. If you do not already have an MLOps practice, this decision creates the need for one.

An air-gapped site needs a self-contained identity, logging and update story. Cloud-based authentication and remote API calls break the moment you cut the external network, so local authentication, immutable audit logs and a validated offline pipeline for weights and patches all have to exist before the first production request.

Portability has to be built in rather than bolted on: classify workloads by sovereignty and risk profile, put the gateway in front of everything, and test each exit path instead of assuming it. And plan the integration surface, because an agent is only as useful as the systems it can reach, and moving inference in-house does nothing on its own to connect it to them.

Should You Rent, or Own Inference?

Teams that get this right treat inference as a portfolio decision, not a platform decision. Different workloads sit at different points on the control-versus-convenience curve, and the architecture that survives a changing regulatory and vendor picture is the one where moving a workload between those points is a routine operation.

That is what the gateway, the OpenAI-compatible interface and a documented exit path all buy you. The hardware question then answers itself, because you can settle it with measurements from your own traffic instead of a forecast. Build the reversibility first, then let the volume tell you where each workload belongs.

Monterail builds agentic systems for teams working under real data-control constraints, from architecture through to production. If you are weighing where your inference should run, our AI engineering team can help you model it against your actual workload.