Every few months a new model drops and someone on the team asks the same question: should we just call an API, or should we run this ourselves? I've been on both sides of that decision more times than I can count, and the honest answer is that it's rarely about which model is "smarter." It's about what kind of infrastructure, risk, and operational burden you're willing to take on. This article walks through what closed-source and open-source AI models actually mean at the infrastructure level, how each one behaves once it's in production, and where teams tend to get the tradeoffs wrong.
What It Is
Closed-source AI models are models whose weights, training data, and architecture details are not published. You interact with them through an API endpoint owned and operated by the vendor. Think of providers exposing models over HTTPS where you send a prompt, get a completion back, and pay per token. You never touch the weights, you never see the training pipeline, and you have no ability to run the model outside their infrastructure.
Open-source AI models are the opposite: the weights are downloadable, often the training methodology is documented to some degree, and you can run inference wherever you have the compute to support it. Some open-source models come with permissive licenses (Apache 2.0, MIT), others come with usage restrictions bundled into an otherwise open weight release. That distinction matters more than people think — "open weights" and "open source" are not the same thing, and I've seen procurement teams get burned assuming a downloadable model was free to use commercially when the license said otherwise.
From an infrastructure standpoint, the real dividing line isn't code visibility. It's who controls the compute. Closed-source means someone else's GPUs, someone else's scaling decisions, someone else's uptime SLA. Open-source means you own that problem, for better or worse.
How It Works
With a closed-source model, your integration surface is an API call. Something like:
POST https://api.solvethenetwork.com/v1/chat/completions
Authorization: Bearer sk-prod-xxxxxxxxxxxx
Content-Type: application/json
{
"model": "vendor-model-large",
"messages": [{"role": "user", "content": "Summarize this incident report."}],
"max_tokens": 512
}Your job as the infra engineer is to manage rate limits, retries, timeouts, and cost tracking. You're building around a black box. You don't control model versioning beyond pinning a model ID, and vendors periodically deprecate or silently update models behind the same endpoint name, which has broken more than one of my pipelines without warning.
With an open-source model, the workflow looks completely different. You pull weights, usually several gigabytes to hundreds of gigabytes depending on parameter count, and you serve them yourself using something like vLLM, TGI, or llama.cpp on a GPU host you provision and patch.
$ ssh infrarunbook-admin@sw-infrarunbook-01
$ docker run --gpus all -p 8000:8000 \
-v /data/models/llama-3-70b:/model \
vllm/vllm-openai:latest \
--model /model --tensor-parallel-size 4
INFO: Uvicorn running on http://10.20.4.15:8000Now you own the whole stack: GPU provisioning, driver versions, CUDA compatibility, model sharding across cards, load balancing across replicas, and monitoring for OOM kills when someone sends a prompt that's too long for your context window budget. It's a completely different job than calling an API, and it requires infrastructure skills that a lot of ML teams don't have in-house.
Why It Matters
The tradeoffs show up in four places I care about most: cost, latency, data control, and operational overhead.
Cost. Closed-source pricing is usage-based and predictable in the small, unpredictable at scale. A team doing light experimentation pays almost nothing. A team pushing millions of tokens a day through a hosted API can end up with a bill that rivals a mid-size GPU cluster's amortized cost. I've watched a batch summarization job that seemed cheap in a proof of concept turn into a five-figure monthly line item once it hit production volume. Open-source flips this: high upfront capital or reserved-instance cost, but marginal cost per request drops sharply once you're past the break-even point. Where that break-even point sits depends entirely on your request volume and how efficiently you can pack GPU utilization, which is its own ongoing effort.
Latency and availability. A hosted API means you're at the mercy of someone else's queueing and regional routing. During traffic spikes I've seen response times triple with zero visibility into why, and no lever to pull besides opening a support ticket. Self-hosted inference puts latency in your hands, but only if you've actually engineered for it — batching requests, tuning KV cache size, and picking the right tensor parallelism for your GPU topology. Get that wrong and your "faster because it's local" model ends up slower than the vendor API you were trying to replace.
Data control. This is the one that gets compliance and security teams involved. With closed-source APIs, your prompts leave your network boundary. Even with a no-training-on-data agreement from the vendor, you're trusting a third party's security posture and their contractual promises. For regulated data — health records, financial details, anything under a strict data residency requirement — that's often a nonstarter. Self-hosted open-source models keep everything inside your VPC. You can run inference entirely on 10.20.0.0/16 with no external egress at all, which is the deciding factor for a lot of the compliance-driven projects I've worked on.
Operational overhead. This is the tradeoff people underestimate the most. Running your own inference stack means you now own GPU driver upgrades, CUDA version drift, model version rollouts, autoscaling policy, and incident response when a node OOMs at 2 a.m. A closed-source API means someone else owns all of that, and your on-call rotation gets to sleep through model infrastructure incidents that aren't yours to fix.
Real-World Examples
A fintech client I worked with needed document classification on internal contracts. Data residency rules ruled out any external API outright, so we deployed an open-source model on-prem, serving from a small GPU pool behind an internal load balancer:
Client Request
|
v
10.20.4.10 (nginx, internal LB)
|
+--> 10.20.4.15 sw-infrarunbook-01 (vLLM, GPU 0-3)
+--> 10.20.4.16 sw-infrarunbook-02 (vLLM, GPU 0-3)
No external DNS resolution. No egress rule beyond package mirrors.Total onboarding took roughly six weeks, mostly spent tuning batch sizes and getting the security team comfortable with the patching cadence for the inference containers. But once it was running, marginal cost per document was close to zero, and there was no data ever leaving the network.
Contrast that with a customer support summarization tool I built for a much smaller team with no in-house ML infra staff. There, a closed-source API was the obvious call. We had a working integration in an afternoon, no GPU procurement, no driver management, and the cost at their volume (a few thousand requests a day) stayed under what a single reserved GPU instance would have cost even at zero utilization. Trying to self-host there would have been infrastructure for infrastructure's sake — nobody on the team wanted to own a GPU fleet for a workload that small.
The pattern I keep seeing: data sensitivity and volume are the two variables that actually decide this, not model quality. Both paths can get you a good model. Only one of them fits your compliance and cost constraints.
Common Misconceptions
The first misconception is that open-source always means cheaper. It's cheaper per-token at scale, sometimes dramatically so, but the fixed cost of standing up and maintaining GPU infrastructure is real and often underestimated. I've seen teams justify a self-hosted deployment on cost grounds, then discover that the engineering hours spent keeping the inference stack healthy erased most of the savings.
The second misconception is that closed-source APIs are inherently more secure because a big vendor runs them. Security isn't about who's bigger, it's about where your data goes and what contractual and technical guarantees exist around it. A well-configured self-hosted deployment inside your own network boundary can have a smaller attack surface than sending data to an external endpoint, even a reputable one.
The third misconception, and one I run into constantly, is that you have to pick one path for the whole organization. In practice, most infrastructure teams end up running both: a closed-source API for fast-moving, low-sensitivity workloads, and self-hosted open-source models for anything regulated or high-volume enough to justify the operational investment. Treating this as a single company-wide decision instead of a per-workload one is how teams end up either overpaying for API calls they didn't need to make, or building GPU infrastructure for a workload that never needed it.
Finally, people assume open-source models are a fire-and-forget deployment once the weights are downloaded. They're not. Model files get updated, security patches land in serving frameworks, GPU drivers need version bumps that can break tensor parallelism configs you tuned months ago. Self-hosting a model is not a one-time infrastructure project, it's an ongoing service you now operate, with the same lifecycle discipline you'd apply to any other production system.
When I get asked which one to use, my honest answer is: figure out your data sensitivity requirements first, then your request volume, then your team's appetite for owning GPU infrastructure. The model choice usually falls out of those three answers on its own.