A quiet shift has taken place in how companies pay for artificial intelligence. For several years the default assumption was that frontier capability would remain behind an API, priced per token and controlled by a handful of vendors. Open-weight models have undermined that assumption by making capable systems downloadable, modifiable and, crucially, hostable on hardware the buyer already owns.
The change is less about ideology than about unit economics. When a model can be run on a dedicated server, the marginal cost of a request becomes a function of electricity, utilisation and staff time rather than a vendor’s price list. For high-volume, predictable workloads that arithmetic frequently favours self-hosting, even after accounting for the engineering effort involved.
Where self-hosting wins and where it does not
Self-hosting tends to make sense when traffic is steady, the task is narrow, and latency requirements are strict. Customer support classification, document extraction and internal search are common examples. It makes less sense when demand spikes unpredictably, when the task changes often, or when the very best available model is required for a small number of difficult requests.
Many teams therefore end up with a mixed estate. They route the bulk of simple requests to a model they host and send the remainder to a hosted API. The engineering work moves from prompt writing to routing, monitoring and evaluation, which is a different skill set from the one that dominated the first wave of adoption.
The licence question is not settled
Open weights do not always mean open use. Licences vary widely, and some restrict commercial deployment above a revenue threshold or require attribution and naming conditions. Legal teams have had to read them carefully, because a licence that permits research use may not permit embedding a model in a product sold to customers.
This matters for smaller companies that lack in-house counsel. A model that is free to download can still carry obligations that are expensive to satisfy. Groups such as the Linux Foundation and various open source offices have published guidance, but the practical answer usually depends on the specific terms attached to the specific checkpoint.
Hardware access becomes the real constraint
The ability to download a model does not guarantee the ability to serve it. Memory capacity determines how large a model can fit on a given accelerator, and high bandwidth memory remains expensive and in short supply. Quantisation helps, but each reduction in precision carries a risk of quality loss that must be measured rather than assumed.
Some organisations solve this by pooling capacity or renting it by the hour, which reintroduces a dependency but at a different layer. Others invest in smaller models that fit comfortably on commodity hardware. The MLCommons consortium has been building benchmarks that reflect these practical constraints rather than idealised conditions.
Evaluation becomes the control point
When a company hosts its own model, nobody else is responsible for quality. That pushes evaluation from a nice-to-have into a production requirement. Teams need regression tests, drift detection and a way to compare a new checkpoint against the one currently serving traffic.
The tools for this are improving but remain fragmented. Many organisations assemble their own harness from open components, which is workable but adds maintenance burden. The gap between teams that measure carefully and teams that do not shows up quickly in user complaints.
What vendors are doing in response
Commercial providers have not stood still. Several now offer smaller, cheaper tiers, per-hour reserved capacity and fine-tuning services. The competitive pressure from open weights is visible in pricing pages even where the underlying strategy has not changed.
The result is a market with more options and more complexity. Procurement decisions that once involved a single vendor now involve a portfolio, and the ability to switch between options has become a form of bargaining power.
Latency, reliability and the operational burden
Hosting a model means owning its uptime. A hosted API fails and the vendor’s status page carries the blame. A self-hosted endpoint fails and the team is paged. That difference is easy to underestimate during a pilot and difficult to ignore at scale.
Latency also changes. A local deployment can be faster when the hardware is close to the user and slower when it is not, and the queueing behaviour of a busy accelerator differs from that of a managed service. Both must be measured under realistic load rather than assumed from a single request.
Practical guidance for teams starting now
Start by measuring the workload. If a small model reaches acceptable quality on the actual task, the cost difference is usually decisive. Keep the ability to move between hosted and self-hosted inference by separating the interface from the implementation. Track total cost including staff time, not just compute.
Finally, read the licence before the model goes anywhere near a product. The savings from an open-weight deployment can be erased by a compliance problem discovered late, and the terms are rarely as simple as the download page suggests.
Image: Derrick Coetzee from Berkeley, CA, USA · CC0 · via Wikimedia Commons.