oss-model-playbook
A deployment playbook for teams that have decided to run open-weight models themselves and now have to answer the questions that follow: which model, on what hardware, how many of them, and what does it cost when it’s actually serving traffic.
What it covers
- Model picker — an interactive way to narrow the open-weight field against real constraints: context length, latency budget, memory footprint, licence.
- Capacity planning — GPU sizing, KV-cache headroom, batching behaviour, and where the concurrency ceiling actually sits.
- Deployment targets — on-prem Kubernetes, AKS, and Azure AI Foundry, with the trade-offs between them stated rather than implied.
- Gateway and security — auth, quota, routing, and the boundary an enterprise needs in front of a self-hosted model.
- TCO — the total cost picture, including the parts that don’t appear on the GPU invoice.
Why it exists
Open-weight serving guidance tends to stop at “here is how to start vLLM.” The hard part comes after: sizing a cluster for a latency SLO, deciding what happens when the KV cache fills, and justifying the bill against a hosted API. This collects the decisions that come up in every one of those conversations, in the order they come up.