How do I size a dedicated server for AI inference?
An inference model fitting into GPU memory is only the starting point. Runtime workspaces, caches and concurrent requests also consume resources, and acceptable throughput depends on latency targets. Size and test the complete serving stack with the intended model and request distribution.
Before you start
- The exact model, precision or quantisation, framework version and expected request characteristics.
Instructions
1. Estimate the working memory
Account for model weights, runtime overhead and the cache used for active sequences where relevant. Longer prompts, generated outputs and concurrent requests can increase memory demand. Check the framework’s supported quantisation and device architecture; smaller weight files do not guarantee equal quality or compatibility.
2. Define user-visible performance
Measure time to first output, subsequent generation speed, total response time and error rate under representative concurrency. Include long prompts, short prompts and the intended output lengths. A high aggregate tokens-per-second figure can coexist with poor latency for an individual request.
3. Validate the complete server
Check CPU preprocessing, system RAM, model-loading storage and network delivery as well as GPU utilisation. If the model is split across devices, include interconnect and synchronisation overhead. Use supported batching and admission limits to keep memory pressure controlled, then rehearse startup and recovery before exposing the endpoint.
Check the result
- Representative requests meet the defined latency and throughput targets within resource limits.
- The serving stack handles peak concurrency without repeated allocation failures or unbounded queues.
Troubleshooting
- The model loads successfully but requests fail under load
- Inspect cache growth, sequence length, concurrent batch limits and competing GPU processes. Successful loading does not prove sufficient memory for the maximum serving workload.
Official references
Use the documentation for your installed version when a flag, service name or platform feature differs.