How do I check GPU memory and utilisation on a dedicated server?
GPU memory occupancy and GPU utilisation measure different things. An application can reserve substantial device memory while doing little compute, and a brief sample may miss a burst. Compare the readings with application throughput and latency over the same interval.
Before you start
- The GPU identity from the device inventory and permission to inspect its telemetry.
Instructions
1. Read a focused snapshot
Use the identified device; index 0 is only an example. Review total, used and free framebuffer memory separately from the utilisation section. Unsupported telemetry can appear as N/A, particularly for some device modes or partitioned configurations.
nvidia-smi -i 0 -q -d MEMORY,UTILIZATIONQueries memory and utilisation information for one selected GPU without changing its configuration.
2. Compare with the workload
Record the time, request load and process identity while the application is active. Distinguish allocator-reserved memory from the model or dataset’s nominal size. Memory accounting and available fields can vary across GPU architecture, driver and virtualisation mode.
3. Investigate the limiting resource
If utilisation is low during a slow job, inspect CPU preparation, storage reads, network transfers and batching. If allocation fails, inspect other users of the selected GPU and the application’s concurrency or cache requirements. Treat these as evidence for a scoped workload change, not a reason to terminate every GPU process.
Check the result
- The measured GPU and time interval match the affected application.
- Memory pressure and utilisation are interpreted alongside the application’s actual performance.
Troubleshooting
- Memory is nearly full but utilisation is low
- Check whether the framework retains allocations or caches between requests. Reserved memory can remain occupied even when no kernel is executing at the sampling moment.
Official references
Use the documentation for your installed version when a flag, service name or platform feature differs.