How do I investigate GPU ECC errors or Xid events on a dedicated server?
ECC counters and NVIDIA Xid events provide diagnostic evidence with device-specific meanings. An Xid can reflect an application, driver, interconnect or hardware problem; the number and surrounding context matter. Preserve the observed state before considering a reset or hardware replacement.
Before you start
- The affected GPU identity, incident time and authorised access to relevant logs.
Instructions
1. Read the device error state
Select the affected device and inspect supported counters. Volatile and aggregate counters can cover different periods; N/A does not mean zero errors. Record the driver version and whether the server has restarted since the event.
nvidia-smi -i 0 -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPERQueries ECC and applicable memory-retirement/remapping state. Support varies with GPU and driver.
2. Find the bounded kernel events
Adjust the time interval to the incident. Record the complete relevant Xid line, timestamp and GPU bus identifier, then consult NVIDIA’s catalogue for that event rather than treating every code alike.
sudo journalctl -k --since '1 hour ago' --no-pager | grep -F 'NVRM: Xid'Reads recent kernel journal messages matching NVIDIA Xid events. No matching line means none was found in this selected available interval.
3. Coordinate further diagnostics
Provide selected redacted evidence to the hardware or GPU support owner. Active GPU diagnostics can consume resources or require stopping jobs, so schedule them separately. Do not clear counters or reset the device simply to make the displayed error disappear.
Check the result
- The evidence identifies the device, event code and time period needed for vendor-specific diagnosis.
Troubleshooting
- No Xid appears but the application reports a GPU error
- Check application logs, earlier boots and journal retention. Not every runtime error produces an Xid in the available kernel-log interval.
Official references
Use the documentation for your installed version when a flag, service name or platform feature differs.