How should I choose server alert thresholds and durations?
Choose alert conditions that require a useful response and allow enough time to act. A universal CPU or memory percentage cannot suit every workload. Combine service-impact signals with sustained resource pressure and distinguish a brief burst from a persistent problem.
Before you start
- A representative baseline, service objectives and a named responder for each alert.
Instructions
1. Define the condition and action
Specify the metric, scope, units and affected service. Explain what the responder should investigate or change. Immediate customer-impact failures may justify a fast alert, while capacity trends may be better routed as planned work. Avoid paging solely because a metric looks unusual.
2. Account for time and recovery
Choose a pending duration appropriate to the failure and sampling interval. Prometheus, for example, supports a for duration before an active condition becomes firing. Consider hysteresis or supported recovery timing to avoid repeated flapping, and treat missing measurements separately from healthy values.
3. Test with realistic evidence
Evaluate the rule against known incidents and normal peaks, then use an approved test to confirm firing, routing and resolution. Check whether provider-managed notifications support the desired metric and duration. Review noisy rules with the service owner rather than silently extending their delay until real outages disappear.
Check the result
- A known actionable condition fires within the response window and normal peaks do not create excessive noise.
- Missing data and resolved conditions produce the intended distinct behaviour.
Troubleshooting
- The alert never fires although the graph crosses the threshold
- Check aggregation, labels, sampling and pending duration. A condition that repeatedly clears before the duration expires may remain pending by design.
Official references
Use the documentation for your installed version when a flag, service name or platform feature differs.