10.3.1 Notes on Stopping and Restarting
- Organization of this subsection
(1) Notes on Stopping and Restarting Prometheus server
If Prometheus server is restarted or stopped while Alertmanager is running with the alert firing, a normal JP1 event may be issued to indicate the alert is recovered.
A normal JP1 event is issued when the following conditions are satisfied:
-
When the time# set for for in the definition of a firing alert plus the amount of time that the Prometheus server service is not running due to hesitation or restart is greater than the value of the evaluation_interval in the Prometheus configuration file.
#: If the alart for is not set, 0 is assumed.
(2) Notes on restarting Exporter
-
When the monitoring target does not exist during operation
If there are no Exporter targets while Exporter is running or stopped, metric for that target may not be generated after restarting Exporter.
For example, if Process A is being monitored by Process exporter, while Process A is running, Process exporter generates metric, but if Process A is stopped and does not exist, Process exporter continues to generate metric for Process A until Process exporter is restarted.
However, if you restart Process exporter, metric will not be generated for Process A that is not running at that time.
When Process A is started, metric for Process A is generated again.
If you view time-series data in integrated operation viewer, the data for periods when no metrics have been generated is not displayed.
If an alert has been set, the alert that was in the firing state may be recovered.
-
If scrape occurs during reboot
During Exporter restart process, if Prometheus Server performs an scrape, metric for that Exporter is not generated until the boot is complete and scrape is enabled.
If you view time-series data in integrated operation viewer, you will not see the duration of scrape failure.
Also, if you have set an alert, the state might change temporarily depending on the alert conditions.