Node Disk IO Saturation

[frit0-rb]

Hello everybody,

I have 3 clusters deployed in virtual machines hosted in Hyper-V clusters.

During last weeks I have been received alerts from Prometheus in a puntual hour, 00:00 and 12:00 about NodeDiskIOSaturation

Labels alertname = NodeDiskIOSaturation container = node-exporter device = dm-2 endpoint = http-metrics instance = 9100 job = node-exporter namespace = prometheus pod = prometheus-prometheus-node-exporter-pkrmv prometheus = prometheus/prometheus-kube-prometheus-prometheus service = prometheus-prometheus-node-exporter severity = warning Annotations description = Disk IO queue (aqu-sq) is high on dm-2 at :9100, has been above 10 for the last 30 minutes, is currently at 11.80. This symptom might indicate disk saturation. runbook_url = https://runbooks.prometheus-operator.dev/runbooks/node/nodediskiosaturation summary = Disk IO queue is high.

Sometimes this saturation goes up and up and some deploys stop working and collapse. This is a big problem because I don't understand what is happening.

I check if we got some backups or jobs during that time but nothing there. Only I see ETCD backups run in that hours.

Are thee any way to prevent that problem or something I can do ?

Thanks in adavance