Disk IO queue (aqu-sq) is high. Disk saturation

[frit0-rb]

Environmental Info:
RKE2 Version: 1.27.15

Node(s) CPU architecture, OS, and Version: LinuxRHEL 4.18.0-553.22.1.el8_10.x86_64 #1 SMP Wed Sep 11 18:02:00 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 3 Controllers and 5 workers

Describe the bug: During last month we see several everyday Disk I/O saturation inside of workers during some jobs backup, jobs from longhorn or runners from gitlab.

this ever happend in partitions dm- / sda

When this saturation is higher and higher, many of my pods dead and if is the case that pod got a Volume, the volume regards attached from that pod and that pod never restart. I need to kill but I need to restart the service from rke2-agent from that worker.

it is so frustating this ever happend everyday and I dont know what to do

Steps To Reproduce:

• Installed RKE2: 1.27.15 via Rancher
• Create a cluster with 7 nodes (3 controllers and 5 workers)
• Install apps like gitlab, mongo, minio, etc, create volumes for all that workloads.
• Run pipelines to gitlab or launch a snapshot job from Longhorn

Expected behavior: Low disk I/O saturation and NO worker stop working

Actual behavior: Very high saturation and workers stop working.

Additional context / logs:

[brandond]

Your disk performance is not sufficient to meet the needs of the applications you are deploying. I'm not sure how this is an rke2 issue?

[frit0-rb]

That's not sense.

We have for any worker 120 GB RAM, 24 vCPUS.

The infra is working in 5 HP servers Proliant DL385 Gen10 Plus and storage DELL/EMC Unity 480 40TB (SSD disks and Flash Disks)

My virtual machines are RHEL 8.10 behind Hyper-v running in those servers.

But the main point is we were working with this cluster during last 2 years and its only during last month when I see that's problems.