Metrics Server: RKE2

[pirouet]

Hi,

I'm probably doing something daft however each time I run kubectl I see this output before the command output

E0422 19:29:49.069552 2248000 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0422 19:29:49.075344 2248000 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0422 19:29:49.076725 2248000 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0422 19:29:49.077887 2248000 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

This did not appear after initial install but I am wondering if I am missing any command line arguments? Do I need to configure metrics server at all?

Only noticed this since v.1.26. For reference, this is the current cluster stat:

Client Version: v1.26.4+rke2r1
Kustomize Version: v4.5.7
Server Version: v1.26.4+rke2r1

Any help would be appreciated

[pirouet]

Digging bit deeper, metrics-server pod is unable to connect to the other servers in the cluster

E0422 17:48:15.734511 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.105.54.159:10250/metrics/resource\": dial tcp 10.105.54.159:10250: connect: connection refused" node="xxx"

No firewalls exist between the nodes at the moment (all ports open)

[brandond]

Can you curl that URL from the node that the metrics-server pod is running on? Something is blocking access to that port on the node; a firewall of some sort (either external, or local iptables rules) is the most likely cause.

[pirouet]

Apologies!

Running a simple curl -k https://10.105.54.159:10250/metrics/resource from the same returns Unauthorised response

[pirouet]

Further bit of digging I can see from kubectl get apiservices the following line:

v1beta1.metrics.k8s.io kube-system/rke2-metrics-server False (FailedDiscoveryCheck)

kubectl describe svc rke2-metrics-server

Name: rke2-metrics-server
Namespace: kube-system
Labels: app=rke2-metrics-server
app.kubernetes.io/managed-by=Helm
chart=rke2-metrics-server-2.11.100-build2022101107
heritage=Helm
release=rke2-metrics-server
Annotations: meta.helm.sh/release-name: rke2-metrics-server
meta.helm.sh/release-namespace: kube-system
Selector: app=rke2-metrics-server,release=rke2-metrics-server
Type: ClusterIP