rke2-metrics-server throws the following error. Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1" #6358

cengizhanR · 2024-07-17T10:45:19Z

cengizhanR
Jul 17, 2024

rke2-metrics-server throws the following errors. These error about certificates?

I0717 11:21:08.222855       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-07-17T14:21:08.339001974+03:00 E0717 11:21:08.338109       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.6:10250/metrics/resource\": dial tcp 10.0.0.6:10250: connect: connection refused" node="rke2agent2"
2024-07-17T14:21:08.339020974+03:00 E0717 11:21:08.338109       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1"
2024-07-17T14:21:08.371552484+03:00 I0717 11:21:08.365979       1 secure_serving.go:213] Serving securely on [::]:10250
2024-07-17T14:21:08.371578985+03:00 I0717 11:21:08.366053       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0717 11:21:08.366062       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
2024-07-17T14:21:08.371585485+03:00 I0717 11:21:08.366114       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
2024-07-17T14:21:08.371587985+03:00 I0717 11:21:08.366216       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0717 11:21:08.366342       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
2024-07-17T14:21:08.371593385+03:00 I0717 11:21:08.366351       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2024-07-17T14:21:08.371597185+03:00 I0717 11:21:08.366370       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0717 11:21:08.366376       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E0717 11:21:08.370139       1 scraper.go:149] "Failed to scrape node" err="request failed, status: \"401 Unauthorized\"" node="rke23"
2024-07-17T14:21:08.486479086+03:00 I0717 11:21:08.466957       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0717 11:21:08.466986       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2024-07-17T14:21:08.486508086+03:00 I0717 11:21:08.466967       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
2024-07-17T14:21:23.331730405+03:00 E0717 11:21:23.331629       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1"
E0717 11:21:23.343822       1 scraper.go:149] "Failed to scrape node" err="request failed, status: \"401 Unauthorized\"" node="rke23"
2024-07-17T14:21:23.371289225+03:00 E0717 11:21:23.371021       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.6:10250/metrics/resource\": dial tcp 10.0.0.6:10250: connect: connection refused" node="rke2agent2"
2024-07-17T14:21:38.338668359+03:00 E0717 11:21:38.338499       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1"
2024-07-17T14:21:38.353056984+03:00 E0717 11:21:38.352970       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.6:10250/metrics/resource\": dial tcp 10.0.0.6:10250: connect: connection refused" node="rke2agent2"
2024-07-17T14:21:53.331808097+03:00 E0717 11:21:53.331676       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1"
E0717 11:21:53.335267       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.0.0.6:10250/metrics/resource\": dial tcp 10.0.0.6:10250: connect: connection refused" node="rke2agent2"

My host File is following.

[root@rke21 kube-state-metrics]# cat /etc/hosts
127.0.0.1               localhost.localdomain    localhost
::1                     localhost6.localdomain6    localhost6
# cluster nodes.
10.0.0.2          rke21.rancher.exemple.com    rke21
10.0.0.3          rke22.rancher.exemple.com    rke22
10.0.0.4          rke23.rancher.exemple.com    rke23
10.0.0.5          rke2agent1.rancher.exemple.com    rke2agent1
10.0.0.6          rke2agent2.rancher.exemple.com    rke2agent2
10.0.0.14         nfs.exemple.com                   nfs

This is Server config.yaml

[root@rke21 bilge]# cat /etc/rancher/rke2/config.yaml
system-default-registry: 10.0.0.10:5000
secrets-encryption: true
token: fuzzybunnyslippers
tls-san:
  - rancher.exemple.com
node-ip: "10.0.0.2"
etcd-expose-metrics: true
write-kubeconfig-mode: "0600"
use-service-account-credentials: true
kube-controller-manager-arg:
- bind-address=127.0.0.1
- use-service-account-credentials=true
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-scheduler-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-proxy-arg:
- metrics-bind-address=0.0.0.0:10249
kube-apiserver-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- authorization-mode=RBAC,Node
- anonymous-auth=false
- audit-policy-file=/etc/rancher/rke2/audit-policy.yaml
- audit-log-mode=blocking-strict
- audit-log-maxage=30
- audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- streaming-connection-idle-timeout=5m
- minimum-container-ttl-duration=10s
- maximum-dead-containers-per-container=2
- maximum-dead-containers=240
- image-gc-high-threshold=85
- image-gc-low-threshold=80

cengizhanR · 2024-07-18T05:33:29Z

cengizhanR
Jul 18, 2024
Author

I edit rke2-metric-server deploymnet yaml and add following flag.

Try adding --kubelet-insecure-tls

kubectl edit deploy metrics-server -n kube-system

      containers:
      - args:
        - --kubelet-insecure-tls

0 replies

brandond · 2024-07-22T21:04:47Z

brandond
Jul 22, 2024
Maintainer

Make sure you've opened the required ports between all of your nodes.
Ref: https://docs.rke2.io/install/requirements#inbound-network-rules

TCP 10250 All RKE2 nodes All RKE2 nodes kubelet metrics

3 replies

cengizhanR Jul 22, 2024
Author

Thank you for your return. I dont have firewalls or something blocking my ports. I have configured kube-api server and kubelet with TLS. When i add this config. Everything is okey.

args:
- --kubelet-insecure-tls

cengizhanR Jul 22, 2024
Author

@brandond this is my server config.

``system-default-registry: 10.0.0.10:5000
secrets-encryption: true
token: fuzzybunnyslippers
tls-san:
  - rancher.exemple.com
node-ip: "10.0.0.2"
etcd-expose-metrics: true
write-kubeconfig-mode: "0600"
use-service-account-credentials: true
kube-controller-manager-arg:
- bind-address=127.0.0.1
- use-service-account-credentials=true
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-scheduler-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-apiserver-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- authorization-mode=RBAC,Node
- anonymous-auth=false
- audit-policy-file=/etc/rancher/rke2/audit-policy.yaml
- audit-log-mode=blocking-strict
- audit-log-maxage=30
- audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- streaming-connection-idle-timeout=5m
- minimum-container-ttl-duration=10s
- maximum-dead-containers-per-container=2
- maximum-dead-containers=240
- image-gc-high-threshold=85
- image-gc-low-threshold=80

brandond Jul 22, 2024
Maintainer

The error you were getting was about the connection being refused, not about the certificate not being trusted. Unless there was some other error about self-signed certs that you didn't share, changing the kubelet-insecure-tls option shouldn't have any effect...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rke2-metrics-server throws the following error. Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1" #6358

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 3 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

rke2-metrics-server throws the following error. Failed to scrape node" err="Get \"https://10.0.0.5:10250/metrics/resource\": dial tcp 10.0.0.5:10250: connect: connection refused" node="rke2agent1" #6358

cengizhanR Jul 17, 2024

Replies: 2 comments · 3 replies

cengizhanR Jul 18, 2024 Author

brandond Jul 22, 2024 Maintainer

cengizhanR Jul 22, 2024 Author

cengizhanR Jul 22, 2024 Author

brandond Jul 22, 2024 Maintainer

cengizhanR
Jul 17, 2024

Replies: 2 comments 3 replies

cengizhanR
Jul 18, 2024
Author

brandond
Jul 22, 2024
Maintainer

cengizhanR Jul 22, 2024
Author

cengizhanR Jul 22, 2024
Author

brandond Jul 22, 2024
Maintainer