After updating config.yaml, kubelet service exits and the node is in not ready state

[richamishra006]

Hi All,
i updated the config.yaml to add a parameter for max pods, but then the when checked the logs, getting error as

kubelet service exits with status code 1

Even i reverted my changes, but still getting the same error and the node is in not ready state, since i am doing POC so used a single node for now
Also could you please let me know how to troubleshoot such issues in RKE2 when node is not ready as i can't see any kubelet service, tried restarting containerd service and rke2-server.service was stuck in activating state

[brandond]

Please fill out the issue template, describing what version you're using and on what OS. Including the content of your configuration file would also be helpful.

Also, we don't have any errors about kubelet service exits; are you sure you're using rke2, or are you just paraphrasing the error message? Please post the complete log output that includes the error you're seeing. Paraphrased errors are not particularly useful for troubleshooting.

[richamishra006]

OS- Debian 11
Rancher version- v2.7
i am sure that i am using RKE2

May 10 06:11:05 c9bm0 rke2[3678406]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 10 06:11:05 c9bm0 rke2[3678406]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 10 06:11:05 c9bm0 rke2[3678406]: Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 10 06:11:05 c9bm0 rke2[3678406]: Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 10 06:11:05 c9bm0 rke2[3678406]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 10 06:11:06 c9bm0 rke2[3551639]: time="2023-05-10T06:11:06-07:00" level=error msg="Kubelet exited: exit status 1"

I updated the config.yaml

root@c9bm0:~# cat /etc/rancher/rke2/config.yaml
token: my-shared-secret
tls-san:
  - contextu.al
  - streethawk.com
kubelet-args:
- config: /etc/kubernetes/kubelet.yaml

and kubelet.yaml looks like

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250

i reverted this change then later updated the config.yaml as

token: my-shared-secret
tls-san:
  - contextu.al
  - streethawk.com
 kubelet: 
    extra_args:
      max-pods: 500

i reverted both the chnages and when restarted the rke2-server.service, the service was stuck in activating state and throwing the error i shared above.
The node was in notready state

[brandond]

Neither of those are valid, please check the docs at https://docs.rke2.io/install/configuration#configuration-file.

You probably want something like:

token: my-shared-secret
tls-san:
  - contextu.al
  - streethawk.com
 kubelet-arg:
  - max-pods=500