calls to api for rke2 config map

[xhejtman]

Hello,

grep in kube-api audit log, I can see that there is huge number of calls for rke2 configmap:

    614 /api/v1/events?fieldSelector=involvedObject.kind%3DPod\u0026labelSelector=
    621 /api/v1/namespaces/default
    621 /api/v1/namespaces/default/services/kubernetes
    621 /api/v1/namespaces/kube-system/configmaps/rke2-coredns-rke2-coredns-autoscaler
    891 /apis/batch/v1/namespaces/mff-prod-ns/jobs?labelSelector=app%3Dorca-orca-rdtscp-16853692639824305\u0026limit=500
    951 /api/v1/namespaces/gpu-operator/configmaps/53822513.nvidia.com
   1930 /api/v1/namespaces/kube-system/configmaps/rke2-etcd
   5145 /api/v1/namespaces/cattle-logging-system/configmaps/logging-operator.logging.banzaicloud.io
   6102 /api/v1/namespaces/kube-system/configmaps/cattle-controllers?timeout=15m0s
   6118 /api/v1/namespaces/cattle-fleet-system/configmaps/fleet-agent-lock
   6120 /api/v1/namespaces/kube-system/configmaps/rke2

is this something expected to be happening on 36 node cluster? (rke2 1.24.13)

the output is result from command:

sed -e 's/.*requestURI":"//' -e 's/".*//' audit.log  | sort | uniq -c | sort -n

[brandond]

I don't know what that's coming from, but it's not RKE2. As far as I know we don't create nor do we look for a configmap by that name. You have the audit logs, can you tell what's requesting it, and what type of requests those are (get/watch/etc)?

[brandond]

Oh actually, I should have asked what version of RKE2 you're on. This is probably for the old ConfigMap-based leader election code; newer releases should use leases instead. The leader will renew a lock on that object every 15 seconds, while all the other servers will attempt to take over the lock in case the current leader has failed.

Out of your 36 nodes, how many are servers?

[xhejtman]

1.24.13 and 1.25.9. Perpahs some migration is needed?

[xhejtman]

5 servers + 31 workers. The cluster with 1.25.9 has only 4 servers and nothing else.

[xhejtman]

This is the content of the rke2 config map:

metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: >-
      {"holderIdentity":"kub-a6.priv.cerit-sc.cz","leaseDurationSeconds":45,"acquireTime":"2023-05-16T09:45:15Z","renewTime":"2023-05-30T19:44:14Z","leaderTransitions":73}

and this is the complete audit event:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"e03d60b7-7dd9-492d-9276-48985ee3b3d4","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/configmaps/rke2","verb":"update","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"deploy@kub-a6.priv.cerit-sc.cz/v1.24.13+rke2r1 (linux/amd64) rke2/05a2e96937f8032630ac8bdfa6c79ba6e28f3da3","objectRef":{"resource":"configmaps","namespace":"kube-system","name":"rke2","uid":"256a67f1-90b9-4747-b0c2-36a3a2b554a6","apiVersion":"v1","resourceVersion":"1011209319"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-05-30T17:47:23.787846Z","stageTimestamp":"2023-05-30T17:47:23.791921Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}

[xhejtman]

A cluster with 4 nodes, all rke2 server, version 1.25.9, something keeps etcd pretty busy:

     47 /apis/batch/v1/namespaces/development-aicope/jobs/postgres-db-init
    294 /apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent
    294 /apis/apps/v1/namespaces/cattle-system/deployments/cattle-cluster-agent
    731 /api/v1/services
   1095 /api/v1/namespaces/kube-node-lease
   1095 /api/v1/namespaces/kube-public
   1095 /api/v1/namespaces/kube-system
   2283 /api/v1/namespaces/research-aicope/pods?fieldSelector=\u0026labelSelector=component%3Dsingleuser-server
   4283 /api/v1/namespaces/research-aicope/events?fieldSelector=involvedObject.kind%3DPod\u0026labelSelector=
   6573 /api/v1/namespaces/default/services/kubernetes
   6575 /api/v1/namespaces/default
  20486 /api/v1/namespaces/kube-system/configmaps/rke2
  20513 /api/v1/namespaces/kube-system/configmaps/rke2-etcd

[brandond]

This is all normal given the version you're on. Kubernetes 1.25 uses the ConfigMapsLeasesResourceLock type by default. Newer versions will use pure Leases instead, as the legacy lock types become deprecated. You'll still see the same amount of traffic to the leader election resources, but they'll be Leases instead of ConfigMaps.

• https://github.com/rancher/wrangler/blob/v1.1.1/pkg/leader/leader.go#L34
• https://github.com/kubernetes/kubernetes/blob/release-1.25/staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/interface.go#L203-L207

Is there an actual problem that you're trying to solve?

[xhejtman]

The problem seems to be, that etcd and kube api overloaded so I thought this might be a problem. Just wanted to know whether this is normal rate, and you are saying it is, so I will search for something else. Thank you very much for your support.