Conections hangs

[zeitler]

Hi all

Two different clusters having the same issue.
Both in debian 11 with kernel 5.10

Applications started to hang. After refresh the page sometimes it was ok
I've found some info about a bug on debian, but fixed after kernel 5.7

I wasn't aware that the issue was only on calico cni, or the fix only works on calico cni.
So, in one of the clusters, I've changed config file to start using cni. Same result

Changed mtu in to 1450, same result.

Found this
and used ethtool -K flannel.1 tx-checksum-ip-generic off, same thing.

I've seen in tcpdump some checksum errors, that's why I've tried this fixes. But normally people say that they have some issues with coredns.
I don't have that.

And I've installed a daemonset in the cluster, the one with canal cni. And when I curl the the endpoint from the same node there is no problems. From another cluster, it hangs but after a lot of curls.
If I curl to the service it self, never hangs also.

This tests are for's curling the endpoint, several at a time.

I don't find a padron, and I'm out of ideas how to solve this.

Someone has some pointers?
Thank you

[manuelbuil]

How are you installing rke2? If it is by config file, could you please share it?

[zeitler]

Hi

Thank you for your reply

I'm installing curling install.sh and, defining:

export INSTALL_RKE2_VERSION=v1.24.10+rke2r1
export INSTALL_RKE2_TYPE=server
sh install.sh

The config in one of the clusters:

tls-san:
  - <frontend IP>
  - <backend IP>
  - <Hostname>
node-label:
  - "node=master"
write-kubeconfig-mode: "0644"
debug: true
disable:
  - rke2-ingress-nginx

The other cluster, the same but with
cni: calico

Installed ingress-nginx with:

helm upgrade --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--version 4.5.2 \
--set controller.kind=DaemonSet --set controller.service.type=NodePort --set controller.hostPort.enabled=true --set controller.hostPort.ports.http=80 --set controller.hostPort.ports.https=443 \

I have the exact same configuration in another three clusters, but the OS is RHEL 8.6, with kernel 4.18.0 and it works very nicely

[manuelbuil]

Can you clarify: when I curl the the endpoint from the same node there is no problems. From another cluster, it hangs but after a lot of curls.

Does that mean two pods in the same cluster can communicate but two pods in different clusters can't communicate?

[manuelbuil]

can you ping between nodes and check if you see that 1.5 seconds?

[zeitler]

From 0.215 to 0.350 ms.

[zeitler]

all nodes to the others

[manuelbuil]

Performance issues is always a bit complicated to debug. If I were you, I'd try to understand the path that the packet is following. Perhaps it is not using a direct route but being moved around without any purpose. Perhaps it is being natted multiple times accidentally

[zeitler]

Hi

The issue is due vxlan checksum bug
The solution in projectcalico/calico#3145 fixed it

I had tried it before. But in the wrong interface.

Thanks for your help