[BUG] dial tcp 127.0.0.1:10261: connect: connection refused

[TaibiaoGuo]

What happened:

• dial tcp 127.0.0.1:10261: connect: connection refused
• The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

• yurtadm v1.5.0 release
• containerd v1.6.4 release
• kubernetes v1.23.10
• openyurt cloud installed
• complie openyurt v1.5.0 release
• join edge node with offical command

Anything else we need to know?:

# kubectl get pod -A
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-74dbdc644f-rp754    1/1     Running   0          10h
kube-system   calico-node-tlsx7                           1/1     Running   0          10h
kube-system   coredns-b7c47bcdc-ckvmr                     1/1     Running   0          10h
kube-system   coredns-b7c47bcdc-pnzkf                     1/1     Running   0          10h
kube-system   kube-apiserver-k8ace-gateway-sgp            1/1     Running   0          10h
kube-system   kube-controller-manager-k8ace-gateway-sgp   1/1     Running   0          10h
kube-system   kube-proxy-zktm6                            1/1     Running   0          10h
kube-system   kube-scheduler-k8ace-gateway-sgp            1/1     Running   0          10h
kube-system   nodelocaldns-cpqn7                          1/1     Running   0          10h
kube-system   raven-agent-ds-gqdp7                        1/1     Running   0          9h
kube-system   yurt-manager-58d55fb8b5-vrclk               1/1     Running   0          10h

Environment:

• OpenYurt version: v1.5.0
• Kubernetes version (use kubectl version): v1.23.10
• OS (e.g: cat /etc/os-release): Ubuntu 22.04.4 LTS
• Kernel (e.g. uname -a): Linux k8gateway-hutb 5.15.0-91-generic # 101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubekey
• Others:

others

root@*****:/home/x/openyurt# _output/local/bin/linux/amd64/yurtadm  join *****:6443 --token=***** --node-type=edge --discovery-token-unsafe-skip-ca-verification --cri-socket=/run/containerd/containerd.sock --v=5
I0818 04:28:23.263614   20546 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "*****:6443"
I0818 04:28:23.263650   20546 token.go:230] [discovery] kube client is nil, use bootstrap config(&{  {false map[]} map[kubernetes:0xc000040980] map[] map[token-bootstrap-client@kubernetes:0xc0002deff0] token-bootstrap-client@kubernetes map[]}) to create client
I0818 04:28:23.473401   20546 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "*****:6443"
I0818 04:28:23.667918   20546 kubernetes.go:369] kubernetes version: v1.23.10
I0818 04:28:23.667933   20546 join.go:396] node join data info: join.joinData{cfgPath:"", joinNodeData:(*joindata.NodeRegistration)(0xc0002defa0), apiServerEndpoint:"*****:6443", token:"*****", tlsBootstrapCfg:(*api.Config)(0xc0001c72c0), clientSet:(*kubernetes.Clientset)(0xc0001b4d00), ignorePreflightErrors:sets.Set[string]{"DirAvailable--etc-kubernetes-manifests":sets.Empty{}, "FileAvailable--etc-kubernetes-kubelet.conf":sets.Empty{}}, organizations:"", pauseImage:"registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2", yurthubImage:"registry.cn-hangzhou.aliyuncs.com/openyurt/yurthub:latest", yurthubTemplate:"", yurthubManifest:"", kubernetesVersion:"v1.23.10", caCertHashes:[]string{}, nodeLabels:map[string]string{}, kubernetesResourceServer:"dl.k8s.io", yurthubServer:"127.0.0.1", reuseCNIBin:false, namespace:"kube-system", staticPodTemplateList:[]string(nil), staticPodManifestList:[]string(nil)}
I0818 04:28:23.722511   20546 system.go:43] Setting ipv4 forward
I0818 04:28:23.722545   20546 system.go:52] Setting bridge settings for kubernetes.
I0818 04:28:23.722632   20546 system.go:74] Disabling SELinux.
I0818 04:28:23.722637   20546 kubernetes.go:117] Check and install kubelet v1.23.10
I0818 04:28:23.757815   20546 kubernetes.go:126] kubelet --version: v1.23.10
I0818 04:28:23.757845   20546 kubernetes.go:138] Kubelet v1.23.10 already exist, skip install.
I0818 04:28:23.757853   20546 kubernetes.go:208] Check and install kubeadm v1.23.10
I0818 04:28:23.773324   20546 kubernetes.go:215] "kubeadm" version=<
        {
          "clientVersion": {
            "major": "1",
            "minor": "23",
            "gitVersion": "v1.23.10",
            "gitCommit": "7e54d50d3012cf3389e43b096ba35300f36e0817",
            "gitTreeState": "clean",
            "buildDate": "2022-08-17T18:31:47Z",
            "goVersion": "go1.17.13",
            "compiler": "gc",
            "platform": "linux/amd64"
          }
        }
 >
I0818 04:28:23.773379   20546 kubernetes.go:232] Kubeadm v1.23.10 already exist, skip install.
I0818 04:28:23.773392   20546 kubernetes.go:183] Skip download cni, use already exist file: /tmp/cni-plugins-linux-amd64-v0.8.0.tgz
I0818 04:28:24.150263   20546 kubernetes.go:258] Setting kubelet service.
I0818 04:28:24.153453   20546 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "*****:6443"
I0818 04:28:24.153466   20546 token.go:230] [discovery] kube client is nil, use bootstrap config(&{  {false map[]} map[kubernetes:0xc0004fe080] map[] map[token-bootstrap-client@kubernetes:0xc0000fc0f0] token-bootstrap-client@kubernetes map[]}) to create client
I0818 04:28:24.212160   20546 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "*****:6443"
I0818 04:28:24.212511   20546 yurthub.go:45] [join-node] Adding edge hub static yaml
I0818 04:28:24.212560   20546 yurthub.go:89] yurthub template: /etc/kubernetes/manifests/yurthub.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    openyurt.io/static-pod-hash: 74c4554785
  creationTimestamp: null
  labels:
    k8s-app: yurt-hub
  name: yurt-hub
  namespace: kube-system
spec:
  containers:
  - command:
    - yurthub
    - --v=2
    - --bind-address=127.0.0.1
    - --server-addr=https://*****:6443
    - --node-name=$(NODE_NAME)
    - --bootstrap-file=/var/lib/yurthub/bootstrap-hub.conf
    - --working-mode=edge
    - --namespace=kube-system
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    image: openyurt/yurthub:v1.4.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        host: 127.0.0.1
        path: /v1/healthz
        port: 10267
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    name: yurt-hub
    resources:
      limits:
        memory: 300Mi
      requests:
        cpu: 150m
        memory: 150Mi
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/yurthub
      name: hub-dir
    - mountPath: /etc/kubernetes
      name: kubernetes
  dnsPolicy: ClusterFirst
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  terminationGracePeriodSeconds: 30
  volumes:
  - hostPath:
      path: /var/lib/yurthub
      type: DirectoryOrCreate
    name: hub-dir
  - hostPath:
      path: /etc/kubernetes
      type: Directory
    name: kubernetes
status: {}
I0818 04:28:24.212598   20546 yurthub.go:94] [join-node] Add hub agent static yaml is ok
I0818 04:28:24.212609   20546 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "*****:6443"
I0818 04:28:24.212615   20546 token.go:230] [discovery] kube client is nil, use bootstrap config(&{  {false map[]} map[kubernetes:0xc0004fe900] map[] map[token-bootstrap-client@kubernetes:0xc0004fc5f0] token-bootstrap-client@kubernetes map[]}) to create client
I0818 04:28:24.269986   20546 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "*****:6443"
[preflight] Running pre-flight checks
        [WARNING FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0818 04:28:30.333229   20582 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [*****]
W0818 04:28:30.335509   20582 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
error execution phase kubelet-start: error uploading crisocket: Get "http://127.0.0.1:10261/api/v1/nodes/*****?timeout=10s": dial tcp 127.0.0.1:10261: connect: connection refused
To see the stack trace of this error execute with --v=5 or higher
Error: exit status 1

/kind bug

[rambohe-ch]

@TaibiaoGuo Thank you for raising the issue. The address 127.0.0.1:10261 is listened by Yurthub component, please check the logs of Yurthub and find why yurthub can not startup correctly?