EKS Anywhere baremetal, ubuntu, CAPI control plane controller: cannot get remote client to workload cluster

[pdamianov]

What happened: Tring to deploy EKS anywhere cluster on bare metal, using created with the image-builder ubuntu image. Cluster hangs on "Creating new workload cluster". Thinkerbell boot finishes successfully. OS is installed and the cluster is created on the bare metal machine.
On the admin machine:
k logs -n capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-xxxxxxxxx

gives a bunch of:
I0720 13:44:24.746087 1 controller.go:221] "Could not connect to workload cluster to fetch status" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" KubeadmControlPlane="eksa-system/mgmt-ubuntu" namespace="eksa-system" name="mgmt-ubuntu" reconcileID="deba6704-27d2-4852-b235-05f27327ea3b" Cluster="eksa-system/mgmt-ubuntu" err="failed to create remote cluster client: eksa-system/mgmt-ubuntu: failed to create cluster accessor: error creating client for remote cluster "eksa-system/mgmt-ubuntu": error getting rest mapping: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get "https://192.168.1.251:6443/api/v1?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
I0720 13:44:24.759627 1 controller.go:429] "Reconcile KubeadmControlPlane" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" KubeadmControlPlane="eksa-system/mgmt-ubuntu" namespace="eksa-system" name="mgmt-ubuntu" reconcileID="5a79440a-2ce0-4bd8-a1e5-ac3365901e32" Cluster="eksa-system/mgmt-ubuntu"

k describe pod -n capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager:

Warning FailedMount 25m (x4 over 25m) kubelet MountVolume.SetUp failed for volume "cert" : secret "capi-kubeadm-control-plane-webhook-service-cert" not found

What you expected to happen: Successfully deploy EKS Anywhere cluster

How to reproduce it (as minimally and precisely as possible):
image-builder: 0.4.0
Node image build command: image-builder build --os ubuntu --hypervisor baremetal --release-channel 1-28
eksctl anywhere version: 0.19.7
hardware: attached
cluster: attached
eksctl create command:
eksctl anywhere create cluster
--hardware-csv hardware.csv --per-machine-wait-timeout 30m
-f eksa-mgmt-cluster-ubuntu.yaml

eksa-mgmt-cluster-ubuntu.yaml:

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: mgmt
spec:
  clusterNetwork:
    cniConfig:
      cilium: {}
    pods:
      cidrBlocks:
      - 10.112.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneConfiguration:
    count: 1
    endpoint:
      host: "192.168.1.253"
  machineGroupRef:
      kind: TinkerbellMachineConfig
      name: mgmt-cp
  datacenterRef:
    kind: TinkerbellDatacenterConfig
    name: mgmt
  kubernetesVersion: "1.28"
  managementCluster:
    name: mgmt

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: TinkerbellDatacenterConfig
metadata:
  name: mgmt
spec:
  osImageURL: "http://192.168.1.106:8000/ubuntu-2204-kube-1-28-eks-a-19-7.gz"
  hookImagesURLPath: "http://192.168.1.106:8000/hook"
  tinkerbellIP: "192.168.1.252"

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: TinkerbellMachineConfig
metadata:
  name: mgmt-cp
spec:
  hardwareSelector:
    type: "cp"
  osFamily: ubuntu
  osImageURL: ""
  hookImagesURLPath: ""
  templateRef: {}
  users:
  - name: ubuntu
    sshAuthorizedKeys:
    - ssh-rsa AAAAB.....
 ---   

Anything else we need to know?:

Environment: 1.28
hardware.csv

• EKS Anywhere Release: 0.19.7
• EKS Distro Release: 1.28

[pdamianov]

I successfully created a cluster with this configuration using EKS anywhere version 0.18.7 (Kubernetes 1.26) following the same procedure as in the documentation. After that, I successfully upgraded the cluster to 1.28 (successfully scaled horizontally to 3 cp nodes). Tests with 0.19.7, 0.20.0, and 0.20.1 led to the above. Am I doing something wrong?