Multiple hardware types and install disks in a serverclass, serverclass inheritance, or conditional serverclasses?

[japtain-cack]

I have specified the install disk as /dev/nvme0n1 but I'm getting a disk does not exist error, on provisioning of a bare metal system. I have confirmed that during the initial boot, the dmesg displays Fast wiped /dev/nvme0n1, and Wipe complete. So /dev/nvme0n1 should be the correct disk.

I believe the issue is that the serverclass talos-masters is the only one being applied as it's the only one defined in the talos-cp MetalMachineTemplate and referenced by the TalosControlPlane's infrastructureTemplate.

How do ServerClasses get applied? Does any matching serverclass contribute to the configuration of the matched server? How can I have a talos-masters serverclass and supply install disks, or other configPatches, conditionally based on hardware type/version/etc?

As you can see below, I have defined a talos-masters serverclass which is referenced by the cluster. You can only set one infrastructureTemplate on the TalosControlPlane to my knowledge. If you look at my serverclasses, you can see that the talos-lenovo-m710q serverclass defines the install disk and is matched according to the describe serverclasses below.

Resources

kubectl get serverclasses

NAME                    AVAILABLE   IN USE                                                                                                                   AGE
any                     []          ["0778b200-9684-11e8-8f88-e0e722df6d00","b80c6841-fb72-4fa2-9fec-7bc8eb7faab3","f2f49248-b3c3-4403-9207-a819bfc86b49"]   6d16h
talos-lenovo-m710q      []          ["0778b200-9684-11e8-8f88-e0e722df6d00"]                                                                                 155m
talos-masters           []          ["0778b200-9684-11e8-8f88-e0e722df6d00","b80c6841-fb72-4fa2-9fec-7bc8eb7faab3","f2f49248-b3c3-4403-9207-a819bfc86b49"]   6d16h
talos-virtual-esx       []          []                                                                                                                       155m
talos-virtual-proxmox   []          ["b80c6841-fb72-4fa2-9fec-7bc8eb7faab3","f2f49248-b3c3-4403-9207-a819bfc86b49"]                                          155m
talos-workers           []          []                                                                                                                       6d16h

kubectl describe metalmachine talos-cp-hl729

Name:         talos-cp-hl729
Namespace:    sidero-system
Labels:       cluster.x-k8s.io/cluster-name=talos
Annotations:  cluster.x-k8s.io/cloned-from-groupkind: MetalMachineTemplate.infrastructure.cluster.x-k8s.io
              cluster.x-k8s.io/cloned-from-name: talos-cp
API Version:  infrastructure.cluster.x-k8s.io/v1alpha3
Kind:         MetalMachine
Metadata:
  Creation Timestamp:  2023-04-08T21:16:28Z
  Finalizers:
    metalmachine.infrastructure.cluster.x-k8s.io
  Generation:  2
  Managed Fields:
    API Version:  infrastructure.cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cluster.x-k8s.io/cloned-from-groupkind:
          f:cluster.x-k8s.io/cloned-from-name:
        f:finalizers:
          .:
          v:"metalmachine.infrastructure.cluster.x-k8s.io":
        f:labels:
          .:
          f:cluster.x-k8s.io/cluster-name:
        f:ownerReferences:
          .:
          k:{"uid":"f2c55399-28af-4b93-8647-659ca4912202"}:
      f:spec:
        .:
        f:providerID:
        f:serverClassRef:
        f:serverRef:
    Manager:      manager
    Operation:    Update
    Time:         2023-04-08T21:22:08Z
    API Version:  infrastructure.cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:addresses:
        f:conditions:
        f:ready:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-04-08T21:23:06Z
  Owner References:
    API Version:           cluster.x-k8s.io/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Machine
    Name:                  talos-cp-2f46r
    UID:                   f2c55399-28af-4b93-8647-659ca4912202
  Resource Version:        2429599
  UID:                     bb708684-3858-4f8b-a127-bb27a7dc3d9d
Spec:
  Provider ID:  sidero://0778b200-9684-11e8-8f88-e0e722df6d00
  Server Class Ref:
    API Version:  metal.sidero.dev/v1alpha1
    Kind:         ServerClass
    Name:         talos-masters
  Server Ref:
    Kind:  Server
    Name:  0778b200-9684-11e8-8f88-e0e722df6d00
Status:
  Addresses:
    Address:  master-03
    Type:     Hostname
  Conditions:
    Last Transition Time:  2023-04-08T21:22:08Z
    Message:               no matching nodes found
    Reason:                ProviderUpdateFailed
    Severity:              Warning
    Status:                False
    Type:                  ProviderSet
    Last Transition Time:  2023-04-08T21:23:06Z
    Message:               failed to validate config: 2 errors occurred:
                           * an install disk is required in "metal" mode
                           * specified install disk does not exist: ""


    Reason:    TalosConfigLoadFailed
    Severity:  Error
    Status:    False
    Type:      TalosConfigLoaded
  Ready:       true
Events:        <none>

ServerClasses

### sidero/kustomize/serverClasses.yaml ###
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
  name: talos-virtual-proxmox
  namespace: default
spec:
  qualifiers:
    systemInformation:
      - manufacturer: QEMU
        version: pc-i440fx-6.2
  configPatches:
    - op: replace
      path: /machine/install/disk
      value: /dev/sda
  bootFromDiskMethod: ipxe-exit
---
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
  name: talos-virtual-esx
  namespace: default
spec:
  qualifiers:
    systemInformation:
      - manufacturer: VMware, Inc.
  configPatches:
    - op: replace
      path: /machine/install/disk
      value: /dev/sda
  bootFromDiskMethod: ipxe-exit
---
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
  name: talos-lenovo-m710q
  namespace: default
spec:
  qualifiers:
    systemInformation:
      - manufacturer: LENOVO
        version: ThinkCentre M710q
  configPatches:
    - op: replace
      path: /machine/install/disk
      value: /dev/nvme0n1
  bootFromDiskMethod: ipxe-exit
---
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
  name: talos-masters
  namespace: default
spec:
  selector:
    matchLabels:
      node-role.kubernetes.io/control-plane: "true"
  qualifiers:
    systemInformation:
      - manufacturer: VMware, Inc.
      - manufacturer: QEMU
      - manufacturer: LENOVO
        family: ThinkCentre M710q
  bootFromDiskMethod: ipxe-exit
---
apiVersion: metal.sidero.dev/v1alpha1
kind: ServerClass
metadata:
  name: talos-workers
  namespace: default
spec:
  selector:
    matchLabels:
      node-role.kubernetes.io/worker: "true"
  qualifiers:
    systemInformation:
      - manufacturer: VMware, Inc.
      - manufacturer: QEMU
      - manufacturer: LENOVO
        family: ThinkCentre M710q
  bootFromDiskMethod: ipxe-exit

Talos Cluster

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: talos
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.10.0.0/16
    services:
      cidrBlocks:
      - 10.11.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: TalosControlPlane
    name: talos-cp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: MetalCluster
    name: talos
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: MetalCluster
metadata:
  name: talos
  namespace: default
spec:
  controlPlaneEndpoint:
    host: cp.talos.example.com
    port: 6443
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: MetalMachineTemplate
metadata:
  name: talos-cp
  namespace: default
spec:
  template:
    spec:
      serverClassRef:
        apiVersion: metal.sidero.dev/v1alpha1
        kind: ServerClass
        name: talos-masters
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  name: talos-cp
  namespace: default
spec:
  controlPlaneConfig:
    controlplane:
      generateType: controlplane
      talosVersion: v1.3.6
      configPatches:
      - op: add
        path: /machine/network
        value:
          interfaces:
          - interface: eth0
            dhcp: true
            vip:
              ip: 10.100.50.52
      - op: add
        path: /machine/certSANs/-
        value:
          "*.talos.example.com"
      - op: add
        path: /cluster/apiServer/certSANs/-
        value:
          "*.talos.example.com"
      - op: add
        path: /machine/kubelet
        value:
          extraArgs:
            rotate-server-certificates: true
      - op: add
        path: /machine/install/extensions
        value:
          - image: ghcr.io/siderolabs/drbd:9.2.0-v1.3.6
      - op: add
        path: /machine/kernel
        value:
          modules:
          - name: drbd
          - name: drbd_transport_tcp
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: MetalMachineTemplate
    name: talos-cp
  replicas: 3
  version: v1.26.2
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: TalosConfigTemplate
metadata:
  name: talos-workers
  namespace: default
spec:
  template:
    spec:
      generateType: join
      talosVersion: v1.3.6
      configPatches:
      - op: add
        path: /machine/kubelet
        value:
          extraArgs:
            rotate-server-certificates: true
      - op: add
        path: /machine/install/extensions
        value:
          - image: ghcr.io/siderolabs/drbd:9.2.0-v1.3.6
      - op: add
        path: /machine/kernel
        value:
          modules:
          - name: drbd
          - name: drbd_transport_tcp
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: talos-workers
  namespace: default
spec:
  clusterName: talos
  replicas: 0
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: TalosConfigTemplate
          name: talos-workers
      clusterName: talos
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: MetalMachineTemplate
        name: talos-workers
      version: v1.26.2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: MetalMachineTemplate
metadata:
  name: talos-workers
  namespace: default
spec:
  template:
    spec:
      serverClassRef:
        apiVersion: metal.sidero.dev/v1alpha1
        kind: ServerClass
        name: talos-workers

[japtain-cack]

I also noticed that the talosVersion in the TalosControlPlane is not respected. I've installed 3 nodes so far, and each one shows version v1.3.0. I had to upgrade each node manually. I can see that it is set to the correct version in the TalosControlPlane resource when describing it.

[japtain-cack]

I was going through the documentation again and noticed this page, verry bottom, https://www.sidero.dev/v0.5/resource-configuration/metadata/. How can we specify conditional patches, based on differences in hardware, for a StorageClass? I solved my original issue by patching the server resource. However, this isn't ideal, because there should be a way to make this automatic, instead of having to edit every server object specifically.

I'm a strong believer in infrastructure as code, this is one of my primary objectives professionally and part of my duties as an engineer where I work. If we were to adopt this and run it in production, this would be a deal breaker. For instance, say we have a mixture of arm and intel based worker nodes with nvme and standard disks. We can easily use node selectors or affinities to direct workloads onto specific nodes easily enough, but without being able to set nvme/regular disks conditionally on the serverclass, we wouldn't be able to do that. I would agree that it's probably a good idea to keep your control plane running on the same hardware, but this is an evaluation in my personal lab. However, from what I understand currently, all control plane and worker hardware must be similar because the target serviceclass can't handle a mixture of hardware specific patches.

It would be ideal if there were serverclass inheritance, or some other way to have a mixture of hardware, which may require different but conflicting patch definitions (like install disks).

[japtain-cack]

Also, outside of the basic setup, there seems to be no production oriented deployment documentation. It would be useful for those of us that are evaluating this for professional reasons to have documentation oriented for a production ready deployment, with recommended hardening procedures and the like. The goal of my evaluation is to create a production ready, HA deployment of Sidero/Talos.

[japtain-cack]

I think the easiest solution that comes to mind is, being able to select multiple ServerClasses when defining the MetalMachineTemplate. Possibly using a ServerClass label, that exists on multiple ServerClasses, or something similar. That way you can have various ServerClasses, with a mixture of hardware and install disk configurations, and it could pull from any of the StorageClasses with the common label.

[smira]

There are many questions in this issue, so not sure if the original problem is still relevant?

The server class the server was picked from can be seen in ServerBinding resource once the server is allocated.

As install disk is usually specific to the node, it might make sense to make it Server-level. If you have many similar machines, you can use ServerClass to provide specific patches.

Also please keep in mind Talos allows picking up system disk using disk selectors: https://www.talos.dev/v1.3/reference/configuration/#installconfig

[japtain-cack]

/I wasn't aware of the disk selectors. That will be helpful regarding install disks. However, let me lay out a scenario and maybe that will help explain what my issue is with serverclasses.

In this scenario, the control plane is using similar hardware setup in the same way. So a single serverclass will work, all the disks are the same, everything works automatically with no user interaction via automatic acceptance.

Workload cluster has arm and intel machines. Arm has additional iscsi (/dev/sdb) volumes for persistent data for low i/o workloads. Intel nodes have additional nvme (/dev/nvme0n1) drives for persistent workload data. These nodes may have any number of additional drives, but since this is a production POC, all hardware will be set up similarly. All intel nodes will be similar, all arm nodes will be similar. However I have two ServerClasses that define each set of hardware.

In this scenario, on the workload cluster, I can only choose a single ServerClass. This serverclass can't define disks or errors will occur on the hardware that the defined disks differ from. So, I have to modify every Server resource by hand and add the additional disks every time. Since you can only specify a single serverclass, there seems to be no way I can set this up in such a way, that it doesn't require user interaction.

There are a couple solutions I've been mulling over:

1. Somehow apply multiple ServerClasses to my workload cluster which, from what I understand, doesn't seem to be possible.
2. Have a single ServerClass conditionally set up hardware which also doesn't seem possible.
3. Deploy multiple workload clusters, each using a single ServerClass and each type of hardware will be in it's own cluster. This is not ideal, because you should be using taints/tolerations/selectors to determine where the workloads run, so it breaks away from the design patterns of kubernetes. I could get around this by deploying a canopy network and basically doing multi-cluster workloads, but adds a lot of complexity for such a simple issue.
4. Use kustomize to patch all Server resources with the additional disks, and/or install disks. This doesn't seem to work, because these changes don't do anything for existing nodes. You would have to decom the node and add it again, but that requires deleting the Server resource, so you have a chicken before the egg issue here.

TLDR:
There needs to be a way to have a mixture of hardware/disks and still be able to deploy a cluster, and ingest nodes, without user intervention.

[smira]

Just in case, you can have multiple worker machine deployments for a single cluster this way you could have multiple worker sets coming from multiple ServerClasses. You can also use that to e.g. label them different way.

Server-level patching should work as well, as long as you can automate it. The whole idea of CAPI is that it is extensible - you can build a controller which ensures each Server has proper config patches based on the hardware info in the resource, or some external inventory database you might have.

[japtain-cack]

Hmm, I think that will work, making another machine deployment. I should be able to duplicate the below resources for each set of hardware. Not exactly ideal, a lot of code duplication, but this may be a viable solution. This is just a thought so far, I haven't tested anything and won't be able to for a couple of days.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: talos-workers-arm
  namespace: default
spec:
  clusterName: talos
  replicas: 0
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: TalosConfigTemplate
          name: talos-workers
      clusterName: talos
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: MetalMachineTemplate
        name: talos-workers-arm
      version: v1.26.2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: MetalMachineTemplate
metadata:
  name: talos-workers-arm
  namespace: default
spec:
  template:
    spec:
      serverClassRef:
        apiVersion: metal.sidero.dev/v1alpha1
        kind: ServerClass
        name: talos-workers-arm

[japtain-cack]

Just in case, you can have multiple worker machine deployments for a single cluster this way you could have multiple worker sets coming from multiple ServerClasses. You can also use that to e.g. label them different way.

Server-level patching should work as well, as long as you can automate it. The whole idea of CAPI is that it is extensible - you can build a controller which ensures each Server has proper config patches based on the hardware info in the resource, or some external inventory database you might have.

I'm still figuring out what can be updated on the fly and what can't. I recently patched node labels at the TalosControlPlane level, and that didn't do anything. I also tried patching the ServerClass/Server resources as well. Updates to the machine/cluster config don't seem to get applied until you decommission/re-deploy the node. I tested this 2 days ago, rebooting/upgrading, deleting the Machine/Node and not the Server resource, didn't work. Nothing short of decommissioning the node, and re-deploying it actually applied all the configs. This would be fine, if the talosVersion that is set in my cluster.yaml was respected. However, after decom/re-deploy, I have to do a node upgrade on top of all that.

[smira]

CAPI only natively supports upgrades by replacing nodes. Sidero will deliver config to the machine only once.

[japtain-cack]

CAPI only natively supports upgrades by replacing nodes. Sidero will deliver config to the machine only once.

Gotcha, not a big deal, Sidero/Talos makes that process easy and I like it because it's in line with ephemeral hardware practices. However, is there any way to get the talosVersion: v1.3.6 to stick? Every time I roll a node, it's back to 1.3.0 and kubernetes is a lower version as well.

This brings up the question though, how can I roll a node without deleting the Server resource? When I left the Server resource alone, deleting only the Machine, the Node/ServerBinding were removed, but it didn't seem to update the node's config, once it came back up. I believe this is because it wasn't caught by Sidero on the next boot and booted from disk. Once I deleted the Server resource, everything worked as expected, except the version mismatch of course. If rolling the nodes to update configs is the pattern Sidero/Talos has chosen, it would be great if talosctl had a way of "rolling" a node for reconfiguration, which would decom it, and readd it again automatically. Deleting the server resource is not ideal, because I have to define stuff in there by hand.

[smira]

talosVersion is not the Talos version you're installing, it's the machine config generation contract: https://github.com/siderolabs/cluster-api-bootstrap-provider-talos/#usage

The version of Talos you're installing is defined by the Environment in Sidero.

[japtain-cack]

Ok, good to know. I will revist that. I appreciate the clarification you have provided so far. This has been really helpful.

For the rolling of a node to update it's machine config, will this process work?

Essentially talosctl -n master-01.talos.mimir-tech.org reset --graceful?

[japtain-cack]

talosVersion is not the Talos version you're installing, it's the machine config generation contract: https://github.com/siderolabs/cluster-api-bootstrap-provider-talos/#usage

The version of Talos you're installing is defined by the Environment in Sidero.

Hmm, this document seems to state that the TalosControlPlane & the TalosConfigTemplate set the talosVersion. However it does say

talosVersion: version of Talos to generate machine configuration for (e.g. v1.0, patch version might be omitted).
So I bet what is happening, it's ignoring the patch version. It specifically states it "may" be omitted, but under which circumstances I do not know.

This doc seems to corroborate this as well.
https://www.sidero.dev/v0.5/guides/upgrades/#upgrading-talos-08---09

And I see no mention of talosVersion in the environments documentation.
https://www.sidero.dev/v0.5/resource-configuration/environments/

[smira]

For the rolling of a node to update it's machine config, will this process work?

Essentially talosctl -n master-01.talos.mimir-tech.org reset --graceful?

By rolling update I mean something which is done by CAPI itself. You don't need to do anything for it except for managing the resources. You can still out of CAPI control change Talos machine configuration, but it might not match CAPI-level settings.

[smira]

And I see no mention of talosVersion in the environments documentation.
https://www.sidero.dev/v0.5/resource-configuration/environments/

Environment is what gets booted up, and that (by default) defines the installed version of Talos.

talosVersion is a machine config version contract. CAPI doesn't update the fields you're setting in the spec on its own.

[blackliner]

talosVersion is not the Talos version you're installing, it's the machine config generation contract: https://github.com/siderolabs/cluster-api-bootstrap-provider-talos/#usage

The version of Talos you're installing is defined by the Environment in Sidero.

and

Environment is what gets booted up, and that (by default) defines the installed version of Talos.

talosVersion is a machine config version contract. CAPI doesn't update the fields you're setting in the spec on its own.

So does this mean that the talosVersion field here is not meant to be specified by the user, but by CAPI?

Got it, it is used to render the talosconfig, not to specify the installed talos version

[smira]

Got it, it is used to render the talosconfig, not to specify the installed talos version

It is used to generate proper base Talos machine configuration. It should be fixed at the moment of cluster creation (e.g. created using Talos 1.6.2, so talosVersion: v1.6), and it will stay this way.