Update Multus docs

README.md

@@ -20,7 +20,7 @@ This command starts a local development server and opens up a browser window. Mo
 
 You can also use Docker to launch the website without needing to install and configure yarn:
 ```
-docker run --rm -it -v $PWD:$PWD -w $PWD -p 3000:3000 node yarn start -h 0.0.0.0
+docker run --rm -d --name k3s-docs -v $PWD:$PWD -w $PWD -p 3000:3000 node sh -c 'yarn; yarn start -h 0.0.0.0'
 ```
 
 ### Build

docs/advanced.md

@@ -135,7 +135,7 @@ The config.Node golang struct is defined [here](https://github.com/k3s-io/k3s/bl
 ### Base template
 
 :::info Version Gate
-Available as of September 2023 releases: v1.24.17+k3s1, v1.25.13+k3s1, v1.26.8+k3s1, v1.27.5+k3s1, v1.28.1+k3s1
+Available as of the September 2023 releases: v1.24.17+k3s1, v1.25.13+k3s1, v1.26.8+k3s1, v1.27.5+k3s1, v1.28.1+k3s1
 :::
 
 You can extend the K3s base template instead of copy-pasting the complete stock template out of the K3s source code. This is useful if you need to build on the existing configuration, and add a few extra lines at the end.

docs/cli/token.md

@@ -146,7 +146,7 @@ Flag | Description
 #### `k3s token rotate`
 
 :::info Version Gate
-Available as of 2023-10 releases (v1.28.2+k3s1, v1.27.7+k3s1, v1.26.10+k3s1, v1.25.15+k3s1).
+Available as of the October 2023 releases (v1.28.2+k3s1, v1.27.7+k3s1, v1.26.10+k3s1, v1.25.15+k3s1).
 :::
 
 Rotate original server token with a new bootstrap token. After running this command, all servers and any agents that originally joined with the old token must be restarted with the new token.

docs/installation/uninstall.md

@@ -3,12 +3,14 @@ title: Uninstalling K3s
 ---
 
 :::warning
-Uninstalling K3s deletes the local cluster data, configuration, and all of the scripts and CLI tools.  
-It does not remove any data from external datastores, or created by pods using external Kubernetes storage volumes.
+Uninstalling K3s may cause data loss!
 :::
 
 If you installed K3s using the installation script, a script to uninstall K3s was generated during installation.
 
+Running the uninstall script stops K3s and all running pods, and deletes the local cluster datastore, [Local Storage](../storage.md#setting-up-the-local-storage-provider) Persistent Volume data, node configuration, and all of the scripts and CLI tools.
+It does not remove any data from external datastores, or created by pods using external Kubernetes Persistent Volumes.
+
 If you are planning on rejoining a node to an existing cluster after uninstalling and reinstalling, be sure to delete the node from the cluster to ensure that the node password secret is removed. See the [Node Registration](../architecture.md#how-agent-node-registration-works) documentation for more information.
 
 ### Uninstalling Servers

docs/networking/multus-ipams.md

@@ -4,77 +4,162 @@ title: "Multus and IPAM plugins"
 
 [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) is a CNI plugin that enables attaching multiple network interfaces to pods. Multus does not replace CNI plugins, instead it acts as a CNI plugin multiplexer. Multus is useful in certain use cases, especially when pods are network intensive and require extra network interfaces that support dataplane acceleration techniques such as SR-IOV.
 
+For more information about Multus, refer to the [multus-cni](https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs) documentation.
+
 Multus can not be deployed standalone. It always requires at least one conventional CNI plugin that fulfills the Kubernetes cluster network requirements. That CNI plugin becomes the default for Multus, and will be used to provide the primary interface for all pods. When deploying K3s with default options, that CNI plugin is Flannel.
 
-### Add the Helm Repository
+:::info Version Gate
+K3s uses a fixed CNI binary path as of the October 2024 releases: v1.28.15+k3s1, v1.29.10+k3s1, v1.30.6+k3s1, v1.31.2+k3s1.
+:::
 
-To deploy Multus, we recommend using the following helm repo:
-```
-helm repo add rke2-charts https://rke2-charts.rancher.io
-helm repo update
-```
+K3s looks at `$DATA_DIR/data/cni` for CNI plugin binaries. By default this is `/var/lib/rancher/k3s/data/cni`. Additional CNI plugins should be installed to this location.
 
-### Configure the IPAM plugin
+Prior to the October 2024 releases, CNI binaries were part of the K3s userspace bundle at `$DATA_DIR/data/$HASH/bin`, where the hash is unique to each release of K3s.
+This made it difficult to deploy additional CNI plugins, as the path would change every time K3s was upgraded.
+If deploying Multus to an older release of K3s, you should use `/var/lib/rancher/k3s/data/current/bin/` as the CNI bin dir, but expect that the plugins will need to be re-deployed whenever K3s is upgraded.
 
-An IPAM plugin is required to assign IP addresses on the extra interfaces created by Multus.
+### Deploy with an IPAM plugin
+
+An IP Address Manager (IPAM) plugin is required to assign IP addresses on the extra interfaces created by Multus. One or more IPAMs can be installed; the examples below each show use of a single IPAM plugin but they may be combined as needed.
+
+The helm deployment examples below will deploy a DaemonSet to create Multus pods to install the required CNI binaries in `/var/lib/rancher/k3s/data/cni/` and Multus CNI config in `/var/lib/rancher/k3s/agent/etc/cni/net.d`.
 
 <Tabs groupId = "MultusIPAMplugins">
 <TabItem value="host-local" default>
 The host-local IPAM plugin allocates ip addresses out of a set of address ranges. It stores the state locally on the host filesystem, hence ensuring uniqueness of IP addresses on a single host. Therefore, we don't recommend it for multi-node clusters. This IPAM plugin does not require any extra deployment. For more information: https://www.cni.dev/plugins/current/ipam/host-local/.
 
-To use the host-local plugin, please create a file called `multus-values.yaml` with the following content:
-```
-config:
-  cni_conf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
-    kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
+To use the host-local plugin, deploy Multus with the following configuration:
+```yaml
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: multus
+  namespace: kube-system
+spec:
+  repo: https://rke2-charts.rancher.io
+  chart: rke2-multus
+  targetNamespace: kube-system
+  valuesContent: |-
+    config:
+      fullnameOverride: multus
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
 ```
 
 </TabItem>
 <TabItem value="Whereabouts" default>
 [Whereabouts](https://github.com/k8snetworkplumbingwg/whereabouts) is an IP Address Management (IPAM) CNI plugin that assigns IP addresses cluster-wide.
 
-To use the Whereabouts IPAM plugin, please create a file called `multus-values.yaml` with the following content:
+To use the Whereabouts IPAM plugin, deploy Multus with the following configuration:
+```yaml
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: multus
+  namespace: kube-system
+spec:
+  repo: https://rke2-charts.rancher.io
+  chart: rke2-multus
+  targetNamespace: kube-system
+  valuesContent: |-
+    config:
+      fullnameOverride: multus
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
+    rke2-whereabouts:
+      fullnameOverride: whereabouts
+      enabled: true
+      cniConf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
 ```
-config:
-  cni_conf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
-    kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
-rke2-whereabouts:
-  fullnameOverride: whereabouts
-  enabled: true
-  cniConf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
+
+When using whereabouts on K3s, `configuration_path` must be set to `/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.conf` in the NetworkAttachmentDefinition's `ipam` configuration.
+For example, when using whereabouts as the IPAM with the macvlan plugin:
+```yaml
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-whereabouts
+spec:
+  config: |-
+    {
+      "cniVersion": "1.0.0",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "whereabouts",
+        "range": "172.17.0.0/24",
+        "gateway": "172.17.0.1",
+        "configuration_path": "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.conf"
+      }
+    }
 ```
 
 </TabItem>
 <TabItem value="Multus DHCP daemon" default>
 The dhcp IPAM plugin can be deployed when there is already a DHCP server running on the network. This daemonset takes care of periodically renewing the DHCP lease. For more information please check the official docs of [DHCP IPAM plugin](https://www.cni.dev/plugins/current/ipam/dhcp/).
 
-To use the DHCP plugin, please create a file called `multus-values.yaml` with the following content:
-```
-config:
-  cni_conf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
-    kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
-manifests:
-  dhcpDaemonSet: true
+To use the DHCP plugin, deploy Multus with the following configuration:
+```yaml
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: multus
+  namespace: kube-system
+spec:
+  repo: https://rke2-charts.rancher.io
+  chart: rke2-multus
+  targetNamespace: kube-system
+  valuesContent: |-
+    config:
+      fullnameOverride: multus
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
+    manifests:
+      dhcpDaemonSet: true
 ```
 
 </TabItem>
 </Tabs>
 
-### Deploy Multus
-
-After creating the `multus-values.yaml` file, everything is ready to install Multus:
+### Using Multus
+
+Once Multus has been deployed, you can create NetworkAttachmentDefinition resources, and reference these in Pod specs to attach additional interfaces.
+For example, using the whereabouts example above, you can create an `eth1` interface on a Pod using the `k8s.v1.cni.cncf.io/networks` annotation:
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: multus-demo
+  labels:
+    app: multus-demo
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: multus-demo
+  template:
+    metadata:
+      annotations:
+        k8s.v1.cni.cncf.io/networks: macvlan-whereabouts@eth1
+      labels:
+        app: multus-demo
+    spec:
+      containers:
+      - name: shell
+        image: docker.io/rancher/mirrored-library-busybox:1.36.1
+        imagePullPolicy: IfNotPresent
+        command:
+          - sleep
+          - "3600"
 ```
-helm install multus rke2-charts/rke2-multus -n kube-system --kubeconfig /etc/rancher/k3s/k3s.yaml --values multus-values.yaml
-```
-
-The helm chart install will deploy a DaemonSet to create Multus pods for installing the required CNI binaries in `/var/lib/rancher/k3s/data/current/` and Multus CNI config in `/var/lib/rancher/k3s/agent/etc/cni/net.d`.
 
-For more information about Multus, refer to the [multus-cni](https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs) documentation.
+See the upstream documentation for additional information and examples.

docs/networking/networking-services.md

@@ -97,7 +97,7 @@ In order to reduce binary size, K3s removes all "in-tree" (built-in) cloud provi
 - Hosts the ServiceLB LoadBalancer controller.
 - Clears the `node.cloudprovider.kubernetes.io/uninitialized` taint that is present when the cloud-provider is set to `external` 
 
-Before deploying an external CCM, you must start all K3s servers with the `--disable-cloud-controller` flag to disable to embedded CCM. 
+Before deploying an external CCM, you must start all K3s servers with the `--disable-cloud-controller` flag to disable the embedded CCM. 
 
 :::note
 If you disable the built-in CCM and do not deploy and properly configure an external substitute, nodes will remain tainted and unschedulable.