ImageVolume broken

[laurivosandi]

Environmental Info:
K3s Version:
k3s version v1.31.0+k3s1 (34be6d9)
go version go1.22.5

Node(s) CPU architecture, OS, and Version:
Linux kubetest 6.8.0-44-generic #44-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 13 13:35:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single node

Describe the bug:
The freshly arrived ImageVolume feature seems broken

Steps To Reproduce:

Deployed k3s 1.31 with:

INSTALL_K3S_EXEC="--kubelet-arg feature-gates=ImageVolume=true --kube-apiserver-arg feature-gates=ImageVolume=true"
INSTALL_K3S_CHANNEL=latest

Applied manifest:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
          volumeMounts:
            - name: model
              mountPath: /model
      volumes:
        - name: model
          image:
            pullPolicy: IfNotPresent
            reference: registry.k8s.io/conformance:v1.31.0

Expected behavior:

Pod starts with registry.k8s.io/conformance:v1.31.0 image contents mounted at /model

Actual behavior:

In Kubernetes Lens following error message is shown

Error: failed to generate container "558bb3f947d9e1a70c697143ba10b2795c58ee7bde9c4fa0e3b317acb02dbe7b" spec: failed to generate spec: failed to mkdir "": mkdir : no such file or directory

Additional context / logs:

Sep 12 07:33:18 kubetest k3s[2270]: E0912 07:33:18.926553    2270 log.go:32] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to generate container \"536ef130eb99ae8a9f69c175a410b6400a576c514f07ea5faf68b6dc638a8860\" spec: failed to generate spec: failed to mkdir \"\": mkdir : no such file or directory" podSandboxID="f785734aa1099aa8557e8d4e6cabe4de5d889901d31880caf2a3f972946d7f78"
Sep 12 07:33:18 kubetest k3s[2270]: E0912 07:33:18.926664    2270 kuberuntime_manager.go:1272] "Unhandled Error" err="container &Container{Name:nginx,Image:nginx:1.14.2,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:80,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:model,ReadOnly:false,MountPath:/model,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-rjk5r,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod ai-test-67984f78cd-n8xqd_default(e55f166a-1f0f-4728-96e2-fde67da551e3): CreateContainerError: failed to generate container \"536ef130eb99ae8a9f69c175a410b6400a576c514f07ea5faf68b6dc638a8860\" spec: failed to generate spec: failed to mkdir \"\": mkdir : no such file or directory"
Sep 12 07:33:18 kubetest k3s[2270]: E0912 07:33:18.927733    2270 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"nginx\" with CreateContainerError: \"failed to generate container \\\"536ef130eb99ae8a9f69c175a410b6400a576c514f07ea5faf68b6dc638a8860\\\" spec: failed to generate spec: failed to mkdir \\\"\\\": mkdir : no such file or directory\"" pod="default/ai-test-67984f78cd-n8xqd" podUID="e55f166a-1f0f-4728-96e2-fde67da551e3"

[brandond]

I don't know where this error is coming from, but it is not code that lives within this repo. I suspect that the problem is either in the kubelet, or containerd.

The error itself is coming back from containerd (the runtime service), but I don't know if the container spec generated by the kubelet is incorrect, or if there's a bug in containerd, or what.

[brandond]

Have you tried this with any other Kubernetes distro with containerd 1.7, or k3s with a different container runtime (cri-dockerd perhaps)?

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.