network create fail: route ip+net: no such network interface

[apostasie]

Description

After some repetitive stress testing, network create may fail with
route ip+net: no such network interface

I am not sure where this message is coming from - could not pinpoint it in nerdctl, cni nor containerd.

You might need to run the test suite x100+ times to trigger this.

As mentioned in #3086 and other tickets, recent patches have made our networking situation better - which means we are now seeing deep seated / complex issues that we were not seeing before (because we failed before hitting them...).

Network gurus around, do you have an idea on this route ip+net: no such network interface thing? :-)

Steps to reproduce the issue

while true; do go test ./cmd/nerdctl/network -count 1; done

ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.344s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.839s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.909s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.584s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.892s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.084s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.021s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.733s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.533s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.528s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.508s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.397s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.858s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.819s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.436s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.610s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.991s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.163s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.088s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.500s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.142s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.860s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.805s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.833s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.384s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.152s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	3.821s
test target: "nerdctl"
--- FAIL: TestNetworkPrune (4.11s)
    helpers.go:77: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 ps -aq
    helpers.go:57: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 rm -f
    helpers.go:57: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 system prune -f --all --volumes
    helpers.go:64: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 namespace remove testnetworkprune-private-7a46bb40
    --- FAIL: TestNetworkPrune/Prune_does_not_collect_started_container_network (0.51s)
        helpers.go:64: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 rm -f testnetworkprune-prune-does-not-collect-started-container-network-cf15bc20
        helpers.go:64: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 network rm testnetworkprune-prune-does-not-collect-started-container-network-cf15bc20
        helpers.go:57: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 network create testnetworkprune-prune-does-not-collect-started-container-network-cf15bc20
        helpers.go:57: assertion failed: expect.ExitCode is not result.ExitCode: Expected exit code: 0

            Command:  /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 network create testnetworkprune-prune-does-not-collect-started-container-network-cf15bc20
            ExitCode: 1
            Error:    exit status 1
            Stdout:
            Stderr:   time="2024-10-05T12:08:59-07:00" level=fatal msg="route ip+net: no such network interface"

            Env:
            SHELL=/bin/bash
            LOGNAME=dmp
            XDG_SESSION_TYPE=tty
            HOME=/home/dmp.linux
            LANG=C.UTF-8
            SSH_CONNECTION=192.168.5.2 51297 192.168.5.15 22
            LESSCLOSE=/usr/bin/lesspipe %s %s
            XDG_SESSION_CLASS=user
            TERM=xterm-256color
            LESSOPEN=| /usr/bin/lesspipe %s
            USER=dmp
            SHLVL=2
            XDG_SESSION_ID=2
            XDG_RUNTIME_DIR=/run/user/501
            SSH_CLIENT=192.168.5.2 51297 22
            XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
            DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/501/bus
            SSH_TTY=/dev/pts/1
            OLDPWD=/Users/dmp
            _=/usr/local/go/bin/go
            PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin:/sbin:/usr/sbin:/sbin:/usr/sbin:/sbin:/usr/local/go/bin
            PWD=/Users/dmp/Projects/go/nerd/nerdctl/cmd/nerdctl/network
            DOCKER_CONFIG=/tmp/TestNetworkPrunePrune_does_not_collect_started_container_network643097071/001
            NERDCTL_TOML=/tmp/TestNetworkPrunePrune_does_not_collect_started_container_network643097071/001/nerdctl.toml
        helpers.go:64: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 rm -f testnetworkprune-prune-does-not-collect-started-container-network-cf15bc20
        helpers.go:64: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 network rm testnetworkprune-prune-does-not-collect-started-container-network-cf15bc20
    helpers.go:77: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 ps -aq
    helpers.go:57: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 rm -f
    helpers.go:57: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 system prune -f --all --volumes
    helpers.go:64: /usr/local/bin/nerdctl --namespace=testnetworkprune-private-7a46bb40 namespace remove testnetworkprune-private-7a46bb40
FAIL
FAIL	github.com/containerd/nerdctl/v2/cmd/nerdctl/network	4.121s
FAIL
FAIL

Describe the results you received and expected

na

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response