.NET SDK 8.0.401: random "multiple attempts to download the nupkg have failed"

[metoule]

NuGet Product Used

dotnet.exe

Product Version

SDK Version: 8.0.401

Worked before?

8.0.304

Impact

It's more difficult to complete my work

Repro Steps & Context

Our CI randomly fails during the dotnet restore phase, with errors like the following:

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Grpc.Tools.2.65.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

Other random examples:

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Grpc.AspNetCore.Server.Reflection.2.65.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Dapper.2.1.35' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

Each time, the package that's not found is a different one, so it seems totally random.

Of course, all these packages exists:

• Grpc.Tools.2.65.0
• Grpc.AspNetCore.Server.Reflection.2.65.0
• Dapper.2.1.35

Verbose Logs

No response

[nicholasreinlein-broadsign]

It seems completely random, and occurs with theses packages as well:

• Microsoft.NET.Test.Sdk
• Serilog.Formatting.Compact.1.1.0

Reruns are successful

[nkolev92]

Looks very much like: #13545.

This has been in the .NET 9 previews, maybe you can update to the latest .NET 9 previews to confirm whether you're still seeing issues?

[Andrenhn]

Will the fix be applied to .NET8 as well? Not that easy to just jump up to NET9.

[nkolev92]

We haven't gotten these reports with .NET 8 SDKs much.
There's no immediate plan to port it to .NET SDK 8.0.4xx, but that could change.

It'd be great if people can try out the .NET 9 SDK and help confirm that the root cause is the same. If you do that, please comment here.

[andrstor]

We have struggled with this for the last week or so. The problems began after the release of the latest .net 8 version. At first we just thought it was something with azure devops artifacts, but now we suspect that this is the bug we are seeing. The packages that fails are totally random, and it is hard to reproduce. It happens several times a day and is starting to become really annoying.

[matoomx]

We use the hosted agent ubuntu-latest (Current image version: '20240818.1.0') in our CI pipeline and we get this error on a daily basis. Right now we have to have an extra manual step in our process where failed pipline-runs are re-scheduled if we see this error in the log. I really hope this will be solved in .net 8 fix

Update:
We have another pipeline that uses the windows-latest image (Current image version: '20240811.1.0') and that pipeline have never failed with this error and it uses .net core sdk 8.0.303

[mabergstrom]

I can just confirm that this issue has started to become really frustrating and happens several times during the day now. Seems to be completely random as people mentioned earlier.

[WalissonPires]

Today I came across the same issue in the pipeline on Azure DevOs when generating a docker image. Here the error was:

error : The feed '***.org [https://api.***.org/v3/index.json]' lists package 'Microsoft.EntityFrameworkCore.Design.8.0.4' but multiple attempts to download the nupkg have failed.

After running the pipeline again manually or restoring it ran successfully.

.NET SDK: 8.X

AzurePipeline:
vmImage: ubuntu-latest
task: DockerInstaller@0
inputs:
dockerVersion: '20.10.9'

[olaherrdahl]

I just ran into this as well when testing the deploy-image workflow described here:
https://docs.github.com/en/packages/managing-github-packages-using-github-actions-workflows/publishing-and-installing-a-package-with-github-actions#upgrading-a-workflow-that-accesses-a-registry-using-a-personal-access-token

#17 [build  9/12] RUN dotnet restore "./...csproj"
#17 1.063   Determining projects to restore...
#17 2.404 /usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Ninject.3.3.6' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [/src/....csproj]

No issues when running it manually a second time.

[borland]

Since .NET SDK 8.0.4xx has arrived, we've been seeing these NuGet download failure attempts intermittently too, wreaking havoc on our CI pipelines. It's good to see a fix is potentially in .NET 9, but that's an STS release which not everyone will be able to uptake. It'd be good to see a fix in 8.0.x as well please.

[NicholasFields]

TY@Niko for keeping us in the know.

CI has become gambling for my team as well.

Alongside @Andrenhn and @borland,
We would very much appreciate consideration of a .NET8 patch.

Till then, good luck to all :)

[iskiselev]

We are affected by the issue for about last two weeks after updating to newer SDK on both Linux/Windows.

[telb99]

We have started seeing this some time over the last two weeks also... feels like it is getting worse each day.
This is on MS Hosted agents in ADO running ubuntu, and we are pulling all packages from a private package feed in ADO.
We have seen this in docker builds also where we are using the 8.0-alpine image.
Please port this back to 8.x, we are not able to jump to a preview version of .NET for our production applications.

[BrettMillerDealX]

We've been seeing this for over 2 weeks when running on BitBucket pipelines.

Multiple instances per day and different libraries each time all with same message.

/usr/share/dotnet/sdk/8.0.401/NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'runtime.any.System.Text.Encoding.4.3.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.

It's been very disruptive.

[Pucmaster]

We have same problem in bitbucket pipelines after migration to net 8

[sergei-lobanov]

Same issue with .NET 8 again and again every day, its a blocker issue (!). We need a hotfix.

[zivkan]

We still think that NuGet/NuGet.Client#5905 will fix the issue, and I'm in the process of trying to get approval for it. If we have customers report here saying they tested the .NET 9 preview SDK and the problem no longer occurs, then it gives the people who approve/reject servicing requests more confidence that we're fixing the right thing, and hopefully not risking new bugs.

It's good to see a fix is potentially in .NET 9, but that's an STS release which not everyone will be able to uptake.

Firstly, it was a request to temporarily test the .NET 9 SDK and provide feedback if it works. Not a request to permanently use the .NET 9 SDK.

Secondly, you can continue to target .NET 8 in your projects, when using the .NET 9 SDK. .NET 9 SDK are the build tools, not the runtime that projects being built has to use. It's no different to using Visual Studio 17.11 even though .NET 8 originally came out with Visual Studio 17.8, or using either of those versions of Visual Studio while running your apps on the .NET 6 LTS runtime. runtime version != SDK version.

Anyway, as mentioned, NuGet/NuGet.Client#5905 is our best guess about the problems, so I'm working on trying to get it added. If it turns out that this won't fix the problem, then even when the .NET 8.0.4xx SDK is updated with a new version of NuGet, the problem might continue. Hence why we'd like customer validation that the problem is resolved in the preview version, to be confident that it's the right hotfix for the release version.

[cpsnowden]

Thanks @vanavaraVL!

Any chance anyone has found a docker sdk image for 8.0.3xx-alpine for containerised builds?

[zivkan]

The .NET SDK 8.0.402 was released today: https://dotnet.microsoft.com/en-us/download/dotnet/8.0

Can someone please update their pipeline to use this version, and report back if the problem is resolved?

[auslavs]

I reran one of our Azure DevOps pipelines that failed yesterday with this issue.
The pipeline picked up the latest SDK and the pipeline completed successfully with no issues.
I'll report back if the issue returns.

[uchitha]

I have moved to the latest SDK few hours ago and a handful of builds after that. So far so good. Thank you.

[wfdevopsteam]

When will this fix be released for Microsoft package repository for Ubuntu? It keeps saying that SDK 8.0.401 is the latest...

Thank you in advance!

[mawl]

When will this get fixed for https://dotnet.microsoft.com/en-us/download/visual-studio-sdks and https://github.com/microsoft/dotnet-framework-docker/blob/2d75ca9b3546b69e511e36a4b5d6a897a805506c/src/sdk/4.8/windowsservercore-ltsc2022/Dockerfile#L37 ?

Thank you very much for your effort in advance.

[baronfel]

When will this fix be released for Microsoft package repository for Ubuntu? It keeps saying that SDK 8.0.401 is the latest...

Thank you in advance!

@leecow do you know if the PMC Ubuntu packages got the 401 hotfix?

[baronfel]

When will this get fixed for https://dotnet.microsoft.com/en-us/download/visual-studio-sdks and https://github.com/microsoft/dotnet-framework-docker/blob/2d75ca9b3546b69e511e36a4b5d6a897a805506c/src/sdk/4.8/windowsservercore-ltsc2022/Dockerfile#L37 ?

Thank you very much for your effort in advance.

These out-of-band hotfixes don't get inserted into Visual Studio, but the normal monthly updates will. I'd expect 8.0.402, which will release in October and ship with Visual Studio, to solve this. In the meantime you can always download and install the SDK as a standalone install outside of Visual Studio to get the latest fixes.

I don't have an answer for the dotnet framework Dockerfile - I can dig into that a bit today.

[leecow]

Which version of Ubuntu?

We're in the process of aligning a policy where we do not publish .NET versions to packages.microsoft.com that are natively available on supported distros. For Ubuntu, this means we only publish .NET 8 up to 23.10.

https://packages.microsoft.com/ubuntu/23.10/prod/pool/main/d/dotnet-sdk-8.0/

[zivkan]

I'm closing this as fixed, since there have been multiple reports of the problem going away, and nobody reporting that they still have the problem.

[mawl]

When will this get fixed for https://dotnet.microsoft.com/en-us/download/visual-studio-sdks and https://github.com/microsoft/dotnet-framework-docker/blob/2d75ca9b3546b69e511e36a4b5d6a897a805506c/src/sdk/4.8/windowsservercore-ltsc2022/Dockerfile#L37 ?
Thank you very much for your effort in advance.

These out-of-band hotfixes don't get inserted into Visual Studio, but the normal monthly updates will. I'd expect 8.0.402, which will release in October and ship with Visual Studio, to solve this. In the meantime you can always download and install the SDK as a standalone install outside of Visual Studio to get the latest fixes.

I don't have an answer for the dotnet framework Dockerfile - I can dig into that a bit today.

Thanks. Yes, we have solved it in our dockerfiles by using the commands from https://github.com/dotnet/dotnet-docker/blob/b9bca757f5f6135feab16b7d7aac9e0e3f101e3b/src/sdk/8.0/windowsservercore-ltsc2022/amd64/Dockerfile#L35

[akshayg47]

I am still getting the build failures with the random packages missing error intermittently.
I confirmed that we are using 8.0.402 SDK build.
I am not sure if this works differently when we do containerized builds using docker. Can anyone provide any feedback on it?

[zivkan]

First step is to either get a binlog of the restore, or get the logs at normal verbosity, and you'll be able to see NuGet's logs when making HTTP requests. There will be lines like GET https://... when HTTP requests are made, or CACHE https://.. when files are read out of the http cache directory. Then, responses will have lines like OK https://... or NOTFOUND https://...

The previous problem was causing NuGet to not even make those requests. NuGet would think that the server didn't have the package, but it didn't even make the request.

If your logs show you that NuGet is making HTTP requests to the server, but the server is responding with NOTFOUND, or some other kind of error, then it's a server-side problem.

If NuGet is still not making requests, or if it's getting a successful response, then it sounds like there is some kind of problem, but without being able debug locally (and therefore reproduce reliably), it's going to be hard to figure out.

[akshayg47]

I managed to get the diagnostic logs of dotnet restore command for the project.
I do see the package being downloaded and installed but still it shows the error which is weird or maybe I am overlooking something. Here are the screenshots.

[akshayg47]

@zivkan do you have any feedback?

[zivkan]

@akshayg47 the error message you got in your last screenshot is not the same as the error message in the title, or the first comment, in this issue. Please create a new issue.

Other things to try:

• set environment variable NUGET_SHOW_STACK to true. There are a few temp files that NuGet create in package directories, then rename to the final file name. It would be interesting to know which one is causing this error.
  • if that doesn't work, try dotnet-trace. I know that perfview on Windows will capture stack traces on exceptions, but I'm not sure if dotnet-trace does
• try running the same dotnet publish command on your host machine, rather than within a docker container. Maybe docker is doing something weird
  • I see that the nuget packages folder is being mounted from somewhere else. Is this shared with any other builds happening at the same time? All processes using the same global packages folder MUST use the same temp, or NuGetScratch, directory, in order for NuGet's file/directory locking to work. Otherwise if two different restores happening at the same time try to extract the same package at the same time, they'll conflict and cause all sorts of problems.
• consider having the .NET SDK create the container for you, rather than building from within a docker image itself, especially if the previous step shows this only fails in docker, not on the host.