NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) #11723

jkotas · 2022-11-28T04:35:48Z

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=300923&view=results

Build leg reported

Build / Installer Build and Test coreclr windows_x86 Debug / Build

Pull Request

dotnet/runtime#78801

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

Open the failing build above and investigate
Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

Add the label "Known Build Error"
Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Failed to retrieve information about",
   "BuildRetry": true
}

Release Note Category

Feature changes/additions
Bug fixes
Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build	Definition	Step Name	Console log	Pull Request
2571095	dotnet-sdk	Build	Log	#44480
853704	dotnet/sdk	Build	Log	dotnet/sdk#44455
2567469	dotnet-sdk	Build	Log	#44224
2567133	dotnet-sdk	Build	Log	#44177
850902	dotnet/runtime	Build product sources	Log	dotnet/runtime#109140
2566966	dotnet-sdk	Build	Log
2566964	dotnet-sdk	Build	Log
2566676	dotnet-sdk	Build	Log
2566108	dotnet-sdk	Build	Log	#44060
2566073	dotnet-sdk	Build	Log	#44078
2565908	dotnet-sdk	Build	Log
2565752	dotnet-sdk	Build	Log	#44060
2565627	dotnet-sdk	Build	Log	#44025
2565267	dotnet-sdk	Build	Log
2565269	dotnet-sdk	Build	Log
2563936	dotnet-sdk	Build	Log	#43939
2563791	dotnet-sdk	Build	Log	#43939
2561706	dotnet-sdk	Build	Log
2561577	dotnet-sdk	Build	Log
840005	dotnet/runtime	Build product	Log	dotnet/runtime#107378
839009	dotnet/runtime	Build product	Log	dotnet/runtime#108767
2557030	dotnet-runtime	Build product	Log
837380	dotnet/sdk	Build	Log	dotnet/sdk#44050
836969	dotnet/runtime	Build product	Log
835859	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#57923
835843	dotnet/runtime	Build managed product components and packages	Log	dotnet/runtime#108495
835918	dotnet/machinelearning	Build	Log
835820	dotnet/sdk	Build	Log	dotnet/sdk#44028
835861	dotnet/aspnetcore	Build shared fx	Log	dotnet/aspnetcore#57923
835587	dotnet/sdk	Build	Log	dotnet/sdk#43987
2554616	dotnet-installer	Build	Log
2554615	dotnet-installer	Build	Log
2554570	dotnet-sdk	Build	Log
2554558	dotnet-installer	Build	Log
2554557	dotnet-installer	Build	Log
2554542	dotnet-installer	Build	Log
2554541	dotnet-installer	Build	Log
2553037	dotnet-installer	Build	Log
2552045	dotnet-dotnet-monitor	Build	Log	#43336
2552039	dotnet-dotnet-monitor	Build	Log	#43335

Build	Definition	Test	Pull Request
2565939	dotnet-sdk	Microsoft.NET.Publish.Tests.dll.1.WorkItemExecution	#44070
2563197	dotnet-runtime	x64.micro.net8.0.Partition17.WorkItemExecution
2560638	dotnet-runtime	release.Partition28.WorkItemExecution
2560645	dotnet-runtime	release.Partition15.WorkItemExecution
2560239	dotnet-runtime	release.Partition14.WorkItemExecution
2554547	dotnet-dotnet	Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario
2554528	dotnet-dotnet	Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario
2551307	dotnet-runtime	x64.micro.net6.0.Partition25.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	48

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=300923
Result validation: ✅ Known issue matched with the provided build.

andriipatsula · 2022-11-28T13:32:51Z

Updated an ErrorMessage

jkotas · 2022-11-28T15:12:54Z

Updated an ErrorMessage

This error message is too specific. The package name does not matter for 503 errors. You can see it in the log file attached to this issue. The build failure was caused by multiple different packages failing to be retrieved - for example, look for Failed to retrieve information about 'Microsoft.Build.Tasks.Git'.

andriipatsula · 2022-11-28T15:17:35Z

Yeh, most probably Failed to retrieve information about pattern should work.

ulisesh · 2022-12-06T00:22:13Z

15 hits in the last 24 hours, I'm going to enable build retry

MattGal · 2022-12-07T20:05:00Z

Created IcM to ask for investigation here as well, https://portal.microsofticm.com/imp/v3/incidents/details/353857134/home

ulisesh · 2022-12-09T21:22:50Z

More hits today.

Update from the IcM: "Will check throttling limits to see if that's kicking in here"

epananth · 2023-02-06T23:07:22Z

I updated the ICM and received no response from them.

dougbu · 2023-02-16T23:47:21Z

Where are 500s from the public AzDO feeds tracked❔ I don't see efcore-ci build #20230213.2 mentioned here for example. That build matches the ErrorMessage in the description above.

ulisesh · 2023-02-16T23:50:00Z

Does the EFCore repo uses build analysis? Known issues tracking is part of build analysis

dougbu · 2023-02-16T23:52:17Z

Might need a separate Known Build Error for the dotnet-public-npm feed errors we're seeing in dotnet/aspnetcore (a lot). For example,

  error An unexpected error occurred: "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-npm/npm/registry/istanbul-lib-report/-/istanbul-lib-report-3.0.0.tgz: Request failed \"500 Internal Server Error\"".
  info If you think this is a bug, please open a bug report with the information provided in "/Users/runner/work/1/s/src/SignalR/clients/ts/common/yarn-error.log".
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
  Process stalled
  Active handles:
    - Socket
    - Socket
    - Socket
    - TLSSocket
    - TLSSocket
    - TLSSocket
    - TLSSocket
    - TLSSocket
/Users/runner/work/1/s/eng/targets/Npm.Common.targets(45,5): error MSB6006: "yarn" exited with code 1. [/Users/runner/work/1/s/src/SignalR/clients/ts/common/common.npmproj]
##[error]eng/targets/Npm.Common.targets(45,5): error MSB6006: (NETCORE_ENGINEERING_TELEMETRY=Restore) "yarn" exited with code 1.

Affected aspnetcore-ci rolling builds over the last week:

dougbu · 2023-02-16T23:53:51Z

Does the EFCore repo uses build analysis? Known issues tracking is part of build analysis

No.

/cc @bricelam and @ajcvickers because I don't remember why they didn't opt in and don't know if those reasons remain valid.

MattGal · 2023-02-16T23:55:08Z

@dougbu this one's on me, the NPM failures you're talking about are #11737. While the IcM for it is unanswered, the known issues system said no repros for a week so I had closed it. Feel free to reactivate. It won't make Azure Devops any more responsive, but it will keep it on our radar.

dougbu · 2023-02-16T23:59:43Z

Would that issue have picked up the efcore-ci failure given it occurred in a rolling build (where build analysis isn't relevant) for a repo where build analysis isn't enabled anyhow❔

dougbu · 2023-02-17T00:01:06Z

In any case, the ErrorMessage above means this issue will capture a superset of those tracked in #11737. I'll reopen the other one just because 500 and 503 aren't the same number 😁

dougbu · 2023-02-17T00:02:20Z

Oh, ignore my last comment. You were talking about the dotnet-public-npm failures. Right @MattGal❔

dougbu · 2023-02-17T00:10:01Z

Side note: The Known Build Errors infrastructure is tilted pretty far toward PR failures. It's far too manual (AFAICT) to use for rolling build failures and the https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/f0702582-7c04-47ca-a145-6ac37fd25813/ReportSectioncb62e8e5baebca8883e1?experience=power-bi doesn't seem to automatically report matches for such issues (nor help enough in creating them).

MattGal · 2023-02-17T00:16:54Z

Oh, ignore my last comment. You were talking about the dotnet-public-npm failures. Right @MattGal❔

Correct I was keying off you talking about "pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-npm/npm/registry/istanbul-lib-report/-/istanbul-lib-report-3.0.0.tgz" since that's not nuget-y

markwilkie · 2023-02-21T16:18:11Z

Side note: The Known Build Errors infrastructure is tilted pretty far toward PR failures. It's far too manual (AFAICT) to use for rolling build failures and the https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/f0702582-7c04-47ca-a145-6ac37fd25813/ReportSectioncb62e8e5baebca8883e1?experience=power-bi doesn't seem to automatically report matches for such issues (nor help enough in creating them).

@AlitzelMendez - any feedback here?

epananth · 2023-04-06T19:45:35Z

@ilyas1974 assigning this to you, cos you were looking at a similar issue

AlitzelMendez · 2023-05-05T17:12:21Z

Side note: The Known Build Errors infrastructure is tilted pretty far toward PR failures. It's far too manual (AFAICT) to use for rolling build failures and the https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/f0702582-7c04-47ca-a145-6ac37fd25813/ReportSectioncb62e8e5baebca8883e1?experience=power-bi doesn't seem to automatically report matches for such issues (nor help enough in creating them).

@AlitzelMendez - any feedback here?

so after some months I saw this comment :) I think we explored this option in the past but I never found the dashboard in which we wanted to add a link to create known issue, but I think is a great and reasonable idea. reopening: #8794

jkotas added Ops - First Responder Detected By - Customer Issue was reported by a customer Known Build Error labels Nov 28, 2022

jkotas mentioned this issue Nov 28, 2022

Convert pointers in a couple tables to relative dotnet/runtime#78801

Merged

Chrisboh assigned jonfortescue Dec 12, 2022

ilyas1974 assigned epananth and unassigned jonfortescue Feb 1, 2023

dougbu mentioned this issue Feb 16, 2023

HTTP 503s from Azure Devops Test Result API in dnceng-public/public #11872

Closed

5 tasks

epananth assigned ilyas1974 and unassigned epananth Apr 6, 2023

ilyas1974 added Build Analysis Retry Known issue that is not actionable but needs to stay open to enable re-try logic. and removed Ops - First Responder Detected By - Customer Issue was reported by a customer labels May 18, 2023

ilyas1974 removed their assignment May 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) #11723

NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) #11723

jkotas commented Nov 28, 2022 •

edited by build-analysis bot

Loading

andriipatsula commented Nov 28, 2022

jkotas commented Nov 28, 2022

andriipatsula commented Nov 28, 2022

ulisesh commented Dec 6, 2022

MattGal commented Dec 7, 2022

ulisesh commented Dec 9, 2022

epananth commented Feb 6, 2023

dougbu commented Feb 16, 2023

ulisesh commented Feb 16, 2023

dougbu commented Feb 16, 2023

dougbu commented Feb 16, 2023

MattGal commented Feb 16, 2023

dougbu commented Feb 16, 2023

dougbu commented Feb 17, 2023

dougbu commented Feb 17, 2023

dougbu commented Feb 17, 2023

MattGal commented Feb 17, 2023

markwilkie commented Feb 21, 2023

epananth commented Apr 6, 2023

AlitzelMendez commented May 5, 2023

NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) #11723

NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) #11723

Comments

jkotas commented Nov 28, 2022 • edited by build-analysis bot Loading

Build

Build leg reported

Pull Request

Action required for the engineering services team

Release Note Category

Release Note Description

Additional information about the issue reported

Report

Summary

Known issue validation

andriipatsula commented Nov 28, 2022

jkotas commented Nov 28, 2022

andriipatsula commented Nov 28, 2022

ulisesh commented Dec 6, 2022

MattGal commented Dec 7, 2022

ulisesh commented Dec 9, 2022

epananth commented Feb 6, 2023

dougbu commented Feb 16, 2023

ulisesh commented Feb 16, 2023

dougbu commented Feb 16, 2023

dougbu commented Feb 16, 2023

MattGal commented Feb 16, 2023

dougbu commented Feb 16, 2023

dougbu commented Feb 17, 2023

dougbu commented Feb 17, 2023

dougbu commented Feb 17, 2023

MattGal commented Feb 17, 2023

markwilkie commented Feb 21, 2023

epananth commented Apr 6, 2023

AlitzelMendez commented May 5, 2023

jkotas commented Nov 28, 2022 •

edited by build-analysis bot

Loading