Fix race condition in EnhancedHttpRetryHelper

[akoeplinger]

Bug

Fixes: NuGet/Home#13545

Regression? Last working version: 9.0-preview5

Description

When multiple threads try to access the RetryCount property we sometimes hit a case where RetryCount returned 0 (*).
This caused the loop in ProcessHttpSourceResultAsync() to be skipped completely because retry <= maxRetries:

NuGet.Client/src/NuGet.Core/NuGet.Protocol/Utility/FindPackagesByIdNupkgDownloader.cs

Lines 258 to 328 in 6c642c2

	int maxRetries = _enhancedHttpRetryHelper.IsEnabled ? _enhancedHttpRetryHelper.RetryCount : 3;
	for (var retry = 1; retry <= maxRetries; ++retry)
	{
	var httpSourceCacheContext = HttpSourceCacheContext.Create(cacheContext, isFirstAttempt: retry == 1);
	try
	{
	return await _httpSource.GetAsync(
	new HttpSourceCachedRequest(
	url,
	"nupkg_" + identity.Id.ToLowerInvariant() + "." + identity.Version.ToNormalizedString(),
	httpSourceCacheContext)
	{
	EnsureValidContents = stream => HttpStreamValidation.ValidateNupkg(url, stream),
	IgnoreNotFounds = true,
	MaxTries = 1,
	IsRetry = retry > 1,
	IsLastAttempt = retry == maxRetries
	},
	async httpSourceResult => await processAsync(httpSourceResult),
	logger,
	token);
	}
	catch (TaskCanceledException) when (retry < maxRetries)
	{
	// Requests can get cancelled if we got the data from elsewhere, no reason to warn.
	var message = string.Format(CultureInfo.CurrentCulture, Strings.Log_CanceledNupkgDownload, url);
	logger.LogMinimal(message);
	}
	catch (Exception ex) when (retry < maxRetries)
	{
	var message = string.Format(
	CultureInfo.CurrentCulture,
	Strings.Log_FailedToDownloadPackage,
	identity,
	url)
	+ Environment.NewLine
	+ ExceptionUtilities.DisplayMessage(ex);
	logger.LogMinimal(message);
	if (_enhancedHttpRetryHelper.IsEnabled &&
	ex.InnerException != null &&
	ex.InnerException is IOException &&
	ex.InnerException.InnerException != null &&
	ex.InnerException.InnerException is System.Net.Sockets.SocketException)
	{
	// An IO Exception with inner SocketException indicates server hangup ("Connection reset by peer").
	// Azure DevOps feeds sporadically do this due to mandatory connection cycling.
	// Delaying gives Azure more of a chance to recover.
	logger.LogVerbose("Enhanced retry: Encountered SocketException, delaying between tries to allow recovery");
	await Task.Delay(TimeSpan.FromMilliseconds(_enhancedHttpRetryHelper.DelayInMilliseconds), token);
	}
	}
	catch (Exception ex) when (retry == maxRetries)
	{
	var message = string.Format(
	CultureInfo.CurrentCulture,
	Strings.Log_FailedToDownloadPackage,
	identity,
	url)
	+ Environment.NewLine
	+ ExceptionUtilities.DisplayMessage(ex);
	logger.LogError(message);
	}
	}
	return await processAsync(null);

Fix this by using Lazy<T> for initializing the fields which is thread-safe.

(*) The reason is that code like int RetryCount => _retryCount ??= 6; gets turned into:

int valueOrDefault = _retryCount.GetValueOrDefault();
if (!_retryCount.HasValue)
{
    valueOrDefault = 6;
    _retryCount = valueOrDefault;
    return valueOrDefault;
}
return valueOrDefault;

Suppose Thread A arrives first and calls GetValueOrDefault() (which is 0 for int) but then Thread B interjects and sets the value, now when Thread A resumes _retryCount.HasValue is true so we skip the if block and return valueOrDefault i.e. 0.

PR Checklist

• PR has a meaningful title

• PR has a linked issue.

• Described changes

• Tests

  • Automated tests added
  • OR
  • Test exception
  • OR
  • N/A

• Documentation

  • Documentation PR or issue filled
  • OR
  • N/A

[donnie-msft]

Thanks for the contribution! I'm running our tests pipeline now...

[nkolev92]

🤝

[wuhkuh]

_maxRetyAfterDelay -> _maxRetryAfterDelay

[akoeplinger]

thanks: #6025