The proportion of timeouts increases after migrating project from .net6 to .net8

[Coke-Cole]

We are encountering more timeout after migrating project from .net6 to .net8.

Here is the error message in .net8:

StackExchange.Redis.RedisTimeoutException: Timeout performing TTL (1000ms), next: DECR ********_Query_count_rate_limit, inst: 9, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 33, last-in: 2, cur-in: 0, sync-ops: 8950, async-ops: 1, serverEndpoint: xxxxxxxxxxxxxx.redis.cache.windows.net:6380, conn-sec: 30705.42, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: AW0SDWK000000000000000(SE.Redis-v2.7.33.41805), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), POOL: (Threads=7,QueuedItems=4,CompletedItems=212384,Timers=17), v: 2.7.33.41805 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Message indicates that the worker was in busy despite that there are lots of free worker. If we don't define the minimal thread pool number, the number of worker equals the number of cpu core.

In .net6, if the default workers are in busy state, it usually return RedisConnectionException due to SocketClosed, while in .net8, all RedisConnectionException become RedisTimeoutException, I suppose this because .net8 change the action when create new thread and .net8 need more delay every time the client program starts a new thread (due to .NET thread pool throttle the creation of new threads) which doesn't match our expectation for the smooth migration of upgrading to .net8.

Despite we can mitigate this issue by set the number of minimal thread pool larger, However, this still shows that this version is not better compatible with the upgrade .net6 to net8

[NickCraver]

It seems like you're running 100% sync operations against Redis, have you tried making async calls here to not tie up thread pool especially in spikes?

[dxynnez]

Hi @NickCraver , we understand the sync operation is something we definitely should improve; however, we don't really understand why only upgrading to .net 8 would cause such a big difference as nothing else was changed - does the thread pool implementation changed in .net 7 / net 8 (windows and not AOT)?

I also see a similar issue posted in the runtime repo a while back - dotnet/runtime#102312

[NickCraver]

Yes, the thread pool implementation changed in that time window (and has gone completely portable in .NET9+ I believe). Though this is a net win for the vast majority of use cases, it's not 100% and some setups could see worse behavior due to how they're locking. At 1 second, it can also be server lag and the thread pool getting more performant.

This may seem counter-intuitive, but when the timer fires more reliably, the check for timeouts can run more promptly (e.g. every 1 second in your case) and the evaluation of what's timed out just triggers more reliably, resulting in more errors, though there hasn't been an increase in overall request duration.