Add partial packet detection and fixup #2714
Conversation
|
I've added comments to the Packet class as requested. The CI was green apart from some ubuntu legs which timed out, many other ubuntu legs succeeded so I don't see any direct inference on from that. Ready for review @David-Engel @saurabh500 @cheenamalhotra |
|
@Wraith2 We are reviewing this and hope to get faster traction towards EOW. |
|
Pasting test failure for reference: This test should be looked at carefully. It failed on Ubuntu with .NET 6 and 8 , and also hung up on Windows when ran with Managed SNI, link to logs 1 link to logs 2. @Wraith2 can you confirm if this is something you're able to repro in Windows with Managed SNI? Please make sure config file is configured to enable Managed SNI on Windows. |
| { | ||
| // Do nothing with callback if closed or broken and error not 0 - callback can occur | ||
| // after connection has been closed. PROBLEM IN NETLIB - DESIGN FLAW. | ||
| return; |
There was a problem hiding this comment.
Can you add a Debug Assert here and check if this is taking any hit?
|
Isn't this the set of tests that @David-Engel pointed out in #2608 (comment) ? If so we discussed it at length on the teams call. I don't believe that those tests are reliable. Setup a breakpoint or Debug.WriteLine where an exception is added to the state object and run the test. You should find that an exception is always added to the state object but that the test will usually succeed. That should not be possible, an exception if added should be thrown. The test is missing failures and if that's the case then the test is unreliable. |
|
When you work past the terrible code in SNITCPHandle and make the test run for long enough it settles into a steady state where it can't reach the end. There is no indication why yet. those 9 in flight items just don't seem to complete but i don't know why. |
|
After a few more hours investigation I know what the problem is but I have no clue what change has caused it. In SqlDataReader when an async method is called we use a context object to contain some state and pass that context object to all the async methods that are used to implement the async read machinery. Part of this state is the TaskCompletionSource. What I don't understand is how cancel is supposed to work. I'm unable to run the tests in native sni mode because the native sni can't be initialized (can't find the sni dll). So I can't compare the managed to unmanaged implementations here. I don't believe that I have made any change that should affect cancellation. I have verified that there are no partial packets in the state objects when the async tasks get stuck. I don't understand how async cancellation is supposed to work at all. |
Wraith2
force-pushed
the
operation-status-part2
branch
from
88fbb93
to
8b57818
Compare
|
Can someone with CI access rerun the failed legs? the failures are random or CI resources not being available as far as i can tell. |
|
The current failures are interesting. They're in the test that was failing before but they new ones are only detected because i made the test more accurate. The previous version of the test accepted any exception when it was expecting a cancellation exception. It was passing on netfx with my previous changes because timeout exceptions were being thrown. I judged that accepting a timeout when we were supposed to be testing whether cancellation had occurred was not correct. If we retained the previous version of the test then everything would have passed cleanly. In the current situation since the test completed correctly without hanging the result is equivalent to what we would have experienced in all test runs in the past, all started threads that we expected to be cancelled exited with an exception. [edit] |
|
@Wraith2 You might be banging your head against an unrelated issue in the driver. IIRC, the test was only introduced to ensure we don't regress "The MARS TDS header contained errors." issue. (The test code came from the repro.) If you isolate your test changes and run them against main code, does it still fail? Yes, the correct exception is probably "Operation cancelled by user." where the exception is being caught. But if it's unrelated to your other changes, I would leave that part of the test as it was and file a new issue with repro code. As it is, it's unclear if and how this behavior is impacting users and I wouldn't hold up your perf improvements for it. |
|
There was definitely a real problem. The results differed between main and my branch. I've solved that issue now and the current state is that we're seeing a real failure because I've made the test more sensitive. I think it's probably safe to lower the sensitivity of the test again now because the new test that I've added covers the specific scenario in the multiplexer that I had missed and everything else is pass. I'll try that and see how the CI likes it. I think the current state on this branch is that it is as stable as live. We need to have confidence that this set of changes is correct before we can merge it. It's high risk and high complexity code. Even understanding it very deeply it has taken me a week to actively debug a very important behaviour change that I missed. |
|
Can someone re-run the failed legs? the only failing test is something to do with event counters which I've been no-where near. |
|
The failing test is EventCounter_ReclaimedConnectionsCounter_Functional. It's doing something with GC specific to net6. It's failing sporadically on net6 managed sni runs but not deterministically. I can't make it fail locally to trace what might be happening. |
|
Any thoughts? |
|
I'm not seeing the failures you mentioned in EventCounter_ReclaimedConnectionsCounter_Functional [in the CI results]. I mainly see fairly consistent failures of CancelAsyncConnections on Linux. It seems to pass on Windows managed SNI, so there might be something that is Linux/Ubuntu network specific. Can you run/debug the test against a local WSL or Docker instance? |
|
If i click through the failure i get to this page https://sqlclientdrivers.visualstudio.com/public/_build/results?buildId=95784&view=ms.vss-test-web.build-test-results-tab The cancel tests are passing now, those failed in the previous runs but not the current ones. |
|
I think there is something wrong with the Tests tab. I don't usually reference it. I scroll down the summary tab to see what jobs had failures:
Then drill into the job and the task that failed to see the log:
|
|
If it's AsyncCancelledConnectionsTest again then there isn't anything further I can do. That test is multithreaded and timing dependent. I've traced the individual packets through the entire call stack. I've run it for 1000 iterations successfully after fixing a reproducible error in it. If someone can isolate a reproducible problem from it then i'll investigate. |
|
I chatted with @saurabh500 and I just want to add that this is definitely something we all want to see get merged. It'll just take someone finding time (could take a few days dedicated time) to get their head wrapped around the new code and be able to help repro/debug to find the issue. |
|
I'm happy to make myself available to talk through the code with anyone that needs it. |
|
@Wraith2 and @David-Engel I was looking at the lifecycle of the snapshots and something that stood out in NetCore vs NetFx is that SqlDataReader for NetCore is storing the cached snapshot with the SqlInternalConnectionTds which is a shared resource among all the SqlDataReader(s) running on a MARS connection. This means that we are saving the reader snapshot on the shared resource, which can be overwritten by any other reader. @Wraith2 have you had a chance to pursue this line of investigation for hanging test? I wonder if the timing is causing the wrong cached snapshot to be provided to a SqlDataReader, causing data corruption and likely causing a hang. |
|
SqlInternalConnection.cs |
|
@Wraith2 I see that you had made the changes in the first place. Can you try another PR where you remove the storage of these contexts and snapshots on SqlInternalConnection and with the multiplexing change, try to see if this solves the problem. Also, I am Happy to be told that my theory is wrong, but I would like to understand how in MARS cases, the shared Cached contexts on InternalConnection is a safe design choice. |
|
Thanks both. I always have this problem. I manage the complex stuff ok but make silly errors, in this case omitting a Use of MichealZ, is there any chance you could reset the state of the SqlClient repo? on the machine I've been using? I tried getting it up to date with the current repo when i pushed previous changes but it somehow ended up identifying changes in >100 files from netfx, not sure what went wrong... |
Sure, give me a few minutes. I always have this problem as well, not sure why :) |
Done |
|
Changes pushed, lets see what the CI thinks now. This includes your Wraith2#5 PR @edwardneal Thanks @MichelZ ! |
I've just tested |
|
I've just identified the problem with the Here's what I found: The connection timeout when running the test suite is infinite. I've hit problems with that before where I've forgotten to start the sql server and the tests don't fail they just hang indefinitely (and confusingly). I believe that we've hit another manifestation of #422 where there just aren't enough resources to do everything and stuff stops working, one of the thing we've seen in 422 is that things that should succeed or error just hang. Combine that with the fact that it's linux networking and it can only be reproduced on low processor count machines and you've got enough suggestive correlation to convince me. I still cannot reproduce the failure at all on any machine I own even in ubuntu limited to 2 processors. Can anyone else think of any reason this can't be the case? Can anyone think of a way to validate it? I can't get from the DbInternalConnection instance in the debugger to the TdsInternalConnection to inspect any of the TdsStateParserObject state to confirm that the multiplexer related state it contains is consistent. However since we're looking at login code here if there were a problem with it then it would be quite likely to be hit in every other test that requires a connection, which is everything in the manual testing project. I've attached the updated test file with the tracker and time changed so you can try to replicate and inspect. |
|
Building on that, I've added a couple more tracking points: AsyncCancelledConnectionsTest.zip I think the connection was closed because it was reset already.
From the image above we can see that it has done All items that remain in the tracker feature the same pattern. |
|
I've gathered a set of memory dumps and I'm fairly certain you're correct - it's thread pool exhaustion. I've left process 405155 running on the testbed @MichelZ, along with the various dotnet-dump/dotnet-gcdump/etc. utilities. I have a heap dump which includes the TdsParserStateObject instances, and I'm trying to load the PDBs so that I can interpret it. In my hanging instance, I can see 8 thread pool threads. One is in use by XUnit, another is in use by the thread waiting on Parallel.ForEach; in both cases, these are waiting on the other six. I think Parallel.ForEach has started four instances, (or at least, I can see four TdsParser/TdsParserStateObjectManaged/SNITCPHandle instances) so there's not a lot of room for manoeuvre if we hit a code path which runs async-over-sync, and it looks like this is what happened - it the remaining six seem to have deadlocked on one another. I haven't yet tested with a wider thread pool, but I'll do that after work. |
|
For what it's worth, I can still reproduce it when setting MinThreads to 32, altough less Tasks seem to hang (i.e. I get 3-8 during the default MinThreads, and 1-2 with 32 MinThreads) I can see that a callback is registered here (code executed in test): But the Continuation is never executed for the hanging tasks here: Maybe that helps getting to the bottom of it? It also seems that most calls get short-circuited and completed here, without needing a continuation: e.g. in my latest test (with 32 MinThreads) I have 3 Register Continuation, and 1 CompleteAsyncCallback, aka 2 hanging. |
|
Your traces concerned me, they indicated that the connection was somehow open which would invalidate my theory. So the connection is initially closed at the preopen point, then transitions from closed to open (the |
|
I have no idea if I'm on the wrong track or not, but I have added the following simple code after this line here: (also capturing the Unwrapped task with Which aborts the Async operation after 1 minute. It also means that the Or I could be completely off of course and it has nothing to do with that |
|
Yes, that means it waiting on network data. A way to catch that read being started in the debugger would be ideal. |
I tested |
|
So what I can see from a packet capture is that for "hanging" tasks, the packets are: You can find the full capture here of all tasks: In this case there were 3 hanging Tasks, 3, 50 and 51. Here is the memory dump The _partialPacket._buffer is: indicating the 0xFD20 DONE DONE_ATTN arrived. |
|
So it's stuck on a network read with a non-null partial packet? |
I'm not sure. I can't see any related stacks in the Parallel Stacks view that would suggest it's waiting on the network. |
|
Here's the memory dump |
|
I can't see any tasks at all when I break into the stuck test with the debugger. It's all very complicated and very confusing. |
|
After cleaning up the debugging i've found that all the stuck connections have an _pendingCallbacks==2 which means 1 from the ctor and 1 from an async read and that there is a complete packet stuck in the _partialPacket. Now to work out how we got there... |
|
Pull and try the latest please. |
|
This has definitely improved things. |
|
Additionally, I do see a callstack in the Parallel Stacks view now that is waiting here: |
|
So this hang seems to be cause by the Debug build from the Assert of This causes this call to fail: Which means that the My fix for this is to use another try/finally block to encapsulate everything in the outer finally block - right after the While this makes the test work, I wonder if that gets ignored here, because it's a is this the correct way of doing it? |
|
How did we end up with an unbalanced pendingCount? all the locations I changed I attempted to maintain the invariants like that by changing as little as possible. Even with a read from the partialPacket the count should be incremented and decremented correctly. |
|
Also note the comment in the function: Tests in this repo are not normally run in Debug mode so this could have existed and been happening for a long time and we wouldn't see it. I've got the test running through your script in a loop with this PR changes and i'll see if it gets to 100. If it does then we'll see what people think. |
|
I've also managed to get a with this Stack trace: This again should only affect Debug builds as far as I can see, but probably still worth fixing? |
|
Yes, worth fixing even if it only explains the invariants at that point in the code. Changed to: |
Split out from #2608 per discussion detailed in #2608 (comment)
Adds packet multiplexer and covering tests.