Linker to NativeAOT sync #71485

vitek-karas · 2022-06-30T11:56:39Z

This brings changes from the linker repo to ilc roughly as of a week ago. This required some non-trivial changes to the code either already in ilc or the one coming from linker:

Port of CompilerGeneratedState
- Modify it to use lock free hashtable (since it's a global cache and thus needs to be thread safe)
- In ILC it requires to operate on generic definitions of everything - so add asserts and code to guarantee this
- It's not a static class anymore, so pass the instance around as necessary
- Requires IL access, so needs ILProvider on input
Cyclic dependency between Logger and CompilerGeneratedState, one needs the instance of the other to work. For now solved by setting it through a property on CompilerGeneratedState
Split of ReflectionMethodBodyScanner into AttributeDataFlow and GenericParameterDataFlow - mostly direct port from linker
- The AttributeDataFlow works slightly differently in ilc so adopted the code there
Changes to DiagnosticContext
- Can selectively disable Trim, AOT, Single-File diagnostics - used to correctly apply RUC, RDC and RAF suppressions
Improve handling of RUC and similar warnings in ReflectionMarker
Added ILOffset to MessageOrigin
- Even though we already store file/column, the ILOffset is necessary since we use the MessageOrigin as a key in dictionary for recorded patterns and thus we need a way to make the origin unique for each callsite (and file/line/column might not be unique or if there's no PDB those will be null anyway).
Enable Equality/HashCode implementation on MessageOrigin, still no CompareTo since we don't need it for anything right now.

…s handling

ILC is multi-threaded and CompilerGeneratedState stores a global cache. So the cache has to be synchronized. This rewrites it to use LockFreeReaderHashtable.

With this it passes smoketests

vitek-karas · 2022-06-30T11:57:36Z

@tlakollo could you please review the changes to CompilerGeneratedState specifically? In general the code should be the same as in linker, but it's now thread safe, so it required quite a few refactorings.

vitek-karas · 2022-06-30T12:11:13Z

I made some changes to the linker repo to make this port easier: dotnet/linker#2873

* Move GetDiagnosticCategory into shared code * Describe the intent around usage of DiagnosticContext in a comment * Minor fix in MessageOrigin GetHasCode * Add tests for patterns which NativeAOT had trouble with Linker changes made for dotnet/runtime#71485

tlakollo

Still reviewing but leaving some comments for now

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/AttributeDataFlow.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/CompilerGeneratedNames.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/DiagnosticContext.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/FlowAnnotations.cs

src/coreclr/tools/aot/ILLink.Shared/DiagnosticId.cs

src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj

tlakollo · 2022-07-01T20:49:52Z

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Logging/MessageOrigin.cs

 #if false
-        : IComparable<MessageOrigin>, IEquatable<MessageOrigin>
+        IComparable<MessageOrigin>,


This seems intentional, but not sure I follow why we need a statement that seems to never be executed because false is always false so it should always derive from IEquatable. Is it to show a difference between Linker and NativeAOT code?

It was there before, so I kept it for now. I agree that we may eventually remove it, depends on how exact we want to be with the linker. Linker sorts all warnings before printing them out. I think it would make sense to do the same for ILC - even more so, since ILC is multi-threaded and thus more likely to produce warnings in random order. We should catch this as we port more tests over.

The #if false is equivalent to commenting out code. I used #if false for larger blocks of code here. I tend to keep commented out code for stuff that consciously wasn't ported over for now.

We track determinism as "the output files should be the same". We haven't really bothered with ordering the warnings. Sometimes it's nice to get warnings from the tool even before the tool finishes. One can just "Ctrl-C" the tool and go fix the warning if it's relevant.

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Logger.cs

vitek-karas · 2022-07-01T21:46:22Z

FYI: I will have to port the most recent changes from Sven over as well - with those it should fix the failures seen in CI. Ultimately we're hitting this bug: #85464. The linker doesn't crash due to changes from Sven, but NativeAOT currently crashes on it.

MichalStrehovsky

I'm still looking through this, but this is how much I was able to look at today (I took a day off yesterday instead of today's public holiday).

MichalStrehovsky · 2022-07-05T06:46:27Z

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/FlowAnnotations.cs

+                if (genericArgumentType.IsTypeOf(ILLink.Shared.TypeSystemProxy.WellKnownType.System_Nullable_T))
+                {
+                    var innerGenericArgument = genericArgumentType.Instantiation.Length == 1 ? genericArgumentType.Instantiation[0] : null;
+                    switch (innerGenericArgument)
+                    {
+                        case GenericParameterDesc gp:
+                            return new NullableValueWithDynamicallyAccessedMembers(genericArgumentType,
+                                new GenericParameterValue(gp, GetGenericParameterAnnotation(gp)));
+
+                        case TypeDesc underlyingType:
+                            return new NullableSystemTypeValue(genericArgumentType, new SystemTypeValue(underlyingType));
+                    }
+                }
+                // All values except for Nullable<T>, including Nullable<> (with no type arguments)
+                return new SystemTypeValue(genericArgumentType);


I see that this is a pattern copied from ILLink so I don't expect it to be fixed here, but I'm not a big fan of code handling impossible states. They tend to hide bugs and make root causing bugs a lot harder.

Two things here:

We "handle" situation when Nullable<T> doesn't have exactly one generic argument.

We "handle" when the instantiation argument is something bogus (null) - by unnecessarily using a non-exhaustive switch instead of a simple if/else.

I see this pattern in many places so I'm not going to comment on them, but it's something worth re-evaluating in the linker repo. Code that crashes when it hits invalid state is a lot easier to work with than code that papers over it and then crashes later (or worse - produces invalid outputs). We have zero test coverage for this invalid state handling.

A lot of the invalid state handling (not this one in particular) also seems to be forced by nullable reference types - I assume it wasn't easy to make something non-null so now we have code handling an impossible null scenario. Honesly, I would prefer hitting a NullReferenceException. Those exceptions are super easy to debug and often can be debugged from nothing but a stack trace. Complex code taking wrong turns is everything but easy to debug.

In the maxim "if I can't cover a branch in testing, the branch shouldn't exist". The null propagating operator ?. is a branch too. We use them very liberally in this codebase. Many of those uses look as questionable as the question mark.

There are two main reasons for some of this weird "handlings", but it doesn't excuse most of them:

Cecil's reference->definition resolution returns null if the resolution fails (missing assembly dependency, wrong dependency, ...). So unlike in NativeAOT's type system, there are potentially lot of places where null can appear unfortunately.

Historically we've seen quite a few cases where weird IL/metadata we never anticipated (like obfuscated code) crashed the linker. While I agree it's easy to diagnose the problem, there's typically absolutely no workaround for the user (other than disabling trimming which in Blazor/Xamarin scenarios is not really viable).

We are guilty of adding ?. and similar in places where we couldn't immediately reason about the input being null or not. This is exaggerated by the fact that we turned on nullable reference types (which was a good thing as it found several issues), but Cecil is not annotated, so everything looks nullable (even if in really it might not be).

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Logger.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/CompilerGeneratedCallGraph.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/AttributeDataFlow.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/CompilerGeneratedCallGraph.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/EcmaExtensions.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/FlowAnnotations.cs

MichalStrehovsky

Looks great. Just a couple opportunities to delete some lines of code.

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/CompilerGeneratedState.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/EcmaExtensions.cs

This was needed to fix failures in libraries build with AOT. The underlying problem is not fully fixed: https://github.com/dotnet/linker/issues/2874 But with these changes it doesn't crash the compiler, it just leads to sometimes imprecise warnings.

vitek-karas · 2022-07-06T19:31:42Z

The latest commit 53800b7 brings recent linker changes over. I suggest this is reviewed incrementally.

I didn't have a chance to apply PR feedback from Michal yet.

As is right now it passes smoke tests and libraries AOT build.

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMarker.cs

MichalStrehovsky · 2022-07-07T05:01:22Z

src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj

@@ -20,6 +20,11 @@
    <ProjectReference Include="..\ILCompiler.DependencyAnalysisFramework\ILCompiler.DependencyAnalysisFramework.csproj" />
    <ProjectReference Include="..\ILCompiler.MetadataTransform\ILCompiler.MetadataTransform.csproj" />
    <ProjectReference Include="..\ILCompiler.TypeSystem\ILCompiler.TypeSystem.csproj" />
+
+    <PackageReference Include="StaticCs" Version="$(StaticCsVersion)">


I think this will be a problem for source build.

It looks like this is just some analyzer with a custom attribute? I would be fine not running the analyzer in source build (or ever) and we can fake the custom attribute so that we don't have to make acrobatics in the shared files.

@agocke - could you please comment on this? I basically just copied this from linker, I don't know the details behind it.

Yup, it's an analyzer and a source file with attributes. I think we can condition it out in source build.

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs

…time into LinkerToAotSync2

MichalStrehovsky · 2022-07-14T23:59:35Z

For the /_/src/libraries/System.Reflection/tests/AssemblyTests.cs(191,0): error IL3002: (NETCORE_ENGINEERING_TELEMETRY=Build) System.Reflection.Tests.AssemblyTests.<>c__DisplayClass20_0.<GetFile_InMemory>b__2(): Using member 'System.Reflection.Assembly.GetFiles()' which has 'RequiresAssemblyFilesAttribute' can break functionality when embedded in a single-file app. This member throws an exception for assemblies embedded in a single-file app., just add IL3002 to the list here:

runtime/eng/testing/tests.singlefile.targets

Line 29 in 81a3dcd

<NoWarn>$(NoWarn);IL3050;IL3051;IL3052;IL3055;IL1005</NoWarn>

The test assemblies produce a lot of analysis warnings. We also don't care.

MichalStrehovsky

Thank you!

...at the cost of a small compat break. We return `int[]` instead of `SomeInt32Enum[]`. Fixes dotnet#72140. We can also delete intrinsic handling of `Enum.GetValues` in dataflow analysis but I don't want to conflict with Vitek's dotnet#71485.

…time into LinkerToAotSync2

If the user method has no intersting code in it, but it has a lambda (for example) which does have an interesting code which needs data flow - we now need to add the user method for data flow analysis (since the user method and all of its compiler generated methods will be analyzed together). This also adds a few smoke tests covering these scenarios.

vitek-karas · 2022-07-18T23:02:46Z

/azp run runtime-extra-platforms

azure-pipelines · 2022-07-18T23:03:08Z

Azure Pipelines successfully started running 1 pipeline(s).

vitek-karas · 2022-07-19T08:18:28Z

The CoreCLR libraries tests are failing on #72429 and probably #68289.
The extra platforms NativeAOT failure is #72149.

vitek-karas added 13 commits June 30, 2022 04:23

First

9e5dd8f

Progress

13e6565

Small fix

59e5ddd

Improvements to warning suppression and general cleanup of diagnostic…

6702417

…s handling

Progress on CompilerGeneratedState

56db859

Finish CompilerGeneratedState port (at least first pass)

b9eac18

Make the simple stuff work

177de44

Fixes

222acda

Formatting

d772322

Formatting

1cd000f

Fixes - rewrite compiler generated state

c34901f

ILC is multi-threaded and CompilerGeneratedState stores a global cache. So the cache has to be synchronized. This rewrites it to use LockFreeReaderHashtable.

Fixes

38520ce

With this it passes smoketests

Cleanup and fixes

f29afe0

vitek-karas added the area-NativeAOT-coreclr label Jun 30, 2022

vitek-karas requested a review from tlakollo June 30, 2022 11:56

vitek-karas self-assigned this Jun 30, 2022

vitek-karas requested a review from MichalStrehovsky as a code owner June 30, 2022 11:56

vitek-karas mentioned this pull request Jun 30, 2022

Small updates to help with linker->aot port dotnet/linker#2873

Merged

tlakollo reviewed Jul 1, 2022

View reviewed changes

PR Feedback - just minor tweaks

10bbc6d

MichalStrehovsky reviewed Jul 5, 2022

View reviewed changes

MichalStrehovsky reviewed Jul 6, 2022

View reviewed changes

Port almost latest linker changes

53800b7

This was needed to fix failures in libraries build with AOT. The underlying problem is not fully fixed: https://github.com/dotnet/linker/issues/2874 But with these changes it doesn't crash the compiler, it just leads to sometimes imprecise warnings.

Merge branch 'main' into LinkerToAotSync2

0da4638

vitek-karas commented Jul 6, 2022

View reviewed changes

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs Outdated Show resolved Hide resolved

tlakollo reviewed Jul 7, 2022

View reviewed changes

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMarker.cs Show resolved Hide resolved

MichalStrehovsky reviewed Jul 7, 2022

View reviewed changes

vitek-karas added 3 commits July 14, 2022 12:47

Mostly PR feedback and small cleanup

af6d00b

Merge branch 'LinkerToAotSync2' of https://github.com/vitek-karas/run…

3a48d93

…time into LinkerToAotSync2

Merge branch 'main' into LinkerToAotSync2

db89134

MichalStrehovsky approved these changes Jul 15, 2022

View reviewed changes

MichalStrehovsky mentioned this pull request Jul 15, 2022

Make Enum.GetValues AOT-safe #72236

Closed

vitek-karas added 3 commits July 15, 2022 12:23

Ignore single-file warnings in AOT tests

d39a7f7

Merge branch 'LinkerToAotSync2' of https://github.com/vitek-karas/run…

381e077

…time into LinkerToAotSync2

runfoapp bot mentioned this pull request Jul 19, 2022

system.net.quic.functional.tests failing with stack buffer overflow #72429

Closed

vitek-karas mentioned this pull request Jul 19, 2022

matrix4x4createfromaxisangletest failing on NativeAOT #72149

Closed

vitek-karas merged commit 56b22b5 into dotnet:main Jul 19, 2022

vitek-karas deleted the LinkerToAotSync2 branch July 19, 2022 08:19

MichaelSimons mentioned this pull request Jul 28, 2022

Resolve StaticCS.0.1.0 source-build prebuilt #73025

Closed

MichalStrehovsky mentioned this pull request Jul 28, 2022

[NativeAOT] Assert when compiling System.Runtime.Tests #73027

Closed

ghost locked as resolved and limited conversation to collaborators Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linker to NativeAOT sync #71485

Linker to NativeAOT sync #71485

vitek-karas commented Jun 30, 2022

vitek-karas commented Jun 30, 2022

vitek-karas commented Jun 30, 2022

tlakollo left a comment

tlakollo Jul 1, 2022

vitek-karas Jul 4, 2022

MichalStrehovsky Jul 6, 2022

vitek-karas commented Jul 1, 2022

MichalStrehovsky left a comment

MichalStrehovsky Jul 5, 2022

vitek-karas Jul 5, 2022

MichalStrehovsky left a comment

vitek-karas commented Jul 6, 2022

MichalStrehovsky Jul 7, 2022

vitek-karas Jul 14, 2022

agocke Jul 15, 2022 •

edited

Loading

MichalStrehovsky commented Jul 14, 2022

MichalStrehovsky left a comment

vitek-karas commented Jul 18, 2022

azure-pipelines bot commented Jul 18, 2022

vitek-karas commented Jul 19, 2022

Linker to NativeAOT sync #71485

Linker to NativeAOT sync #71485

Conversation

vitek-karas commented Jun 30, 2022

vitek-karas commented Jun 30, 2022

vitek-karas commented Jun 30, 2022

tlakollo left a comment

Choose a reason for hiding this comment

tlakollo Jul 1, 2022

Choose a reason for hiding this comment

vitek-karas Jul 4, 2022

Choose a reason for hiding this comment

MichalStrehovsky Jul 6, 2022

Choose a reason for hiding this comment

vitek-karas commented Jul 1, 2022

MichalStrehovsky left a comment

Choose a reason for hiding this comment

MichalStrehovsky Jul 5, 2022

Choose a reason for hiding this comment

vitek-karas Jul 5, 2022

Choose a reason for hiding this comment

MichalStrehovsky left a comment

Choose a reason for hiding this comment

vitek-karas commented Jul 6, 2022

MichalStrehovsky Jul 7, 2022

Choose a reason for hiding this comment

vitek-karas Jul 14, 2022

Choose a reason for hiding this comment

agocke Jul 15, 2022 • edited Loading

Choose a reason for hiding this comment

MichalStrehovsky commented Jul 14, 2022

MichalStrehovsky left a comment

Choose a reason for hiding this comment

vitek-karas commented Jul 18, 2022

azure-pipelines bot commented Jul 18, 2022

vitek-karas commented Jul 19, 2022

agocke Jul 15, 2022 •

edited

Loading