Files uploaded via arcade log commands aren't included in SDL scans by 1ES

[akoeplinger]

• This issue is blocking
• This issue is causing unreasonable pain

Arcade uploads assets via AzDO log commands here:

arcade/src/Microsoft.DotNet.Build.Tasks.Feed/src/PushToBuildStorage.cs

Lines 302 to 319 in 7808236

	// Push to AzDO artifacts storage
	switch (itemType)
	{
	case ItemType.AssetManifest:
	Log.LogMessage(MessageImportance.High,
	$"##vso[artifact.upload containerfolder=AssetManifests;artifactname=AssetManifests]{path}");
	break;
	case ItemType.PackageArtifact:
	Log.LogMessage(MessageImportance.High,
	$"##vso[artifact.upload containerfolder=PackageArtifacts;artifactname=PackageArtifacts]{path}");
	break;
	case ItemType.BlobArtifact:
	Log.LogMessage(MessageImportance.High,
	$"##vso[artifact.upload containerfolder=BlobArtifacts;artifactname=BlobArtifacts]{path}");
	break;

The problem is that 1ES PT doesn't see these and so it doesn't inject running the SDL on these artifacts. The 1ES team is aware and will fix it at some point:

Hey! We know about this. We are working on enforcing scanning for this scenario as well. Please use the output or 1ES publish task for now.

Not sure if we can wait until 1ES addresses it or if we need to do some stop-gap solution for now to be compliant.

[mmitche]

This changes required for this are non-trivial (and somewhat unknown at this point). Also, fixing this will not run SDL tooling over the binaries that .NET typically ships. Binskim does not unpack archives. .NET deals in archives.

[mmitche]

I don't think this is blocking because solving it will not solve the issue at hand

[akoeplinger]

Would it work if instead of uploading via the log command the PushToBuildStorage task would copy the assets to some location that we later upload via 1ES.PublishPipelineArtifact@1? We could also unpack the archives as part of that until 1ES handles that

[mmitche]

I don't think we want every artifact loose coming out of the pipeline. For runtime that might be a 100G of data in addition to the normal set of data. That is not a good workaround IMO.

The other problem is that using PublishPipelineArtifact doesn't work with the way we publish today. Maybe we could get away with PublishBuildArtifacts, but today we join all artifacts together in PackageArtifacts and BlobArtifacts. The downstream Maestro publishing only knows how to go and find those.

The current setup is built on the logging commands. When you pass -publish to your build leg, it figures out what artifacts are to be published (based on msbuild project properties and ItemsToPushToBlobFeed), creates a manifest for it, and then uses the logging commands to immediately upload these artifacts. The manifest goes to AssetManifests and the rest go to PackageArtifacts and BlobArtifacts. Then, when all of your jobs that produce outputs that go to customers or to other repos are complete, you have a join job that takes all those asset manifests, merges them together, and then publishes the metadata to BAR. Then if the build should go to a channel, Maestro publishing comes along, downloads the merge manifest and artifacts, and pushes them to feeds. These steps require that you know the location of those artifacts.

What makes this especially non-trivial to solve is that you may need to change hundreds of branches to switch. Let's say you wanted to use build artifacts still. So the -publish switch creates a manifest and stages a PackageArtifacts, BlobArtifacts, and AssetManifests dir with the contents. You then need to go and add an explicit upload step to all the pipelines to upload this data. This might be possible in templates, but I'm not sure it will cleanly apply everywhere.

If you want to use pipeline artifacts, then things get hairy, because all jobs would have to have a unique artifact name, and you'd need to propagate that through the system. It would have to end up in the BAR for publishing to work. This probably requires changes to the DB, for instance.