Enable testing of internal servicing builds

[NikolaMilosavljevic]

Contributes to: https://github.com/dotnet/dotnet-docker-internal/issues/5952

This PR includes changes in 4 distinct areas, with its own commits:

• Template changes
• Changes in dependency update code
• Pipeline changes
• Test infra changes

Summary of changes:

• Add support for internal artifact URLs with necessary authorization
• Update templates with support for internal testing requirements
• Add new pipeline yaml
• Update pipeline stages/steps/jobs for support of internal testing scenario
• Update pipeline for internal testing so it behaves as if it was a Pull request
• Update test infra to support internal testing requirements

Workflow

This pipeline runs build and test on a branch that contains the updated dockerfiles (that reference internal build artifacts). Another pipeline will be implemented, to run the dependency (and dockerfile) update step, commit changes to a feature branch, named appropriately (i.e. include BuildId in the name), and push the branch. Trigger would be added to initiate the run of the build/test pipeline as soon as new testing branch is pushed.

[MichaelSimons]

@NikolaMilosavljevic, could you please link to a test run or something that illustrates the generated Dockerfiles?

[mthalman]

Can you describe the overall workflow for how this will work? It seems that these changes rely on the Dockerfiles already being regenerated to target internal drops. How will that happen in an automated way? I'd like to understand the overall plan before proceeding with these changes.

[NikolaMilosavljevic]

@NikolaMilosavljevic, could you please link to a test run or something that illustrates the generated Dockerfiles?

Test run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2548267&view=results

[NikolaMilosavljevic]

Can you describe the overall workflow for how this will work? It seems that these changes rely on the Dockerfiles already being regenerated to target internal drops. How will that happen in an automated way? I'd like to understand the overall plan before proceeding with these changes.

I'm adding a paragraph to the PR description. In essence, to avoid adding additional complexity with updating dependencies in each leg, we've decided to run dependency (and dockerfiles) update in a separate pipeline that would commit those changes in a new branch and trigger the build/test pipeline.

[MichaelSimons]

How will we ensure regressions aren't introduced into the templates for this internal testing scenario with "normal" changes going forward? Is there a way to validate in PR validation? At a minimum we should ensure that the internal servicing validation pipeline is run anytime there are impactful infrastructure changes made (e.g. Dockerfile template changes) in order to flush out regressions right away versus in the release endgame when we are trying to validate product builds.

[NikolaMilosavljevic]

How will we ensure regressions aren't introduced into the templates for this internal testing scenario with "normal" changes going forward? Is there a way to validate in PR validation? At a minimum we should ensure that the internal servicing validation pipeline is run anytime there are impactful infrastructure changes made (e.g. Dockerfile template changes) in order to flush out regressions right away versus in the release endgame when we are trying to validate product builds.

That is a concern and something needs to be done. We've had this issue already it was on to developer to ensure template changes do not regress any of the scenarios. I've created an issue to track any verification work: #5935

[mthalman]

I didn't realize until now but this should be targeting the nightly branch.

Please ensure that execution of the modified scripts in the eng directory and update-dependencies continue to work correctly for normal, non-internal scenarios.

[NikolaMilosavljevic]

I didn't realize until now but this should be targeting the nightly branch.

Please ensure that execution of the modified scripts in the eng directory and update-dependencies continue to work correctly for normal, non-internal scenarios.

There was one bug in templates that I fixed on Friday, it affected non-internal scenarios - it was caught by PR tests. I believe I have tested all non-internal scenarios, but I'll double-check.

Should I retarget this to nightly branch?

For docker-tools PR - should that target main or a different branch?

[mthalman]

Should I retarget this to nightly branch?

Yes

For docker-tools PR - should that target main or a different branch?

Just main.

[NikolaMilosavljevic]

I didn't realize until now but this should be targeting the nightly branch.
Please ensure that execution of the modified scripts in the eng directory and update-dependencies continue to work correctly for normal, non-internal scenarios.

There was one bug in templates that I fixed on Friday, it affected non-internal scenarios - it was caught by PR tests. I believe I have tested all non-internal scenarios, but I'll double-check.

Should I retarget this to nightly branch?

For docker-tools PR - should that target main or a different branch?

I've pushed a fix that fixes the regression in existing dev scenarios.

[NikolaMilosavljevic]

Cherry-picked all commits locally to nightly, fixed one incorrect git merge and force pushed to this PR. Changed the target branch.

[NikolaMilosavljevic]

Updated run-tests.ps1 parameters for internal access token.

[mthalman]

Let's wait to merge this until later this week, after servicing.

[NikolaMilosavljevic]

I've pushed another change: fb25e1a

It removes an obsolete comment - I had originally made more changes to this file, that were not needed so later reverted, but the change in comment was left behind.

I noticed this while porting changes to docker-tools.

[NikolaMilosavljevic]

Pushed a fix (32ae5cd) that ensures we always use nuget.config.internal for internal testing.