add pinned_packages option in conda-forge.yml

[jaimergp]

Checklist

• Added a news entry

Comes from discussion at #1773

[jaimergp]

Needs some live testing in a demo feedstock.

[isuruf]

Can we move this to https://github.com/conda-forge/conda-forge-ci-setup-feedstock/blob/main/recipe/conda_forge_ci_setup/build_utils.py#L136?

[jaimergp]

Sadly not. We need the pins before conda-forge-ci-setup has been installed.

[h-vetinari]

Thanks for tackling all this so quickly! I'm wondering if this isn't effectively duplicating remote_ci_setup functionality in a slightly different way? Not saying that's bad, but if so we should probably consider getting rid of one or the other (at least in the medium term)?

Or is there some non-overlapping task that remote_ci_setup satisfies, but pinned_packages doesn't?

Looking at the history, remote_ci_setup was introduced in b0d8879, and updated to take a list in 83fb02e. It was also designed to allow <other_channel>::<pkg> dependencies, but I'm not sure those are in use anywhere.

[jaimergp]

AFAIU, remote_ci_setup is meant to add packages to the base environment. We were also using it to constrain existing ones, but that doesn't guarantee it's not overridden later in the workflow (e.g. those update commands would rely on how the history is handled).

So my idea here is to have two mechanisms:

• remote_ci_setup to add extra stuff to your base environment. A different name these days would better clarify the intent here, but I don't mind too much.
• pinned_packages to constrain the base environment. I guess this can even be used à la run_constrained with impossible specs to prevent certain packages from being installed, but I don't think we have seen that need yet.

[h-vetinari]

AFAIU, remote_ci_setup is meant to add packages to the base environment.

Yeah, though stuff in conda-meta/pinned is effectively added to the base environment as well. 🤷

I don't mind too much, but thought I'd point out the overlap.

[jaimergp]

Yeah, though stuff in conda-meta/pinned is effectively added to the base environment as well.

I'm not sure I follow. I can add tensorflow=2 to pinned and if I don't request it in the CLI it won't be installed.

[h-vetinari]

Hm, my bad. At the time when I last used it, the information I had found said (paraphrasing) "is injected upon every install/update". Wanted to double-check the docs, but they're empty. Further searches in the code did not produce more insights either, so I guess I had misunderstood that part. Tested it now locally and indeed it doesn't get added. Sorry for the noise.

[jaimergp]

No probs, happy to clarify. The key line in classic is this one. The "explicit pool", AFAIK, is the subset of the index that contains all the possible packages in the dependency tree of roughly {*requested, *installed}. So, if the tree contains a pin, then it's enforced. Otherwise it's ignored.

[isuruf]

I don't think this is necessary after #1774 right?

[jaimergp]

We don't need it as a fix, but I'd say that this feature could be useful or at least the intent is more clear (perhaps with a different name). For example, right now we might need to use remote_ci_setup to nudge the solver into more recent versions: conda-forge/compiler-rt-feedstock#83 (comment)

I explained the idea in more detail at #1776 (comment)