How to improve interface for accessing nested Flux instances

[SteVwonder]

Given the hierarchical nature of Flux (both nesting within itself and inside other RMs), the ability to navigate easily between instances is going to be key. When launching Flux under Slurm/LSF, many (all?) of our workflow users currently leverage an initial-program that dumps out the flux URI to a file, then they do the equivalent of FLUX_URI=$(cat flux.info) flux .... Many of our users find this flow confusing, and the script to generate that file is ad-hoc.

A discussion started in #3726 as to how to make it easier to grab a FLUX_URI from flux instances nested within Slurm. On our ☕ call, we also discussed ways to make that even easier (e.g., flux proxy slurm://JOBID). I wanted to open this GH discussion to capture these discussions and hopefully come up with a formal proposal for how to move forward (and then we can solicit feedback from our workflow users).

[SteVwonder]

One component of this seems to be spinning up Flux in the first place. Currently we provide examples of what to run in our documentation for the various schedulers (i.e., Slurm and LSF). It would probably still be good to push forward with our earlier flux-bootstrap command that attempts to figure out what system you are running on and what the right launch command is. If we had that setup, we could then use that command to abstract away some of the other tricks we come up to make this more seamless.

[SteVwonder]

Another component of this is where to storethe URIs of the nested instances. For flux nesting within flux case, we already have the rc1 script that dumps the nested URI into the parent's KVS. For the flux within Slurm case, @grondo has a proof-of-concept rc1 script in #3726 to store the URI in the comment field of the slurm job. At least at LLNL, we will also probably want to figure out how to handle this for flux within LSF.

[SteVwonder]

Another component of this is how users can discover/access the URIs of the nested instances. For the flux-in-flux, we need to ensure the flux jobs command lists those URIs. For the flux-in-other-RM case, it would be really nice if we could abstract away the RM-specific methods we use for storing the URIs. One suggestion on the ☕ call was changes to the flux-proxy command, either a new argument or a new URI scheme. Strawmen examples:

• flux proxy --slurm JOBID
• flux proxy slurm://JOBID

The slurm option could then be extend/copied for flux and lsf.

[grondo]

I believe some new additions to flux-core have mostly solved this problem:

1. A new extensible Python FluxURIResolver class can be used to resolve high-level URIs to native Flux URIs from Python, and includes resolvers for Flux jobids (jobid scheme), process ids (pid scheme), and Slurm jobs (slurm scheme) in flux-core. There is the potential for new plugins for other foreign RMs or even site-specific URI resolution if necessary.
2. The new flux uri command can be utilized to resolve Flux jobids and other high-level URIs to native URIs from the command line and in scripts, when necessary.
3. PR cmd: support high-level URIs and JOBID arguments in flux-top and flux-proxy #4004 extends flux proxy to call out to flux uri to resolve its JOBID argument, so that flux proxy JOBID or flux proxy slurm:SLURM_JOB_ID will now work.