[Feature Request] Add variables to HydraConfig during callback that are usable for naming output directory

[gerard-sanroma]

🚀 Feature Request

It would be useful to add variables to the HydraConfig object from within user-defined Callbacks that can be referenced in config.yaml for naming the output directory

Motivation

This would allow to add the short git SHA to the name of the output directory.

There is GitInfo callback available [1], however this only prints the hash on screen ([1] https://github.com/paquiteau/hydra-callbacks)

It would be much more useful to have it persistent in the directory name (including a dirty flag) to be able to see immediately outputs from dirty / clean runs.

Beyond addition of git revs, this would extend to use-cases requiring to "burn" names on output directory that depend on variables defined at runtime (like the currently supported time variable now).

Pitch

Describe the solution you'd like

After adding git_rev into HydraConfig on hook on_run_start then it should be possible to define output folder name in config.yaml as follows:

hydra:
  run:
    dir: output_${now:%Y%m%d}_${now:%H%M%S}__${git_rev}

Describe alternatives you've considered

There are currently two issues preventing this from happening:

1. the only hook that seems to run before output_dir creation is on_run_start(). However in this hook there seems to be no HydraConfig available yet (raised ValueError: HydraConfig was not set).

2. most importantly, it seems that it is not possible at all to add variables to HydraConfig neither with open_dict() [2] or read_write [3] tricks.

[2] https://omegaconf.readthedocs.io/en/latest/usage.html#struct-flag
[3] https://omegaconf.readthedocs.io/en/2.1_branch/usage.html

My current solution consists in creating an empty file named after the git revision with a dirty flag within the output directory, similarly as suggested in this related (but not the same) feature request [4]. This is suboptimal since it is not immediately visible from the output directory name.

[4] #2199

Thanks!

[odelalleau]

What prevents you from using a custom resolver, similar to how ${now:} is currently used to define the output dir?

[gerard-sanroma]

hi @odelalleau, I have not found a way of adding a variable to the HydraConfig at runtime that I can reference like so, ${git_rev}, when defining the output_dir name.

Regarding the addition of the variable, the following applies:

1. It must be added at runtime (so cannot be pre-defined in config.yaml) because the git SHA has to be computed.
2. It must be added before output directory creation (eg, within callback hook on_run_start).

I may be missing something, but I found 2 concrete issues preventing me to achieve this (see end of original post).
Thanks!
G.

[odelalleau]

So what I'm suggesting is something like this (untested):

def get_git_rev():
    ... # code that checks the git repo
    return f"{git_revision}_{is_dirty}"

OmegaConf.register_new_resolver("git_rev", get_git_rev, use_cache=True)

@hydra.main(...):
def main(cfg):
    ....

and in your config use

hydra:
  run:
    dir: output_${now:%Y%m%d}_${now:%H%M%S}__${git_rev:}

[gerard-sanroma]

hi @odelalleau, thanks very much for your proposal. It works like a charm! :D
This solution is indeed much leaner than using callbacks (all user-code, no extra config required).
Besides, the variable (ie, "resovler") is available and usable at output_dir creation (not the case with callbacks).
It would maybe worthwile to add a couple of lines in the doc pointing to the possibility of adding custom resolvers. Perhaps a new subsection "Custom resolvers" under "Hydra resolvers" in https://hydra.cc/docs/1.0/configure_hydra/intro/#hydra-resolvers

[odelalleau]

hi @odelalleau, thanks very much for your proposal. It works like a charm! :D This solution is indeed much leaner than using callbacks (all user-code, no extra config required). Besides, the variable (ie, "resovler") is available and usable at output_dir creation (not the case with callbacks). It would maybe worthwile to add a couple of lines in the doc pointing to the possibility of adding custom resolvers. Perhaps a new subsection "Custom resolvers" under "Hydra resolvers" in https://hydra.cc/docs/1.0/configure_hydra/intro/#hydra-resolvers

Glad to hear! Yeah I guess it'd be good to link to https://omegaconf.readthedocs.io/en/latest/custom_resolvers.html#custom-resolvers on that page.

[gerard-sanroma]

Added a link to omegaconf custom resolvers to the doc in PR #2881