-
-
Notifications
You must be signed in to change notification settings - Fork 628
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This reverts commit d780d76.
- Loading branch information
Showing
6 changed files
with
313 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
--- | ||
id: internal-fb-cluster | ||
title: Hydra on the internet FB Cluster | ||
--- | ||
|
||
Support for launching jobs to the AI cluster is currently still experimental and is expected to evolve over | ||
the coming months. | ||
|
||
## flow-cli | ||
|
||
flow-cli integration is hacky at the moment. | ||
See the sample f6.sample_projects.classy_hydra_project.workflow.main for details. | ||
|
||
```bash title="Example run" | ||
$ CFG='{"config": {"overrides": ["trainer=multi_gpu","trainer.max_epochs=90","+lr_scheduler=multi_step"]}}' | ||
$ ENTITLEMENT=cv_images_gpu_prod | ||
$ TEAM=team_computer_vision | ||
$ WORKFLOW=f6.sample_projects.classy_hydra_project.workflow.main | ||
$ flow-cli canary $WORKFLOW --run-as-secure-group $TEAM --parameters-json=$CFG --entitlement $ENTITLEMENT | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
--- | ||
id: fbcode-configerator-config-source | ||
title: Configerator Config Source Plugin | ||
--- | ||
|
||
The ConfigeratorConfigSource plugin makes it possible for Hydra applications to consume a domain of configs from configerator. | ||
|
||
### Dependency | ||
|
||
Add the following to your `TARGET` file | ||
```commandline | ||
//fair_infra/fbcode_hydra_plugins/configerator_config_source:configerator_config_source | ||
``` | ||
|
||
### Usage | ||
|
||
1. The Configerator Config Source plugin requires that you place the configs under a [domain](https://fburl.com/wiki/n5cgchxe). | ||
You can find an example domain [here](https://fburl.com/diffusion/ms50g5hu) | ||
|
||
:::important | ||
Due to the limitations of Configerator APIs, matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names. | ||
For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) | ||
::: | ||
|
||
|
||
2. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. | ||
The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) | ||
|
||
:::info | ||
Adding a new search path will become much easier once [#274](https://github.com/facebookresearch/hydra/issues/274) is resolved, which is planned for Hydra 1.1. | ||
::: | ||
|
||
|
||
### Example: | ||
|
||
#### Example SearchPathPlugin | ||
[`ConfigeratorExampleSearchPathPlugin`](https://fburl.com/diffusion/vwa82fbg) adds the example configerator domain to the search path of the example applications. | ||
|
||
#### Reading primary config from configerator | ||
This example reads its primary config from configerator [here](https://fburl.com/diffusion/twk3smkj) which has a default list defined. | ||
|
||
```commandline | ||
$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/primary_config:my_app | ||
... | ||
Parsing buck files: finished in 1.1 sec | ||
... | ||
{'driver': 'mysql', 'user': 'alau'} | ||
``` | ||
|
||
#### Compose config with configerator | ||
This example reads its primary config from local yaml file `primary_config.yaml` but reads config groups info from configerator. | ||
|
||
```commandline | ||
$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/config_group:my_app -- +db=mysql | ||
... | ||
Parsing buck files: finished in 1.1 sec | ||
... | ||
{'foo': 'bar', 'driver': 'mysql', 'user': 'alau'} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,119 @@ | ||
--- | ||
id: fair-cluster | ||
title: Hydra on the FAIR cluster | ||
--- | ||
|
||
import GithubLink from "@site/src/components/GithubLink" | ||
|
||
Hydra 1.0rc is available on FAIR Cluster. The recommended way for installation is via meta package [hydra-fair-plugin](https://github.com/fairinternal/hydra-fair-plugins). | ||
|
||
## Hydra FAIR Plugins | ||
1. It brings the correct Hydra dependency and has been tested on the FAIR Cluster. | ||
2. It provides FAIR Cluster specific defaults overrides (for example, ```hydra.sweep.dir``` is set to be ```/checkpoint/${oc.env:USER}/outputs/${now:%Y-%m-%d}/${now:%H-%M-%S}```) | ||
3. It provides a [fairtask](https://github.com/fairinternal/fairtask) launcher plugin. | ||
4. It installs [Submitit](https://github.com/facebookincubator/submitit) launcher plugin by default. | ||
|
||
### Installation | ||
<details> | ||
<summary>0.3.1 (stable), compatible with Hydra 0.11</summary> | ||
|
||
### Clean Install | ||
|
||
```commandline | ||
pip install hydra-fair-plugins | ||
``` | ||
|
||
The dependency installed looks like | ||
```commandline | ||
$ pip freeze | grep hydra | ||
hydra-core==0.11.3 | ||
hydra-fair-cluster==0.1.4 | ||
hydra-fair-plugins==0.3.1 | ||
hydra-fairtask==0.1.8 | ||
hydra-submitit==0.2.0 | ||
``` | ||
</details> | ||
|
||
<details> | ||
<summary>1.0 (Release candidate), compatible with Hydra 1.0rc</summary> | ||
|
||
With [`Submitit`](https://github.com/facebookincubator/submitit) open sourced, the corresponding plugin has been moved | ||
<GithubLink to="plugins/hydra_submitit_launcher">here</GithubLink>. Read this [doc](/docs/plugins/submitit_launcher) on installation/usage info. | ||
|
||
### Clean Install | ||
|
||
```commandline | ||
pip install hydra-fair-plugins --pre --upgrade --upgrade-strategy=eager | ||
``` | ||
|
||
### Upgrade from stable | ||
|
||
```commandline | ||
# Remove legacy fair internal submitit launcher plugin | ||
pip uninstall hydra-submitit -y | ||
pip install hydra-fair-plugins --pre --upgrade --upgrade-strategy=eager | ||
``` | ||
Check out [Hydra documentation](/docs/plugins/submitit_launcher) for more info on ```Submitit``` launcher plugin. | ||
|
||
|
||
The depedency looks like | ||
```commandline | ||
$ pip freeze | grep hydra | ||
hydra-core==1.0.0rc1 | ||
hydra-fair-cluster==1.0.0rc1 | ||
hydra-fair-plugins==1.0.0rc1 | ||
hydra-fairtask==1.0.0rc1 | ||
hydra-submitit-launcher==1.0.0rc3 | ||
``` | ||
|
||
Please refer to [Hydra upgrades](/docs/upgrades/0.11_to_1.0/config_path_changes) on what changes are needed for your app for upgrading to Hydra 1.0 | ||
</details> | ||
|
||
<details> | ||
<summary>Downgrade From 1.0rc to stable</summary> | ||
|
||
Downgrade to stable in case you run into issues and need to be unblocked immediately. | ||
|
||
```commandline | ||
pip freeze | grep hydra | xargs pip uninstall -y | ||
pip install hydra-fair-plugins | ||
``` | ||
</details> | ||
|
||
### Usage | ||
|
||
<details> | ||
<summary>0.3.1 (stable)</summary> | ||
|
||
Once the plugins are installed, you can launch to the FAIR cluster by appending hydra/launcher=fairtask or hydra/launcher=submitit | ||
for example: | ||
|
||
``` | ||
python my_app.py -m hydra/launcher=submitit db=mysql,postgresql | ||
# or | ||
python my_app.py -m hydra/launcher=fairtask db=mysql,postgresql | ||
``` | ||
|
||
Both hydra-submitit and hydra-fairtask are providing sensible defaults for their configuration ([Submitit](https://github.com/fairinternal/hydra-fair-plugins/blob/master/plugins/hydra-submitit/hydra_plugins/submitit/conf/hydra/launcher/submitit.yaml), [fairtask](https://github.com/fairinternal/hydra-fair-plugins/blob/master/plugins/hydra-fairtask/hydra_plugins/fairtask/conf/hydra/launcher/fairtask.yaml)) | ||
|
||
You can customize fairtask/submitit behavior much like you can customize anything else, from the command line or by overriding in your config file or composing in alternative launcher configuration. | ||
You can view the Hydra config (which includes the config for submitit or fairtask) with this command: | ||
``` | ||
python my_app.py hydra/launcher=submitit --cfg=hydra | ||
``` | ||
</details> | ||
|
||
|
||
<details> | ||
<summary>1.0 (Release Candidate)</summary> | ||
|
||
|
||
For 1.0, ```fairtask``` usage remains the same. To use ```Submitit```, the command changes to: | ||
|
||
```commandline | ||
python my_app.py -m hydra/launcher=submitit_slurm db=mysql,postgresql | ||
``` | ||
|
||
More info on ```Submitit``` launcher can be found [here](https://hydra.cc/docs/plugins/submitit_launcher) | ||
|
||
</details> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
--- | ||
id: fbcode | ||
title: Hydra at fbcode | ||
--- | ||
|
||
## Differences in fbcode | ||
|
||
### Open source plugins | ||
#### Supported: | ||
- hydra_ax_sweeper | ||
- hydra_colorlog | ||
- hydra_nevergrad_sweeper | ||
|
||
#### Unsupported: | ||
- joblib launcher: Joblib's Loki backend does not work correctly when executed from a par file. | ||
|
||
### Facebook specified plugins | ||
- fbcode_defaults : Changes configuration defaults to be appropriate for fbcode (e.g: Output directories are in `fbcode/outputs` and `fbcode/multirun`) | ||
- configerator_config_source: See [doc](/fb/configerator-config-source.md) for more information. | ||
|
||
#### TARGETS | ||
Hydra includes buck TARGETS you can use in fbcode. In general, if there is TARGET there are two options: | ||
1. You can depend on the TARGETS to use Hydra or a plugin. | ||
2. The TARGETS contains a runnable example. | ||
|
||
targets are under `github/facebookresearch/hydra`: | ||
- [:`hydra`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra) : Primary target to use in most cases. Includes `hydra_oss` and the `fbcode_defaults`. | ||
- `:hydra_oss` : Vanilla Hydra without any Facebook specific targets. | ||
- [`plugins`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/plugins): Plugins that have a TARGETS file are runnable in fbcode. | ||
- [`examples`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/examples): Examples that have a TARGETS file are runnable in `fbcode`. All tutorials (in [`examples/tutorials`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/examples)) are supported. An example TARGET file can be found [here](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/examples/tutorials/basic/your_first_hydra_app/5_composition). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
--- | ||
id: flow-launcher | ||
title: Flow Launcher | ||
--- | ||
|
||
The Flow Launcher plugin provides a way to launch application via `flow`. | ||
|
||
## Dependency | ||
To use the Flow Launcher, add the following to your `TARGETS` | ||
```commandline | ||
//github/facebookresearch/hydra/plugins/hydra_flow_launcher:hydra_flow_launcher | ||
``` | ||
|
||
## Usage | ||
Add hydra/launcher=flow to your command line. Alternatively, override hydra/launcher in your config: | ||
```commandline | ||
defaults: | ||
- hydra/launcher: flow | ||
``` | ||
|
||
<details><summary>Discover Flow Launcher's config</summary> | ||
|
||
```yaml title="$ buck run @mode/opt //path:my_app -- --cfg hydra -p hydra.launcher" | ||
|
||
# @package hydra.launcher | ||
_target_: hydra_plugins.flow_launcher_plugin.flow_launcher.FlowLauncher | ||
mode: flow | ||
owner: ${oc.env:USER} | ||
entitlement: gpu_pnb_fair | ||
pkg_version: fblearner.flow.canary:19e63cbf9945467281cf681bc8902c50 | ||
driver_path: '' | ||
resource_requirements: | ||
gpu: 0 | ||
cpu: 1 | ||
memory: 10g | ||
region: null | ||
capabilities: [] | ||
percent_cpu: null | ||
run_as_secure_group: fair_research_and_engineering | ||
retries: 2 | ||
tags: [] | ||
``` | ||
</details> | ||
The Launcher currently support both `par` and `xar` style. You can override `resource_requirements` just like how you would via `flow-cli`. | ||
|
||
|
||
:::info NOTE | ||
Flow launcher only supports `@mode/opt`. | ||
::: | ||
|
||
To run the example application: | ||
```commandline | ||
buck run @mode/opt //github/facebookresearch/hydra/plugins/hydra_flow_launcher/example:my_app -- --multirun | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
--- | ||
id: intro | ||
title: Hydra at Facebook | ||
--- | ||
|
||
### Intro | ||
Facebook has multiple different environments, such as the **Internal FB Cluster**, the **FAIR Cluster** etc. | ||
|
||
The FB specific docs are describing the differences. | ||
|
||
### Release strategy | ||
Hydra's source of truth is the [GitHub repo](https://github.com/facebookresearch/hydra). | ||
|
||
Hydra is developed using release branches. Once a new major is released, it is maintained in patch only mode. | ||
Primary development is happening on the `master` branch. | ||
|
||
When a new major version of Hydra is released, a new release branch is created in Hydra repo. A corresponding Hydra version will be created inside `github/facebookresearch/hydra_VERSION` to track | ||
the release branch. | ||
|
||
Hydra is trying hard to remain backward compatible between two subsequent versions and in most cases the upgrade will be smooth. | ||
There could be some new deprecations warnings that should be fixed before the next major version. | ||
|
||
### Maintaining this documentation | ||
This documentation lives in in the Hydra repo which is publicly accessible. The pages will only normally render on the internal | ||
copy of the docs, but keep in mind that everyone can read those docs in the repo if they want to. | ||
|
||
1. Do not put anything sensitive here, no root passwords or launch codes. | ||
2. If you are in need to have sensitive Hydra related documentation please reach out to the maintainers of Hydra for help. |