From fa0486bf3245ca18ee4a8cd94b25d8b9e9549d77 Mon Sep 17 00:00:00 2001 From: "Jessica Zhang (NY)" Date: Tue, 30 Jul 2024 15:11:27 -0400 Subject: [PATCH] Restore fb doc This reverts commit d780d767bbeadeb72cffac76d4c7a1ccfeb5a722. --- website/docs/fb/ai-cluster.md | 20 +++ website/docs/fb/configerator-config-source.md | 59 +++++++++ website/docs/fb/fair-cluster.md | 119 ++++++++++++++++++ website/docs/fb/fbcode.md | 30 +++++ website/docs/fb/flow-launcher.md | 57 +++++++++ website/docs/fb/intro.md | 28 +++++ 6 files changed, 313 insertions(+) create mode 100644 website/docs/fb/ai-cluster.md create mode 100644 website/docs/fb/configerator-config-source.md create mode 100644 website/docs/fb/fair-cluster.md create mode 100644 website/docs/fb/fbcode.md create mode 100644 website/docs/fb/flow-launcher.md create mode 100644 website/docs/fb/intro.md diff --git a/website/docs/fb/ai-cluster.md b/website/docs/fb/ai-cluster.md new file mode 100644 index 00000000000..fa843777e36 --- /dev/null +++ b/website/docs/fb/ai-cluster.md @@ -0,0 +1,20 @@ +--- +id: internal-fb-cluster +title: Hydra on the internet FB Cluster +--- + +Support for launching jobs to the AI cluster is currently still experimental and is expected to evolve over +the coming months. + +## flow-cli + +flow-cli integration is hacky at the moment. +See the sample f6.sample_projects.classy_hydra_project.workflow.main for details. + +```bash title="Example run" +$ CFG='{"config": {"overrides": ["trainer=multi_gpu","trainer.max_epochs=90","+lr_scheduler=multi_step"]}}' +$ ENTITLEMENT=cv_images_gpu_prod +$ TEAM=team_computer_vision +$ WORKFLOW=f6.sample_projects.classy_hydra_project.workflow.main +$ flow-cli canary $WORKFLOW --run-as-secure-group $TEAM --parameters-json=$CFG --entitlement $ENTITLEMENT +``` diff --git a/website/docs/fb/configerator-config-source.md b/website/docs/fb/configerator-config-source.md new file mode 100644 index 00000000000..65589ca335d --- /dev/null +++ b/website/docs/fb/configerator-config-source.md @@ -0,0 +1,59 @@ +--- +id: fbcode-configerator-config-source +title: Configerator Config Source Plugin +--- + +The ConfigeratorConfigSource plugin makes it possible for Hydra applications to consume a domain of configs from configerator. + +### Dependency + +Add the following to your `TARGET` file +```commandline +//fair_infra/fbcode_hydra_plugins/configerator_config_source:configerator_config_source +``` + +### Usage + +1. The Configerator Config Source plugin requires that you place the configs under a [domain](https://fburl.com/wiki/n5cgchxe). +You can find an example domain [here](https://fburl.com/diffusion/ms50g5hu) + +:::important +Due to the limitations of Configerator APIs, matching the name of your domain and directory of configs is necessary for the plugin to extract information on the full config names. +For example, the config paths returned by Configerator API could look like `fair_infra/hydra_plugins/configerator_config_source/example/db/mysql`. The plugin needs to know where the directory of configs begins ([`example`](https://fburl.com/diffusion/7c0c5tig)), in order to determine the full config name (`db/mysql`). So in this case the domain should be named [`example.cconf`](https://fburl.com/diffusion/pyymoo1t) +::: + + +2. Create a [SearchPathPlugin](https://hydra.cc/docs/next/advanced/search_path) to add the Configerator path to the list of search paths. + The path you add in your SearchPathPlugin should be the name of your domain of configs, such as in this [example SearchPathPlugin](https://fburl.com/diffusion/ljggtux5) + +:::info +Adding a new search path will become much easier once [#274](https://github.com/facebookresearch/hydra/issues/274) is resolved, which is planned for Hydra 1.1. +::: + + +### Example: + +#### Example SearchPathPlugin +[`ConfigeratorExampleSearchPathPlugin`](https://fburl.com/diffusion/vwa82fbg) adds the example configerator domain to the search path of the example applications. + +#### Reading primary config from configerator +This example reads its primary config from configerator [here](https://fburl.com/diffusion/twk3smkj) which has a default list defined. + +```commandline +$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/primary_config:my_app +... +Parsing buck files: finished in 1.1 sec +... +{'driver': 'mysql', 'user': 'alau'} +``` + +#### Compose config with configerator +This example reads its primary config from local yaml file `primary_config.yaml` but reads config groups info from configerator. + +```commandline +$ buck run //fair_infra/fbcode_hydra_plugins/configerator_config_source/example/config_group:my_app -- +db=mysql +... +Parsing buck files: finished in 1.1 sec +... +{'foo': 'bar', 'driver': 'mysql', 'user': 'alau'} +``` diff --git a/website/docs/fb/fair-cluster.md b/website/docs/fb/fair-cluster.md new file mode 100644 index 00000000000..6bf0f44fb12 --- /dev/null +++ b/website/docs/fb/fair-cluster.md @@ -0,0 +1,119 @@ +--- +id: fair-cluster +title: Hydra on the FAIR cluster +--- + +import GithubLink from "@site/src/components/GithubLink" + +Hydra 1.0rc is available on FAIR Cluster. The recommended way for installation is via meta package [hydra-fair-plugin](https://github.com/fairinternal/hydra-fair-plugins). + +## Hydra FAIR Plugins +1. It brings the correct Hydra dependency and has been tested on the FAIR Cluster. +2. It provides FAIR Cluster specific defaults overrides (for example, ```hydra.sweep.dir``` is set to be ```/checkpoint/${oc.env:USER}/outputs/${now:%Y-%m-%d}/${now:%H-%M-%S}```) +3. It provides a [fairtask](https://github.com/fairinternal/fairtask) launcher plugin. +4. It installs [Submitit](https://github.com/facebookincubator/submitit) launcher plugin by default. + +### Installation +
+0.3.1 (stable), compatible with Hydra 0.11 + +### Clean Install + +```commandline +pip install hydra-fair-plugins +``` + +The dependency installed looks like +```commandline +$ pip freeze | grep hydra +hydra-core==0.11.3 +hydra-fair-cluster==0.1.4 +hydra-fair-plugins==0.3.1 +hydra-fairtask==0.1.8 +hydra-submitit==0.2.0 +``` +
+ +
+1.0 (Release candidate), compatible with Hydra 1.0rc + +With [`Submitit`](https://github.com/facebookincubator/submitit) open sourced, the corresponding plugin has been moved +here. Read this [doc](/docs/plugins/submitit_launcher) on installation/usage info. + +### Clean Install + +```commandline +pip install hydra-fair-plugins --pre --upgrade --upgrade-strategy=eager +``` + +### Upgrade from stable + +```commandline +# Remove legacy fair internal submitit launcher plugin +pip uninstall hydra-submitit -y +pip install hydra-fair-plugins --pre --upgrade --upgrade-strategy=eager +``` +Check out [Hydra documentation](/docs/plugins/submitit_launcher) for more info on ```Submitit``` launcher plugin. + + +The depedency looks like +```commandline +$ pip freeze | grep hydra +hydra-core==1.0.0rc1 +hydra-fair-cluster==1.0.0rc1 +hydra-fair-plugins==1.0.0rc1 +hydra-fairtask==1.0.0rc1 +hydra-submitit-launcher==1.0.0rc3 +``` + +Please refer to [Hydra upgrades](/docs/upgrades/0.11_to_1.0/config_path_changes) on what changes are needed for your app for upgrading to Hydra 1.0 +
+ +
+Downgrade From 1.0rc to stable + +Downgrade to stable in case you run into issues and need to be unblocked immediately. + +```commandline +pip freeze | grep hydra | xargs pip uninstall -y +pip install hydra-fair-plugins +``` +
+ +### Usage + +
+ 0.3.1 (stable) + +Once the plugins are installed, you can launch to the FAIR cluster by appending hydra/launcher=fairtask or hydra/launcher=submitit +for example: + +``` + python my_app.py -m hydra/launcher=submitit db=mysql,postgresql +# or + python my_app.py -m hydra/launcher=fairtask db=mysql,postgresql +``` + +Both hydra-submitit and hydra-fairtask are providing sensible defaults for their configuration ([Submitit](https://github.com/fairinternal/hydra-fair-plugins/blob/master/plugins/hydra-submitit/hydra_plugins/submitit/conf/hydra/launcher/submitit.yaml), [fairtask](https://github.com/fairinternal/hydra-fair-plugins/blob/master/plugins/hydra-fairtask/hydra_plugins/fairtask/conf/hydra/launcher/fairtask.yaml)) + +You can customize fairtask/submitit behavior much like you can customize anything else, from the command line or by overriding in your config file or composing in alternative launcher configuration. +You can view the Hydra config (which includes the config for submitit or fairtask) with this command: +``` +python my_app.py hydra/launcher=submitit --cfg=hydra +``` +
+ + +
+ 1.0 (Release Candidate) + + +For 1.0, ```fairtask``` usage remains the same. To use ```Submitit```, the command changes to: + +```commandline +python my_app.py -m hydra/launcher=submitit_slurm db=mysql,postgresql +``` + +More info on ```Submitit``` launcher can be found [here](https://hydra.cc/docs/plugins/submitit_launcher) + +
\ No newline at end of file diff --git a/website/docs/fb/fbcode.md b/website/docs/fb/fbcode.md new file mode 100644 index 00000000000..de176c20d62 --- /dev/null +++ b/website/docs/fb/fbcode.md @@ -0,0 +1,30 @@ +--- +id: fbcode +title: Hydra at fbcode +--- + +## Differences in fbcode + +### Open source plugins +#### Supported: + - hydra_ax_sweeper + - hydra_colorlog + - hydra_nevergrad_sweeper + +#### Unsupported: + - joblib launcher: Joblib's Loki backend does not work correctly when executed from a par file. + +### Facebook specified plugins + - fbcode_defaults : Changes configuration defaults to be appropriate for fbcode (e.g: Output directories are in `fbcode/outputs` and `fbcode/multirun`) + - configerator_config_source: See [doc](/fb/configerator-config-source.md) for more information. + +#### TARGETS +Hydra includes buck TARGETS you can use in fbcode. In general, if there is TARGET there are two options: +1. You can depend on the TARGETS to use Hydra or a plugin. +2. The TARGETS contains a runnable example. + +targets are under `github/facebookresearch/hydra`: +- [:`hydra`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra) : Primary target to use in most cases. Includes `hydra_oss` and the `fbcode_defaults`. +- `:hydra_oss` : Vanilla Hydra without any Facebook specific targets. +- [`plugins`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/plugins): Plugins that have a TARGETS file are runnable in fbcode. +- [`examples`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/examples): Examples that have a TARGETS file are runnable in `fbcode`. All tutorials (in [`examples/tutorials`](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/examples)) are supported. An example TARGET file can be found [here](https://www.internalfb.com/intern/diffusion/FBS/browsedir/master/fbcode/github/facebookresearch/hydra/examples/tutorials/basic/your_first_hydra_app/5_composition). \ No newline at end of file diff --git a/website/docs/fb/flow-launcher.md b/website/docs/fb/flow-launcher.md new file mode 100644 index 00000000000..01de19bb4c1 --- /dev/null +++ b/website/docs/fb/flow-launcher.md @@ -0,0 +1,57 @@ +--- +id: flow-launcher +title: Flow Launcher +--- + +The Flow Launcher plugin provides a way to launch application via `flow`. + +## Dependency +To use the Flow Launcher, add the following to your `TARGETS` +```commandline +//github/facebookresearch/hydra/plugins/hydra_flow_launcher:hydra_flow_launcher +``` + +## Usage +Add hydra/launcher=flow to your command line. Alternatively, override hydra/launcher in your config: +```commandline +defaults: + - hydra/launcher: flow +``` + +
Discover Flow Launcher's config + +```yaml title="$ buck run @mode/opt //path:my_app -- --cfg hydra -p hydra.launcher" + +# @package hydra.launcher +_target_: hydra_plugins.flow_launcher_plugin.flow_launcher.FlowLauncher +mode: flow +owner: ${oc.env:USER} +entitlement: gpu_pnb_fair +pkg_version: fblearner.flow.canary:19e63cbf9945467281cf681bc8902c50 +driver_path: '' +resource_requirements: + gpu: 0 + cpu: 1 + memory: 10g + region: null + capabilities: [] + percent_cpu: null +run_as_secure_group: fair_research_and_engineering +retries: 2 +tags: [] +``` +
+ + + +The Launcher currently support both `par` and `xar` style. You can override `resource_requirements` just like how you would via `flow-cli`. + + +:::info NOTE +Flow launcher only supports `@mode/opt`. +::: + +To run the example application: +```commandline +buck run @mode/opt //github/facebookresearch/hydra/plugins/hydra_flow_launcher/example:my_app -- --multirun +``` \ No newline at end of file diff --git a/website/docs/fb/intro.md b/website/docs/fb/intro.md new file mode 100644 index 00000000000..e72afde153f --- /dev/null +++ b/website/docs/fb/intro.md @@ -0,0 +1,28 @@ +--- +id: intro +title: Hydra at Facebook +--- + +### Intro +Facebook has multiple different environments, such as the **Internal FB Cluster**, the **FAIR Cluster** etc. + +The FB specific docs are describing the differences. + +### Release strategy +Hydra's source of truth is the [GitHub repo](https://github.com/facebookresearch/hydra). + +Hydra is developed using release branches. Once a new major is released, it is maintained in patch only mode. +Primary development is happening on the `master` branch. + +When a new major version of Hydra is released, a new release branch is created in Hydra repo. A corresponding Hydra version will be created inside `github/facebookresearch/hydra_VERSION` to track +the release branch. + +Hydra is trying hard to remain backward compatible between two subsequent versions and in most cases the upgrade will be smooth. +There could be some new deprecations warnings that should be fixed before the next major version. + +### Maintaining this documentation +This documentation lives in in the Hydra repo which is publicly accessible. The pages will only normally render on the internal +copy of the docs, but keep in mind that everyone can read those docs in the repo if they want to. + +1. Do not put anything sensitive here, no root passwords or launch codes. +2. If you are in need to have sensitive Hydra related documentation please reach out to the maintainers of Hydra for help.