From b171b51c117f694d93ae0c23bc41c556c085abcf Mon Sep 17 00:00:00 2001 From: pryce-turner <31577879+pryce-turner@users.noreply.github.com> Date: Wed, 10 Jul 2024 10:18:40 -0700 Subject: [PATCH] Pryce/doc 434 clarify how code is pushed into a given image during pyflyte (#5548) * Removed experimental designation for array node Signed-off-by: pryce-turner * Added .flyteignore as an ignore option Signed-off-by: pryce-turner * Added notes on where registered workflows land Signed-off-by: pryce-turner * spelling and formatting Signed-off-by: pryce-turner * more formatting Signed-off-by: pryce-turner --------- Signed-off-by: pryce-turner --- docs/flyte_fundamentals/registering_workflows.md | 13 +++++++++++-- docs/user_guide/advanced_composition/map_tasks.md | 6 ------ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/flyte_fundamentals/registering_workflows.md b/docs/flyte_fundamentals/registering_workflows.md index 9e503e4127..1c53a78020 100644 --- a/docs/flyte_fundamentals/registering_workflows.md +++ b/docs/flyte_fundamentals/registering_workflows.md @@ -72,6 +72,8 @@ run it with the supplied arguments. As you can see from the expected output, you can visit the link to the Flyte console to see the progress of your running execution. +You may also run `run --remote --copy-all`, which is very similar to the above command. As the name suggests, this will copy the source tree rooted at the top-level `__init__.py` file. With this strategy, any modules discoverable on the `PYTHONPATH` will be importable. + ```{note} `pyflyte run` supports Flyte workflows that import any other user-defined modules that contain additional tasks or workflows. @@ -177,8 +179,15 @@ all user-defined code at the root of your project. In some cases, your project directory may contain datasets, model files, and other potentially large artifacts that you want to exclude from the tarball. -You can do so by specifying these files in a `.gitignore` or `.dockerignore` -file in the root directory of your project. +You can do so by specifying these files in a `.flyteignore` file in the root +of your project. You can also use `.gitignore` or `.dockerignore` if you'd like +to avoid adding another file. +``` + +```{note} +`WORKDIR`, `PYTHONPATH`, and `PATH` + +When executing any of the above commands, the archive that gets creates is extracted wherever the WORKDIR is set. This can be handled directly via the WORKDIR directive in a Dockerfile, or specified via `source_root` if using ImageSpec. This is important for discovering code and executables via `PATH` or `PYTHONPATH`. A common pattern for making your Python packages fully discoverable is to have a top-level `src` folder, adding that to your `PYTHONPATH`, and making all your imports absolute. This avoids having to "install" your Python project in the image at any point e.g. via `pip install -e`. ``` ### Productionizing your workflows diff --git a/docs/user_guide/advanced_composition/map_tasks.md b/docs/user_guide/advanced_composition/map_tasks.md index bdd339b954..c57f37b95b 100644 --- a/docs/user_guide/advanced_composition/map_tasks.md +++ b/docs/user_guide/advanced_composition/map_tasks.md @@ -162,12 +162,6 @@ pyflyte run --remote \ ## ArrayNode -:::{important} -This feature is experimental and the API is subject to breaking changes. -If you encounter any issues, please submit a -[bug report](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=bug%2Cuntriaged&projects=&template=bug_report.yaml&title=%5BBUG%5D+). -::: - ArrayNode map tasks serve as a seamless substitution for regular map tasks, differing solely in the submodule utilized to import the `map_task` function. Specifically, you will need to import `map_task` from the experimental module as illustrated below: