From 3c7c21a3d5c2177672ccaad9ad02b4a29946f50f Mon Sep 17 00:00:00 2001 From: Jenny G <40272060+zhen0@users.noreply.github.com> Date: Tue, 28 Nov 2023 11:18:17 -0500 Subject: [PATCH] Release notes for 2.14.7 (#11262) Co-authored-by: Jeff Hale Co-authored-by: Serina Grill <42048900+serinamarie@users.noreply.github.com> Co-authored-by: Jeff Hale Co-authored-by: urimandujano --- RELEASE-NOTES.md | 3609 ++++++++++++++++++++++++++-------------------- 1 file changed, 2068 insertions(+), 1541 deletions(-) diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md index 756b3b75067b..dd20ed6e7833 100644 --- a/RELEASE-NOTES.md +++ b/RELEASE-NOTES.md @@ -1,5 +1,30 @@ # Prefect Release Notes +## Release 2.14.7 + +This release fixes a bug introduced in 2.14.6 where deployments with default Docker image builds looked for images tagged `v2.14.6` instead of `2.14.6`. Users of `2.14.6` should upgrade if planning to create deployments with an image other than a custom image. + +### Enhancements + +- Use a new route to read work pool types when connected to Prefect Cloud — +- Add `parent_flow_run_id` as a new API filter for flow runs — + +### Fixes + +- Allow more than one dependency package in the requirements of a push or pull step — + +### Documentation + +- Add serverless work pool landing page — +- Update Azure Container Instance guide to reflect current Azure Portal interface and Prefect UI — +- Update imports in **Flows** concept page example — + +### New Contributors + +- @oakbramble made their first contribution in + +**All changes**: + ## Release 2.14.6 ### View the next run for a deployment at a glance @@ -11,7 +36,7 @@ Click below to see it in action! ](https://github.com/PrefectHQ/prefect/assets/12350579/c1658f50-512a-4cd4-9d36-a523d3cc9ef0) See the following pull request for implementation details: -— https://github.com/PrefectHQ/prefect/pull/11230 +— ### Automatic project configuration for Cloud Run push work pools @@ -61,25 +86,29 @@ If you have yet to try using a Cloud Run run push pool, now is a great time! If you use another cloud provider, don't fret; we will add support for ECS and Azure Container Instances push work pools in future releases! See the following pull request for implementation details: -— https://github.com/PrefectHQ/prefect/pull/11204 +— ### Enhancements -- Add ability to search for block documents by name in the Prefect UI and API — https://github.com/PrefectHQ/prefect/pull/11212 -- Add pagination to the Blocks page in the Prefect UI for viewing/filtering more than 200 blocks — https://github.com/PrefectHQ/prefect/pull/11214 -- Include concurrency controls in `prefect-client` — https://github.com/PrefectHQ/prefect/pull/11227 + +- Add ability to search for block documents by name in the Prefect UI and API — +- Add pagination to the Blocks page in the Prefect UI for viewing/filtering more than 200 blocks — +- Include concurrency controls in `prefect-client` — ### Fixes -- Fix SQLite migration to work with older SQLite versions — https://github.com/PrefectHQ/prefect/pull/11215 -- Fix Subflow Runs tab filters and persist to URL in the Flow Runs page of the Prefect UI — https://github.com/PrefectHQ/prefect/pull/11218 + +- Fix SQLite migration to work with older SQLite versions — +- Fix Subflow Runs tab filters and persist to URL in the Flow Runs page of the Prefect UI — ### Documentation -- Improve formatting in deployment guides — https://github.com/PrefectHQ/prefect/pull/11217 -- Add instructions for turning off the flow run logger to the unit testing guide — https://github.com/PrefectHQ/prefect/pull/11223 + +- Improve formatting in deployment guides — +- Add instructions for turning off the flow run logger to the unit testing guide — ### Contributors + - @ConstantinoSchillebeeckx -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.5...2.14.6 +**All changes**: ## Release 2.14.5 @@ -122,42 +151,47 @@ if __name__ == "__main__": Note that a storage block must be saved before deploying a flow, but not if you're serving a remotely stored flow. See the following pull request for implementation details: -- https://github.com/PrefectHQ/prefect/pull/11092 + +- ### Enhancements -- Add customizable host and port settings for worker webserver — https://github.com/PrefectHQ/prefect/pull/11175 -- Safely retrieve `flow_run_id` in `EventsWorker` while finding related events — https://github.com/PrefectHQ/prefect/pull/11182 -- Add client-side setting for specifying a default work pool — https://github.com/PrefectHQ/prefect/pull/11137 -- Allow configuration of task run tag concurrency slot delay transition time via setting — https://github.com/PrefectHQ/prefect/pull/11020 -- Enable enhanced flow run cancellation by default - https://github.com/PrefectHQ/prefect/pull/11192 + +- Add customizable host and port settings for worker webserver — +- Safely retrieve `flow_run_id` in `EventsWorker` while finding related events — +- Add client-side setting for specifying a default work pool — +- Allow configuration of task run tag concurrency slot delay transition time via setting — +- Enable enhanced flow run cancellation by default - ### Fixes -- Fix access token retrieval when using `GitRepository` with a private repo and `.deploy` — https://github.com/PrefectHQ/prefect/pull/11156 -- Fix bug where check for required packages fails incorrectly during `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/11111 -- Fix routing to the Flows page from a flow run in the Prefect UI — https://github.com/PrefectHQ/prefect/pull/11190 -- Ensure the Prefect UI Flow Runs page reacts to filter changes - https://github.com/PrefectHQ/prefect-ui-library/pull/1874 -- Optimize memory usage by clearing `args/kwargs` in a Prefect `Call` post-execution - https://github.com/PrefectHQ/prefect/pull/11153 -- Allow logs to handle un-`uuid`-like flow_run_ids - https://github.com/PrefectHQ/prefect/pull/11191 -- Only run unit tests for Python file changes — https://github.com/PrefectHQ/prefect/pull/11159 -- Add `codespell` config and add to pre-commit — https://github.com/PrefectHQ/prefect/pull/10893 -- Update token regex in release notes generation script for VSCode compatibility - https://github.com/PrefectHQ/prefect/pull/11195 + +- Fix access token retrieval when using `GitRepository` with a private repo and `.deploy` — +- Fix bug where check for required packages fails incorrectly during `prefect deploy` — +- Fix routing to the Flows page from a flow run in the Prefect UI — +- Ensure the Prefect UI Flow Runs page reacts to filter changes - +- Optimize memory usage by clearing `args/kwargs` in a Prefect `Call` post-execution - +- Allow logs to handle un-`uuid`-like flow_run_ids - +- Only run unit tests for Python file changes — +- Add `codespell` config and add to pre-commit — +- Update token regex in release notes generation script for VSCode compatibility - ### Documentation -- Add Terraform Provider guide, update and simplify guides navigation — https://github.com/PrefectHQ/prefect/pull/11170 -- Clarify and harmonize Prefect Cloud documentation to reflect nomenclature and UX changes — https://github.com/PrefectHQ/prefect/pull/11157 -- Add information on Prefect Cloud to README — https://github.com/PrefectHQ/prefect/pull/11167 -- Update work pool-based deployment guide to include `.deploy` — https://github.com/PrefectHQ/prefect/pull/11174 -- Add Github information to auth-related Prefect Cloud documentation — https://github.com/PrefectHQ/prefect/pull/11178 -- Update workers tutorial — https://github.com/PrefectHQ/prefect/pull/11185 -- Update mkdocs material pin — https://github.com/PrefectHQ/prefect/pull/11160 -- Fix typo in audit log documentation — https://github.com/PrefectHQ/prefect/pull/11161 -- Fix typo in workers tutorial example — https://github.com/PrefectHQ/prefect/pull/11183 + +- Add Terraform Provider guide, update and simplify guides navigation — +- Clarify and harmonize Prefect Cloud documentation to reflect nomenclature and UX changes — +- Add information on Prefect Cloud to README — +- Update work pool-based deployment guide to include `.deploy` — +- Add Github information to auth-related Prefect Cloud documentation — +- Update workers tutorial — +- Update mkdocs material pin — +- Fix typo in audit log documentation — +- Fix typo in workers tutorial example — ### Contributors -- @yarikoptic made their first contribution in https://github.com/PrefectHQ/prefect/pull/10893 + +- @yarikoptic made their first contribution in - @taljaards -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.4...2.14.5 +**All changes**: ## Release 2.14.4 @@ -170,38 +204,43 @@ The flow run graph in the Prefect UI has been rebuilt from the ground up, offeri

See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/11112 -- https://github.com/PrefectHQ/prefect/pull/11105 -- https://github.com/PrefectHQ/prefect/pull/11113 -- https://github.com/PrefectHQ/prefect/pull/11132 -- https://github.com/PrefectHQ/prefect/pull/11138 + +- +- +- +- +- ### Enhancements -- Add API route for block counts — https://github.com/PrefectHQ/prefect/pull/11090 + +- Add API route for block counts — - Improved tag handling on `DeploymentImage` for `.deploy`: - - https://github.com/PrefectHQ/prefect/pull/11115 - - https://github.com/PrefectHQ/prefect/pull/11119 -- Allow `image` passed into `.deploy` to be optional if loading flow from storage — https://github.com/PrefectHQ/prefect/pull/11117 -- Ensure client avoids image builds when deploying to managed work pools — https://github.com/PrefectHQ/prefect/pull/11120 -- Add `SIGTERM` handling to runner to gracefully handle timeouts — https://github.com/PrefectHQ/prefect/pull/11133 -- Allow tasks to use `get_run_logger` w/o parent flow run — https://github.com/PrefectHQ/prefect/pull/11129 -- Allow `ResultFactory` creation `from_task` when no `flow_run_context` — https://github.com/PrefectHQ/prefect/pull/11134 + - + - +- Allow `image` passed into `.deploy` to be optional if loading flow from storage — +- Ensure client avoids image builds when deploying to managed work pools — +- Add `SIGTERM` handling to runner to gracefully handle timeouts — +- Allow tasks to use `get_run_logger` w/o parent flow run — +- Allow `ResultFactory` creation `from_task` when no `flow_run_context` — ### Fixes -- Avoid printing references to workers when deploying to managed pools — https://github.com/PrefectHQ/prefect/pull/11122 + +- Avoid printing references to workers when deploying to managed pools — ### Documentation -- Fix docstring for `flow.deploy` method example — https://github.com/PrefectHQ/prefect/pull/11108 -- Add warning about image architecture to push pool guide — https://github.com/PrefectHQ/prefect/pull/11118 -- Move webhooks guide to `Development` section in guides index — https://github.com/PrefectHQ/prefect/pull/11141 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.3...2.14.4 +- Fix docstring for `flow.deploy` method example — +- Add warning about image architecture to push pool guide — +- Move webhooks guide to `Development` section in guides index — + +**All changes**: ## Release 2.14.3 ### Observability with deployment status You can now track the status of your deployments in the Prefect UI, which is especially useful when serving flows as they have no associated work pool or worker. If you see a flow run enter a `LATE` state (it isn’t running), you can click into the deployment for that flow run and see a red indicator next to your deployment. The worker, runner, or agent polling that deployment or its associated work queue is offline. + - Deployments created from served flows will have a `READY` status if its associated process is running. - Deployments created in a work pool will have a `READY` status when a worker is `ONLINE` and polling the associated work queue. - Deployments created in a push work pool (Prefect Cloud) will always have a `READY` status. @@ -222,10 +261,11 @@ You can also create an automation triggered by deployment status changes on the

See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect-ui-library/pull/1801 -- https://github.com/PrefectHQ/prefect/pull/10969 -- https://github.com/PrefectHQ/prefect/pull/10951 -- https://github.com/PrefectHQ/prefect/pull/10949 + +- +- +- +- ### Additional storage options for `flow.from_source` @@ -259,39 +299,44 @@ if __name__ == "__main__": ``` See the following pull request for implementation details: -- https://github.com/PrefectHQ/prefect/pull/11072 + +- ### Enhancements -- Add option to skip building a Docker image with `flow.deploy` — https://github.com/PrefectHQ/prefect/pull/11082 -- Display placeholder on the variables page when no variables are present — https://github.com/PrefectHQ/prefect/pull/11044 -- Allow composite sort of block documents by `block_type_name` and name — https://github.com/PrefectHQ/prefect/pull/11054 -- Add option to configure a warning via `PREFECT_TASK_INTROSPECTION_WARN_THRESHOLD` if task parameter introspection takes a long time — https://github.com/PrefectHQ/prefect/pull/11075 + +- Add option to skip building a Docker image with `flow.deploy` — +- Display placeholder on the variables page when no variables are present — +- Allow composite sort of block documents by `block_type_name` and name — +- Add option to configure a warning via `PREFECT_TASK_INTROSPECTION_WARN_THRESHOLD` if task parameter introspection takes a long time — ### Fixes -- Update cancellation cleanup service to allow for infrastructure teardown — https://github.com/PrefectHQ/prefect/pull/11055 -- Allow `password` to be provided in `credentials` for `GitRespository` — https://github.com/PrefectHQ/prefect/pull/11056 -- Enable page refresh loading for non dashboard pages — https://github.com/PrefectHQ/prefect/pull/11065 -- Allow runner to load remotely stored flows when running hooks — https://github.com/PrefectHQ/prefect/pull/11077 -- Fix reading of flow run graph with unstarted runs — https://github.com/PrefectHQ/prefect/pull/11070 -- Allow Pydantic V2 models in flow function signatures — https://github.com/PrefectHQ/prefect/pull/10966 -- Run `prefect-client` build workflow on reqs.txt updates — https://github.com/PrefectHQ/prefect/pull/11079 -- Skips unsupported Windows tests — https://github.com/PrefectHQ/prefect/pull/11076 -- Avoid yanked `pytest-asyncio==0.22.0` — https://github.com/PrefectHQ/prefect/pull/11064 + +- Update cancellation cleanup service to allow for infrastructure teardown — +- Allow `password` to be provided in `credentials` for `GitRespository` — +- Enable page refresh loading for non dashboard pages — +- Allow runner to load remotely stored flows when running hooks — +- Fix reading of flow run graph with unstarted runs — +- Allow Pydantic V2 models in flow function signatures — +- Run `prefect-client` build workflow on reqs.txt updates — +- Skips unsupported Windows tests — +- Avoid yanked `pytest-asyncio==0.22.0` — ### Documentation -- Add guide to daemonize a worker or `.serve` process with systemd — https://github.com/PrefectHQ/prefect/pull/11008 -- Add clarification of term `task` in Global Concurrency docs — https://github.com/PrefectHQ/prefect/pull/11085 -- Update Global Concurrency guide to highlight general purpose use of concurrency limits — https://github.com/PrefectHQ/prefect/pull/11074 -- Update push work pools documentation to mention concurrency — https://github.com/PrefectHQ/prefect/pull/11068 -- Add documentation on Prefect Cloud teams — https://github.com/PrefectHQ/prefect/pull/11057 -- Update 2.14.2 release notes — https://github.com/PrefectHQ/prefect/pull/11053 -- Fix rendering of marketing banner on the Prefect dashboard — https://github.com/PrefectHQ/prefect/pull/11069 -- Fix typo in `README.md` — https://github.com/PrefectHQ/prefect/pull/11058 + +- Add guide to daemonize a worker or `.serve` process with systemd — +- Add clarification of term `task` in Global Concurrency docs — +- Update Global Concurrency guide to highlight general purpose use of concurrency limits — +- Update push work pools documentation to mention concurrency — +- Add documentation on Prefect Cloud teams — +- Update 2.14.2 release notes — +- Fix rendering of marketing banner on the Prefect dashboard — +- Fix typo in `README.md` — ## New Contributors -* @vatsalya-vyas made their first contribution in https://github.com/PrefectHQ/prefect/pull/11058 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.2...2.14.3 +- @vatsalya-vyas made their first contribution in + +**All changes**: ## Release 2.14.2 @@ -320,22 +365,26 @@ assert data == {"my_arg": "custom_value", "state": state} This can be used in conjunction with the `.with_options` method on tasks and flows to dynamically provide extra kwargs to your hooks, like [this example](https://docs.prefect.io/latest/concepts/states/#pass-kwargs-to-your-hooks) in the docs. See the following pull request for implementation details: -- https://github.com/PrefectHQ/prefect/pull/11022 + +- ### Fixes -- Moves responsibility for running `on_cancellation` and `on_crashed` flow hooks to runner when present — https://github.com/PrefectHQ/prefect/pull/11026 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.1...2.14.2 +- Moves responsibility for running `on_cancellation` and `on_crashed` flow hooks to runner when present — + +**All changes**: ## Release 2.14.1 ### Documentation -- Add Python `serve` and `deploy` options to the `schedules` concepts documentation — https://github.com/PrefectHQ/prefect/pull/11000 + +- Add Python `serve` and `deploy` options to the `schedules` concepts documentation — ### Fixes -- Refine flow parameter validation to use the correct form of validation depending on if the parameter is a pydantic v1 or v2 model. — https://github.com/PrefectHQ/prefect/pull/11028 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.14.0...2.14.1 +- Refine flow parameter validation to use the correct form of validation depending on if the parameter is a pydantic v1 or v2 model. — + +**All changes**: ## Release 2.14.0 @@ -360,26 +409,32 @@ hello_world() ``` See implementation details in the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10988 + +- ### Enhancements -- Add flow name to the label for subflow runs in the Prefect UI — https://github.com/PrefectHQ/prefect/pull/11009 + +- Add flow name to the label for subflow runs in the Prefect UI — ### Fixes -- Fix ability to pull flows and build deployments in Windows environments - https://github.com/PrefectHQ/prefect/pull/10989 -- Remove unnecessary work queue health indicator from push pools in the Prefect UI dashboard - https://github.com/PrefectHQ/prefect-ui-library/pull/1813 -- Rename mismatched alembic file — https://github.com/PrefectHQ/prefect/pull/10888 + +- Fix ability to pull flows and build deployments in Windows environments - +- Remove unnecessary work queue health indicator from push pools in the Prefect UI dashboard - +- Rename mismatched alembic file — ### Documentation -- Standardize heading capitalization in guide to developing a new worker type — https://github.com/PrefectHQ/prefect/pull/10999 -- Update Docker guide to mention image builds with `prefect.yaml` and `flow.deploy` — https://github.com/PrefectHQ/prefect/pull/11012 -- Update Kubernetes guide to mention and link to Python-based flow `deploy` creation method — https://github.com/PrefectHQ/prefect/pull/11010 + +- Standardize heading capitalization in guide to developing a new worker type — +- Update Docker guide to mention image builds with `prefect.yaml` and `flow.deploy` — +- Update Kubernetes guide to mention and link to Python-based flow `deploy` creation method — ## New Contributors -* @m-steinhauer made their first contribution in https://github.com/PrefectHQ/prefect/pull/10888 -* @maitlandmarshall made their first contribution in https://github.com/PrefectHQ/prefect/pull/10989 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.8...2.14.0 +- @m-steinhauer made their first contribution in + +- @maitlandmarshall made their first contribution in + +**All changes**: ## Release 2.13.8 @@ -441,38 +496,43 @@ if __name__ == "__main__": You can also use `deploy` as a replacement for `serve` if you want to deploy multiple flows at once. -For more information, check out our tutorial's newly updated [Worker & Work Pools](https://docs.prefect.io/latest/tutorial/workers/) section! +For more information, check out our tutorial's newly updated [Worker & Work Pools](https://docs.prefect.io/latest/tutorial/workers/) section! See implementation details in the following pull requests: -- https://github.com/PrefectHQ/prefect/pull/10957 -- https://github.com/PrefectHQ/prefect/pull/10975 -- https://github.com/PrefectHQ/prefect/pull/10993 + +- +- +- ### Enhancements -- Add `last_polled` column to deployment table — https://github.com/PrefectHQ/prefect/pull/10949 -- Add `status` and `last_polled` to deployment API responses — https://github.com/PrefectHQ/prefect/pull/10951 -- Add flow run graph v2 endpoint tuned for UI applications — https://github.com/PrefectHQ/prefect/pull/10912 -- Add ability to convert `GitRepository` into `git_clone` deployment step — https://github.com/PrefectHQ/prefect/pull/10957 -- Update `/deployments/get_scheduled_flow_runs` endpoint to update deployment status — https://github.com/PrefectHQ/prefect/pull/10969 + +- Add `last_polled` column to deployment table — +- Add `status` and `last_polled` to deployment API responses — +- Add flow run graph v2 endpoint tuned for UI applications — +- Add ability to convert `GitRepository` into `git_clone` deployment step — +- Update `/deployments/get_scheduled_flow_runs` endpoint to update deployment status — ### Fixes -- Clarify CLI prompt message for missing integration library for worker — https://github.com/PrefectHQ/prefect/pull/10990 -- Renamed `ruamel-yaml` to `ruamel.yaml` in `requirements.txt` — https://github.com/PrefectHQ/prefect/pull/10987 -- Clarify work pool banner on Work Pool page UI — https://github.com/PrefectHQ/prefect/pull/10992 + +- Clarify CLI prompt message for missing integration library for worker — +- Renamed `ruamel-yaml` to `ruamel.yaml` in `requirements.txt` — +- Clarify work pool banner on Work Pool page UI — ### Documentation -- Clean up `Using the Prefect Orchestration Client` guide — https://github.com/PrefectHQ/prefect/pull/10968 -- Add link to Coiled's documentation for hosting served flows — https://github.com/PrefectHQ/prefect/pull/10977 -- Clarify that access control lists do not affect related objects — https://github.com/PrefectHQ/prefect/pull/10934 -- Improve block-based deployment concept page metadata and admonitions — https://github.com/PrefectHQ/prefect/pull/10970 -- Update docs to prioritize workers over agents — https://github.com/PrefectHQ/prefect/pull/10904 -- Update work pools and workers tutorial to use `flow.deploy` — https://github.com/PrefectHQ/prefect/pull/10985 -- Move Docker image discussion to Docker guide — https://github.com/PrefectHQ/prefect/pull/10910 + +- Clean up `Using the Prefect Orchestration Client` guide — +- Add link to Coiled's documentation for hosting served flows — +- Clarify that access control lists do not affect related objects — +- Improve block-based deployment concept page metadata and admonitions — +- Update docs to prioritize workers over agents — +- Update work pools and workers tutorial to use `flow.deploy` — +- Move Docker image discussion to Docker guide — ### Contributors -* @lpequignot made their first contribution in https://github.com/PrefectHQ/prefect/pull/10987 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.7...2.13.8 +- @lpequignot made their first contribution in + +**All changes**: ## Release 2.13.7 @@ -481,16 +541,18 @@ See implementation details in the following pull requests: In 2.13.5 we released experimental support for Pydantic V2, which made it co-installable via forced install. In this release, we are enabling co-installation by default which will allow you to leverage Pydantic V2 in your flows and tasks. Additionally, you can choose to update to Pydantic V2 on your own timeline as we maintain compatibility with V1 within flows and tasks. See implementation details in the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10946 + +- ### Documentation -- Fix typo in release notes - https://github.com/PrefectHQ/prefect/pull/10950 +- Fix typo in release notes - ### Contributors + - @taljaards -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.6...2.13.7 +**All changes**: ## Release 2.13.6 @@ -510,14 +572,16 @@ deployments: env: PREFECT_DEFAULT_RESULT_STORAGE_BLOCK: s3/my-s3-bucket-block-name ``` + This enhancement enables easier swapping of result storages by just updating the environment in the UI or in your `prefect.yaml`, eliminating the need to alter your flow source code. See the following pull request for details: -- https://github.com/PrefectHQ/prefect/pull/10925 + +- ### Experimental support for enhanced cancellation -We're introducing a new experimental feature that will enable more consistent and reliable cancellation of flow runs. +We're introducing a new experimental feature that will enable more consistent and reliable cancellation of flow runs. To enable enhanced cancellation, set the `PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION` setting on your worker or agents to `True`: @@ -530,34 +594,40 @@ When enabled, you can cancel flow runs where cancellation can fail, such as when Note: If you are using the Kubernetes worker, you will need to update your `prefect-kubernetes` installation to `0.3.1`. If you are using the Cloud Run or Vertex AI workers, you will need to update your `prefect-gcp` installation to `0.5.1`. See the following pull requests for details: -- https://github.com/PrefectHQ/prefect/pull/10920 -- https://github.com/PrefectHQ/prefect/pull/10944 + +- +- ### Enhancements -- Add link to Prefect Cloud information in the Prefect UI — https://github.com/PrefectHQ/prefect/pull/10909 + +- Add link to Prefect Cloud information in the Prefect UI — ### Fixes -- Avoid `prefect deploy` prompt for remote storage if a global pull step is already defined - https://github.com/PrefectHQ/prefect/pull/10941 + +- Avoid `prefect deploy` prompt for remote storage if a global pull step is already defined - ### Documentation -- Add a guide for using the Prefect client — https://github.com/PrefectHQ/prefect/pull/10924 -- Remove icons from side navigation for improved readability — https://github.com/PrefectHQ/prefect/pull/10908 -- Update deployments tutorial for consistent styling — https://github.com/PrefectHQ/prefect/pull/10911 -- Fix typo in CLI command in deployments tutorial — https://github.com/PrefectHQ/prefect/pull/10937 -- Fix typo in logging guide — https://github.com/PrefectHQ/prefect/pull/10936 -- Update documentation styling — https://github.com/PrefectHQ/prefect/pull/10913 + +- Add a guide for using the Prefect client — +- Remove icons from side navigation for improved readability — +- Update deployments tutorial for consistent styling — +- Fix typo in CLI command in deployments tutorial — +- Fix typo in logging guide — +- Update documentation styling — ### Contributors -* @Sun-of-a-beach made their first contribution in https://github.com/PrefectHQ/prefect/pull/10937 + +- @Sun-of-a-beach made their first contribution in + - @manaw -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.5...2.13.6 +**All changes**: ## Release 2.13.5 ### Load and serve remotely stored flows -You can now load and serve flows from a git repository! +You can now load and serve flows from a git repository! With the new `flow.from_source` method, you can specify a git repository and a path to a flow file in that repository. This method will return a flow object that can be run or served with `flow.serve()`. @@ -578,119 +648,140 @@ When you load and serve a flow from a git repository, the serving process will p To learn more about loading and serving flows from a git repository, check out [the docs](https://docs.prefect.io/latest/concepts/flows/#retrieve-a-flow-from-remote-storage)! See the following pull requests for details: -- https://github.com/PrefectHQ/prefect/pull/10884 -- https://github.com/PrefectHQ/prefect/pull/10850 + +- +- ### Experimental Pydantic 2 Compatibility We're working eagerly toward having `prefect` installable with either `pydantic<2` or `pydantic>2`. As a first step toward compatibility, we've ensured that Prefect's use of `pydantic` is isolated from _your_ use of `pydantic` in as many ways as possible. As of this release, `prefect` still has a stated `pydantic` requirement of `<2`, but we are testing against `pydantic>2` in our continuous integration tests. If you're feeling adventurous, feel free to manually install `pydantic>2` and run some flows with it. If you do, please let us know how it's going with a note in Slack or with a Github issue. See the following pull requests for details -- https://github.com/PrefectHQ/prefect/pull/10860 -- https://github.com/PrefectHQ/prefect/pull/10867 -- https://github.com/PrefectHQ/prefect/pull/10868 -- https://github.com/PrefectHQ/prefect/pull/10870 -- https://github.com/PrefectHQ/prefect/pull/10873 -- https://github.com/PrefectHQ/prefect/pull/10891 -- https://github.com/PrefectHQ/prefect/pull/10876 + +- +- +- +- +- +- +- ### Enhancements -- Use flow run context for default values in task run logger — https://github.com/PrefectHQ/prefect/pull/10334 -- Default `PREFECT_UI_API_URL` to relative path /api — https://github.com/PrefectHQ/prefect/pull/10755 -- Add blob storage options to `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10656 -- Add retries on responses with a 408 status code — https://github.com/PrefectHQ/prefect/pull/10883 + +- Use flow run context for default values in task run logger — +- Default `PREFECT_UI_API_URL` to relative path /api — +- Add blob storage options to `prefect deploy` — +- Add retries on responses with a 408 status code — ### Fixes -- Ensure agents only query work queues in `default-agent-pool` work pool if no pool is specified — https://github.com/PrefectHQ/prefect/pull/10804 -- Update `Runner` to correctly handle spaces in Python executable path — https://github.com/PrefectHQ/prefect/pull/10878 -- Update `PREFECT__FLOW_RUN_ID` environment variable to dash-delimited UUID format — https://github.com/PrefectHQ/prefect/pull/10881 -- Fix bug preventing importing `prefect` in a thread — https://github.com/PrefectHQ/prefect/pull/10871 + +- Ensure agents only query work queues in `default-agent-pool` work pool if no pool is specified — +- Update `Runner` to correctly handle spaces in Python executable path — +- Update `PREFECT__FLOW_RUN_ID` environment variable to dash-delimited UUID format — +- Fix bug preventing importing `prefect` in a thread — ### Documentation -- Add GCP Vertex AI worker to worker types list in work pools documentation — https://github.com/PrefectHQ/prefect/pull/10858 -- Expound upon rate limit info and global concurrency use cases in concurrency guide — https://github.com/PrefectHQ/prefect/pull/10886 -- Point docker guide link to tutorial on workers — https://github.com/PrefectHQ/prefect/pull/10872 -- Clarify workers and work pools as an alternative to `.serve()` in tutorials — https://github.com/PrefectHQ/prefect/pull/10861 -- Fix typo in deployments concept page — https://github.com/PrefectHQ/prefect/pull/10857 -- Remove beta label from push work pool documentation — https://github.com/PrefectHQ/prefect/pull/10848 + +- Add GCP Vertex AI worker to worker types list in work pools documentation — +- Expound upon rate limit info and global concurrency use cases in concurrency guide — +- Point docker guide link to tutorial on workers — +- Clarify workers and work pools as an alternative to `.serve()` in tutorials — +- Fix typo in deployments concept page — +- Remove beta label from push work pool documentation — ### Contributors -* @alexmojaki made their first contribution in https://github.com/PrefectHQ/prefect/pull/10334 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.4...2.13.5 +- @alexmojaki made their first contribution in + +**All changes**: ## Release 2.13.4 ### Enhancements -- Lift API and database constraints that require task runs to have an associated flow run id — https://github.com/PrefectHQ/prefect/pull/10816 + +- Lift API and database constraints that require task runs to have an associated flow run id — ### Fixes -- Fix an issue with infinite scrolling on the sub flow runs tab in the UI - https://github.com/PrefectHQ/prefect-ui-library/pull/1788 + +- Fix an issue with infinite scrolling on the sub flow runs tab in the UI - ### Documentation -- Add dark mode base job template screenshot to work pools documentation — https://github.com/PrefectHQ/prefect/pull/10849 -- Drop beta tag from push work pools documentation — https://github.com/PrefectHQ/prefect/pull/10799 -- Improve logo sizing and general housekeeping - https://github.com/PrefectHQ/prefect/pull/10830 + +- Add dark mode base job template screenshot to work pools documentation — +- Drop beta tag from push work pools documentation — +- Improve logo sizing and general housekeeping - ## Release 2.13.3 ## Allow configuration of a work pool's base job template via the CLI + Previously, the creation and modification of work pools, including editing the base job template, were done through the Prefect UI. Now you can alter the base job template through CLI commands: Retrieve the default base job template for a given work pool: + ```bash prefect work-pool get-default-base-job-template --type kubernetes ``` You can customize the base job template by passing a JSON file to the `--base-job-template` flag: + ```bash prefect work-pool create my-k8s-pool --type kubernetes --base-job-template ./path/template.yaml ``` Useful for version control, you can now make updates to a work pool's base job template via the CLI: + ```bash prefect work-pool update my-work-pool --base-job-template base-job-template.json --description "My work pool" --concurrency-limit 10 ``` See the documentation on [work pools](https://docs.prefect.io/latest/concepts/work-pools/) for more information, or see the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/10793 -- https://github.com/PrefectHQ/prefect/pull/10797 -- https://github.com/PrefectHQ/prefect/pull/10796 -- https://github.com/PrefectHQ/prefect/pull/10798 -- https://github.com/PrefectHQ/prefect/pull/10844 + +- +- +- +- +- ## Allow users to customize their default flow runs view in the Prefect UI -You can now set your own default filter view on your Flow Runs page! You must first save and name a view before you can set it as your default. This setting is only stored locally so it will not be shared across machines/browsers. + +You can now set your own default filter view on your Flow Runs page! You must first save and name a view before you can set it as your default. This setting is only stored locally so it will not be shared across machines/browsers. image Note: The previous default view ("Default view") has been renamed to "Past week". ## New Google Vertex AI work pool and worker -- Run flows in containers on Google Vertex AI. + +- Run flows in containers on Google Vertex AI. - Requires a Google Cloud Platform account and prefect-gcp library installed. Read more [here](https://prefecthq.github.io/prefect-gcp/vertex_worker/). ### Enhancements -- Display `pull_steps` on Deployments page in the Prefect UI — https://github.com/PrefectHQ/prefect/pull/10819 -- Add `/deployments/get_scheduled_flow_runs` endpoint for retrieving scheduled flow runs from deployments — https://github.com/PrefectHQ/prefect/pull/10817 -- Add flow run filter for fetching the first-level subflows for a given flow — https://github.com/PrefectHQ/prefect/pull/10806 + +- Display `pull_steps` on Deployments page in the Prefect UI — +- Add `/deployments/get_scheduled_flow_runs` endpoint for retrieving scheduled flow runs from deployments — +- Add flow run filter for fetching the first-level subflows for a given flow — ### Fixes -- Raise `RuntimeError` error if `pip_install_requirements` step fails — https://github.com/PrefectHQ/prefect/pull/10823 -- Use a fixed list of known collection registry views - https://github.com/PrefectHQ/prefect/pull/10838 + +- Raise `RuntimeError` error if `pip_install_requirements` step fails — +- Use a fixed list of known collection registry views - ### Documentation -- Fix typos in documentation and codebase — https://github.com/PrefectHQ/prefect/pull/10813 -- Fix example in tasks concept documentation — https://github.com/PrefectHQ/prefect/pull/10833 -- Update `git_clone` deployment step example in documentation — https://github.com/PrefectHQ/prefect/pull/10827 -- Add `prefect deploy` guide to guide index for visibility — https://github.com/PrefectHQ/prefect/pull/10828 -- Fix warning in deployment storage guide documentation — https://github.com/PrefectHQ/prefect/pull/10825 + +- Fix typos in documentation and codebase — +- Fix example in tasks concept documentation — +- Update `git_clone` deployment step example in documentation — +- Add `prefect deploy` guide to guide index for visibility — +- Fix warning in deployment storage guide documentation — ### Contributors -* @arthurgtllr made their first contribution in https://github.com/PrefectHQ/prefect/pull/10833 + +- @arthurgtllr made their first contribution in + - @mj0nez -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.2...2.13.3 +**All changes**: ## Release 2.13.2 @@ -724,6 +815,7 @@ if __name__ == "__main__": ``` An attempt to run the created deployment with invalid parameters will fail and give a reason the flow run cannot be created: + ```bash > prefect deployment run 'my-flow/testing-params' -p person='{"name": 1}' @@ -733,124 +825,143 @@ Error creating flow run: Validation failed for field 'person.name'. Failure reas You can enable parameter enforcement via `prefect deploy` with the `--enforce-parameter-schema` flag or by setting `enforce_parameter_schema` to `True` in your `prefect.yaml` file. See the following pull request for details: -- https://github.com/PrefectHQ/prefect/pull/10773 + +- ### Enhanced deployment flexibility with pattern-based deploying In an effort to increase flexibility and provide more powerful deployment options, this enhancement enables users to deploy flows based on a variety of patterns, facilitating versatile and dynamic deployment management: **Deploy all deployments for a specific flow:** + ```bash prefect deploy -n flow-a/* ``` **Deploy all deployments for a specific deployment:** + ```bash prefect deploy -n */prod ``` + Note: This was previously possible in non-interactive mode with `prefect --no-prompt deploy -n prod` **Deploy all deployments containing a specified string in the flow name:** + ```bash prefect deploy -n *extract*/* ``` **Deploy deployments with a mix of pattern matching styles** + ```bash prefect deploy -n flow-a/* -n */prod ``` **Deploy deployments with a mix of pattern matching and without:** + ```bash prefect deploy -n flow-a/* -n flow-b/default ``` See the following pull request for details: -- https://github.com/PrefectHQ/prefect/pull/10772 + +- ### Enhancements -- Add API route for work pool counts — https://github.com/PrefectHQ/prefect/pull/10770 -- Add CLI command to get default base job template — https://github.com/PrefectHQ/prefect/pull/10776 + +- Add API route for work pool counts — +- Add CLI command to get default base job template — ### Fixes -- Make paths relative rather than absolute in the `prefect dev build-ui` command — https://github.com/PrefectHQ/prefect/pull/10390 -- Lower the upper bound on pinned pendulum library — https://github.com/PrefectHQ/prefect/pull/10752 -- Fix command handling in `run_shell_script` deployment step on Windows — https://github.com/PrefectHQ/prefect/pull/10719 -- Fix validation on concurrency limits — https://github.com/PrefectHQ/prefect/pull/10790 -- Fix Prefect variable resolution in deployments section of `prefect.yaml` — https://github.com/PrefectHQ/prefect/pull/10783 + +- Make paths relative rather than absolute in the `prefect dev build-ui` command — +- Lower the upper bound on pinned pendulum library — +- Fix command handling in `run_shell_script` deployment step on Windows — +- Fix validation on concurrency limits — +- Fix Prefect variable resolution in deployments section of `prefect.yaml` — ### Documentation -- Update UI screenshot for role creation — https://github.com/PrefectHQ/prefect/pull/10732 -- Add `push work pools` tag to push work pools guide to raise visibility — https://github.com/PrefectHQ/prefect/pull/10739 -- Update docs with recent brand changes — https://github.com/PrefectHQ/prefect/pull/10736 -- Update Prefect Cloud quickstart guide to include new features — https://github.com/PrefectHQ/prefect/pull/10742 -- Fix broken diagram in workers tutorial — https://github.com/PrefectHQ/prefect/pull/10762 -- Add screenshots to artifacts concept page — https://github.com/PrefectHQ/prefect/pull/10748 -- Remove boost from block-based deployments page in documentation and improve visibility of `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10775 -- Add example of retrieving default base job template to work pools concept documentation — https://github.com/PrefectHQ/prefect/pull/10784 -- Add references to `enforce_parameter_schema` to docs — https://github.com/PrefectHQ/prefect/pull/10782 -- Add documentation for pattern matching in `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10791 + +- Update UI screenshot for role creation — +- Add `push work pools` tag to push work pools guide to raise visibility — +- Update docs with recent brand changes — +- Update Prefect Cloud quickstart guide to include new features — +- Fix broken diagram in workers tutorial — +- Add screenshots to artifacts concept page — +- Remove boost from block-based deployments page in documentation and improve visibility of `prefect deploy` — +- Add example of retrieving default base job template to work pools concept documentation — +- Add references to `enforce_parameter_schema` to docs — +- Add documentation for pattern matching in `prefect deploy` — ### New contributors -* @danielhstahl made their first contribution in https://github.com/PrefectHQ/prefect/pull/10390 -* @morremeyer made their first contribution in https://github.com/PrefectHQ/prefect/pull/10759 -* @NikoRaisanen made their first contribution in https://github.com/PrefectHQ/prefect/pull/10719 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.1...2.13.2 +- @danielhstahl made their first contribution in + +- @morremeyer made their first contribution in +- @NikoRaisanen made their first contribution in + +**All changes**: ## Release 2.13.1 ### Hide subflow runs in the Prefect UI + We’ve added the ability to filter out subflow runs from the list on the Flow Runs page! This feature is especially beneficial for those who frequently use subflows, making it easier to focus on parent flows with less clutter. ![Hide subflows in UI demo](https://github.com/PrefectHQ/prefect/assets/31014960/7f6a9473-8003-4a90-8ff7-4d766623b38b) See the following for implementation details: -- https://github.com/PrefectHQ/prefect/pull/10708 + +- ### Enhancements -- Add `run_count` to `prefect.runtime.flow_run` — https://github.com/PrefectHQ/prefect/pull/10676 -- Add `run_count` to `prefect.runtime.task_run` — https://github.com/PrefectHQ/prefect/pull/10676 -- Allow passing deployment triggers via CLI with `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10690 -- Add `is_null` filter for deployments to `/flows/filter` endpoint — https://github.com/PrefectHQ/prefect/pull/10724 -- Show associated flow name on Custom Run page in the Prefect UI - https://github.com/PrefectHQ/prefect-ui-library/pull/1744 -- Add ability to reset a task-based concurrency limit from the UI - https://github.com/PrefectHQ/prefect-ui-library/pull/1746 -- Display error `details` returned by API - https://github.com/PrefectHQ/prefect-ui-library/pull/1712 -- Add pagination to Deployments and Flows pages in the Prefect UI - https://github.com/PrefectHQ/prefect-ui-library/pull/1732 -- Add opt-in to display large flow run graphs in Prefect UI - https://github.com/PrefectHQ/prefect-ui-library/pull/1739 -- Add Prefect logo to UI sidebar and fix dashboard padding — https://github.com/PrefectHQ/prefect/pull/10684 -- Add ability to update existing deployment configurations with `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10718 + +- Add `run_count` to `prefect.runtime.flow_run` — +- Add `run_count` to `prefect.runtime.task_run` — +- Allow passing deployment triggers via CLI with `prefect deploy` — +- Add `is_null` filter for deployments to `/flows/filter` endpoint — +- Show associated flow name on Custom Run page in the Prefect UI - +- Add ability to reset a task-based concurrency limit from the UI - +- Display error `details` returned by API - +- Add pagination to Deployments and Flows pages in the Prefect UI - +- Add opt-in to display large flow run graphs in Prefect UI - +- Add Prefect logo to UI sidebar and fix dashboard padding — +- Add ability to update existing deployment configurations with `prefect deploy` — ### Fixes -- Avoid creating unpersisted blocks remotely — https://github.com/PrefectHQ/prefect/pull/10649 -- Handling DST in `CronSchedules` — https://github.com/PrefectHQ/prefect/pull/10678 -- Allow Python classes as flow/task type hints — https://github.com/PrefectHQ/prefect/pull/10711 -- Fix formatting of `SendgridEmail.to_emails` example in notifications API reference — https://github.com/PrefectHQ/prefect/pull/10669 -- Streamline Artifact search filters to match other pages in the Prefect UI - https://github.com/PrefectHQ/prefect-ui-library/pull/1689 -- Improve the mobile navigation in the Prefect UI — https://github.com/PrefectHQ/prefect/pull/10686 + +- Avoid creating unpersisted blocks remotely — +- Handling DST in `CronSchedules` — +- Allow Python classes as flow/task type hints — +- Fix formatting of `SendgridEmail.to_emails` example in notifications API reference — +- Streamline Artifact search filters to match other pages in the Prefect UI - +- Improve the mobile navigation in the Prefect UI — ### Documentation -- Add object ACL documentation — https://github.com/PrefectHQ/prefect/pull/10695 -- Use better arrow icon for `Try Cloud` button — https://github.com/PrefectHQ/prefect/pull/10675 -- Improves bash output format in code blocks on concepts/agents page — https://github.com/PrefectHQ/prefect/pull/10680 -- Update concepts screen shots to reflect improved Prefect UI — https://github.com/PrefectHQ/prefect/pull/10670 -- Update event feed screenshot in concepts pages — https://github.com/PrefectHQ/prefect/pull/10685 -- Update Prefect Cloud index screenshots and remove Prefect Cloud quickstart — https://github.com/PrefectHQ/prefect/pull/10692 -- Add error summaries section to Prefect Cloud index — https://github.com/PrefectHQ/prefect/pull/10698 -- Clarify supported artifact types — https://github.com/PrefectHQ/prefect/pull/10706 -- Update Prefect Cloud pages screenshots — https://github.com/PrefectHQ/prefect/pull/10700 -- Fix broken links in events concept docs and variables guide — https://github.com/PrefectHQ/prefect/pull/10726 + +- Add object ACL documentation — +- Use better arrow icon for `Try Cloud` button — +- Improves bash output format in code blocks on concepts/agents page — +- Update concepts screen shots to reflect improved Prefect UI — +- Update event feed screenshot in concepts pages — +- Update Prefect Cloud index screenshots and remove Prefect Cloud quickstart — +- Add error summaries section to Prefect Cloud index — +- Clarify supported artifact types — +- Update Prefect Cloud pages screenshots — +- Fix broken links in events concept docs and variables guide — ### New Contributors -* @odoublewen made their first contribution in https://github.com/PrefectHQ/prefect/pull/10706 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.13.0...2.13.1 +- @odoublewen made their first contribution in + +**All changes**: ## Release 2.13.0 ### Introducing global concurrency limits -Control task execution and system stability with Prefect's new global concurrency and rate limits. +Control task execution and system stability with Prefect's new global concurrency and rate limits. - **Concurrency Limits:** Manage task execution efficiently, controlling how many tasks can run simultaneously. Ideal for optimizing resource usage and customizing task execution. @@ -859,13 +970,15 @@ Control task execution and system stability with Prefect's new global concurrenc Choose concurrency limits for resource optimization and task management, and opt for rate limits to maintain system stability and fair access to services. To begin using global concurrency limits check out our [guide](https://docs.prefect.io/guides/global-concurrency-limits/). See the following pull request for details: -- https://github.com/PrefectHQ/prefect/pull/10496 + +- ### Introducing work pool and worker status Work pools and workers are critical components of Prefect's distributed execution model. To help you monitor and manage your work pools and workers, we've added status indicators to the Prefect UI. Work pools can now have one of three statuses: + - `Ready` - at least one online worker is polling the work pool and the work pool is ready to accept work. - `Not Ready` - no online workers are polling the work pool and indicates that action needs to be taken to allow the work pool to accept work. - `Paused` - the work pool is paused and work will not be executed until it is unpaused. @@ -873,6 +986,7 @@ Work pools can now have one of three statuses: ![Prefect dashboard showing work pool health](https://user-images.githubusercontent.com/12350579/265874237-7fae81e0-1b1a-460b-9fc5-92d969326d22.png) Workers can now have one of two statuses: + - `Online` - the worker is polling the work pool and is ready to accept work. - `Offline` - the worker is not polling the work pool and is not ready to accept work. Indicates that the process running the worker has stopped or crashed. @@ -883,22 +997,26 @@ With the introduction of work pool and worker status, we are deprecating work qu See the documentation on [work pool status](https://docs.prefect.io/latest/concepts/work-pools/#work-pool-status) and [worker status](https://docs.prefect.io/latest/concepts/work-pools/#worker-status) for more information. See the following pull request for details: -- https://github.com/PrefectHQ/prefect/pull/10636 -- https://github.com/PrefectHQ/prefect/pull/10654 + +- +- ### Removing deprecated Orion references -Six months ago, we deprecated references to `orion` in our codebase. In this release, we're removing those references. If you still have references to `ORION` in your profile, run `prefect config validate` to automatically convert all of the settings in your *current* profile to the new names! +Six months ago, we deprecated references to `orion` in our codebase. In this release, we're removing those references. If you still have references to `ORION` in your profile, run `prefect config validate` to automatically convert all of the settings in your _current_ profile to the new names! For example: + ```bash ❯ prefect config validate Updated 'PREFECT_ORION_DATABASE_CONNECTION_URL' to 'PREFECT_API_DATABASE_CONNECTION_URL'. Configuration valid! ``` -#### Below is a full guide to the changes: -##### Settings renamed: +#### Below is a full guide to the changes + +##### Settings renamed + - `PREFECT_LOGGING_ORION_ENABLED` → `PREFECT_LOGGING_TO_API_ENABLED` - `PREFECT_LOGGING_ORION_BATCH_INTERVAL` → `PREFECT_LOGGING_TO_API_BATCH_INTERVAL` - `PREFECT_LOGGING_ORION_BATCH_SIZE` → `PREFECT_LOGGING_TO_API_BATCH_SIZE` @@ -933,7 +1051,9 @@ Configuration valid! - `PREFECT_ORION_SERVICES_PAUSE_EXPIRATIONS_ENABLED` → `PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_ENABLED` - `PREFECT_ORION_TASK_CACHE_KEY_MAX_LENGTH` → `PREFECT_API_TASK_CACHE_KEY_MAX_LENGTH` - `PREFECT_ORION_SERVICES_CANCELLATION_CLEANUP_ENABLED` → `PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED` -##### Changes: + +##### Changes + - Module `prefect.client.orion` → `prefect.client.orchestration` - Command group `prefect orion` → `prefect server` - Module `prefect.orion` → `prefect.server` @@ -951,48 +1071,55 @@ Configuration valid! - Class `OrionClient` → `PrefectClient` See the following pull request for details: -- Remove deprecated `orion` references — https://github.com/PrefectHQ/prefect/pull/10642 + +- Remove deprecated `orion` references — ### Fixes -- Fix an issue with `prefect server start` on Windows - https://github.com/PrefectHQ/prefect/pull/10547 + +- Fix an issue with `prefect server start` on Windows - ### Documentation -- Update deployment concept documentation to emphasize server-side deployment — https://github.com/PrefectHQ/prefect/pull/10615 -- Add Kubernetes guide for deploying worker to Azure AKS — https://github.com/PrefectHQ/prefect/pull/10575 -- Add information on `--no-prompt` and `PREFECT_CLI_PROMPT` to deployment documentation — https://github.com/PrefectHQ/prefect/pull/10600 -- Fix broken link to docker guide with redirect and harmonize naming — https://github.com/PrefectHQ/prefect/pull/10624 -- Remove invalid link in API keys documentation — https://github.com/PrefectHQ/prefect/pull/10658 -- Update screenshots and CLI log output in quickstart documentation — https://github.com/PrefectHQ/prefect/pull/10659 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.12.1...2.13.0 +- Update deployment concept documentation to emphasize server-side deployment — +- Add Kubernetes guide for deploying worker to Azure AKS — +- Add information on `--no-prompt` and `PREFECT_CLI_PROMPT` to deployment documentation — +- Fix broken link to docker guide with redirect and harmonize naming — +- Remove invalid link in API keys documentation — +- Update screenshots and CLI log output in quickstart documentation — + +**All changes**: ## Release 2.12.1 This release includes some important fixes and enhancements. In particular, it resolves an issue preventing the flow run graph from rendering correctly in some cases. ### Enhancements -- Reduce logging noise on QueueServices startup failures and item processing failures — https://github.com/PrefectHQ/prefect/pull/10564 -- Expose a setting for configuring a process limit on served flows — https://github.com/PrefectHQ/prefect/pull/10602 + +- Reduce logging noise on QueueServices startup failures and item processing failures — +- Expose a setting for configuring a process limit on served flows — ### Fixes -- Improve failure recovery for websockets — https://github.com/PrefectHQ/prefect/pull/10597 -- Fix flow run graph rendering issues — https://github.com/PrefectHQ/prefect/pull/10606 + +- Improve failure recovery for websockets — +- Fix flow run graph rendering issues — ### Documentation -- Update Docker guide to include with `flow.serve()` — https://github.com/PrefectHQ/prefect/pull/10596 + +- Update Docker guide to include with `flow.serve()` — ### Contributors -* @urimandujano made their first contribution in https://github.com/PrefectHQ/prefect/pull/10564 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.12.0...2.12.1 +- @urimandujano made their first contribution in + +**All changes**: ## Release 2.12.0 ### Introducing `Flow.serve()` -We're excited to introduce a radically simple way to deploy flows. +We're excited to introduce a radically simple way to deploy flows. -The new `.serve()` method available on every flow allows you to take your existing flows and schedule or trigger runs via the Prefect UI and CLI. +The new `.serve()` method available on every flow allows you to take your existing flows and schedule or trigger runs via the Prefect UI and CLI. This addition makes it easier than it's ever been to deploy flows with Prefect: @@ -1026,11 +1153,11 @@ Running this script will start a process that will run the `hello` flow every ho To start serving your flows, check out our newly updated [quickstart](https://docs.prefect.io/latest/getting-started/quickstart/) and [tutorial](https://docs.prefect.io/latest/tutorial/). See the following pull requests for details: -- https://github.com/PrefectHQ/prefect/pull/10534 -- https://github.com/PrefectHQ/prefect/pull/10549 -- https://github.com/PrefectHQ/prefect/pull/10574 -- https://github.com/PrefectHQ/prefect/pull/10585 +- +- +- +- ### A fresh look and feel @@ -1039,50 +1166,56 @@ The Prefect UI just got a fresh coat of paint! We've carefully updated colors th ![Updated Prefect UI in light and dark modes](https://github.com/PrefectHQ/prefect/assets/42048900/c526619c-22d3-44e6-82ee-255ae1233035) See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/10546 -- https://github.com/PrefectHQ/prefect/pull/10578 -- https://github.com/PrefectHQ/prefect/pull/10584 -- https://github.com/PrefectHQ/prefect/pull/10583 -- https://github.com/PrefectHQ/prefect/pull/10588 + +- +- +- +- +- ### Enhancements -- Allow JSON infra overrides via `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10355 -- Improve validation for `Flow.name` — https://github.com/PrefectHQ/prefect/pull/10463 -- Add a Docker image for conda for Python 3.11 — https://github.com/PrefectHQ/prefect/pull/10532 -- Increase default `PREFECT_API_REQUEST_TIMEOUT` setting to 60 seconds — https://github.com/PrefectHQ/prefect/pull/10543 -- Remove missing work queue warning from the deployment page — https://github.com/PrefectHQ/prefect/pull/10550 -- Add `PREFECT_SQLALCHEMY_POOL_SIZE` and `PREFECT_SQLALCHEMY_MAX_OVERFLOW` settings to configure SQLAlchemy connection pool size — https://github.com/PrefectHQ/prefect/pull/10348 -- Improve format handling of `GitLab` and `Bitbucket` tokens during `git_clone` deployment step — https://github.com/PrefectHQ/prefect/pull/10555 -- Persist active tabs in Prefect UI pages upon refresh — https://github.com/PrefectHQ/prefect/pull/10544 -- Add ability to view subflows in the UI that are linked from `run_deployment` with `DaskTaskRunner` and `RayTaskRunner` — https://github.com/PrefectHQ/prefect/pull/10541 -- Improve CLI output for push work pools https://github.com/PrefectHQ/prefect/pull/10582 + +- Allow JSON infra overrides via `prefect deploy` — +- Improve validation for `Flow.name` — +- Add a Docker image for conda for Python 3.11 — +- Increase default `PREFECT_API_REQUEST_TIMEOUT` setting to 60 seconds — +- Remove missing work queue warning from the deployment page — +- Add `PREFECT_SQLALCHEMY_POOL_SIZE` and `PREFECT_SQLALCHEMY_MAX_OVERFLOW` settings to configure SQLAlchemy connection pool size — +- Improve format handling of `GitLab` and `Bitbucket` tokens during `git_clone` deployment step — +- Persist active tabs in Prefect UI pages upon refresh — +- Add ability to view subflows in the UI that are linked from `run_deployment` with `DaskTaskRunner` and `RayTaskRunner` — +- Improve CLI output for push work pools ### Fixes -- Pin `anyio` to < 4 in `requirements.txt` — https://github.com/PrefectHQ/prefect/pull/10570 -- Add upper bounds to core requirements to prevent major version upgrades https://github.com/PrefectHQ/prefect/pull/10592 -- Fix race condition in concurrent subflow runs involving `AsyncWaiters` — https://github.com/PrefectHQ/prefect/pull/10533 -- Fix `cloud login` false success when `PREFECT_API_KEY` set as environment variable or expired — https://github.com/PrefectHQ/prefect/pull/8641 -- Fix ability to view deployments page tags on larger screens - https://github.com/PrefectHQ/prefect/pull/10566 -- Properly indent `docker-git` recipe `prefect.yaml` — https://github.com/PrefectHQ/prefect/pull/10519 -- Fix Slack community invitation link — https://github.com/PrefectHQ/prefect/pull/10509 + +- Pin `anyio` to < 4 in `requirements.txt` — +- Add upper bounds to core requirements to prevent major version upgrades +- Fix race condition in concurrent subflow runs involving `AsyncWaiters` — +- Fix `cloud login` false success when `PREFECT_API_KEY` set as environment variable or expired — +- Fix ability to view deployments page tags on larger screens - +- Properly indent `docker-git` recipe `prefect.yaml` — +- Fix Slack community invitation link — ### Experimental -- Serialize concurrency requests — https://github.com/PrefectHQ/prefect/pull/10545 + +- Serialize concurrency requests — ### Documentation -- Detail Kubernetes work pool usage in Kubernetes guide — https://github.com/PrefectHQ/prefect/pull/10516 -- Add quickstart documentation, simplify welcome page and API reference overview — https://github.com/PrefectHQ/prefect/pull/10520 -- Add block and agent-based deployments to leftside navigation — https://github.com/PrefectHQ/prefect/pull/10528 -- Add `Try Prefect Cloud` button to documentation header — https://github.com/PrefectHQ/prefect/pull/10537 -- Remove blank menu bar in documentation header — https://github.com/PrefectHQ/prefect/pull/10565 -- Fix link to guide on moving data to and from cloud providers — https://github.com/PrefectHQ/prefect/pull/10521 -- Shorten push work pools description in guides index — https://github.com/PrefectHQ/prefect/pull/10589 -- Organize guides index into sections: Development, Execution, Workers and Agents, and Other Guides — https://github.com/PrefectHQ/prefect/pull/10587 + +- Detail Kubernetes work pool usage in Kubernetes guide — +- Add quickstart documentation, simplify welcome page and API reference overview — +- Add block and agent-based deployments to leftside navigation — +- Add `Try Prefect Cloud` button to documentation header — +- Remove blank menu bar in documentation header — +- Fix link to guide on moving data to and from cloud providers — +- Shorten push work pools description in guides index — +- Organize guides index into sections: Development, Execution, Workers and Agents, and Other Guides — ### Contributors + - @mattklein -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.11.5...2.12.0 +**All changes**: ## Release 2.11.5 @@ -1099,57 +1232,65 @@ Moving data to cloud-based storage and retrieving it is crucial in many data eng For those aiming to optimize their flows using Kubernetes, this guide provides a deep dive on how to efficiently run flows on Kubernetes using containers. Catering to both novices and seasoned experts, [this guide](https://docs.prefect.io/latest/guides/deployment/kubernetes/) offers insights for all proficiency levels. See the following pull requests for details: -- https://github.com/PrefectHQ/prefect/pull/10133 -- https://github.com/PrefectHQ/prefect/pull/10368 -- https://github.com/PrefectHQ/prefect/pull/10591 + +- +- +- ### Enhancements -- Warn users upon setting a misconfigured `PREFECT_API_URL` — https://github.com/PrefectHQ/prefect/pull/10450 -- Show CLI warning if worker is polling a paused work pool or queue — https://github.com/PrefectHQ/prefect/pull/10369 -- Optimize the query generated by the `/task_runs` endpoint — https://github.com/PrefectHQ/prefect/pull/10422 -- Extend optimization on `/task_runs` endpoint to include safety guard — https://github.com/PrefectHQ/prefect/pull/10466 -- Add `DiscordWebhook` notification block — https://github.com/PrefectHQ/prefect/pull/10394 -- Remove reference to deprecated `prefect project ls` in interactive `prefect deploy` command — https://github.com/PrefectHQ/prefect/pull/10473 + +- Warn users upon setting a misconfigured `PREFECT_API_URL` — +- Show CLI warning if worker is polling a paused work pool or queue — +- Optimize the query generated by the `/task_runs` endpoint — +- Extend optimization on `/task_runs` endpoint to include safety guard — +- Add `DiscordWebhook` notification block — +- Remove reference to deprecated `prefect project ls` in interactive `prefect deploy` command — ### Fixes -- Remove base job template validation when work pools are read — https://github.com/PrefectHQ/prefect/pull/10486 + +- Remove base job template validation when work pools are read — ### Experimental -- Codify concurrency context managers and rate limiting with tests — https://github.com/PrefectHQ/prefect/pull/10414 + +- Codify concurrency context managers and rate limiting with tests — ### Documentation -- Add reference to workers in flows documentation admonition — https://github.com/PrefectHQ/prefect/pull/10464 -- Combine Kubernetes worker and flows pages — https://github.com/PrefectHQ/prefect/pull/10448 -- Remove references to `flow_name` from deployments documentation — https://github.com/PrefectHQ/prefect/pull/10477 -- Improve readability of Kubernetes guide — https://github.com/PrefectHQ/prefect/pull/10481 -- Fix typos in contribution and host documentation — https://github.com/PrefectHQ/prefect/pull/10488 -- Raise visibility of push work pools documentation — https://github.com/PrefectHQ/prefect/pull/10497 -- Fix heading size, remove unnecessary link in deployments documentation — https://github.com/PrefectHQ/prefect/pull/10489 -- Add GCP-specific guide for deploying a GKE cluster to host a worker — https://github.com/PrefectHQ/prefect/pull/10490 -- Fix typo in `prefect-gcs` deployment example — https://github.com/PrefectHQ/prefect/pull/10442 -- Move guide on upgrading from agents to workers — https://github.com/PrefectHQ/prefect/pull/10445 -- Fix grammatical errors in documentation — https://github.com/PrefectHQ/prefect/pull/10457 -- Clarify deployments variables and fix `prefect.yaml` example — https://github.com/PrefectHQ/prefect/pull/10474 -- Update `README` header image with new Prefect branding — https://github.com/PrefectHQ/prefect/pull/10493 + +- Add reference to workers in flows documentation admonition — +- Combine Kubernetes worker and flows pages — +- Remove references to `flow_name` from deployments documentation — +- Improve readability of Kubernetes guide — +- Fix typos in contribution and host documentation — +- Raise visibility of push work pools documentation — +- Fix heading size, remove unnecessary link in deployments documentation — +- Add GCP-specific guide for deploying a GKE cluster to host a worker — +- Fix typo in `prefect-gcs` deployment example — +- Move guide on upgrading from agents to workers — +- Fix grammatical errors in documentation — +- Clarify deployments variables and fix `prefect.yaml` example — +- Update `README` header image with new Prefect branding — ## Contributors -- @mattklein made their first contribution in https://github.com/PrefectHQ/prefect/pull/10422 -- @vishalsanfran made their first contribution in https://github.com/PrefectHQ/prefect/pull/10394 + +- @mattklein made their first contribution in +- @vishalsanfran made their first contribution in - @AmanSal1 - @mj0nez -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.11.4...2.11.5 +**All changes**: ## Release 2.11.4 ### Guide to upgrade from agents to workers -Upgrading to workers significantly enhances the experience of deploying flows. It simplifies the specification of each flow's infrastructure and runtime environment. + +Upgrading to workers significantly enhances the experience of deploying flows. It simplifies the specification of each flow's infrastructure and runtime environment. A [worker](/concepts/work-pools/#worker-overview) is the fusion of an [agent](/concepts/agents/) with an [infrastructure block](/concepts/infrastructure/). Like agents, workers poll a work pool for flow runs that are scheduled to start. Like infrastructure blocks, workers are typed - they work with only one kind of infrastructure and they specify the default configuration for jobs submitted to that infrastructure. We've written [a handy guide](https://docs.prefect.io/latest/guides/upgrade-guide-agents-to-workers/) that describes how to upgrade from agents to workers in just a few quick steps. ### Visualize your flow before running it + Until now, the only way to produce a visual schematic of a flow has been to run it and view the corresponding flow run page in the Prefect UI. Some flows, though, are time consuming or expensive to run. Now, you can get a quick sense of the structure of your flow using the `.visualize()` method. Calling this method will attempt to locally produce an image of the flow's schematic diagram without running the flow's code. ![viz-return-value-tracked](https://github.com/PrefectHQ/prefect/assets/3407835/325ef46e-82ce-4400-93d2-b3110c805116) @@ -1157,36 +1298,40 @@ Until now, the only way to produce a visual schematic of a flow has been to run See the [flows documentation](https://docs.prefect.io/latest/concepts/flows/#visualizing-flow-structure) or the [pull request](https://github.com/PrefectHQ/prefect/pull/10417) for more information. ### Enhancements -- Update `prefect deploy` to skip building docker image prompt if `build` key explicitly set to null in `prefect.yaml` — https://github.com/PrefectHQ/prefect/pull/10371 -- Handle spot instance eviction in Kubernetes Infrastructure Block — https://github.com/PrefectHQ/prefect/pull/10426 + +- Update `prefect deploy` to skip building docker image prompt if `build` key explicitly set to null in `prefect.yaml` — +- Handle spot instance eviction in Kubernetes Infrastructure Block — ### Fixes -- Reduce wait time between tasks by adding a clause to the visiting function to raise if it encounters a quote annotation — https://github.com/PrefectHQ/prefect/pull/10370 -- Enable dashboard filters to update with each polling interval so the 24h time span constantly updates — https://github.com/PrefectHQ/prefect/pull/10327 -- Resolve issue with validation of templated variables in base job template of work pool — https://github.com/PrefectHQ/prefect/pull/10385 -- Update CLI to refer to a "work pool" instead of a "worker pool" — https://github.com/PrefectHQ/prefect/pull/10309 + +- Reduce wait time between tasks by adding a clause to the visiting function to raise if it encounters a quote annotation — +- Enable dashboard filters to update with each polling interval so the 24h time span constantly updates — +- Resolve issue with validation of templated variables in base job template of work pool — +- Update CLI to refer to a "work pool" instead of a "worker pool" — ### Documentation -- Elevate Guides in navigation and remove migration guide — https://github.com/PrefectHQ/prefect/pull/10361 -- Update notes about community support — https://github.com/PrefectHQ/prefect/pull/10322 -- Update concepts page to clean up table and remove unnecessary header — https://github.com/PrefectHQ/prefect/pull/10374 -- Improve headings on deployments concept page — https://github.com/PrefectHQ/prefect/pull/10366 -- Update the storage guide for Bitbucket to add `x-token-auth` — https://github.com/PrefectHQ/prefect/pull/10379 -- Add Planetary Computer collection — https://github.com/PrefectHQ/prefect/pull/10387 -- Highlight `@flow` decorator instead of function in tutorial — https://github.com/PrefectHQ/prefect/pull/10401 -- Update tutorial summary list — https://github.com/PrefectHQ/prefect/pull/10403 -- Update Cloud connection guide to include whitelisting URLs — https://github.com/PrefectHQ/prefect/pull/10418 -- Update code snippets and highlighting in tutorial — https://github.com/PrefectHQ/prefect/pull/10391 -- Remove "Reference Material" section from tutorial — https://github.com/PrefectHQ/prefect/pull/10402 -- Fix typo in schedules concept page — https://github.com/PrefectHQ/prefect/pull/10378 -- Fix typo on artifacts concept page — https://github.com/PrefectHQ/prefect/pull/10380 + +- Elevate Guides in navigation and remove migration guide — +- Update notes about community support — +- Update concepts page to clean up table and remove unnecessary header — +- Improve headings on deployments concept page — +- Update the storage guide for Bitbucket to add `x-token-auth` — +- Add Planetary Computer collection — +- Highlight `@flow` decorator instead of function in tutorial — +- Update tutorial summary list — +- Update Cloud connection guide to include whitelisting URLs — +- Update code snippets and highlighting in tutorial — +- Remove "Reference Material" section from tutorial — +- Fix typo in schedules concept page — +- Fix typo on artifacts concept page — ### Contributors -- @shahrukhx01 made their first contribution in https://github.com/PrefectHQ/prefect/pull/10378 + +- @shahrukhx01 made their first contribution in - @giorgiobasile - @marwan116 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.11.3...2.11.4 +**All changes**: ## Release 2.11.3 @@ -1195,6 +1340,7 @@ See the [flows documentation](https://docs.prefect.io/latest/concepts/flows/#vis Previously, to expand environment variables in the `run_shell_script` step, you had to enclose your scripts in `bash -c`. We have optimized this process by introducing a new field: `expand_env_vars`. By setting this field to `true`, you can easily pass environment variables to your script. Consider the following example where the script utilizes the `$USER` environment variable: + ```yaml pull: - prefect.deployments.steps.run_shell_script: @@ -1206,131 +1352,157 @@ pull: ``` For implementation details, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10198 + +- ### Enhancements -- Change language for `--ci` option in `prefect deploy --help`. — https://github.com/PrefectHQ/prefect/pull/10347 + +- Change language for `--ci` option in `prefect deploy --help`. — ### Experimental -- Port concurrency limit v2 API and modeling from Prefect Cloud — https://github.com/PrefectHQ/prefect/pull/10363 + +- Port concurrency limit v2 API and modeling from Prefect Cloud — ### Documentation -- Add Prefect Cloud quickstart to navigation menu — https://github.com/PrefectHQ/prefect/pull/10350 -- Fix typo in deployments documentation — https://github.com/PrefectHQ/prefect/pull/10353 -- Reorganize concepts pages — https://github.com/PrefectHQ/prefect/pull/10359 + +- Add Prefect Cloud quickstart to navigation menu — +- Fix typo in deployments documentation — +- Reorganize concepts pages — ### Contributors + - @AmanSal1 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.11.2...2.11.3 +**All changes**: ## Release 2.11.2 ### Enhancements -- Explicitly set all calls to `pendulum.now()` to "UTC" — https://github.com/PrefectHQ/prefect/pull/10320 + +- Explicitly set all calls to `pendulum.now()` to "UTC" — ### Documentation -- Add guide for specifying storage for deployments — https://github.com/PrefectHQ/prefect/pull/10150 -- Add ACI push work pool guide — https://github.com/PrefectHQ/prefect/pull/10323 -- Move some concepts and cloud pages to guides section — https://github.com/PrefectHQ/prefect/pull/10328 + +- Add guide for specifying storage for deployments — +- Add ACI push work pool guide — +- Move some concepts and cloud pages to guides section — ### Deprecations -- Deprecate `FlowRunCreate.deployment_id` — https://github.com/PrefectHQ/prefect/pull/10324 + +- Deprecate `FlowRunCreate.deployment_id` — ### Contributors -- @psofiterol made their first contribution in https://github.com/PrefectHQ/prefect/pull/10320 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.11.1...2.11.2 +- @psofiterol made their first contribution in + +**All changes**: ## Release 2.11.1 ### Enhancements -- Add `work_queue_name` field when creating a flow run for a deployment, enabling the queue setting to be overridden on a per-run basis — https://github.com/PrefectHQ/prefect/pull/10276 -- Prevent accidental credential logging on BindFailure by logging only a list of key names, but not the values — https://github.com/PrefectHQ/prefect/pull/10264 -- Allow task runs to explicitly return `Paused` states, therefore pausing the flow run using the same settings — https://github.com/PrefectHQ/prefect/pull/10269 + +- Add `work_queue_name` field when creating a flow run for a deployment, enabling the queue setting to be overridden on a per-run basis — +- Prevent accidental credential logging on BindFailure by logging only a list of key names, but not the values — +- Allow task runs to explicitly return `Paused` states, therefore pausing the flow run using the same settings — ### Fixes -- Hide links to work queues for push work pools — https://github.com/PrefectHQ/prefect-ui-library/pull/1603 -- Fix issue with `Pause` state fields — https://github.com/PrefectHQ/prefect-ui-library/pull/1606 -- Fix issue with flow run logs missing until after refresh — https://github.com/PrefectHQ/prefect-ui-library/pull/1594 + +- Hide links to work queues for push work pools — +- Fix issue with `Pause` state fields — +- Fix issue with flow run logs missing until after refresh — ### Experimental -- Add a general use concurrency context manager — https://github.com/PrefectHQ/prefect/pull/10267 -- Add `rate_limit` function to block execution while acquiring slots — https://github.com/PrefectHQ/prefect/pull/10299 + +- Add a general use concurrency context manager — +- Add `rate_limit` function to block execution while acquiring slots — ### Documentation -- Add redirect to quickstart page — https://github.com/PrefectHQ/prefect/pull/10292 -- Add missing quotation mark in docstring — https://github.com/PrefectHQ/prefect/pull/10286 -- Fix `run_deployment` docstring rendering — https://github.com/PrefectHQ/prefect/pull/10310 -- Fix type in deployment docs — https://github.com/PrefectHQ/prefect/pull/10303 + +- Add redirect to quickstart page — +- Add missing quotation mark in docstring — +- Fix `run_deployment` docstring rendering — +- Fix type in deployment docs — ### Contributors -- @Sche7 made their first contribution in https://github.com/PrefectHQ/prefect/pull/10286 -- @LennyArdiles made their first contribution in https://github.com/PrefectHQ/prefect/pull/10264 -- @Akshat0410 made their first contribution in https://github.com/PrefectHQ/prefect/pull/10303 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.11.0...2.11.1 +- @Sche7 made their first contribution in +- @LennyArdiles made their first contribution in +- @Akshat0410 made their first contribution in + +**All changes**: ## Release 2.11.0 ### Flow summary graphs and stats + Each flow page now includes graphs of its recent flow runs, task runs, and (in Prefect Cloud) related events, as well as summary statistics! Screenshot 2023-07-20 at 3 42 51 PM Flow details have been moved to a dedicated tab. For implementation details, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10242 + +- ### Work pools and workers are now generally available + Since first being introduced in Prefect 2.10.0, Prefect [workers and work pools](https://docs.prefect.io/2.10.21/concepts/work-pools/) have come a long way. There are now work pools for every major infrastructure type. Work pools expose rich configuration of their infrastructure. Every work pool type has a base configuration with sensible defaults such that you can begin executing work with just a single command. The infrastructure configuration is fully customizable from the Prefect UI. Push work pools, recently released in Prefect Cloud, remain a beta feature. For implementation details, see the following pull requests: -- https://github.com/PrefectHQ/prefect/pull/10244 -- https://github.com/PrefectHQ/prefect/pull/10243 + +- +- ### Enhancements -- Use `orjson_dumps_extra_compatible` when serializing in `build_from_flow` — https://github.com/PrefectHQ/prefect/pull/10232 + +- Use `orjson_dumps_extra_compatible` when serializing in `build_from_flow` — ### Fixes -- Make `resolve_futures_to_data` function raise on failure by default — https://github.com/PrefectHQ/prefect/pull/10197 -- Fix flow runs page not polling for new runs and not loading more flow runs when scrolling — https://github.com/PrefectHQ/prefect/pull/10247 -- Don't create DB default during settings load — https://github.com/PrefectHQ/prefect/pull/10246 -- Fix issues causing flow runs to be incorrectly marked as failed — https://github.com/PrefectHQ/prefect/pull/10249 -- Fix incorrect path in error message — https://github.com/PrefectHQ/prefect/pull/10255 -- Fix `LocalFileSystem.get_directory` with basepath behaviour — https://github.com/PrefectHQ/prefect/pull/10258 -- Fix Dashboard refresh cadence — https://github.com/PrefectHQ/prefect/pull/10227 + +- Make `resolve_futures_to_data` function raise on failure by default — +- Fix flow runs page not polling for new runs and not loading more flow runs when scrolling — +- Don't create DB default during settings load — +- Fix issues causing flow runs to be incorrectly marked as failed — +- Fix incorrect path in error message — +- Fix `LocalFileSystem.get_directory` with basepath behaviour — +- Fix Dashboard refresh cadence — ### Documentation -- Add undocumented runtime parameters — https://github.com/PrefectHQ/prefect/pull/10229 -- Add Deployment Quickstart — https://github.com/PrefectHQ/prefect/pull/9985 -- Add guide for setting up a push work pool — https://github.com/PrefectHQ/prefect/pull/10248 -- Add guide for deploying a flow using Docker — https://github.com/PrefectHQ/prefect/pull/10252 -- Edit install and quickstart pages for clarity — https://github.com/PrefectHQ/prefect/pull/10231 -- Update automations screenshots — https://github.com/PrefectHQ/prefect/pull/10245 -- Fix typos on Deployment Management page — https://github.com/PrefectHQ/prefect/pull/10241 -- Fix flow retries example — https://github.com/PrefectHQ/prefect/pull/10233 -- Fix missing document title and adding terminal login section — https://github.com/PrefectHQ/prefect/pull/10256 + +- Add undocumented runtime parameters — +- Add Deployment Quickstart — +- Add guide for setting up a push work pool — +- Add guide for deploying a flow using Docker — +- Edit install and quickstart pages for clarity — +- Update automations screenshots — +- Fix typos on Deployment Management page — +- Fix flow retries example — +- Fix missing document title and adding terminal login section — ### Contributors -- @dbentall made their first contribution in https://github.com/PrefectHQ/prefect/pull/10258 + +- @dbentall made their first contribution in - @mesejo -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.21...2.11.0 +**All changes**: ## Release 2.10.21 ### The Prefect Dashboard - your heads up display + The response to the experimental Prefect dashboard was so enthusiastic that we've made it generally available as the default landing page in the Prefect UI. The dashboard provides an overview of all Prefect activity, surfaces the urgent information, and provides the context to understand that information. With the dashboard, you can: + - Confirm that all flows run in the past 24 hours behaved as expected - Identify a flow run that recently failed and jump directly to its page - See a work pool that is unhealthy and the work that is impacted -### Deploy deployments prefixed by flow name during `prefect deploy` +### Deploy deployments prefixed by flow name during `prefect deploy` + You can now specify the deployment to be executed by prefixing the deployment name with the flow name. For example, the following command creates a deployment with the name `my-deployment` for a flow with the name `my-flow`: + ```bash prefect deploy --name my-flow/my-deployment ``` @@ -1338,10 +1510,12 @@ prefect deploy --name my-flow/my-deployment This is especially useful when you have several flows with deployments that have the same name. For implementation details, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10189 + +- ### Use environment variables in deployment steps -Prefect now supports the usage of environment variables in deployment steps, allowing you to access environment variables during the `pull` action at runtime or during the `build` and `push` actions when running `prefect deploy`. Particularly useful for CI/CD builds, this makes Prefect deployments more versatile. + +Prefect now supports the usage of environment variables in deployment steps, allowing you to access environment variables during the `pull` action at runtime or during the `build` and `push` actions when running `prefect deploy`. Particularly useful for CI/CD builds, this makes Prefect deployments more versatile. For example, you can now use the following syntax to set an image tag of a Dockerized build by loading an environment variable during the `build` action: @@ -1353,9 +1527,10 @@ build: tag: '{{ $CUSTOM_TAG }}' ``` -You can also use environment variables inside of steps. +You can also use environment variables inside of steps. For example: + ```yaml - prefect.deployments.steps.run_shell_script: script: echo "test-'{{ $PREFECT_API_URL }}'" @@ -1363,12 +1538,15 @@ For example: ``` For implementation details, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10199 + +- ### Use `prefect deploy` with multiple deployments with the same name + When there are multiple deployments with the same name, the `prefect deploy` command now prompts you to choose which one to deploy: For example, if you have the following `prefect.yaml`: + ```yaml deployments: - name: "default" @@ -1377,67 +1555,78 @@ deployments: - name: "default" entrypoint: "flows/hello.py:hello_parallel" ``` + running `prefect deploy -n default` will now prompt you to choose which flow to create a deployment for: prompt choose a deployment For implementation details, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10189 + +- ### Enhancements -- Enable workspace dashboard by default — https://github.com/PrefectHQ/prefect/pull/10202 -- Add `SendgridEmail` notification block — https://github.com/PrefectHQ/prefect/pull/10118 -- Raise state change hook errors during creation if not correctly formatted — https://github.com/PrefectHQ/prefect/pull/9692 -- Improve `prefect deploy` nonexistent entrypoint `ValueError` - https://github.com/PrefectHQ/prefect/pull/10210 -- Truncate row length in interactive `prefect deploy` table display - https://github.com/PrefectHQ/prefect/pull/10209 -- Add `prefect.runtime.flow_run.parent_flow_run_id` and `prefect.runtime.flow_run.parent_deployment_id` - https://github.com/PrefectHQ/prefect/pull/10204 + +- Enable workspace dashboard by default — +- Add `SendgridEmail` notification block — +- Raise state change hook errors during creation if not correctly formatted — +- Improve `prefect deploy` nonexistent entrypoint `ValueError` - +- Truncate row length in interactive `prefect deploy` table display - +- Add `prefect.runtime.flow_run.parent_flow_run_id` and `prefect.runtime.flow_run.parent_deployment_id` - ### Fixes -- Adds handling for failed Kubernetes jobs — https://github.com/PrefectHQ/prefect/pull/10125 + +- Adds handling for failed Kubernetes jobs — ### Documentation -- Fix formatting in `mkdocs.yml` — https://github.com/PrefectHQ/prefect/pull/10187 -- Fix link to API docs in automations documentation — https://github.com/PrefectHQ/prefect/pull/10208 -- Remove the duplicate listing in installation documentation — https://github.com/PrefectHQ/prefect/pull/10200 -- Fix example in proactive trigger documentation — https://github.com/PrefectHQ/prefect/pull/10203 -- Remove references to nonexistent `prefect profile get` - https://github.com/PrefectHQ/prefect/pull/10214 + +- Fix formatting in `mkdocs.yml` — +- Fix link to API docs in automations documentation — +- Remove the duplicate listing in installation documentation — +- Fix example in proactive trigger documentation — +- Remove references to nonexistent `prefect profile get` - ## Contributors -* @rkscodes -* @Ishankoradia made their first contribution in https://github.com/PrefectHQ/prefect/pull/10118 -* @bsenst made their first contribution in https://github.com/PrefectHQ/prefect/pull/10200 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.20...2.10.21 +- @rkscodes + +- @Ishankoradia made their first contribution in +- @bsenst made their first contribution in + +**All changes**: ## Release 2.10.20 ### Resolving UI form input issues This release resolves bugs preventing UI form inputs from being rendered and parsed correctly, including: -- Dates & times — https://github.com/PrefectHQ/prefect-ui-library/pull/1554 -- List values — https://github.com/PrefectHQ/prefect-ui-library/pull/1556 -- JSON fields — https://github.com/PrefectHQ/prefect-ui-library/pull/1557 + +- Dates & times — +- List values — +- JSON fields — ### Prefect no longer supports Python 3.7 Python 3.7 reached end-of-life on 27 Jun 2023. Consistent with our warning, this release drops Python 3.7 support. Prefect now requires Python 3.8 or later. ### Enhancements -- Add UUID validation for webhook CLI commands to raise errors earlier and more clearly — https://github.com/PrefectHQ/prefect/pull/10005 -- Clarify Dockerfile rename prompt in `prefect deploy` — https://github.com/PrefectHQ/prefect/pull/10124 -- Improve `prefect deploy` error message — https://github.com/PrefectHQ/prefect/pull/10175 -- Add `work_pool_name` to `Deployment` docstring — https://github.com/PrefectHQ/prefect/pull/10174 + +- Add UUID validation for webhook CLI commands to raise errors earlier and more clearly — +- Clarify Dockerfile rename prompt in `prefect deploy` — +- Improve `prefect deploy` error message — +- Add `work_pool_name` to `Deployment` docstring — ### Contributors + - @toby-coleman -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.19...2.10.20 +**All changes**: ## Release 2.10.19 ### Peer into the future with the experimental dashboard We're excited to make the new Prefect dashboard available as an experimental feature. The dashboard provides an overview of all Prefect activity, surfaces the urgent information, and provides the context to understand that information. With the dashboard, you can: + - Confirm that all flows run in the past 24 hours behaved as expected - Identify a flow run that recently failed and jump directly to its page - See a work pool that is unhealthy and the work that is impacted @@ -1451,38 +1640,46 @@ See [this pull request](https://github.com/PrefectHQ/prefect/pull/10152) for imp Previously, users had to apply the appropriate format for their service credentials in a `Secret` block using the `access_token` field in `git_clone`. The `git_clone` pull step now includes an additional `credentials` field, allowing users to leverage their existing `GitHubCredentials`, `GitLabCredentials`, or `BitBucketCredentials` blocks when cloning from a private repository. For examples of providing credentials, see the [updated documentation](https://docs.prefect.io/2.10.19/concepts/deployments-ux/#the-pull-action). For implementation details see: -- https://github.com/PrefectHQ/prefect/pull/10157 + +- ### Fixes -- Improve language in `prefect deploy` to not recommend deprecated `-f/--flow` — https://github.com/PrefectHQ/prefect/pull/10121 -- Pin Pydantic to v1 in `requirements.txt` — https://github.com/PrefectHQ/prefect/pull/10144 -- Add default value of `None` for `WorkQueue.work_pool_id` — https://github.com/PrefectHQ/prefect/pull/10106 + +- Improve language in `prefect deploy` to not recommend deprecated `-f/--flow` — +- Pin Pydantic to v1 in `requirements.txt` — +- Add default value of `None` for `WorkQueue.work_pool_id` — ### Documentation -- Update `git_clone` documentation with examples of using credentials field - https://github.com/PrefectHQ/prefect/pull/10168 -- Add documentation on deleting blocks — https://github.com/PrefectHQ/prefect/pull/10115 -- Add docs tabs linking and styling — https://github.com/PrefectHQ/prefect/pull/10113 -- Fix example in `Block.load` docstring — https://github.com/PrefectHQ/prefect/pull/10098 -- Fix task tutorial documentation example — https://github.com/PrefectHQ/prefect/pull/10120 -- Clarify heading in rate limits documentation — https://github.com/PrefectHQ/prefect/pull/10148 -- Fix link in events documentation — https://github.com/PrefectHQ/prefect/pull/10160 -- Remove outdated disclaimer about configuring webhooks with the Prefect Cloud UI — https://github.com/PrefectHQ/prefect/pull/10167 + +- Update `git_clone` documentation with examples of using credentials field - +- Add documentation on deleting blocks — +- Add docs tabs linking and styling — +- Fix example in `Block.load` docstring — +- Fix task tutorial documentation example — +- Clarify heading in rate limits documentation — +- Fix link in events documentation — +- Remove outdated disclaimer about configuring webhooks with the Prefect Cloud UI — ### Integrations -- Add `prefect-earthdata` integration — https://github.com/PrefectHQ/prefect/pull/10151 + +- Add `prefect-earthdata` integration — ### Contributors + - @rkscodes - @StefanBRas -* @JordonMaule made their first contribution in https://github.com/PrefectHQ/prefect/pull/10120 -* @AmanSal1 made their first contribution in https://github.com/PrefectHQ/prefect/pull/10121 -* @giorgiobasile made their first contribution in https://github.com/PrefectHQ/prefect/pull/10151 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.18...2.10.19 +- @JordonMaule made their first contribution in + +- @AmanSal1 made their first contribution in +- @giorgiobasile made their first contribution in + +**All changes**: ## Release 2.10.18 ### Docker image support during flow deployment + We enhanced support for Docker-based infrastructures when deploying flows through the interactive `prefect deploy` experience. Users can now easily custom-build or auto-build Docker images and push them to remote registries if they so choose. The CLI automatically detects if a work pool supports Docker images (e.g., docker, ecs, cloud-run) during `prefect deploy` and will now guide the user through the experience of building and pushing a Docker image if support is detected. @@ -1492,45 +1689,52 @@ This enhancement to managing deployments will greatly simplify the process of cr Not only that, we will also create a `pull` step for you when you choose to build a Docker image through `prefect deploy`. Whether you have your own Dockerfile or you want to use the auto-build feature in `build_docker_image`, we will create a `pull` step for you to help you set the correct path to your flow code. See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/10022 -- https://github.com/PrefectHQ/prefect/pull/10090 + +- +- ### Event-driven deployments with triggers + You can now easily incorporate event-based triggers into your Prefect Cloud deployments - simply add triggers to your `prefect.yaml` file or directly from the Prefect UI deployment page. Deployment triggers utilize automations - any automation that runs flows from a given deployment will be reflected on that deployment page. See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/10049 -- https://github.com/PrefectHQ/prefect/pull/10097 + +- +- ### Enhancements -- Allow saving of updated deployment configurations — https://github.com/PrefectHQ/prefect/pull/10018 -- Add `--install-policy` option to `prefect worker start` - https://github.com/PrefectHQ/prefect/pull/10040 -- Update Docker-based `prefect init` recipes to use `push_docker_image` step — https://github.com/PrefectHQ/prefect/pull/10092 + +- Allow saving of updated deployment configurations — +- Add `--install-policy` option to `prefect worker start` - +- Update Docker-based `prefect init` recipes to use `push_docker_image` step — ### Fixes -- Fix deployment `pull` step saving by preserving placeholders with missing values — https://github.com/PrefectHQ/prefect/pull/10053 -- Fix `prefect server start` and `prefect agent start` on Windows — https://github.com/PrefectHQ/prefect/pull/10059 -- Add ability to use Prefect variables in `job_variables` section of deploy config in `prefect.yaml` — https://github.com/PrefectHQ/prefect/pull/10078 -- Add default option to `new_parameters.pop` in `explode_variadic_parameter` used to handle `**kwargs` in task mapping — https://github.com/PrefectHQ/prefect/pull/10067 -- Skip schedule prompts in `prefect deploy` if schedule is set or null in `prefect.yaml` — https://github.com/PrefectHQ/prefect/pull/10074 -- Fix saving of `pull` and `push` step deployment configuration — https://github.com/PrefectHQ/prefect/pull/10087 -- Fix issue hosting and running the UI in unsecured contexts - https://github.com/PrefectHQ/prefect-design/pull/829 + +- Fix deployment `pull` step saving by preserving placeholders with missing values — +- Fix `prefect server start` and `prefect agent start` on Windows — +- Add ability to use Prefect variables in `job_variables` section of deploy config in `prefect.yaml` — +- Add default option to `new_parameters.pop` in `explode_variadic_parameter` used to handle `**kwargs` in task mapping — +- Skip schedule prompts in `prefect deploy` if schedule is set or null in `prefect.yaml` — +- Fix saving of `pull` and `push` step deployment configuration — +- Fix issue hosting and running the UI in unsecured contexts - ### Documentation -- Adjust docs to reflect Prefect requires Python 3.8 — https://github.com/PrefectHQ/prefect/pull/9853 -- Add custom `pull` step examples to deployment management docs — https://github.com/PrefectHQ/prefect/pull/10073 -- Add troubleshooting guide to docs — https://github.com/PrefectHQ/prefect/pull/10079 -- Add information on finding Prefect Cloud account id and workspace id — https://github.com/PrefectHQ/prefect/pull/10103 -- Reference webhooks documentation from events documentation — https://github.com/PrefectHQ/prefect/pull/10045 -- Simplify deployment description in docs — https://github.com/PrefectHQ/prefect/pull/10050 + +- Adjust docs to reflect Prefect requires Python 3.8 — +- Add custom `pull` step examples to deployment management docs — +- Add troubleshooting guide to docs — +- Add information on finding Prefect Cloud account id and workspace id — +- Reference webhooks documentation from events documentation — +- Simplify deployment description in docs — ### Contributors -- @garylavayou made their first contribution in https://github.com/PrefectHQ/prefect/pull/10060 -- @themattmorris made their first contribution in https://github.com/PrefectHQ/prefect/pull/10056 + +- @garylavayou made their first contribution in +- @themattmorris made their first contribution in - @NodeJSmith - @rpeden -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.17...2.10.18 +**All changes**: ## Release 2.10.17 @@ -1539,26 +1743,31 @@ See the following pull requests for implementation details: Prefect's documentation has an [improved tutorial](https://docs.prefect.io/2.10.17/tutorial/), redesigned to include Prefect's recent enhancements. With the introduction of work pools and the interactive deployment CLI, the new tutorial reflects the elevated experience that these new features offer, alongside the key elements and features of Prefect. You can find content related to more advanced features or less common use cases in the [Guides](https://docs.prefect.io/2.10.17/guides/) section. ### Enhancements -- Update Prefect client to follow redirects by default — https://github.com/PrefectHQ/prefect/pull/9988 -- Always show checkboxes on list items, rather than animating them on hover — https://github.com/PrefectHQ/prefect-ui-library/pull/1490 -- New `CustomWebhookNotificationBlock` for triggering custom webhooks in response to flow run state changes — https://github.com/PrefectHQ/prefect/pull/9547 + +- Update Prefect client to follow redirects by default — +- Always show checkboxes on list items, rather than animating them on hover — +- New `CustomWebhookNotificationBlock` for triggering custom webhooks in response to flow run state changes — ### Fixes -- Limit the number of files concurrently opened by `prefect deploy` when searching for flows — https://github.com/PrefectHQ/prefect/pull/10014 -- Fix `TypeError: crypto.randomUUID is not a function` that caused pages to break — https://github.com/PrefectHQ/prefect-ui-library/pull/1501 + +- Limit the number of files concurrently opened by `prefect deploy` when searching for flows — +- Fix `TypeError: crypto.randomUUID is not a function` that caused pages to break — ### Documentation -- Fix broken link to `prefect-docker` documentation on the deployments UX page — https://github.com/PrefectHQ/prefect/pull/10013 -- Document `--work-queue / -q` arguments to `worker start` command — https://github.com/PrefectHQ/prefect/pull/10027 -- Add link to join Club 42 to Community page — https://github.com/PrefectHQ/prefect/pull/9927 -- Improve Prefect tutorial to be more succinct and purposeful — https://github.com/PrefectHQ/prefect/pull/9940 + +- Fix broken link to `prefect-docker` documentation on the deployments UX page — +- Document `--work-queue / -q` arguments to `worker start` command — +- Add link to join Club 42 to Community page — +- Improve Prefect tutorial to be more succinct and purposeful — ### Contributors -* @eclark9270 made their first contribution in https://github.com/PrefectHQ/prefect/pull/9927 -* @AutumnSun1996 made their first contribution in https://github.com/PrefectHQ/prefect/pull/9547 -* @dianaclarke made their first contribution in https://github.com/PrefectHQ/prefect/pull/9988 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.16...2.10.17 +- @eclark9270 made their first contribution in + +- @AutumnSun1996 made their first contribution in +- @dianaclarke made their first contribution in + +**All changes**: ## Release 2.10.16 @@ -1568,35 +1777,43 @@ We're making it easier than ever to deploy your first flow! Previously, you need ![flow selector example](https://user-images.githubusercontent.com/12350579/247144440-d89916d4-cbf1-408e-9959-45df94a35f8d.png) -For more details on implementation, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/10004 +For more details on implementation, see the following pull request: + +- ### Enhancements -- Add option to specify work queue priority during creation from CLI — https://github.com/PrefectHQ/prefect/pull/9999 -- Improve 'Invalid timezone' error message — https://github.com/PrefectHQ/prefect/pull/10007 + +- Add option to specify work queue priority during creation from CLI — +- Improve 'Invalid timezone' error message — ### Fixes -- Fix wrong key used in generated `git_clone` step — https://github.com/PrefectHQ/prefect/pull/9997 + +- Fix wrong key used in generated `git_clone` step — ### Deprecations -- Deprecate `prefect deploy` `--ci` flag — https://github.com/PrefectHQ/prefect/pull/10002 + +- Deprecate `prefect deploy` `--ci` flag — ### Documentation -- Resolve missing image in Prefect Cloud event documentation — https://github.com/PrefectHQ/prefect/pull/9904 -- Fix typo in webhooks documentation — https://github.com/PrefectHQ/prefect/pull/10003 + +- Resolve missing image in Prefect Cloud event documentation — +- Fix typo in webhooks documentation — ### Integrations -- Fix bug in `KubernetesWorker` where flow runs crashed during submission - https://github.com/PrefectHQ/prefect-kubernetes/pull/76 + +- Fix bug in `KubernetesWorker` where flow runs crashed during submission - ### Contributors -- @kkdenk made their first contribution in https://github.com/PrefectHQ/prefect/pull/9904 + +- @kkdenk made their first contribution in - @rito-sixt -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.15...2.10.16 +**All changes**: ## Release 2.10.15 ## Introducing deployment configuration saving in `prefect deploy` + We are excited to announce a significant enhancement to our `prefect deploy` command to make your deployment process even more intuitive. Previously, users had to recall their deployment configurations each time they wanted to redeploy with the same settings. Recognizing this potential inconvenience, we've now incorporated a feature to save your deployment inputs for future use, thereby streamlining redeployments. @@ -1608,17 +1825,20 @@ The new interactive `prefect deploy` command guides you through the deployment p If you have a `prefect.yaml` file in the same directory where you run your command, running the `deploy` command again allows you to reuse the saved deployment configuration or create a new one. If you choose to create a new deployment, you will again be given the option to save your inputs. This way, you can maintain a list of multiple deployment configurations, ready to be used whenever needed! For more details on implementation, see the following pull request: -- https://github.com/PrefectHQ/prefect/pull/9948 + +- ### Fixes -- Fix error in `prefect deploy` when `.prefect` folder is absent — https://github.com/PrefectHQ/prefect/pull/9972 -- Fix use of deprecated `git_clone_project` — https://github.com/PrefectHQ/prefect/pull/9978 -- Fix exception raised in `prefect init` command when no recipe is selected — https://github.com/PrefectHQ/prefect/pull/9963 + +- Fix error in `prefect deploy` when `.prefect` folder is absent — +- Fix use of deprecated `git_clone_project` — +- Fix exception raised in `prefect init` command when no recipe is selected — ### Documentation -- Fix broken deployments api-ref page — https://github.com/PrefectHQ/prefect/pull/9965 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.14...2.10.15 +- Fix broken deployments api-ref page — + +**All changes**: ## Release 2.10.14 @@ -1634,7 +1854,9 @@ pull: id: clone-step repository: https://github.com/org/repo.git ``` + is now + ```yaml pull: - prefect.deployments.steps.git_clone: @@ -1643,6 +1865,7 @@ pull: ``` An example using the `prefect_gcp` library: + ```yaml build: - prefect_gcp.projects.steps.push_project_to_gcs: @@ -1650,7 +1873,9 @@ build: bucket: my-bucket folder: my-project ``` + is now + ```yaml build: - prefect_gcp.deployments.steps.push_to_gcs: @@ -1664,19 +1889,21 @@ In addition, we've removed the need to use the `project` command group through t ![prefect init recipe interaction](https://github.com/PrefectHQ/prefect/assets/42048900/c2bea9b4-4e1f-4029-8772-50ecde6073a7) We have also deprecated deploying a flow via flow name (`-f`), allowing a single, streamlined way to deploy. + ```python prefect deploy ./path/to/flow.py:flow-fn-name ``` See these pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/9887 -- https://github.com/PrefectHQ/prefect/pull/9930 -- https://github.com/PrefectHQ/prefect/pull/9928 -- https://github.com/PrefectHQ/prefect/pull/9944 -- https://github.com/PrefectHQ/prefect/pull/9942 -- https://github.com/PrefectHQ/prefect/pull/9957 -- https://github.com/PrefectHQ/prefect-gcp/pull/189 -- https://github.com/PrefectHQ/prefect-aws/pull/278 + +- +- +- +- +- +- +- +- ### Prefect Cloud Webhook CLI @@ -1687,6 +1914,7 @@ Even if you have minimal control over the systems you're integrating with, Prefe Webhooks are currently available [via the API and `prefect` CLI](https://docs.prefect.io/2.10.14/cloud/webhooks/#configuring-webhooks). You can create your first Cloud webhook via the CLI like so: + ```bash prefect cloud webhook create your-webhook-name \ --description "Receives webhooks from your system" \ @@ -1694,43 +1922,48 @@ prefect cloud webhook create your-webhook-name \ ``` See the following pull request for implementation details: -- https://github.com/PrefectHQ/prefect/pull/9874 + +- ### Enhancements -- Make related automations visible from `prefect deployment inspect` — https://github.com/PrefectHQ/prefect/pull/9929 -- Enable deleting blocks with Python SDK — https://github.com/PrefectHQ/prefect/pull/9932 -- Enhance ability to delete a single flow on the flows page - https://github.com/PrefectHQ/prefect-ui-library/pull/1478 -- Add `work_pool_name` to work queue API responses — https://github.com/PrefectHQ/prefect/pull/9659 -- Add httpx request method to Prefect Cloud client — https://github.com/PrefectHQ/prefect/pull/9873 -- Mark flow as crashed if infrastructure submission fails — https://github.com/PrefectHQ/prefect/pull/9691 -- Re-enable the retrieval of existing clients from flow and task run contexts when safe — https://github.com/PrefectHQ/prefect/pull/9880 -- Add `prefect --prompt/--no-prompt` to force toggle interactive CLI sessions — https://github.com/PrefectHQ/prefect/pull/9897 -- Return sorted task run ids when inspecting concurrency limit via CLI — https://github.com/PrefectHQ/prefect/pull/9711 -- Use existing thread in `BatchedQueueService` to reduce queue retrieval overhead — https://github.com/PrefectHQ/prefect/pull/9877 + +- Make related automations visible from `prefect deployment inspect` — +- Enable deleting blocks with Python SDK — +- Enhance ability to delete a single flow on the flows page - +- Add `work_pool_name` to work queue API responses — +- Add httpx request method to Prefect Cloud client — +- Mark flow as crashed if infrastructure submission fails — +- Re-enable the retrieval of existing clients from flow and task run contexts when safe — +- Add `prefect --prompt/--no-prompt` to force toggle interactive CLI sessions — +- Return sorted task run ids when inspecting concurrency limit via CLI — +- Use existing thread in `BatchedQueueService` to reduce queue retrieval overhead — ### Fixes -- Provide a default `DTSTART` to anchor `RRULE` schedules to ensure extra schedules not created — https://github.com/PrefectHQ/prefect/pull/9872 -- Fix bug where attribute error raised on service shutdown when the app startup fails — https://github.com/PrefectHQ/prefect/pull/9900 -- Improve retry behavior when SQLite database locked — https://github.com/PrefectHQ/prefect/pull/9938 + +- Provide a default `DTSTART` to anchor `RRULE` schedules to ensure extra schedules not created — +- Fix bug where attribute error raised on service shutdown when the app startup fails — +- Improve retry behavior when SQLite database locked — ### Documentation -- Add tip on `PREFECT_API_URL` setting for workers and agents — https://github.com/PrefectHQ/prefect/pull/9882 -- Add deployment triggers documentation — https://github.com/PrefectHQ/prefect/pull/9886 -- Add more detailed documentation to the engine api-ref — https://github.com/PrefectHQ/prefect/pull/9924 -- Add note on matching on multiple resources when using automations — https://github.com/PrefectHQ/prefect/pull/9867 -- Updates automations examples in docs — https://github.com/PrefectHQ/prefect/pull/9952 -- Update Prefect Cloud users documentation on user settings — https://github.com/PrefectHQ/prefect/pull/9920 -- Boost non-API docs pages to optimize search results — https://github.com/PrefectHQ/prefect/pull/9854 -- Update testing documentation tag — https://github.com/PrefectHQ/prefect/pull/9905 -- Exemplify how to import Prefect client — https://github.com/PrefectHQ/prefect/pull/9671 + +- Add tip on `PREFECT_API_URL` setting for workers and agents — +- Add deployment triggers documentation — +- Add more detailed documentation to the engine api-ref — +- Add note on matching on multiple resources when using automations — +- Updates automations examples in docs — +- Update Prefect Cloud users documentation on user settings — +- Boost non-API docs pages to optimize search results — +- Update testing documentation tag — +- Exemplify how to import Prefect client — ## Contributors + - @Hongbo-Miao -- @rito-sixt made their first contribution in https://github.com/PrefectHQ/prefect/pull/9711 -- @drpin2341 made their first contribution in https://github.com/PrefectHQ/prefect/pull/9905 -- @amansal1 made their first contribution in https://github.com/PrefectHQ/prefect-ui-library/pull/1478 +- @rito-sixt made their first contribution in +- @drpin2341 made their first contribution in +- @amansal1 made their first contribution in -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.13...2.10.14 +**All changes**: ## Release 2.10.13 @@ -1741,12 +1974,14 @@ See the following pull request for implementation details: Project-based deployments are now easier to use, especially for first time users! You can now run `prefect deploy` without first initializing a project. If you run `prefect deploy` without a project initialized, the CLI will generate a default pull step that your worker can use to retrieve your flow code when executing scheduled flow runs. The prefect deploy command will also prompt you with scheduling options, making it even easier to schedule your flows! See these two pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/9832 -- https://github.com/PrefectHQ/prefect/pull/9844 + +- +- This release also adds two new deployment steps: `pip_install_requirements` and `run_shell_script`. Both of these are new 'utility' deployment steps that can be used to automate portions of your deployment process. Use the `pip_install_requirements` step to install Python dependencies before kicking off a flow run: + ```yaml pull: - prefect.projects.steps.git_clone_project: @@ -1759,6 +1994,7 @@ pull: ``` Use the `run_shell_script` step to grab your repository's commit hash and use it to tag your Docker image: + ```yaml build: - prefect.projects.steps.run_shell_script: @@ -1773,41 +2009,46 @@ build: ``` See these two pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/9810 -- https://github.com/PrefectHQ/prefect/pull/9868 + +- +- ### Enhancements -- Allow project `pull` steps to pass step outputs — https://github.com/PrefectHQ/prefect/pull/9861 -- Update work queue health indicators in Prefect UI for greater clarity - https://github.com/PrefectHQ/prefect-ui-library/pull/1464 -- State messages no longer include tracebacks — https://github.com/PrefectHQ/prefect/pull/9835 -- Allow passing a payload to `emit_instance_method_called_event` - https://github.com/PrefectHQ/prefect/pull/9869 + +- Allow project `pull` steps to pass step outputs — +- Update work queue health indicators in Prefect UI for greater clarity - +- State messages no longer include tracebacks — +- Allow passing a payload to `emit_instance_method_called_event` - ### Fixes -- Reference `.prefectignore` files when moving files around locally to - https://github.com/PrefectHQ/prefect/pull/9863 -- Fix typo in warning message raised when flow is called during script loading — https://github.com/PrefectHQ/prefect/pull/9817 -- Allow creation of identical block names between different block types - https://github.com/PrefectHQ/prefect-ui-library/pull/1473 -- Ensure flow timeouts do not override existing alarm signal handlers — https://github.com/PrefectHQ/prefect/pull/9835 -- Ensure timeout tracking begins from the actual start of the call, rather than the scheduled start — https://github.com/PrefectHQ/prefect/pull/9835 -- Ensure timeout monitoring threads immediately exit upon run completion — https://github.com/PrefectHQ/prefect/pull/9835 -- Fix bug where background services could throw logging errors on interpreter exit — https://github.com/PrefectHQ/prefect/pull/9835 -- Fix bug where asynchronous timeout enforcement could deadlock — https://github.com/PrefectHQ/prefect/pull/9835 + +- Reference `.prefectignore` files when moving files around locally to - +- Fix typo in warning message raised when flow is called during script loading — +- Allow creation of identical block names between different block types - +- Ensure flow timeouts do not override existing alarm signal handlers — +- Ensure timeout tracking begins from the actual start of the call, rather than the scheduled start — +- Ensure timeout monitoring threads immediately exit upon run completion — +- Fix bug where background services could throw logging errors on interpreter exit — +- Fix bug where asynchronous timeout enforcement could deadlock — ### Documentation -- Add documentation on Prefect Cloud webhook usage - https://github.com/PrefectHQ/prefect/pull/9857 -- Fix broken link and Prefect server reference in Cloud docs — https://github.com/PrefectHQ/prefect/pull/9820 -- Fix broken link to Docker guide in API reference docs — https://github.com/PrefectHQ/prefect/pull/9821 -- Update subflow run cancellation information in flows concept doc — https://github.com/PrefectHQ/prefect/pull/9753 -- Improve ability to give feedback on documentation — https://github.com/PrefectHQ/prefect/pull/9836 -- Add projects deployment diagram to work pool, workers & agents concept doc — https://github.com/PrefectHQ/prefect/pull/9841 -- Add missing Prefect Server URL in API reference docs — https://github.com/PrefectHQ/prefect/pull/9864 -- Fix code typo in task runners concept doc — https://github.com/PrefectHQ/prefect/pull/9818 -- Add documentation on flow run parameter size limit — https://github.com/PrefectHQ/prefect/pull/9847 -- Fix link to orchestration tutorial in execution tutorial - https://github.com/PrefectHQ/prefect/pull/9862 + +- Add documentation on Prefect Cloud webhook usage - +- Fix broken link and Prefect server reference in Cloud docs — +- Fix broken link to Docker guide in API reference docs — +- Update subflow run cancellation information in flows concept doc — +- Improve ability to give feedback on documentation — +- Add projects deployment diagram to work pool, workers & agents concept doc — +- Add missing Prefect Server URL in API reference docs — +- Fix code typo in task runners concept doc — +- Add documentation on flow run parameter size limit — +- Fix link to orchestration tutorial in execution tutorial - ### Contributors -- @ac1997 made their first contribution in https://github.com/PrefectHQ/prefect/pull/9862 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.12...2.10.13 +- @ac1997 made their first contribution in + +**All changes**: ## Release 2.10.12 @@ -1818,49 +2059,55 @@ We got a lot of positive feedback about the new flows page that was redesigned t See the [pull request](https://github.com/PrefectHQ/prefect/pull/9800) for implementation details. ### Enhancements -- All server-side schemas now have dedicated client-side duplicates — https://github.com/PrefectHQ/prefect/pull/9577 -- Import of `prefect.server` is delayed to improve CLI start time and `import prefect` time — https://github.com/PrefectHQ/prefect/pull/9577 -- Add task run as a related object to emitted events — https://github.com/PrefectHQ/prefect/pull/9759 -- Emit task run state change events when orchestrating a task run — https://github.com/PrefectHQ/prefect/pull/9684 -- Add healthcheck webserver to workers — https://github.com/PrefectHQ/prefect/pull/9687 -- Create files and directories with user-scoped permissions — https://github.com/PrefectHQ/prefect/pull/9789 -- Runtime variables mocked with environment variables for testing are now coerced to the correct type — https://github.com/PrefectHQ/prefect/pull/9561 + +- All server-side schemas now have dedicated client-side duplicates — +- Import of `prefect.server` is delayed to improve CLI start time and `import prefect` time — +- Add task run as a related object to emitted events — +- Emit task run state change events when orchestrating a task run — +- Add healthcheck webserver to workers — +- Create files and directories with user-scoped permissions — +- Runtime variables mocked with environment variables for testing are now coerced to the correct type — ### Fixes -- Show 404 instead of blank page in UI flow run id is invalid or if flow run is missing — https://github.com/PrefectHQ/prefect/pull/9746 -- Fix bug where event loop shutdown hooks could fail due to early garbage collection — https://github.com/PrefectHQ/prefect/pull/9748 -- Fix process worker `documentation_url` — https://github.com/PrefectHQ/prefect/pull/9791 -- Fix bug where given priority was ignored when creating a work queue — https://github.com/PrefectHQ/prefect/pull/9798 -- Fix inconsistent work queue handling by agent when cancelling flow runs — https://github.com/PrefectHQ/prefect/pull/9757 + +- Show 404 instead of blank page in UI flow run id is invalid or if flow run is missing — +- Fix bug where event loop shutdown hooks could fail due to early garbage collection — +- Fix process worker `documentation_url` — +- Fix bug where given priority was ignored when creating a work queue — +- Fix inconsistent work queue handling by agent when cancelling flow runs — ### Experimental -- Add `dashboard` experiment via `ENABLE_WORKSPACE_DASHBOARD` — https://github.com/PrefectHQ/prefect/pull/9802, https://github.com/PrefectHQ/prefect/pull/9799 + +- Add `dashboard` experiment via `ENABLE_WORKSPACE_DASHBOARD` — , ### Deprecations -- Deprecate `create_orion_api` in favor of `create_api_app` — https://github.com/PrefectHQ/prefect/pull/9745 -- Deprecate "send_to_orion" logging option in favor of "send_to_api" — https://github.com/PrefectHQ/prefect/pull/9743 + +- Deprecate `create_orion_api` in favor of `create_api_app` — +- Deprecate "send_to_orion" logging option in favor of "send_to_api" — ### Documentation -- Add descriptions to concept tables — https://github.com/PrefectHQ/prefect/pull/9718 -- Removes unreferenced requests import in 'real-world example' — https://github.com/PrefectHQ/prefect/pull/9760 -- Add state change hooks to guides overview page — https://github.com/PrefectHQ/prefect/pull/9761 -- Fix typo in flows and tasks tutorials — https://github.com/PrefectHQ/prefect/pull/9762 -- Update task docs to reference common params and link to all params — https://github.com/PrefectHQ/prefect/pull/9787 -- Add Google Analytics to documentation — https://github.com/PrefectHQ/prefect/pull/9793 -- Remove outdated announcement — https://github.com/PrefectHQ/prefect/pull/9792 -- Add extra loggers example — https://github.com/PrefectHQ/prefect/pull/9714 -- Clarify work pool priority options — https://github.com/PrefectHQ/prefect/pull/9752 -- Update worker requirements in projects tutorial — https://github.com/PrefectHQ/prefect/pull/9579 -- Fix default value comment in docs/concepts/variables — https://github.com/PrefectHQ/prefect/pull/9771 -- Fix formatting of link to Ray page — https://github.com/PrefectHQ/prefect/pull/9772 -- Add book a rubber duck links — https://github.com/PrefectHQ/prefect/pull/9790 + +- Add descriptions to concept tables — +- Removes unreferenced requests import in 'real-world example' — +- Add state change hooks to guides overview page — +- Fix typo in flows and tasks tutorials — +- Update task docs to reference common params and link to all params — +- Add Google Analytics to documentation — +- Remove outdated announcement — +- Add extra loggers example — +- Clarify work pool priority options — +- Update worker requirements in projects tutorial — +- Fix default value comment in docs/concepts/variables — +- Fix formatting of link to Ray page — +- Add book a rubber duck links — ### Contributors -- @marco-buttu made their first contribution in https://github.com/PrefectHQ/prefect/pull/9771 -- @jcozar87 made their first contribution in https://github.com/PrefectHQ/prefect/pull/9561 + +- @marco-buttu made their first contribution in +- @jcozar87 made their first contribution in - @rmorshea -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.11...2.10.12 +**All changes**: ## Release 2.10.11 @@ -1879,37 +2126,41 @@ Secondly, we've added a work pool creation wizard to streamline the process and Together, these improvements offer an interactive, guided experience that not only simplifies deployments and work pool creation but also empowers users to navigate the process confidently and efficiently. Check out these pull requests for more details: -- https://github.com/PrefectHQ/prefect-ui-library/pull/1431 -- https://github.com/PrefectHQ/prefect/pull/9707 -- https://github.com/PrefectHQ/prefect/pull/9686 + +- +- +- ### Enhancements -- Emit events from deployments, work queues, and work pools — https://github.com/PrefectHQ/prefect/pull/9635 -- Improve SQLite database transaction behavior — https://github.com/PrefectHQ/prefect/pull/9594 -- Add support for SQLAlchemy 2 — https://github.com/PrefectHQ/prefect/pull/9656 -- Add `on_cancellation` flow run state change hook — https://github.com/PrefectHQ/prefect/pull/9389 -- Improve cancellation cleanup service iteration over subflow runs - https://github.com/PrefectHQ/prefect/pull/9731 -- Add request retry support to Prefect Cloud client — https://github.com/PrefectHQ/prefect/pull/9724 -- Add `PREFECT_CLIENT_MAX_RETRIES` for configuration of maximum HTTP request retries - https://github.com/PrefectHQ/prefect/pull/9735 -- Add an `/api/ready` endpoint to the Prefect server to check database connectivity — https://github.com/PrefectHQ/prefect/pull/9701 -- Display URL to flow run on creation - https://github.com/PrefectHQ/prefect/pull/9740 -- Add guard against changing the profile path from `prefect config set` — https://github.com/PrefectHQ/prefect/pull/9696 -- Use flow run logger to report traceback for failed submissions — https://github.com/PrefectHQ/prefect/pull/9733 -- Improve default Prefect image tag when using development versions — https://github.com/PrefectHQ/prefect/pull/9503 -- Emit worker event when a flow run is scheduled to run or cancel — https://github.com/PrefectHQ/prefect/pull/9702 -- Add ability to filter for `Retrying` state in the Task Runs tab of the Prefect UI — https://github.com/PrefectHQ/prefect-ui-library/pull/1410 + +- Emit events from deployments, work queues, and work pools — +- Improve SQLite database transaction behavior — +- Add support for SQLAlchemy 2 — +- Add `on_cancellation` flow run state change hook — +- Improve cancellation cleanup service iteration over subflow runs - +- Add request retry support to Prefect Cloud client — +- Add `PREFECT_CLIENT_MAX_RETRIES` for configuration of maximum HTTP request retries - +- Add an `/api/ready` endpoint to the Prefect server to check database connectivity — +- Display URL to flow run on creation - +- Add guard against changing the profile path from `prefect config set` — +- Use flow run logger to report traceback for failed submissions — +- Improve default Prefect image tag when using development versions — +- Emit worker event when a flow run is scheduled to run or cancel — +- Add ability to filter for `Retrying` state in the Task Runs tab of the Prefect UI — ### Fixes -- Display CLI deprecation warnings to STDERR instead of STDOUT — https://github.com/PrefectHQ/prefect/pull/9690 -- Fix hanging flow runs from deployments when variables retrieved in base scope - https://github.com/PrefectHQ/prefect/pull/9665 -- Fix maximum character length when updating variables — https://github.com/PrefectHQ/prefect/pull/9710 -- Fix bug where agents would fail when processing runs with deleted deployments — https://github.com/PrefectHQ/prefect/pull/9464 -- Fix bug where `uvicorn` could not be found when server was started from an unloaded virtual environment - https://github.com/PrefectHQ/prefect/pull/9734 -- Allow table artifacts `table` argument as list of lists — https://github.com/PrefectHQ/prefect/pull/9732 -- Fix bug where events worker would fail if the API URL includes a trailing `/` — https://github.com/PrefectHQ/prefect/pull/9663 -- Fix bug where flow run timeline crashed when custom state names were used — https://github.com/PrefectHQ/prefect-ui-library/pull/1448 + +- Display CLI deprecation warnings to STDERR instead of STDOUT — +- Fix hanging flow runs from deployments when variables retrieved in base scope - +- Fix maximum character length when updating variables — +- Fix bug where agents would fail when processing runs with deleted deployments — +- Fix bug where `uvicorn` could not be found when server was started from an unloaded virtual environment - +- Allow table artifacts `table` argument as list of lists — +- Fix bug where events worker would fail if the API URL includes a trailing `/` — +- Fix bug where flow run timeline crashed when custom state names were used — ### Collections + - Stream Kubernetes Worker flow run logs to the API - [#72](https://github.com/PrefectHQ/prefect-kubernetes/pull/72) - Stream ECS Worker flow run logs to the API - [#267](https://github.com/PrefectHQ/prefect-aws/pull/267) - Stream Cloud Run Worker flow run logs logs to the API - [#183](https://github.com/PrefectHQ/prefect-gcp/pull/183) @@ -1923,34 +2174,38 @@ Check out these pull requests for more details: - Improve failure message when creating a Kubernetes job fails - [#71](https://github.com/PrefectHQ/prefect-kubernetes/pull/71) ### Deprecations -- Rename `prefect.infrastructure.docker` to `prefect.infrastructure.container` - https://github.com/PrefectHQ/prefect/pull/8788 -- Rename `prefect.docker` to `prefect.utilities.dockerutils` - https://github.com/PrefectHQ/prefect/pull/8788 + +- Rename `prefect.infrastructure.docker` to `prefect.infrastructure.container` - +- Rename `prefect.docker` to `prefect.utilities.dockerutils` - ### Documentation -- Create examples of working with Prefect REST APIs — https://github.com/PrefectHQ/prefect/pull/9661 -- Add state change hook documentation - https://github.com/PrefectHQ/prefect/pull/9721 -- Add tip about private repositories in projects documentation — https://github.com/PrefectHQ/prefect/pull/9685 -- Improve runtime context documentation — https://github.com/PrefectHQ/prefect/pull/9652 -- Simplify the flow and task configuration documentation — https://github.com/PrefectHQ/prefect/pull/9420 -- Clarify task retries documentation — https://github.com/PrefectHQ/prefect/pull/9575 -- Fix typos in cloud documentation — https://github.com/PrefectHQ/prefect/pull/9657 -- Update automations documentation — https://github.com/PrefectHQ/prefect/pull/9680 -- Fix typo in tutorial documentation — https://github.com/PrefectHQ/prefect/pull/9646 -- Add tip on `keys` in artifacts documentation — https://github.com/PrefectHQ/prefect/pull/9666 -- Expand docstrings for artifacts — https://github.com/PrefectHQ/prefect/pull/9704 -- Update description of `image` parameter of `DockerContainer` in infrastructure documentation — https://github.com/PrefectHQ/prefect/pull/9682 -- Lowercase Prefect server where appropriate — https://github.com/PrefectHQ/prefect/pull/9697 -- Remove `Upgrading from Prefect Beta` section of installation page — https://github.com/PrefectHQ/prefect/pull/9726 -- Update rate limit documentation to include `/set_state` and `/flows` endpoint for Prefect Cloud — https://github.com/PrefectHQ/prefect/pull/9694 -- Update documentation links in UI to concepts when possible — https://github.com/PrefectHQ/prefect-ui-library/pull/1351 + +- Create examples of working with Prefect REST APIs — +- Add state change hook documentation - +- Add tip about private repositories in projects documentation — +- Improve runtime context documentation — +- Simplify the flow and task configuration documentation — +- Clarify task retries documentation — +- Fix typos in cloud documentation — +- Update automations documentation — +- Fix typo in tutorial documentation — +- Add tip on `keys` in artifacts documentation — +- Expand docstrings for artifacts — +- Update description of `image` parameter of `DockerContainer` in infrastructure documentation — +- Lowercase Prefect server where appropriate — +- Remove `Upgrading from Prefect Beta` section of installation page — +- Update rate limit documentation to include `/set_state` and `/flows` endpoint for Prefect Cloud — +- Update documentation links in UI to concepts when possible — ## Contributors -* @BitTheByte -* @snikch made their first contribution in https://github.com/PrefectHQ/prefect/pull/9646 -* @rkscodes made their first contribution in https://github.com/PrefectHQ/prefect/pull/9682 -* @sarahmk125 made their first contribution in https://github.com/PrefectHQ/prefect/pull/9694 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.10...2.10.11 +- @BitTheByte + +- @snikch made their first contribution in +- @rkscodes made their first contribution in +- @sarahmk125 made their first contribution in + +**All changes**: ## Release 2.10.10 @@ -1960,37 +2215,40 @@ We wanted the CLI to be as fast as the rest of Prefect. Through a series of enha See the following pull requests for implementation details: -- Delay `apprise` imports — https://github.com/PrefectHQ/prefect/pull/9557 -- Defer import of `dateparser` — https://github.com/PrefectHQ/prefect/pull/9582 -- Defer loading of Prefect integrations until necessary — https://github.com/PrefectHQ/prefect/pull/9571 -- Add `Block.get_block_class_from_key` and replace external uses of `lookup_type` — https://github.com/PrefectHQ/prefect/pull/9621 -- Load collections before auto-registering block types on the server — https://github.com/PrefectHQ/prefect/pull/9626 -- Do not restrict deployment build infrastructure types to types known at import time — https://github.com/PrefectHQ/prefect/pull/9625 +- Delay `apprise` imports — +- Defer import of `dateparser` — +- Defer loading of Prefect integrations until necessary — +- Add `Block.get_block_class_from_key` and replace external uses of `lookup_type` — +- Load collections before auto-registering block types on the server — +- Do not restrict deployment build infrastructure types to types known at import time — ### Enhancements -- Handle `SIGTERM` received by workers gracefully — https://github.com/PrefectHQ/prefect/pull/9530 -- Add ability to view table artifacts with NaN values in the Prefect UI — https://github.com/PrefectHQ/prefect/pull/9585 -- Update `prefect version` command to avoid creating the database if it does not exist — https://github.com/PrefectHQ/prefect/pull/9586 -- Allow client retries when server SQLite database is busy — https://github.com/PrefectHQ/prefect/pull/9632 -- Allow client retries when general database errors are encountered — https://github.com/PrefectHQ/prefect/pull/9633 -- Ensure published Docker images have latest versions of requirements — https://github.com/PrefectHQ/prefect/pull/9640 + +- Handle `SIGTERM` received by workers gracefully — +- Add ability to view table artifacts with NaN values in the Prefect UI — +- Update `prefect version` command to avoid creating the database if it does not exist — +- Allow client retries when server SQLite database is busy — +- Allow client retries when general database errors are encountered — +- Ensure published Docker images have latest versions of requirements — ### Fixes -- Fix bug where `SIGTERM` was not properly captured as a flow run crash for flow runs created by a deployment — https://github.com/PrefectHQ/prefect/pull/9543 -- Fix deadlock when logging is overridden from an asynchronous context — https://github.com/PrefectHQ/prefect/pull/9602 -- Fix orchestration race conditions by adding lock for update to flow run state transitions — https://github.com/PrefectHQ/prefect/pull/9590 -- Fix date range filter on flow runs page — https://github.com/PrefectHQ/prefect/pull/9636 -- Fix bug where ephemeral server raised exceptions client-side — https://github.com/PrefectHQ/prefect/pull/9637 -- Fix bug where ARM64 Docker images had a corrupt database — https://github.com/PrefectHQ/prefect/pull/9587 + +- Fix bug where `SIGTERM` was not properly captured as a flow run crash for flow runs created by a deployment — +- Fix deadlock when logging is overridden from an asynchronous context — +- Fix orchestration race conditions by adding lock for update to flow run state transitions — +- Fix date range filter on flow runs page — +- Fix bug where ephemeral server raised exceptions client-side — +- Fix bug where ARM64 Docker images had a corrupt database — ### Documentation -- Clarify the retry on tasks concept page — https://github.com/PrefectHQ/prefect/pull/9560 -- Improve the navigation structure and clarity of the API docs — https://github.com/PrefectHQ/prefect/pull/9574 -- Add `work_pool_name` to `Deployment.build_from_flow` on deployments concept page — https://github.com/PrefectHQ/prefect/pull/9581 -- Add additional worker types to work pools, workers & agents concept page — https://github.com/PrefectHQ/prefect/pull/9580 -- Add docstrings for all schema filters — https://github.com/PrefectHQ/prefect/pull/9572 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.9...2.10.10 +- Clarify the retry on tasks concept page — +- Improve the navigation structure and clarity of the API docs — +- Add `work_pool_name` to `Deployment.build_from_flow` on deployments concept page — +- Add additional worker types to work pools, workers & agents concept page — +- Add docstrings for all schema filters — + +**All changes**: ## Release 2.10.9 @@ -2000,34 +2258,37 @@ Workers now link relevant logs to specific flow runs, allowing you to view infra Process worker logs - You'll see generic logs from all worker types. Integration worker implementations such as Kubernetes workers will be updated to send additional rich logs to give you insight into the behavior of flow run infrastructure. -See https://github.com/PrefectHQ/prefect/pull/9496 for details. +See for details. ### Enhancements -- Handle `SIGTERM` received by agent gracefully — https://github.com/PrefectHQ/prefect/pull/8691 -- Add global default settings for flow and task retries and retry delay seconds — https://github.com/PrefectHQ/prefect/pull/9171 -- Add support for populating submodules to `git_clone_project` projects step — https://github.com/PrefectHQ/prefect/pull/9504 -- Add wrapper for exceptions encountered while resolving parameter inputs — https://github.com/PrefectHQ/prefect/pull/8584 -- Add flush of logs before exiting deployed flow run processes to ensure messages are not lost — https://github.com/PrefectHQ/prefect/pull/9516 -- Update worker to be able to include itself as a related resource — https://github.com/PrefectHQ/prefect/pull/9531 + +- Handle `SIGTERM` received by agent gracefully — +- Add global default settings for flow and task retries and retry delay seconds — +- Add support for populating submodules to `git_clone_project` projects step — +- Add wrapper for exceptions encountered while resolving parameter inputs — +- Add flush of logs before exiting deployed flow run processes to ensure messages are not lost — +- Update worker to be able to include itself as a related resource — ### Fixes -- Fix bug where `SIGTERM` was not properly captured as a flow run crash — https://github.com/PrefectHQ/prefect/pull/9498 -- Fix pass of optional parameters to API in `client.create_work_queue` — https://github.com/PrefectHQ/prefect/pull/9521 + +- Fix bug where `SIGTERM` was not properly captured as a flow run crash — +- Fix pass of optional parameters to API in `client.create_work_queue` — ### Documentation -- Add tip about flow run level concurrency — https://github.com/PrefectHQ/prefect/pull/9490 -- Add documentation on `on_failure` flow run state change hook — https://github.com/PrefectHQ/prefect/pull/9511 -- Update tutorials landing page — https://github.com/PrefectHQ/prefect/pull/9450 +- Add tip about flow run level concurrency — +- Add documentation on `on_failure` flow run state change hook — +- Update tutorials landing page — ### Contributors -* @andrewbrannan made their first contribution in https://github.com/PrefectHQ/prefect/pull/9521 + +- @andrewbrannan made their first contribution in + - @ddelange -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.8...2.10.9 +**All changes**: ## Release 2.10.8 @@ -2036,41 +2297,44 @@ See https://github.com/PrefectHQ/prefect/pull/9496 for details. A flow run orchestration rule which was previously intended to prevent backwards transitions is updated in this release to allow most transitions. Now, it only prevents some transitions to `PENDING` states to prevent race conditions during handling of runs by multiple agents or workers. This improves orchestration behavior during infrastructure restarts. For example, when a Kubernetes pod is interrupted, the flow run can be rescheduled on a new pod by Kubernetes. Previously, Prefect would abort the run as it attempted to transition from a `RUNNING` to a `RUNNING` state. Now, Prefect will allow this transition and your flow run will continue. In summary, the following rules apply now: + - `CANCELLED` -> `PENDING` is not allowed - `CANCELLING`/`RUNNING` -> `RUNNING` is allowed - `CANCELLING`/`RUNNING`/`PENDING` -> `SCHEDULED` is allowed -See https://github.com/PrefectHQ/prefect/pull/9447 for details. - +See for details. ### Enhancements -- Display message when service back-off is reset to avoid confusion — https://github.com/PrefectHQ/prefect/pull/9463 -- Improve `QueueService` performance — https://github.com/PrefectHQ/prefect/pull/9481 + +- Display message when service back-off is reset to avoid confusion — +- Improve `QueueService` performance — ### Fixes -- Ensure deployment creation does not require write access when a prefectignore file exists — https://github.com/PrefectHQ/prefect/pull/9460 -- Fix bug where `deployment apply` command could hang on exit — https://github.com/PrefectHQ/prefect/pull/9481 + +- Ensure deployment creation does not require write access when a prefectignore file exists — +- Fix bug where `deployment apply` command could hang on exit — ### Deprecations -- Add future warning for Python 3.7 EOL — https://github.com/PrefectHQ/prefect/pull/9469 -### Documentation -- Move creating a new worker type tutorial to guides — https://github.com/PrefectHQ/prefect/pull/9455 -- Fix `name` description in `deployment.yaml` reference — https://github.com/PrefectHQ/prefect/pull/9461 +- Add future warning for Python 3.7 EOL — +### Documentation -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.7...2.10.8 +- Move creating a new worker type tutorial to guides — +- Fix `name` description in `deployment.yaml` reference — +**All changes**: ## Release 2.10.7 ### New and improved Flows page -This release combines the previously separate flows and deployments UI pages into a single, holistic page that brings together flows and deployments, as well as their recent and upcoming runs. You can now see the state of the most recent flow run for each flow and deployment, giving you a snapshot of the status of your workspace. In addition, you can now filter deployments by whether their schedule is active and the work pool to which flow runs are submitted. See https://github.com/PrefectHQ/prefect/pull/9438 for details. +This release combines the previously separate flows and deployments UI pages into a single, holistic page that brings together flows and deployments, as well as their recent and upcoming runs. You can now see the state of the most recent flow run for each flow and deployment, giving you a snapshot of the status of your workspace. In addition, you can now filter deployments by whether their schedule is active and the work pool to which flow runs are submitted. See for details. ![flows-page](https://user-images.githubusercontent.com/3407835/236275227-04944fde-cdc2-4f44-bcae-eb65f4cafa0d.png) ### `on_crashed` state change hook for flows + This release introduces the new `on_crashed` hook for flows, allowing you to add client-side hooks that will be called when your flow crashes. This is useful for cases where you want to execute code without involving the Prefect API, and for custom handling on `CRASHED` terminal states. This callable hook will receive three arguments: `flow`, `flow_run`, and `state`. Here is an example of how to use the `on_crashed` hook in your flow: @@ -2107,46 +2371,50 @@ def my_flow(): See the [pull request](https://github.com/PrefectHQ/prefect/pull/9418) for implementation details. ### Enhancements -- Prevent unnecessarily verbose logs by updating `log_prints` to ignore prints where a custom `file` is used — https://github.com/PrefectHQ/prefect/pull/9358 -- Create a process work pool by default when a new worker is started with a new work pool name and no type — https://github.com/PrefectHQ/prefect/pull/9326 -- Add support for asynchronous project steps — https://github.com/PrefectHQ/prefect/pull/9388 -- Update `critical_service_loop` to retry on all 5XX HTTP status codes — https://github.com/PrefectHQ/prefect/pull/9400 -- Add backoff on failure to agent critical loop services — https://github.com/PrefectHQ/prefect/pull/9402 -- Add print statement to `git pull` to isolate issues between clone and execution — https://github.com/PrefectHQ/prefect/pull/9328 -- Add `on_crashed` flow run state change hook — https://github.com/PrefectHQ/prefect/pull/9418 -- Make build->push step explicit in docker project recipes — https://github.com/PrefectHQ/prefect/pull/9417 -- Add storage blocks to cli `deployment build` help description — https://github.com/PrefectHQ/prefect/pull/9411 -- Add `call_in_...` methods to the concurrency API — https://github.com/PrefectHQ/prefect/pull/9415 -- Add support for `Callable[[], T]` to concurrency API methods — https://github.com/PrefectHQ/prefect/pull/9413 + +- Prevent unnecessarily verbose logs by updating `log_prints` to ignore prints where a custom `file` is used — +- Create a process work pool by default when a new worker is started with a new work pool name and no type — +- Add support for asynchronous project steps — +- Update `critical_service_loop` to retry on all 5XX HTTP status codes — +- Add backoff on failure to agent critical loop services — +- Add print statement to `git pull` to isolate issues between clone and execution — +- Add `on_crashed` flow run state change hook — +- Make build->push step explicit in docker project recipes — +- Add storage blocks to cli `deployment build` help description — +- Add `call_in_...` methods to the concurrency API — +- Add support for `Callable[[], T]` to concurrency API methods — - Add a parameters JSON input option for deployments in the UI — [`#1405`](https://github.com/PrefectHQ/prefect-ui-library/pull/1405) - Improve consistency in UI help modals — [`#1397`](https://github.com/PrefectHQ/prefect-ui-library/pull/1397) ### Fixes -- Add guard against null schedule in `deployment.yaml` — https://github.com/PrefectHQ/prefect/pull/9373 -- Fix issue preventing work pool filter from being applied to the flow runs page — https://github.com/PrefectHQ/prefect/pull/9390 -- Fix project recipe `image_name` and `tag` templating in docker-git, docker-gcs, and docker-s3 — https://github.com/PrefectHQ/prefect/pull/9425 -- Fix bug with work queues showing as unhealthy when a work queue with the same name is unhealthy — https://github.com/PrefectHQ/prefect/pull/9437 -- Fix bug where child flows would not fail the parent when they received invalid arguments — https://github.com/PrefectHQ/prefect/pull/9386 + +- Add guard against null schedule in `deployment.yaml` — +- Fix issue preventing work pool filter from being applied to the flow runs page — +- Fix project recipe `image_name` and `tag` templating in docker-git, docker-gcs, and docker-s3 — +- Fix bug with work queues showing as unhealthy when a work queue with the same name is unhealthy — +- Fix bug where child flows would not fail the parent when they received invalid arguments — - Fix schema values mapping on the create flow run forms to ensure all parameter values can be edited — [`#1407`](https://github.com/PrefectHQ/prefect-ui-library/pull/1407) - Add a check for color scheme to ensure the flow run state favicon is visible — [`#1392`](https://github.com/PrefectHQ/prefect-ui-library/pull/1392) -- Fix deadlock during API log handler flush when logging configuration is overridden — https://github.com/PrefectHQ/prefect/pull/9354 -- Fix send/drain race conditions in queue services — https://github.com/PrefectHQ/prefect/pull/9426 -- Fix bug where missing trailing slash in remote filesystems path would cause download failures — https://github.com/PrefectHQ/prefect/pull/9440 +- Fix deadlock during API log handler flush when logging configuration is overridden — +- Fix send/drain race conditions in queue services — +- Fix bug where missing trailing slash in remote filesystems path would cause download failures — ### Documentation -- Add a link to bug bounty program information — https://github.com/PrefectHQ/prefect/pull/9366 -- Add `Additional Resources` Section to Work Pools, Workers, & Agents page — https://github.com/PrefectHQ/prefect/pull/9393 -- Fix mistaken placement of `result_storage` parameter — https://github.com/PrefectHQ/prefect/pull/9422 -- Add concept list to concept section parent page — https://github.com/PrefectHQ/prefect/pull/9404 -- Add Paused and Cancelling states to states concept page — https://github.com/PrefectHQ/prefect/pull/9435 -- Update docs logos — https://github.com/PrefectHQ/prefect/pull/9365 -- Direct *Prefect Integration template* link to the correct page — https://github.com/PrefectHQ/prefect/pull/9362 -- Update landing page image — https://github.com/PrefectHQ/prefect/pull/9448 + +- Add a link to bug bounty program information — +- Add `Additional Resources` Section to Work Pools, Workers, & Agents page — +- Fix mistaken placement of `result_storage` parameter — +- Add concept list to concept section parent page — +- Add Paused and Cancelling states to states concept page — +- Update docs logos — +- Direct _Prefect Integration template_ link to the correct page — +- Update landing page image — ### New Contributors -- @rmorshea made their first contribution in https://github.com/PrefectHQ/prefect/pull/9422 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.6...2.10.7 +- @rmorshea made their first contribution in + +**All changes**: ## Release 2.10.6 @@ -2158,7 +2426,7 @@ Deployments that are declared in a project are independent of each other and can Because the deployments are all declared in a single YAML file, you can also take advantage of YAML anchors and aliases to avoid duplication in your `deployment.yaml` file. This enables declaring custom projects actions once and reusing them across different deployments or using the same schedule for multiple deployments. To learn more about Projects, check out our [documentation](https://docs.prefect.io/latest/concepts/projects/) and [tutorials](https://docs.prefect.io/latest/tutorials/projects/) to quickly accelerate your flow deployment process! -See https://github.com/PrefectHQ/prefect/pull/9217 for details. +See for details. ### Improve run restart behavior @@ -2176,50 +2444,54 @@ For example, these changes enable the following behaviors: - A flow run may be rescheduled without using the "awaiting retry" name - A flow run that fails and is orchestrated again will run instead of aborting -See https://github.com/PrefectHQ/prefect/pull/9152 for details. +See for details. ### Enhancements -- Add support for recursive flow calls — https://github.com/PrefectHQ/prefect/pull/9342 -- Add support for concurrent runs same flow — https://github.com/PrefectHQ/prefect/pull/9342 -- Add ability for `flow_run_name` and `task_run_name` settings to accept functions — https://github.com/PrefectHQ/prefect/pull/8933 -- Add pending items count to service failure exception message — https://github.com/PrefectHQ/prefect/pull/9306 -- Add `severity` key to JSON-formatted logs for GCP compatibility — https://github.com/PrefectHQ/prefect/pull/9200 -- Update orchestration rules to allow transitions from terminal states — https://github.com/PrefectHQ/prefect/pull/9152 -- Enable filtering flows by work pool at the `/flows/filter` endpoint — https://github.com/PrefectHQ/prefect/pull/9308 -- Add `--tail` option to `prefect flow-run logs` CLI — https://github.com/PrefectHQ/prefect/pull/9028 -- Enhance UI handling of flow run graph and accompanying selection panel — https://github.com/PrefectHQ/prefect/pull/9333 -- Enhance UI rendering of schema-generated forms (used for flow run creation, deployment editing, block configuration, notifications, and work pool job templates) and their values — https://github.com/PrefectHQ/prefect-ui-library/pull/1384 -- Update icons and Prefect logo — https://github.com/PrefectHQ/prefect/pull/9352 -- Add results to task run page — https://github.com/PrefectHQ/prefect-ui-library/pull/1372 -- Add artifacts to task run page — https://github.com/PrefectHQ/prefect/pull/9353 -- Show entrypoint and path in deployment details — https://github.com/PrefectHQ/prefect-ui-library/pull/1364 -- Enhance clarity of error message by raising `UnfinishedRun` instead of `MissingResult` when state is not final — https://github.com/PrefectHQ/prefect-ui-library/pull/9334 + +- Add support for recursive flow calls — +- Add support for concurrent runs same flow — +- Add ability for `flow_run_name` and `task_run_name` settings to accept functions — +- Add pending items count to service failure exception message — +- Add `severity` key to JSON-formatted logs for GCP compatibility — +- Update orchestration rules to allow transitions from terminal states — +- Enable filtering flows by work pool at the `/flows/filter` endpoint — +- Add `--tail` option to `prefect flow-run logs` CLI — +- Enhance UI handling of flow run graph and accompanying selection panel — +- Enhance UI rendering of schema-generated forms (used for flow run creation, deployment editing, block configuration, notifications, and work pool job templates) and their values — +- Update icons and Prefect logo — +- Add results to task run page — +- Add artifacts to task run page — +- Show entrypoint and path in deployment details — +- Enhance clarity of error message by raising `UnfinishedRun` instead of `MissingResult` when state is not final — ### Fixes -- Ensure the Prefect UI displays actual parameters used to kick off a flow run — https://github.com/PrefectHQ/prefect/pull/9293 -- Ensure workers only create one client while running — https://github.com/PrefectHQ/prefect/pull/9302 -- Ensure services are drained on global loop shutdown — https://github.com/PrefectHQ/prefect/pull/9307 -- Show logs on pending flow run pages — https://github.com/PrefectHQ/prefect/pull/9313 -- Fix `flow-run logs --limit` — https://github.com/PrefectHQ/prefect/pull/9314 -- Fix `future.result()` and `future.wait()` calls from async contexts — https://github.com/PrefectHQ/prefect/pull/9316 -- Update `QueueService.send` to wait for the item to be placed in the queue before returning — https://github.com/PrefectHQ/prefect/pull/9318 -- Update `resolve_futures_to_data` and `resolve_futures_to_states` to wait for futures in the correct event loop — https://github.com/PrefectHQ/prefect/pull/9336 -- Fix bug where tasks were not called when debug mode was enabled — https://github.com/PrefectHQ/prefect/pull/9341 -- Fix bug where boolean values for new flow runs created through the UI were not sent if the value matched the deployment's schema default — https://github.com/PrefectHQ/prefect-ui-library/pull/1389 -- Fix race condition in event loop thread start — https://github.com/PrefectHQ/prefect/pull/9343 + +- Ensure the Prefect UI displays actual parameters used to kick off a flow run — +- Ensure workers only create one client while running — +- Ensure services are drained on global loop shutdown — +- Show logs on pending flow run pages — +- Fix `flow-run logs --limit` — +- Fix `future.result()` and `future.wait()` calls from async contexts — +- Update `QueueService.send` to wait for the item to be placed in the queue before returning — +- Update `resolve_futures_to_data` and `resolve_futures_to_states` to wait for futures in the correct event loop — +- Fix bug where tasks were not called when debug mode was enabled — +- Fix bug where boolean values for new flow runs created through the UI were not sent if the value matched the deployment's schema default — +- Fix race condition in event loop thread start — ### Documentation -- Add tutorial for developing a new worker — https://github.com/PrefectHQ/prefect/pull/9179 -- Fix social cards to enable previews when linking documentation — https://github.com/PrefectHQ/prefect/pull/9321 -- Fix rendering of Prefect Server and Cloud feature list — https://github.com/PrefectHQ/prefect/pull/9305 -- Fix a broken link and clarify language — https://github.com/PrefectHQ/prefect/pull/9295 -- Update "Event Feed" screenshot — https://github.com/PrefectHQ/prefect/pull/9349 + +- Add tutorial for developing a new worker — +- Fix social cards to enable previews when linking documentation — +- Fix rendering of Prefect Server and Cloud feature list — +- Fix a broken link and clarify language — +- Update "Event Feed" screenshot — ### New Contributors -- @rsampaths16 made their first contribution in https://github.com/PrefectHQ/prefect/pull/8933 -- @Shubhamparashar made their first contribution in https://github.com/PrefectHQ/prefect/pull/9028 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.5...2.10.6 +- @rsampaths16 made their first contribution in +- @Shubhamparashar made their first contribution in + +**All changes**: ## Release 2.10.5 @@ -2232,28 +2504,34 @@ With the new [Deploy a Prefect flow](https://github.com/marketplace/actions/depl Workers, Prefect's next-generation agents, have dedicated infrastructure types. This week, we are releasing typed workers for each major cloud provider: AWS, GCP, and Azure. You will be able to find them in the [prefect-aws](https://github.com/PrefectHQ/prefect-aws), [prefect-gcp](https://prefecthq.github.io/prefect-gcp/), and [prefect-azure](https://github.com/PrefectHQ/prefect-azure) collections, respectively. See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect-aws/pull/238 -- https://github.com/PrefectHQ/prefect-aws/pull/244 -- https://github.com/PrefectHQ/prefect-gcp/pull/172 -- https://github.com/PrefectHQ/prefect-azure/pull/87 + +- +- +- +- ### Enhancements + - Add `idempotency_key` to flow runs filter — [#8600](https://github.com/PrefectHQ/prefect/pull/8600) - Add `details` tab to flow run page and increase flow run graph width — [#9258](https://github.com/PrefectHQ/prefect/pull/9258) - Add status code to base client log on retry — [#9265](https://github.com/PrefectHQ/prefect/pull/9265) ### Fixes + - Fix issue in which work queues were duplicated in the `default-agent-pool` when creating a deployment — [#9046](https://github.com/PrefectHQ/prefect/pull/9046) - Add `configuration` to `Worker.kill_infrastructure` signature — [#9250](https://github.com/PrefectHQ/prefect/pull/9250) - Update `critical_service_loop` to throw a runtime error on failure — [#9267](https://github.com/PrefectHQ/prefect/pull/9267) - Fix pip requirement inference compatibility with Python 3.11+ and pip 23.1+ — [#9278](https://github.com/PrefectHQ/prefect/pull/9278) - Fix validation error occurring on default values in `variables` schema of `Workpool.base_job_template` [#9282](https://github.com/PrefectHQ/prefect/pull/9282) + ### Experimental + - Add `worker.executed-flow-run` event — [#9227](https://github.com/PrefectHQ/prefect/pull/9227) - Emit events for worker lifecycle — [#9249](https://github.com/PrefectHQ/prefect/pull/9249) - Emit `cancelled-flow-run` event when worker cancels a flow run — [#9255](https://github.com/PrefectHQ/prefect/pull/9255) ### Documentation + - Fix broken link on docs landing page — [#9247](https://github.com/PrefectHQ/prefect/pull/9247) - Remove outdated warning from task run concurrency UI docs — [#9256](https://github.com/PrefectHQ/prefect/pull/9256) - Add `edit` button to docs to improve ability to fix documentation — [#9259](https://github.com/PrefectHQ/prefect/pull/9259) @@ -2261,7 +2539,7 @@ See the following pull requests for implementation details: - Add tutorial for creating a worker — [#9179](https://github.com/PrefectHQ/prefect/pull/9179) - Add GitHub Action to trigger versioned builds in docs repository — [#8984](https://github.com/PrefectHQ/prefect/pull/8984) -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.4...2.10.5 +**All changes**: ## Release 2.10.4 @@ -2269,13 +2547,16 @@ This release further refines Prefect 2.10 with enhancements for [project deploym [workers](https://docs.prefect.io/latest/concepts/work-pools/#worker-overview), fixes for flow run cancellation and the worker CLI, and more. ### More flexible project deployments + Prior to this release, removing keys from a project's `deployment.yaml` caused an error. Thanks to the changes in [#9190](https://github.com/PrefectHQ/prefect/pull/9190), Prefect now uses default values for any required keys missing from your project's configuration. ### Enhancements + - Allow partial `deployment.yaml` files for projects by using defaults for missing values — [#9190](https://github.com/PrefectHQ/prefect/pull/9190) - Add flow run cancellation support for workers — [#9198](https://github.com/PrefectHQ/prefect/pull/9198) ### Fixes + - Prevent scheduled flow runs from getting stuck in `CANCELLING` state — [#8414](https://github.com/PrefectHQ/prefect/pull/8414) - Fix `work_queues` and `worker_type` arguments for the `prefect worker start` CLI command — [#9154](https://github.com/PrefectHQ/prefect/pull/9154) - Fix overflow in flow run logger UI [`#1342`](https://github.com/PrefectHQ/prefect-ui-library/pull/1342) @@ -2283,14 +2564,16 @@ Prior to this release, removing keys from a project's `deployment.yaml` caused a - Improve flow graph UX by suppressing shortcuts when a metakey is active [`#1333`](https://github.com/PrefectHQ/prefect-ui-library/pull/1333) ### Experimental + - Emit an event when a worker submits a flow run for execution — [#9203](https://github.com/PrefectHQ/prefect/pull/9203) ### Documentation + - Fix a broken link by removing an obsolete redirect — [#9189](https://github.com/PrefectHQ/prefect/pull/9189) - Add polling interval information to worker and agent documentation — [#9209](https://github.com/PrefectHQ/prefect/pull/9209) - Update documentation badge styling to improve docs usability — [#9207](https://github.com/PrefectHQ/prefect/pull/9207) -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.3...2.10.4 +**All changes**: ## Release 2.10.3 @@ -2298,53 +2581,58 @@ This release builds on 2.10 to further improve the experience of setting up and This release also includes [a critical fix](https://github.com/PrefectHQ/prefect/pull/9180) for Prefect logs that were sometimes delayed in being sent to the API. - ### Enhancements -- Rename `prefect.__root_path__` to `prefect.__development_base_path__` — https://github.com/PrefectHQ/prefect/pull/9136 -- Include flow run and flow as related resources when emitting events via the events worker — https://github.com/PrefectHQ/prefect/pull/9129 -- Improve Cloud storage Projects recipes — https://github.com/PrefectHQ/prefect/pull/9145 -- Use new sessions and transactions for each query during `CancellationCleanup` — https://github.com/PrefectHQ/prefect/pull/9124 -- Stream `git` output during `git_clone_project` — https://github.com/PrefectHQ/prefect/pull/9149 -- Update deployment defaults with project init — https://github.com/PrefectHQ/prefect/pull/9146 -- Add ability to mock `prefect.runtime` attributes via environment variable — https://github.com/PrefectHQ/prefect/pull/9156 -- Add scheduling options to deploy CLI — https://github.com/PrefectHQ/prefect/pull/9176 -- Add deployment and flow filters to `/artifacts/filter` and `/artifacts/latest/filter` routes — https://github.com/PrefectHQ/prefect/pull/9089 -- Add `/artifacts/latest/count` route — https://github.com/PrefectHQ/prefect/pull/9090 -- Add flow run metadata to task run logger — https://github.com/PrefectHQ/prefect/pull/9170 -- Add pragma statements automatically if sqlite writing database migrations for SQLite — https://github.com/PrefectHQ/prefect/pull/9169 -- Improve Projects `recipe` initialization UX — https://github.com/PrefectHQ/prefect/pull/9158 + +- Rename `prefect.__root_path__` to `prefect.__development_base_path__` — +- Include flow run and flow as related resources when emitting events via the events worker — +- Improve Cloud storage Projects recipes — +- Use new sessions and transactions for each query during `CancellationCleanup` — +- Stream `git` output during `git_clone_project` — +- Update deployment defaults with project init — +- Add ability to mock `prefect.runtime` attributes via environment variable — +- Add scheduling options to deploy CLI — +- Add deployment and flow filters to `/artifacts/filter` and `/artifacts/latest/filter` routes — +- Add `/artifacts/latest/count` route — +- Add flow run metadata to task run logger — +- Add pragma statements automatically if sqlite writing database migrations for SQLite — +- Improve Projects `recipe` initialization UX — ### Fixes -- Update `prefect deploy` to pull `flow_name` and `entrypoint` from deployment.yaml if specified — https://github.com/PrefectHQ/prefect/pull/9157 -- Fix bug where non-zero status codes would be reported when deployed flow runs paused or failed — https://github.com/PrefectHQ/prefect/pull/9175 -- Hide command when access token is provided and `git_clone_project` fails — https://github.com/PrefectHQ/prefect/pull/9150 -- Fix bug where log worker only sent logs to API on flush rather than on an interval — https://github.com/PrefectHQ/prefect/pull/9180 -- Fix apply artifact collection filter — https://github.com/PrefectHQ/prefect/pull/9153 + +- Update `prefect deploy` to pull `flow_name` and `entrypoint` from deployment.yaml if specified — +- Fix bug where non-zero status codes would be reported when deployed flow runs paused or failed — +- Hide command when access token is provided and `git_clone_project` fails — +- Fix bug where log worker only sent logs to API on flush rather than on an interval — +- Fix apply artifact collection filter — ### Documentation -- Add artifacts to API reference — https://github.com/PrefectHQ/prefect/pull/9143 -- Expand upon Projects `steps` documentation — https://github.com/PrefectHQ/prefect/pull/9151 + +- Add artifacts to API reference — +- Expand upon Projects `steps` documentation — ### Collections -- Add `prefect-spark-on-k8s-operator` to integrations catalog list — https://github.com/PrefectHQ/prefect/pull/9029 + +- Add `prefect-spark-on-k8s-operator` to integrations catalog list — ### Contributors -* @tardunge made their first contribution in https://github.com/PrefectHQ/prefect/pull/9029 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.2...2.10.3 +- @tardunge made their first contribution in + +**All changes**: ## Release 2.10.2 -Fixes a bug where deployments were not downloaded from remote storage blocks during flow runs — https://github.com/PrefectHQ/prefect/pull/9138 +Fixes a bug where deployments were not downloaded from remote storage blocks during flow runs — ### Enhancements -- Add httpx.ConnectTimeout to the list of retry exceptions in base client — https://github.com/PrefectHQ/prefect/pull/9125 + +- Add httpx.ConnectTimeout to the list of retry exceptions in base client — ### Contributors -* @sorendaugaard made their first contribution in https://github.com/PrefectHQ/prefect/pull/9125 +- @sorendaugaard made their first contribution in -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.1...2.10.2 +**All changes**: ## Release 2.10.1 @@ -2386,13 +2674,13 @@ Projects are a contract between you and a worker, specifying what you do when yo See the new [project concept doc](https://docs.prefect.io/latest/concepts/projects/) for more information or the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/8930 -- https://github.com/PrefectHQ/prefect/pull/9103 -- https://github.com/PrefectHQ/prefect/pull/9105 -- https://github.com/PrefectHQ/prefect/pull/9112 -- https://github.com/PrefectHQ/prefect/pull/9093 -- https://github.com/PrefectHQ/prefect/pull/9083 -- https://github.com/PrefectHQ/prefect/pull/9041 +- +- +- +- +- +- +- ### Variables @@ -2438,52 +2726,56 @@ Now, when you visit the Prefect documentation site, you'll see a version selecto ![versioned docs](https://user-images.githubusercontent.com/228762/230432235-26fc9406-1390-4c63-9956-b8cdabdfba6f.png) - ### Breaking Changes -- Unused options for sorting logs have been removed from the API — https://github.com/PrefectHQ/prefect/pull/7873 + +- Unused options for sorting logs have been removed from the API — ### Enhancements -- Add artifacts view to flow run page — https://github.com/PrefectHQ/prefect/pull/9109 -- Improve performance of the background event worker — https://github.com/PrefectHQ/prefect/pull/9019 -- Update deployment flow run creation to default to a SCHEDULED state instead of PENDING — https://github.com/PrefectHQ/prefect/pull/9049 -- Add `PREFECT_CLIENT_RETRY_EXTRA_CODES` to allow retry on additional HTTP status codes — https://github.com/PrefectHQ/prefect/pull/9056 -- Improve performance of the background log worker — https://github.com/PrefectHQ/prefect/pull/9048 -- Update agent cancellation check interval to double the scheduled check interval — https://github.com/PrefectHQ/prefect/pull/9084 -- Update default agent query interval from 10s to 15s — https://github.com/PrefectHQ/prefect/pull/9085 -- Add a 10 minute cache to API healthchecks — https://github.com/PrefectHQ/prefect/pull/9069 -- Improve performance of concurrent task runner — https://github.com/PrefectHQ/prefect/pull/9073 -- Improve performance of waiting for task submission — https://github.com/PrefectHQ/prefect/pull/9072 -- Add retry on 502 BAD GATEWAY to client — https://github.com/PrefectHQ/prefect/pull/9102 -- Update local and remote file systems to return path on write — https://github.com/PrefectHQ/prefect/pull/8965 -- Add artifacts `/count` route — https://github.com/PrefectHQ/prefect/pull/9022 -- Improve performance of automatic block registration — https://github.com/PrefectHQ/prefect/pull/8838 -- Improve performance of log retrieval queries — https://github.com/PrefectHQ/prefect/pull/9035 -- Improve performance of artifact retrieval — https://github.com/PrefectHQ/prefect/pull/9061 / https://github.com/PrefectHQ/prefect/pull/9064 -- Add `--type` option to create work-pool CLI — https://github.com/PrefectHQ/prefect/pull/8993 -- Improve flow run timeline performance — https://github.com/PrefectHQ/prefect-ui-library/pull/1315 -- Add flow names to sub flows on the flow run timeline graph — https://github.com/PrefectHQ/prefect-ui-library/pull/1304 + +- Add artifacts view to flow run page — +- Improve performance of the background event worker — +- Update deployment flow run creation to default to a SCHEDULED state instead of PENDING — +- Add `PREFECT_CLIENT_RETRY_EXTRA_CODES` to allow retry on additional HTTP status codes — +- Improve performance of the background log worker — +- Update agent cancellation check interval to double the scheduled check interval — +- Update default agent query interval from 10s to 15s — +- Add a 10 minute cache to API healthchecks — +- Improve performance of concurrent task runner — +- Improve performance of waiting for task submission — +- Add retry on 502 BAD GATEWAY to client — +- Update local and remote file systems to return path on write — +- Add artifacts `/count` route — +- Improve performance of automatic block registration — +- Improve performance of log retrieval queries — +- Improve performance of artifact retrieval — / +- Add `--type` option to create work-pool CLI — +- Improve flow run timeline performance — +- Add flow names to sub flows on the flow run timeline graph — ### Fixes -- Fix bug where iterable defaults were treated as mapped parameters — https://github.com/PrefectHQ/prefect/pull/9021 -- Fix sequential execution with mapped tasks using the SequentialTaskRunner — https://github.com/PrefectHQ/prefect/pull/8473 -- Fix race condition where futures did not wait for submission to complete — https://github.com/PrefectHQ/prefect/pull/9070 -- Fix detection of iterables within `quote` annotations while mapping — https://github.com/PrefectHQ/prefect/pull/9095 -- Fix Dockerfile copy of UI package files on latest Docker version — https://github.com/PrefectHQ/prefect/pull/9077 + +- Fix bug where iterable defaults were treated as mapped parameters — +- Fix sequential execution with mapped tasks using the SequentialTaskRunner — +- Fix race condition where futures did not wait for submission to complete — +- Fix detection of iterables within `quote` annotations while mapping — +- Fix Dockerfile copy of UI package files on latest Docker version — ### Documentation -- Add copy to clipboard button in documentation code blocks — https://github.com/PrefectHQ/prefect/pull/9026 -- Fixed styling of deployments mermaid diagram — https://github.com/PrefectHQ/prefect/pull/9017 -- Add documentation for database migrations — https://github.com/PrefectHQ/prefect/pull/9044 -- Adds documentation for BitBucket to flow code storage types — https://github.com/PrefectHQ/prefect/pull/9080 -- Update rate limit documentation for Cloud — https://github.com/PrefectHQ/prefect/pull/9100 + +- Add copy to clipboard button in documentation code blocks — +- Fixed styling of deployments mermaid diagram — +- Add documentation for database migrations — +- Adds documentation for BitBucket to flow code storage types — +- Update rate limit documentation for Cloud — ### Contributors -- @mianos made their first contribution in https://github.com/PrefectHQ/prefect/pull/9077 -- @dominictarro made their first contribution in https://github.com/PrefectHQ/prefect/pull/8965 + +- @mianos made their first contribution in +- @dominictarro made their first contribution in - @joelluijmes - @john-jam -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.9.0...2.10.0 +**All changes**: ## Release 2.9.0 @@ -2533,12 +2825,13 @@ You can view your artifacts in the Artifacts page of the Prefect UI, easily sear ![Table artifact in a timeline view](https://user-images.githubusercontent.com/27291717/228905740-bd297de9-6381-45ec-aba3-8b72def70a08.png) See [the documentation](https://docs.prefect.io/concepts/artifacts) for more information, as well as the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/9003 -- https://github.com/PrefectHQ/prefect/pull/8832 -- https://github.com/PrefectHQ/prefect/pull/8932 -- https://github.com/PrefectHQ/prefect/pull/8875 -- https://github.com/PrefectHQ/prefect/pull/8874 -- https://github.com/PrefectHQ/prefect/pull/8985 + +- +- +- +- +- +- ### Configure result storage keys @@ -2566,6 +2859,7 @@ my_flow() ``` Which will persist three result files in the storage directory: + ``` $ ls ~/.prefect/storage | grep "hello__" hello__rousing-mushroom__bar.json @@ -2591,43 +2885,49 @@ The `prefect.runtime` module is now the preferred way to access information abou See [the documentation](https://docs.prefect.io/concepts/runtime-context/) for more information. See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/8947 -- https://github.com/PrefectHQ/prefect/pull/8948 -- https://github.com/PrefectHQ/prefect/pull/8949 -- https://github.com/PrefectHQ/prefect/pull/8951 -- https://github.com/PrefectHQ/prefect/pull/8954 -- https://github.com/PrefectHQ/prefect/pull/8956 + +- +- +- +- +- +- ### Enhancements -- Add unique integers to worker thread names for inspection — https://github.com/PrefectHQ/prefect/pull/8908 -- Add support to `JSONSerializer` for serialization of exceptions so they are persisted even on failure — https://github.com/PrefectHQ/prefect/pull/8922 -- Add Gzip middleware to the UI and API FastAPI apps for compressing responses — https://github.com/PrefectHQ/prefect/pull/8931 -- Update the runtime to detect flow run information from task run contexts — https://github.com/PrefectHQ/prefect/pull/8951 + +- Add unique integers to worker thread names for inspection — +- Add support to `JSONSerializer` for serialization of exceptions so they are persisted even on failure — +- Add Gzip middleware to the UI and API FastAPI apps for compressing responses — +- Update the runtime to detect flow run information from task run contexts — ### Fixes -- Fix imports in copytree backport for Python 3.7 — https://github.com/PrefectHQ/prefect/pull/8925 -- Retry on sqlite operational errors — https://github.com/PrefectHQ/prefect/pull/8950 -- Add 30 second timeout to shutdown of the log worker thread — https://github.com/PrefectHQ/prefect/pull/8983 + +- Fix imports in copytree backport for Python 3.7 — +- Retry on sqlite operational errors — +- Add 30 second timeout to shutdown of the log worker thread — ### Documentation -- Disambiguate reference to "Blocks" — https://github.com/PrefectHQ/prefect/pull/8921 -- Fix broken concepts link — https://github.com/PrefectHQ/prefect/pull/8923 -- Add note about fine-grained PAT format — https://github.com/PrefectHQ/prefect/pull/8929 -- Add `UnpersistedResult` type — https://github.com/PrefectHQ/prefect/pull/8953 -- Update docs CSS and config for versioning compatibility — https://github.com/PrefectHQ/prefect/pull/8957 -- Clarify Filesystem package dependencies — https://github.com/PrefectHQ/prefect/pull/8989 -- Update flow runs documentation — https://github.com/PrefectHQ/prefect/pull/8919 -- Fix missing backticks on Work Pools concept page — https://github.com/PrefectHQ/prefect/pull/8942 -- Update links to the release notes in the installation guide — https://github.com/PrefectHQ/prefect/pull/8974 -- Fix `EXTRA_PIP_PACKAGES` info in Docker guide — https://github.com/PrefectHQ/prefect/pull/8995 -- Fix `KubernetesJob.job_watch_timeout_seconds` docstring — https://github.com/PrefectHQ/prefect/pull/8977 -- Add task run runtime to API reference — https://github.com/PrefectHQ/prefect/pull/8998 -- Add documentation for runtime context — https://github.com/PrefectHQ/prefect/pull/8999 + +- Disambiguate reference to "Blocks" — +- Fix broken concepts link — +- Add note about fine-grained PAT format — +- Add `UnpersistedResult` type — +- Update docs CSS and config for versioning compatibility — +- Clarify Filesystem package dependencies — +- Update flow runs documentation — +- Fix missing backticks on Work Pools concept page — +- Update links to the release notes in the installation guide — +- Fix `EXTRA_PIP_PACKAGES` info in Docker guide — +- Fix `KubernetesJob.job_watch_timeout_seconds` docstring — +- Add task run runtime to API reference — +- Add documentation for runtime context — ### Contributors -- @andreadistefano made their first contribution in https://github.com/PrefectHQ/prefect/pull/8942 -- @knl made their first contribution in https://github.com/PrefectHQ/prefect/pull/8974 -- @thomas-te made their first contribution in https://github.com/PrefectHQ/prefect/pull/8959 + +- @andreadistefano made their first contribution in +- @knl made their first contribution in +- @thomas-te made their first contribution in + ## Release 2.8.7 If you have been watching the experimental section of our release notes, you may have noticed a lot of work around concurrency tooling, flow run graph enhancements, and result artifacts. With this release, these experiments have culminated into exciting features! @@ -2640,7 +2940,7 @@ The behavioral changes include: - All orchestration of flows and tasks happens in a dedicated worker thread - Synchronous flows are run on the main thread instead of worker threads - — Solves problems where flow code must be in the main thread e.g. https://github.com/PrefectHQ/prefect/issues/5991 + — Solves problems where flow code must be in the main thread e.g. - Asynchronous flows no longer share an event loop with the Prefect engine - Flow timeouts are now enforced with signals — Allows interrupt of long-running system calls like `sleep` for more effective timeout enforcement @@ -2658,10 +2958,10 @@ The behavioral changes include: See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect/pull/8702 -- https://github.com/PrefectHQ/prefect/pull/8887 -- https://github.com/PrefectHQ/prefect/pull/8903 -- https://github.com/PrefectHQ/prefect/pull/8830 +- +- +- +- ### Results tab on flow run pages @@ -2673,11 +2973,12 @@ This view provides a visual representation of the output of your tasks and flows ![Results grid view](https://user-images.githubusercontent.com/27291717/227274578-35673508-09e2-4b83-bc22-11538f813eea.png) See the following pull requests for implementation details: -- https://github.com/PrefectHQ/prefect-ui-library/pull/1207 -- https://github.com/PrefectHQ/prefect-ui-library/pull/1213 -- https://github.com/PrefectHQ/prefect-ui-library/pull/1223 -- https://github.com/PrefectHQ/prefect/pull/8904 -- https://github.com/PrefectHQ/prefect/pull/8759 + +- +- +- +- +- ### Flow run graph @@ -2687,44 +2988,51 @@ We heard that people loved the simplicity and sleekness of the timeline on the f Subflow run expansion ### Enhancements -- Add `--reverse` option to the flow run logs CLI to view logs in descending order — https://github.com/PrefectHQ/prefect/pull/8625 -- Show all flow runs for deployments rather than just the last 7 days — https://github.com/PrefectHQ/prefect/pull/8837 -- Add jitter to Prefect client request retries — https://github.com/PrefectHQ/prefect/pull/8839 -- Add `deployment.name` and `deployment.version` to `prefect.runtime` — https://github.com/PrefectHQ/prefect/pull/8864 -- Add `flow_run.scheduled_start_time` to `prefect.runtime` — https://github.com/PrefectHQ/prefect/pull/8864 -- Adjust SQLite sync mode for improved performance — https://github.com/PrefectHQ/prefect/pull/8071 -- Add debug level log of active profile on module import — https://github.com/PrefectHQ/prefect/pull/8856 -- Update server to use new FastAPI lifespan context manager — https://github.com/PrefectHQ/prefect/pull/8842 -- Add support for variadic keyword arguments to `Task.map` — https://github.com/PrefectHQ/prefect/pull/8188 -- Show the full run history in the UI — https://github.com/PrefectHQ/prefect/pull/8885 + +- Add `--reverse` option to the flow run logs CLI to view logs in descending order — +- Show all flow runs for deployments rather than just the last 7 days — +- Add jitter to Prefect client request retries — +- Add `deployment.name` and `deployment.version` to `prefect.runtime` — +- Add `flow_run.scheduled_start_time` to `prefect.runtime` — +- Adjust SQLite sync mode for improved performance — +- Add debug level log of active profile on module import — +- Update server to use new FastAPI lifespan context manager — +- Add support for variadic keyword arguments to `Task.map` — +- Show the full run history in the UI — ### Fixes -- Fix `prefect dev start` failure — https://github.com/PrefectHQ/prefect/pull/8850 -- Fix bug where `propose_state` could exceed recursion limits during extended waits — https://github.com/PrefectHQ/prefect/pull/8827 -- Fix configuration of flow run infrastructure when using agent default — https://github.com/PrefectHQ/prefect/pull/8872 -- Fix saving block document secrets that have not been modified — https://github.com/PrefectHQ/prefect/pull/8848 -- Disable SLSA provenance setting in Docker buildx to resolve image pull errors with certain Cloud providers — https://github.com/PrefectHQ/prefect/pull/8889 -- Fix race condition in worker thread start — https://github.com/PrefectHQ/prefect/pull/8886 -- The state message has been returned to the flow run metadata panel on the right side of the flow run page — https://github.com/PrefectHQ/prefect/pull/8885 + +- Fix `prefect dev start` failure — +- Fix bug where `propose_state` could exceed recursion limits during extended waits — +- Fix configuration of flow run infrastructure when using agent default — +- Fix saving block document secrets that have not been modified — +- Disable SLSA provenance setting in Docker buildx to resolve image pull errors with certain Cloud providers — +- Fix race condition in worker thread start — +- The state message has been returned to the flow run metadata panel on the right side of the flow run page — ### Experimental -- Update to worker base job template logic for nested placeholders — https://github.com/PrefectHQ/prefect/pull/8795 -- Require lowercase artifact `key` field — https://github.com/PrefectHQ/prefect/pull/8860 -- Create `emit_event` helper that takes args for an `Event` and emits it via a worker — https://github.com/PrefectHQ/prefect/pull/8867 -- Allow multiple artifacts to have the same key — https://github.com/PrefectHQ/prefect/pull/8855 -- Add common values to job configuration prior to flow run submission — https://github.com/PrefectHQ/prefect/pull/8826 + +- Update to worker base job template logic for nested placeholders — +- Require lowercase artifact `key` field — +- Create `emit_event` helper that takes args for an `Event` and emits it via a worker — +- Allow multiple artifacts to have the same key — +- Add common values to job configuration prior to flow run submission — ### Deprecations -- Creating data documents will now throw deprecation warnings — https://github.com/PrefectHQ/prefect/pull/8760 + +- Creating data documents will now throw deprecation warnings — ### Documentation -- Add documentation for events and resources — https://github.com/PrefectHQ/prefect/pull/8858 + +- Add documentation for events and resources — ### Contributors -* @lounis89 made their first contribution in https://github.com/PrefectHQ/prefect/pull/8625 -* @mesejo made their first contribution in https://github.com/PrefectHQ/prefect/pull/8842 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.6...2.8.7 +- @lounis89 made their first contribution in + +- @mesejo made their first contribution in + +**All changes**: ## Release 2.8.6 @@ -2744,117 +3052,137 @@ def my_flow(): This will create a Dask client whose name mirrors the flow run ID. Similarly, you can use `prefect.runtime` to access parameters that were passed to this deployment run via `prefect.runtime.deployment.parameters`. Note that all of these attributes will be empty if they are not available. -See https://github.com/PrefectHQ/prefect/pull/8790 for details. +See for details. ### Enhancements -- Add deployment id support to `run_deployment` — https://github.com/PrefectHQ/prefect/pull/7958 -- Disable Postgres JIT for performance improvements — https://github.com/PrefectHQ/prefect/pull/8804 + +- Add deployment id support to `run_deployment` — +- Disable Postgres JIT for performance improvements — ### Fixes -- Fix blocking file read in async method `Deployment.load_from_yaml` — https://github.com/PrefectHQ/prefect/pull/8798 -- Allow tasks and flows to make redundant transitions such as `RUNNING` -> `RUNNING` — https://github.com/PrefectHQ/prefect/pull/8802 + +- Fix blocking file read in async method `Deployment.load_from_yaml` — +- Allow tasks and flows to make redundant transitions such as `RUNNING` -> `RUNNING` — ### Experimental -- Enable setting environment variables for worker submitted flow runs — https://github.com/PrefectHQ/prefect/pull/8706 -- Add `--work-queue` option to worker CLI — https://github.com/PrefectHQ/prefect/pull/8771 -- Add artifact description column — https://github.com/PrefectHQ/prefect/pull/8805 -- Format types in result descriptions as code — https://github.com/PrefectHQ/prefect/pull/8808 -- Add artifacts for unpersisted results — https://github.com/PrefectHQ/prefect/pull/8759 -- Update default result descriptions — https://github.com/PrefectHQ/prefect/pull/8772 + +- Enable setting environment variables for worker submitted flow runs — +- Add `--work-queue` option to worker CLI — +- Add artifact description column — +- Format types in result descriptions as code — +- Add artifacts for unpersisted results — +- Update default result descriptions — ### Documentation -- Update workspace roles table to emphasize differences between roles — https://github.com/PrefectHQ/prefect/pull/8787 -- Add webhook block docs — https://github.com/PrefectHQ/prefect/pull/8773 -- Update info on Ray's support for hardware and software — https://github.com/PrefectHQ/prefect/pull/8811 + +- Update workspace roles table to emphasize differences between roles — +- Add webhook block docs — +- Update info on Ray's support for hardware and software — ### Helm chart -- Helm charts are now automatically published on each Prefect release — https://github.com/PrefectHQ/prefect/pull/8776 + +- Helm charts are now automatically published on each Prefect release — ### Contributors + - @devanshdoshi9 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.5...2.8.6 +**All changes**: ## Release 2.8.5 ### Enhancements -- Add an endpoint to retrieve data from the collection registry — https://github.com/PrefectHQ/prefect/pull/8685 -- Remove deployment flow run foreign key to speed up deployment deletion — https://github.com/PrefectHQ/prefect/pull/8684 + +- Add an endpoint to retrieve data from the collection registry — +- Remove deployment flow run foreign key to speed up deployment deletion — ### Fixes -- Fix `prefect cloud login` detection of "ENTER" on some machines — https://github.com/PrefectHQ/prefect/pull/8705 -- Fix Kubernetes job watch timeout request error by rounding floats — https://github.com/PrefectHQ/prefect/pull/8733 -- Fix flow load errors by excluding fsspec `2023.3.0` during requirements installation — https://github.com/PrefectHQ/prefect/pull/8757 -- Fix Deployment and Concurrency Limit pages tabs — https://github.com/PrefectHQ/prefect/pull/8716 -- Add tests for base exceptions and calls — https://github.com/PrefectHQ/prefect/pull/8734 + +- Fix `prefect cloud login` detection of "ENTER" on some machines — +- Fix Kubernetes job watch timeout request error by rounding floats — +- Fix flow load errors by excluding fsspec `2023.3.0` during requirements installation — +- Fix Deployment and Concurrency Limit pages tabs — +- Add tests for base exceptions and calls — ### Experimental -- Refactor supervisor API to allow configuration — https://github.com/PrefectHQ/prefect/pull/8695 -- Consolidate `WorkItem` and `Call` classes — https://github.com/PrefectHQ/prefect/pull/8697 -- Use `PREFECT_API_URL` when initializing the events client — https://github.com/PrefectHQ/prefect/pull/8704 -- Refactor supervisors to interact directly with "Worker" threads — https://github.com/PrefectHQ/prefect/pull/8714 -- Add chaining to cancel contexts — https://github.com/PrefectHQ/prefect/pull/8719 -- Add portal abstract base for worker threads and supervisors — https://github.com/PrefectHQ/prefect/pull/8717 -- Fix bugs in supervisors implementation — https://github.com/PrefectHQ/prefect/pull/8718 -- Refactor concurrency module and add documentation — https://github.com/PrefectHQ/prefect/pull/8724 -- Update block event resource IDs to use block-document id instead of name — https://github.com/PrefectHQ/prefect/pull/8730 -- Add cancellation reporting to calls and waiters — https://github.com/PrefectHQ/prefect/pull/8731 -- Add worker command output when applying deployments with a work pool — https://github.com/PrefectHQ/prefect/pull/8725 -- Add support for float timeouts using alarms — https://github.com/PrefectHQ/prefect/pull/8737 -- Add the ability to discover type from work pool when starting a worker — https://github.com/PrefectHQ/prefect/pull/8711 -- Add basic event instrumentation to blocks — https://github.com/PrefectHQ/prefect/pull/8686 + +- Refactor supervisor API to allow configuration — +- Consolidate `WorkItem` and `Call` classes — +- Use `PREFECT_API_URL` when initializing the events client — +- Refactor supervisors to interact directly with "Worker" threads — +- Add chaining to cancel contexts — +- Add portal abstract base for worker threads and supervisors — +- Fix bugs in supervisors implementation — +- Refactor concurrency module and add documentation — +- Update block event resource IDs to use block-document id instead of name — +- Add cancellation reporting to calls and waiters — +- Add worker command output when applying deployments with a work pool — +- Add support for float timeouts using alarms — +- Add the ability to discover type from work pool when starting a worker — +- Add basic event instrumentation to blocks — ### Documentation -- Corrected typo in Storage.md — https://github.com/PrefectHQ/prefect/pull/8692 -- Fix `prefect flow-run cancel` help — https://github.com/PrefectHQ/prefect/pull/8755 + +- Corrected typo in Storage.md — +- Fix `prefect flow-run cancel` help — ### Contributors -* @Zesky665 made their first contribution in https://github.com/PrefectHQ/prefect/pull/8692 -* @predatorprasad made their first contribution in https://github.com/PrefectHQ/prefect/pull/8755 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.4...2.8.5 +- @Zesky665 made their first contribution in + +- @predatorprasad made their first contribution in + +**All changes**: ## Release 2.8.4 ### Enhancements -- Enable `DefaultAzureCredential` authentication for Azure filesystem block — https://github.com/PrefectHQ/prefect/pull/7513 -- Add support for yaml config strings to `KubernetesClusterConfig` — https://github.com/PrefectHQ/prefect/pull/8643 -- Add `--description` flag to `prefect deployment build` CLI command — https://github.com/PrefectHQ/prefect/pull/8603 -- Handle SIGTERM received by server gracefully — https://github.com/PrefectHQ/prefect/pull/7948 -- Optimize database query performance by changing SQLAlchemy lazy loads from `joined` to `selectin` — https://github.com/PrefectHQ/prefect/pull/8659 -- Add clarifying modal to the task run page in the UI — https://github.com/PrefectHQ/prefect/pull/8295 + +- Enable `DefaultAzureCredential` authentication for Azure filesystem block — +- Add support for yaml config strings to `KubernetesClusterConfig` — +- Add `--description` flag to `prefect deployment build` CLI command — +- Handle SIGTERM received by server gracefully — +- Optimize database query performance by changing SQLAlchemy lazy loads from `joined` to `selectin` — +- Add clarifying modal to the task run page in the UI — ### Fixes -- Ensure flow parameters default values are present during deployment runs — https://github.com/PrefectHQ/prefect/pull/8666 -- Use a monotonic clock for Kubernetes job watch timeout deadline calculation — https://github.com/PrefectHQ/prefect/pull/8680 -- Fix version misaligned on the settings page in the UI — https://github.com/PrefectHQ/prefect/pull/8676 + +- Ensure flow parameters default values are present during deployment runs — +- Use a monotonic clock for Kubernetes job watch timeout deadline calculation — +- Fix version misaligned on the settings page in the UI — ### Experimental -- Refactor supervisors to manage submission — https://github.com/PrefectHQ/prefect/pull/8631 -- Improve supervisor repr for debugging — https://github.com/PrefectHQ/prefect/pull/8633 -- Add timeout support to supervisors — https://github.com/PrefectHQ/prefect/pull/8649 -- Track flow run id when generating task run results — https://github.com/PrefectHQ/prefect/pull/8674 -- Create `EventsWorker` to manage client lifecycle and abstract async nature — https://github.com/PrefectHQ/prefect/pull/8673 + +- Refactor supervisors to manage submission — +- Improve supervisor repr for debugging — +- Add timeout support to supervisors — +- Track flow run id when generating task run results — +- Create `EventsWorker` to manage client lifecycle and abstract async nature — ### Documentation -- Add tutorial for running an agent on Azure Container Instances — https://github.com/PrefectHQ/prefect/pull/8620 -- Add security headers for docs — https://github.com/PrefectHQ/prefect/pull/8655 -- Add markdown link fix in orchestration docs — https://github.com/PrefectHQ/prefect/pull/8660 + +- Add tutorial for running an agent on Azure Container Instances — +- Add security headers for docs — +- Add markdown link fix in orchestration docs — ## New Contributors -* @samdyzon made their first contribution in https://github.com/PrefectHQ/prefect/pull/7513 -* @mjschock made their first contribution in https://github.com/PrefectHQ/prefect/pull/8660 -* @jcorrado76 made their first contribution in https://github.com/PrefectHQ/prefect/pull/8603 -* @scharlottej13 made their first contribution in https://github.com/PrefectHQ/prefect/pull/8669 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.3...2.8.4 +- @samdyzon made their first contribution in + +- @mjschock made their first contribution in +- @jcorrado76 made their first contribution in +- @scharlottej13 made their first contribution in + +**All changes**: ## Release 2.8.3 ### `on_completion` and `on_failure` hooks for flows and tasks + With this release you can now add client-side hooks that will be called when your flow or task enters a `Completed` or `Failed` state. This is great for any case where you want to execute code without involvement of the Prefect API. Both flows and tasks include `on_completion` and `on_failure` options where a list of callable hooks can be provided. The callable will receive three arguments: + - `flow`, `flow_run`, and `state` in the case of a flow hook - `task`, `task_run`, and `state` in the case of a task hook @@ -2908,47 +3236,52 @@ if __name__ == "__main__": ``` ### Enhancements -- Update `quote` handling in input resolution to skip descending into the quoted expression — https://github.com/PrefectHQ/prefect/pull/8576 -- Add light and dark mode color and contrast enhancements to UI — https://github.com/PrefectHQ/prefect/pull/8629 + +- Update `quote` handling in input resolution to skip descending into the quoted expression — +- Add light and dark mode color and contrast enhancements to UI — ### Fixes -- Fix `Task.map` type hint for type-checker compatibility with async tasks — https://github.com/PrefectHQ/prefect/pull/8607 -- Update Docker container name sanitization to handle "ce" and "ee" when checking Docker version — https://github.com/PrefectHQ/prefect/pull/8588 -- Fix Kubernetes Job watch timeout behavior when streaming logs — https://github.com/PrefectHQ/prefect/pull/8618 -- Fix date range filter selection on the flow runs UI page — https://github.com/PrefectHQ/prefect/pull/8616 -- Fix Kubernetes not streaming logs when using multiple containers in Job — https://github.com/PrefectHQ/prefect/pull/8430 + +- Fix `Task.map` type hint for type-checker compatibility with async tasks — +- Update Docker container name sanitization to handle "ce" and "ee" when checking Docker version — +- Fix Kubernetes Job watch timeout behavior when streaming logs — +- Fix date range filter selection on the flow runs UI page — +- Fix Kubernetes not streaming logs when using multiple containers in Job — ### Experimental -- Update worker variable typing for clearer display in the UI — https://github.com/PrefectHQ/prefect/pull/8613 -- Update `BaseWorker` to ignore flow runs with associated storage block — https://github.com/PrefectHQ/prefect/pull/8619 -- Add experimental API for artifacts — https://github.com/PrefectHQ/prefect/pull/8404 + +- Update worker variable typing for clearer display in the UI — +- Update `BaseWorker` to ignore flow runs with associated storage block — +- Add experimental API for artifacts — ### Documentation -- Add documentation for resuming a flow run via the UI — https://github.com/PrefectHQ/prefect/pull/8621 -- Add [`prefect-sifflet`](https://siffletapp.github.io/prefect-sifflet/) to Collections catalog — https://github.com/PrefectHQ/prefect/pull/8599 +- Add documentation for resuming a flow run via the UI — +- Add [`prefect-sifflet`](https://siffletapp.github.io/prefect-sifflet/) to Collections catalog — ### Contributors -- @jefflaporte made their first contribution in https://github.com/PrefectHQ/prefect/pull/8430 -- @AzemaBaptiste made their first contribution in https://github.com/PrefectHQ/prefect/pull/8599 + +- @jefflaporte made their first contribution in +- @AzemaBaptiste made their first contribution in - @darrida -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.2...2.8.3 +**All changes**: ## Release 2.8.2 ### Fixes -- Re-enable plugin loading in `prefect` module init — https://github.com/PrefectHQ/prefect/pull/8569 + +- Re-enable plugin loading in `prefect` module init — ### Documentation -- Fix logging format override example — https://github.com/PrefectHQ/prefect/pull/8565 +- Fix logging format override example — ### Experimental -- Add events client to `PrefectClient` — https://github.com/PrefectHQ/prefect/pull/8546 +- Add events client to `PrefectClient` — -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.1...2.8.2 +**All changes**: ## Release 2.8.1 @@ -2962,27 +3295,31 @@ See the [deprecated section](https://github.com/PrefectHQ/prefect/blob/main/RELE Note: Many settings have been renamed but your old settings will be respected. To automatically convert all of the settings in your current profile to the new names, run the `prefect config validate` command. - ### Enhancements -- Add `MattermostWebhook` notification block — https://github.com/PrefectHQ/prefect/pull/8341 -- Add ability to pass in RRule string to `--rrule` option in `prefect set-schedule` command — https://github.com/PrefectHQ/prefect/pull/8543 + +- Add `MattermostWebhook` notification block — +- Add ability to pass in RRule string to `--rrule` option in `prefect set-schedule` command — ### Fixes -- Fix default deployment parameters not populating in the UI — https://github.com/PrefectHQ/prefect/pull/8518 -- Fix ability to use anchor date when setting an interval schedule with the `prefect set-schedule` command — https://github.com/PrefectHQ/prefect/pull/8524 + +- Fix default deployment parameters not populating in the UI — +- Fix ability to use anchor date when setting an interval schedule with the `prefect set-schedule` command — ### Documentation -- Add table listing available blocks — https://github.com/PrefectHQ/prefect/pull/8443 -- Fix work pools documentation links — https://github.com/PrefectHQ/prefect/pull/8477 -- Add examples for custom automation triggers — https://github.com/PrefectHQ/prefect/pull/8476 -- Add webhooks to Automations docs — https://github.com/PrefectHQ/prefect/pull/8514 -- Document Prefect Cloud API rate limits — https://github.com/PrefectHQ/prefect/pull/8529 + +- Add table listing available blocks — +- Fix work pools documentation links — +- Add examples for custom automation triggers — +- Add webhooks to Automations docs — +- Document Prefect Cloud API rate limits — ### Experimental -- Add metadata fields to `BaseWorker` — https://github.com/PrefectHQ/prefect/pull/8527 -- Add default artifact metadata to `LiteralResults` and `PersistedResults` — https://github.com/PrefectHQ/prefect/pull/8501 + +- Add metadata fields to `BaseWorker` — +- Add default artifact metadata to `LiteralResults` and `PersistedResults` — ### Deprecated + - Default SQLite database name changed from `orion.db` to `prefect.db` - Logger `prefect.orion` renamed to `prefect.server` - Constant `ORION_API_VERSION` renamed to `SERVER_API_VERSION` @@ -3035,12 +3372,12 @@ Note: Many settings have been renamed but your old settings will be respected. T — `PREFECT_ORION_TASK_CACHE_KEY_MAX_LENGTH` → `PREFECT_API_TASK_CACHE_KEY_MAX_LENGTH` — `PREFECT_ORION_SERVICES_CANCELLATION_CLEANUP_ENABLED` → `PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED` - ### Contributors -- @qheuristics made their first contribution in https://github.com/PrefectHQ/prefect/pull/8478 -- @KernelErr made their first contribution in https://github.com/PrefectHQ/prefect/pull/8485 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.8.0...2.8.1 +- @qheuristics made their first contribution in +- @KernelErr made their first contribution in + +**All changes**: ## Release 2.8.0 @@ -3091,30 +3428,34 @@ prefect agent start --pool my-pool --queue high-priority To learn more about work pools, check out the [docs](https://docs.prefect.io/concepts/work-pools/) or see the relevant pull requests: ### Enhancements -- Add ability to filter on work pool and queue when querying flow runs — https://github.com/PrefectHQ/prefect/pull/8459 -- Ensure agent respects work queue priority — https://github.com/PrefectHQ/prefect/pull/8458 -- Add ability to create a flow run from the UI with parameters from a previous run — https://github.com/PrefectHQ/prefect/pull/8405 -- Add generic `Webhook` block — https://github.com/PrefectHQ/prefect/pull/8401 -- Add override customizations functionality to deployments via CLI — https://github.com/PrefectHQ/prefect/pull/8349 -- Add ability to reset concurrency limits in CLI to purge existing runs from taking concurrency slots — https://github.com/PrefectHQ/prefect/pull/8408 -- Ensure matching flow run state information in UI — https://github.com/PrefectHQ/prefect/pull/8441 -- Customize CLI block registration experience based on `PREFECT_UI_URL` — https://github.com/PrefectHQ/prefect/pull/8438 + +- Add ability to filter on work pool and queue when querying flow runs — +- Ensure agent respects work queue priority — +- Add ability to create a flow run from the UI with parameters from a previous run — +- Add generic `Webhook` block — +- Add override customizations functionality to deployments via CLI — +- Add ability to reset concurrency limits in CLI to purge existing runs from taking concurrency slots — +- Ensure matching flow run state information in UI — +- Customize CLI block registration experience based on `PREFECT_UI_URL` — ### Fixes -- Fix `prefect dev start` command — https://github.com/PrefectHQ/prefect/pull/8176 -- Fix display of long log messages when in the UI — https://github.com/PrefectHQ/prefect/pull/8449 -- Update `get_run_logger` to accommodate returning `logging.LoggerAdapter` — https://github.com/PrefectHQ/prefect/pull/8422 -- Restore Prefect wrapper around HTTP errors for nicer error messages — https://github.com/PrefectHQ/prefect/pull/8391 -- Fix display of work pool flow run filter in the UI — https://github.com/PrefectHQ/prefect/pull/8453 + +- Fix `prefect dev start` command — +- Fix display of long log messages when in the UI — +- Update `get_run_logger` to accommodate returning `logging.LoggerAdapter` — +- Restore Prefect wrapper around HTTP errors for nicer error messages — +- Fix display of work pool flow run filter in the UI — ### Documentation -- Update Infrastructure concept documentation with `extra-pip-package` example and updated `deployment.yaml` — https://github.com/PrefectHQ/prefect/pull/8465 -- Add work pools documentation — https://github.com/PrefectHQ/prefect/pull/8377 + +- Update Infrastructure concept documentation with `extra-pip-package` example and updated `deployment.yaml` — +- Add work pools documentation — ### Contributors + - @carderne -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.12...2.8.0 +**All changes**: ## Release 2.7.12 @@ -3154,25 +3495,27 @@ def my_flow(name: str, date: datetime): my_flow() ``` -See [the docs](https://docs.prefect.io/tutorials/tasks/#basic-flow-configuration) or https://github.com/PrefectHQ/prefect/pull/8378 for more details. +See [the docs](https://docs.prefect.io/tutorials/tasks/#basic-flow-configuration) or for more details. ### Enhancements -- Update the deployment page to show the runs tab before the description — https://github.com/PrefectHQ/prefect/pull/8398 + +- Update the deployment page to show the runs tab before the description — ### Fixes -- Fix artifact migration to only include states that have non-null data — https://github.com/PrefectHQ/prefect/pull/8420 -- Fix error when using `prefect work-queue ls` without enabling work pools — https://github.com/PrefectHQ/prefect/pull/8427 + +- Fix artifact migration to only include states that have non-null data — +- Fix error when using `prefect work-queue ls` without enabling work pools — ### Experimental -- Add error when attempting to apply a deployment to a work pool that hasn't been created yet — https://github.com/PrefectHQ/prefect/pull/8413 -- Create queues in the correct work pool when applying a deployment for a queue that hasn't been created yet — https://github.com/PrefectHQ/prefect/pull/8413 -### Contributors -- @NodeJSmith +- Add error when attempting to apply a deployment to a work pool that hasn't been created yet — +- Create queues in the correct work pool when applying a deployment for a queue that hasn't been created yet — -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.11...2.7.12 +### Contributors +- @NodeJSmith +**All changes**: ## Release 2.7.11 @@ -3216,48 +3559,54 @@ Updated profile 'default'. ``` ### Enhancements -- Update default task run name to exclude hash of task key — https://github.com/PrefectHQ/prefect/pull/8292 -- Update Docker images to update preinstalled packages on build — https://github.com/PrefectHQ/prefect/pull/8288 -- Add PREFECT_LOGGING_TO_API_WHEN_MISSING_FLOW to allow loggers to be used outside of flows — https://github.com/PrefectHQ/prefect/pull/8311 -- Display Runs before Deployments on flow pages — https://github.com/PrefectHQ/prefect/pull/8386 -- Clarify output CLI message when switching profiles — https://github.com/PrefectHQ/prefect/pull/8383 + +- Update default task run name to exclude hash of task key — +- Update Docker images to update preinstalled packages on build — +- Add PREFECT_LOGGING_TO_API_WHEN_MISSING_FLOW to allow loggers to be used outside of flows — +- Display Runs before Deployments on flow pages — +- Clarify output CLI message when switching profiles — ### Fixes -- Fix bug preventing agents from properly updating Cancelling runs to a Cancelled state — https://github.com/PrefectHQ/prefect/pull/8315 -- Fix bug where Kubernetes job monitoring exited early when no timeout was given — https://github.com/PrefectHQ/prefect/pull/8350 + +- Fix bug preventing agents from properly updating Cancelling runs to a Cancelled state — +- Fix bug where Kubernetes job monitoring exited early when no timeout was given — ### Experimental + - We're working on work pools, groups of work queues. Together, work pools & queues give you greater flexibility and control in organizing and prioritizing work. - — Add updates to work queue `last_polled` time when polling work pools — https://github.com/PrefectHQ/prefect/pull/8338 - — Add CLI support for work pools — https://github.com/PrefectHQ/prefect/pull/8259 - — Add fields to `work_queue` table to accommodate work pools — https://github.com/PrefectHQ/prefect/pull/8264 - — Add work queue data migration — https://github.com/PrefectHQ/prefect/pull/8327 - — Fix default value for priority on `WorkQueue` core schema — https://github.com/PrefectHQ/prefect/pull/8373 -- Add ability to exclude experimental fields in API calls — https://github.com/PrefectHQ/prefect/pull/8274, https://github.com/PrefectHQ/prefect/pull/8331 -- Add Prefect Cloud Events schema and clients — https://github.com/PrefectHQ/prefect/pull/8357 + — Add updates to work queue `last_polled` time when polling work pools — + — Add CLI support for work pools — + — Add fields to `work_queue` table to accommodate work pools — + — Add work queue data migration — + — Fix default value for priority on `WorkQueue` core schema — +- Add ability to exclude experimental fields in API calls — , +- Add Prefect Cloud Events schema and clients — ### Documentation -- Add git commands to Prefect Recipes contribution page — https://github.com/PrefectHQ/prefect/pull/8283 -- Add `retry_delay_seconds` and `exponential_backoff` examples to Tasks retries documentation — https://github.com/PrefectHQ/prefect/pull/8280 -- Add role permissions regarding block secrets — https://github.com/PrefectHQ/prefect/pull/8309 -- Add getting started tutorial video to Prefect Cloud Quickstart — https://github.com/PrefectHQ/prefect/pull/8336 -- Add tips for re-registering blocks from Prefect Collections — https://github.com/PrefectHQ/prefect/pull/8333 -- Improve examples for Kubernetes infrastructure overrides — https://github.com/PrefectHQ/prefect/pull/8312 -- Add mention of reverse proxy for `PREFECT_API_URL` config — https://github.com/PrefectHQ/prefect/pull/8240 -- Fix unused Cloud Getting Started page — https://github.com/PrefectHQ/prefect/pull/8291 -- Fix Prefect Cloud typo in FAQ — https://github.com/PrefectHQ/prefect/pull/8317 + +- Add git commands to Prefect Recipes contribution page — +- Add `retry_delay_seconds` and `exponential_backoff` examples to Tasks retries documentation — +- Add role permissions regarding block secrets — +- Add getting started tutorial video to Prefect Cloud Quickstart — +- Add tips for re-registering blocks from Prefect Collections — +- Improve examples for Kubernetes infrastructure overrides — +- Add mention of reverse proxy for `PREFECT_API_URL` config — +- Fix unused Cloud Getting Started page — +- Fix Prefect Cloud typo in FAQ — ### Collections -- Add `ShellOperation` implementing `JobBlock` in `v0.1.4` release of `prefect-shell` — https://github.com/PrefectHQ/prefect-shell/pull/55 -- Add `CensusSync` implementing `JobBlock` in `v0.1.1` release of `prefect-census` — https://github.com/PrefectHQ/prefect-census/pull/15 + +- Add `ShellOperation` implementing `JobBlock` in `v0.1.4` release of `prefect-shell` — +- Add `CensusSync` implementing `JobBlock` in `v0.1.1` release of `prefect-census` — ### Contributors + - @chiaberry - @hozn - @manic-miner - @space-age-pete -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.10...2.7.11 +**All changes**: ## Release 2.7.10 @@ -3271,8 +3620,7 @@ We improved our handling of runs that are in the process of cancelling. When a r We added cleanup of tasks and subflows belonging to cancelled flow runs. Previously, these tasks and subflows could be left in a "running" state. This can cause problems with concurrency slot consumption and restarts, so we've added a service that updates the states of the children of recently cancelled flow runs. -See https://github.com/PrefectHQ/prefect/pull/8126 for implementation details. - +See for implementation details. ### Multiarchitecture Docker builds @@ -3281,7 +3629,7 @@ In 2.7.8, we announced that we were publishing development Docker images, includ You can try one of the new images by including the `--platform` specifier, e.g.: ```bash -$ docker run --platform linux/arm64 --pull always prefecthq/prefect:2-latest prefect version +docker run --platform linux/arm64 --pull always prefecthq/prefect:2-latest prefect version ``` We will be publishing images for the following architectures: @@ -3292,41 +3640,47 @@ We will be publishing images for the following architectures: This should provide a significant speedup to anyone running containers on ARM64 machines (I'm looking at you, Apple M1 chips!) and reduce the complexity for our users that are deploying on different platforms. The workflow for building our images was rewritten from scratch, and it'll be easy for us to expand support to include other common platforms. Shoutout to [@ddelange](https://github.com/ddelange) who led implementation of the feature. -See https://github.com/PrefectHQ/prefect/pull/7902 for details. +See for details. ### Enhancements -- Add [`is_schedule_active` option](https://docs.prefect.io/api-ref/prefect/deployments/#prefect.deployments.Deployment) to `Deployment` class to allow control of automatic scheduling — https://github.com/PrefectHQ/prefect/pull/7430 -- Add documentation links to blocks in UI — https://github.com/PrefectHQ/prefect/pull/8210 -- Add Kubernetes kube-system permissions to Prefect agent template for retrieving UUID from kube-system namespace — https://github.com/PrefectHQ/prefect/pull/8205 -- Add support for obscuring secrets in nested block fields in the UI — https://github.com/PrefectHQ/prefect/pull/8246 -- Enable publish of multiarchitecture Docker builds on release — https://github.com/PrefectHQ/prefect/pull/7902 -- Add `CANCELLING` state type — https://github.com/PrefectHQ/prefect/pull/7794 -- Add graceful shutdown of engine on `SIGTERM` — https://github.com/PrefectHQ/prefect/pull/7887 -- Add cancellation cleanup service — https://github.com/PrefectHQ/prefect/pull/8093 -- Add `PREFECT_ORION_API_KEEPALIVE_TIMEOUT` setting to allow configuration of Uvicorn `timeout-keep-alive` setting — https://github.com/PrefectHQ/prefect/pull/8190 +- Add [`is_schedule_active` option](https://docs.prefect.io/api-ref/prefect/deployments/#prefect.deployments.Deployment) to `Deployment` class to allow control of automatic scheduling — + +- Add documentation links to blocks in UI — +- Add Kubernetes kube-system permissions to Prefect agent template for retrieving UUID from kube-system namespace — +- Add support for obscuring secrets in nested block fields in the UI — +- Enable publish of multiarchitecture Docker builds on release — +- Add `CANCELLING` state type — +- Add graceful shutdown of engine on `SIGTERM` — +- Add cancellation cleanup service — +- Add `PREFECT_ORION_API_KEEPALIVE_TIMEOUT` setting to allow configuration of Uvicorn `timeout-keep-alive` setting — ### Fixes -- Fix server compatibility with clients on 2.7.8 — https://github.com/PrefectHQ/prefect/pull/8272 -- Fix tracking of long-running Kubernetes jobs and add handling for connection failures — https://github.com/PrefectHQ/prefect/pull/8189 + +- Fix server compatibility with clients on 2.7.8 — +- Fix tracking of long-running Kubernetes jobs and add handling for connection failures — ### Experimental -- Add functionality to specify a work pool when starting an agent — https://github.com/PrefectHQ/prefect/pull/8222 -- Disable `Work Queues` tab view when work pools are enabled — https://github.com/PrefectHQ/prefect/pull/8257 -- Fix property for `WorkersTable` in UI — https://github.com/PrefectHQ/prefect/pull/8232 + +- Add functionality to specify a work pool when starting an agent — +- Disable `Work Queues` tab view when work pools are enabled — +- Fix property for `WorkersTable` in UI — ### Documentation -- [Add Prefect Cloud Quickstart tutorial](https://docs.prefect.io/ui/cloud-getting-started/) — https://github.com/PrefectHQ/prefect/pull/8227 -- Add `project_urls` to `setup.py` — https://github.com/PrefectHQ/prefect/pull/8224 -- Add configuration to `mkdocs.yml` to enable versioning at a future time — https://github.com/PrefectHQ/prefect/pull/8204 -- Improve [contributing documentation](https://docs.prefect.io/contributing/overview/) with venv instructions — https://github.com/PrefectHQ/prefect/pull/8247 -- Update documentation on [KubernetesJob options](https://docs.prefect.io/concepts/infrastructure/#kubernetesjob) — https://github.com/PrefectHQ/prefect/pull/8261 -- Update documentation on [workspace-level roles](https://docs.prefect.io/ui/roles/#workspace-level-roles) — https://github.com/PrefectHQ/prefect/pull/8263 + +- [Add Prefect Cloud Quickstart tutorial](https://docs.prefect.io/ui/cloud-getting-started/) — +- Add `project_urls` to `setup.py` — +- Add configuration to `mkdocs.yml` to enable versioning at a future time — +- Improve [contributing documentation](https://docs.prefect.io/contributing/overview/) with venv instructions — +- Update documentation on [KubernetesJob options](https://docs.prefect.io/concepts/infrastructure/#kubernetesjob) — +- Update documentation on [workspace-level roles](https://docs.prefect.io/ui/roles/#workspace-level-roles) — ### Collections -- Add [prefect-openai](https://prefecthq.github.io/prefect-openai/) to [Collections catalog](https://docs.prefect.io/collections/catalog/) — https://github.com/PrefectHQ/prefect/pull/8236 + +- Add [prefect-openai](https://prefecthq.github.io/prefect-openai/) to [Collections catalog](https://docs.prefect.io/collections/catalog/) — ### Contributors + - @ddelange - @imsurat - @Laerte @@ -3334,30 +3688,36 @@ See https://github.com/PrefectHQ/prefect/pull/7902 for details. ## Release 2.7.9 ### Enhancements -- Add `--head` flag to `flow-run logs` CLI command to limit the number of logs returned — https://github.com/PrefectHQ/prefect/pull/8003 -- Add `--num_logs` option to `flow-run logs` CLI command to specify the number of logs returned — https://github.com/PrefectHQ/prefect/pull/8003 -- Add option to filter out `.git` files when reading files with the GitHub storage block — https://github.com/PrefectHQ/prefect/pull/8193 + +- Add `--head` flag to `flow-run logs` CLI command to limit the number of logs returned — +- Add `--num_logs` option to `flow-run logs` CLI command to specify the number of logs returned — +- Add option to filter out `.git` files when reading files with the GitHub storage block — ### Fixes -- Fix bug causing failures when spawning Windows subprocesses — https://github.com/PrefectHQ/prefect/pull/8184 -- Fix possible recursive loop when blocks label themselves as both their own parent and reference — https://github.com/PrefectHQ/prefect/pull/8197 + +- Fix bug causing failures when spawning Windows subprocesses — +- Fix possible recursive loop when blocks label themselves as both their own parent and reference — ### Documentation -- Add [recipe contribution page](https://docs.prefect.io/recipes/recipes/#contributing-recipes) and [AWS Chalice](https://docs.prefect.io/recipes/recipes/#recipe-catalog) recipe — https://github.com/PrefectHQ/prefect/pull/8183 -- Add new `discourse` and `blog` admonition types — https://github.com/PrefectHQ/prefect/pull/8202 -- Update Automations and Notifications documentation — https://github.com/PrefectHQ/prefect/pull/8140 -- Fix minor API docstring formatting issues — https://github.com/PrefectHQ/prefect/pull/8196 + +- Add [recipe contribution page](https://docs.prefect.io/recipes/recipes/#contributing-recipes) and [AWS Chalice](https://docs.prefect.io/recipes/recipes/#recipe-catalog) recipe — +- Add new `discourse` and `blog` admonition types — +- Update Automations and Notifications documentation — +- Fix minor API docstring formatting issues — ### Collections + - [`prefect-openai` 0.1.0](https://github.com/PrefectHQ/prefect-openai) newly released with support for authentication and completions ### Experimental -- Add ability for deployment create and deployment update to create work pool queues — https://github.com/PrefectHQ/prefect/pull/8129 + +- Add ability for deployment create and deployment update to create work pool queues — ## New Contributors -* @mj0nez made their first contribution in https://github.com/PrefectHQ/prefect/pull/8201 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.8...2.7.9 +- @mj0nez made their first contribution in + +**All changes**: ## Release 2.7.8 @@ -3371,36 +3731,42 @@ This feature is currently in beta and we have lots of improvements planned in th ![The timeline view visualizes execution of your flow run over time](https://user-images.githubusercontent.com/6200442/212138540-78586356-89bc-4401-a700-b80b15a17020.png) ### Enhancements -- Add [task option `refresh_cache`](https://docs.prefect.io/concepts/tasks/#refreshing-the-cache) to update the cached data for a task run — https://github.com/PrefectHQ/prefect/pull/7856 -- Add logs when a task run receives an abort signal and is in a non-final state — https://github.com/PrefectHQ/prefect/pull/8097 -- Add [publishing of multiarchitecture Docker images](https://hub.docker.com/r/prefecthq/prefect-dev) for development builds — https://github.com/PrefectHQ/prefect/pull/7900 -- Add `httpx.WriteError` to client retryable exceptions — https://github.com/PrefectHQ/prefect/pull/8145 -- Add support for memory limits and privileged containers to `DockerContainer` — https://github.com/PrefectHQ/prefect/pull/8033 + +- Add [task option `refresh_cache`](https://docs.prefect.io/concepts/tasks/#refreshing-the-cache) to update the cached data for a task run — +- Add logs when a task run receives an abort signal and is in a non-final state — +- Add [publishing of multiarchitecture Docker images](https://hub.docker.com/r/prefecthq/prefect-dev) for development builds — +- Add `httpx.WriteError` to client retryable exceptions — +- Add support for memory limits and privileged containers to `DockerContainer` — ### Fixes -- Add support for `allow_failure` to mapped task arguments — https://github.com/PrefectHQ/prefect/pull/8135 -- Update conda requirement regex to support channel and build hashes — https://github.com/PrefectHQ/prefect/pull/8137 -- Add numpy array support to orjson serialization — https://github.com/PrefectHQ/prefect/pull/7912 + +- Add support for `allow_failure` to mapped task arguments — +- Update conda requirement regex to support channel and build hashes — +- Add numpy array support to orjson serialization — ### Experimental -- Rename "Worker pools" to "Work pools" — https://github.com/PrefectHQ/prefect/pull/8107 -- Rename default work pool queue — https://github.com/PrefectHQ/prefect/pull/8117 -- Add worker configuration — https://github.com/PrefectHQ/prefect/pull/8100 -- Add `BaseWorker` and `ProcessWorker` — https://github.com/PrefectHQ/prefect/pull/7996 + +- Rename "Worker pools" to "Work pools" — +- Rename default work pool queue — +- Add worker configuration — +- Add `BaseWorker` and `ProcessWorker` — ### Documentation -- Add YouTube video to welcome page — https://github.com/PrefectHQ/prefect/pull/8090 -- Add social links — https://github.com/PrefectHQ/prefect/pull/8088 -- Increase visibility of Prefect Cloud and Orion REST API documentation — https://github.com/PrefectHQ/prefect/pull/8134 + +- Add YouTube video to welcome page — +- Add social links — +- Increase visibility of Prefect Cloud and Orion REST API documentation — ## New Contributors -* @muddi900 made their first contribution in https://github.com/PrefectHQ/prefect/pull/8101 -* @ddelange made their first contribution in https://github.com/PrefectHQ/prefect/pull/7900 -* @toro-berlin made their first contribution in https://github.com/PrefectHQ/prefect/pull/7856 -* @Ewande made their first contribution in https://github.com/PrefectHQ/prefect/pull/7912 -* @brandonreid made their first contribution in https://github.com/PrefectHQ/prefect/pull/8153 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.7...2.7.8 +- @muddi900 made their first contribution in + +- @ddelange made their first contribution in +- @toro-berlin made their first contribution in +- @Ewande made their first contribution in +- @brandonreid made their first contribution in + +**All changes**: ## Release 2.7.7 @@ -3417,61 +3783,67 @@ Note: you can also view the REST API documentation [embedded in our open source We've also improved the parsing and rendering of reference documentation for our Python API. See the [@flow decorator reference](https://docs.prefect.io/api-ref/prefect/flows/#prefect.flows.flow) for example. ### Enhancements -- Add link to blocks catalog after registering blocks in CLI — https://github.com/PrefectHQ/prefect/pull/8017 -- Add schema migration of block documents during `Block.save` — https://github.com/PrefectHQ/prefect/pull/8056 -- Update result factory creation to avoid creating an extra client instance — https://github.com/PrefectHQ/prefect/pull/8072 -- Add logs for deployment flow code loading — https://github.com/PrefectHQ/prefect/pull/8075 -- Update `visit_collection` to support annotations e.g. `allow_failure` — https://github.com/PrefectHQ/prefect/pull/7263 -- Update annotations to inherit from `namedtuple` for serialization support in Dask — https://github.com/PrefectHQ/prefect/pull/8037 -- Add `PREFECT_API_TLS_INSECURE_SKIP_VERIFY` setting to disable client SSL verification — https://github.com/PrefectHQ/prefect/pull/7850 -- Update OpenAPI schema for flow parameters to include positions for display — https://github.com/PrefectHQ/prefect/pull/8013 -- Add parsing of flow docstrings to populate parameter descriptions in the OpenAPI schema — https://github.com/PrefectHQ/prefect/pull/8004 -- Add `validate` to `Block.load` allowing validation to be disabled — https://github.com/PrefectHQ/prefect/pull/7862 -- Improve error message when saving a block with an invalid name — https://github.com/PrefectHQ/prefect/pull/8038 -- Add limit to task run cache key size — https://github.com/PrefectHQ/prefect/pull/7275 -- Add limit to RRule length — https://github.com/PrefectHQ/prefect/pull/7762 -- Add flow run history inside the date range picker — https://github.com/PrefectHQ/orion-design/issues/994 + +- Add link to blocks catalog after registering blocks in CLI — +- Add schema migration of block documents during `Block.save` — +- Update result factory creation to avoid creating an extra client instance — +- Add logs for deployment flow code loading — +- Update `visit_collection` to support annotations e.g. `allow_failure` — +- Update annotations to inherit from `namedtuple` for serialization support in Dask — +- Add `PREFECT_API_TLS_INSECURE_SKIP_VERIFY` setting to disable client SSL verification — +- Update OpenAPI schema for flow parameters to include positions for display — +- Add parsing of flow docstrings to populate parameter descriptions in the OpenAPI schema — +- Add `validate` to `Block.load` allowing validation to be disabled — +- Improve error message when saving a block with an invalid name — +- Add limit to task run cache key size — +- Add limit to RRule length — +- Add flow run history inside the date range picker — ### Fixes -- Fix bug where flow timeouts started before waiting for upstreams — https://github.com/PrefectHQ/prefect/pull/7993 -- Fix captured Kubernetes error type in `get_job` — https://github.com/PrefectHQ/prefect/pull/8018 -- Fix `prefect cloud login` error when no workspaces exist — https://github.com/PrefectHQ/prefect/pull/8034 -- Fix serialization of `SecretDict` when used in deployments — https://github.com/PrefectHQ/prefect/pull/8074 -- Fix bug where `visit_collection` could fail when accessing extra Pydantic fields — https://github.com/PrefectHQ/prefect/pull/8083 + +- Fix bug where flow timeouts started before waiting for upstreams — +- Fix captured Kubernetes error type in `get_job` — +- Fix `prefect cloud login` error when no workspaces exist — +- Fix serialization of `SecretDict` when used in deployments — +- Fix bug where `visit_collection` could fail when accessing extra Pydantic fields — ### Experimental -- Add pages and routers for workers — https://github.com/PrefectHQ/prefect/pull/7973 + +- Add pages and routers for workers — ### Documentation -- Update API reference documentation to use new parser and renderer — https://github.com/PrefectHQ/prefect/pull/7855 -- Add new REST API reference using Redoc — https://github.com/PrefectHQ/prefect/pull/7503 + +- Update API reference documentation to use new parser and renderer — +- Add new REST API reference using Redoc — ### Collections + - [`prefect-aws` 0.2.2](https://github.com/PrefectHQ/prefect-aws/releases/tag/v0.2.2) released with many improvements to `S3Bucket` ### Contributors -* @j-tr made their first contribution in https://github.com/PrefectHQ/prefect/pull/8013 -* @toby-coleman made their first contribution in https://github.com/PrefectHQ/prefect/pull/8083 -* @riquelmev made their first contribution in https://github.com/PrefectHQ/prefect/pull/7768 -* @joelluijmes +- @j-tr made their first contribution in -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.5...2.7.7 +- @toby-coleman made their first contribution in +- @riquelmev made their first contribution in +- @joelluijmes +**All changes**: ## Release 2.7.6 This release fixes a critical bug in the SQLite database migrations in 2.7.4 and 2.7.5. -See https://github.com/PrefectHQ/prefect/issues/8058 for details. +See for details. -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.5...2.7.6 +**All changes**: ## Release 2.7.5 ### Schedule flow runs and read logs from the CLI You can now specify either `--start-in` or `--start-at` when running deployments from the CLI. + ``` ❯ prefect deployment run foo/test --start-at "3pm tomorrow" Creating flow run for deployment 'foo/test'... @@ -3483,6 +3855,7 @@ Created flow run 'pompous-porpoise'. ``` You can also get the logs for a flow run using `prefect flow-run logs ` + ``` ❯ prefect flow-run logs 7aec7a60-a0ab-4f3e-9f2a-479cd85a2aaf 2022-12-29 20:00:40.651 | INFO | Flow run 'optimal-pegasus' — meow @@ -3492,29 +3865,34 @@ You can also get the logs for a flow run using `prefect flow-run logs +- Add `flow-run logs` to get logs using the CLI — ### Documentation -- Fix task annotation in task runner docs — https://github.com/PrefectHQ/prefect/pull/7977 -- Add instructions for building custom blocks — https://github.com/PrefectHQ/prefect/pull/7979 + +- Fix task annotation in task runner docs — +- Add instructions for building custom blocks — ### Collections + - Added `BigQueryWarehouse` block in `prefect-gcp` v0.2.1 - Added `AirbyteConnection` block in `prefect-airbyte` v0.2.0 - Added dbt Cloud metadata API client to `DbtCloudCredentials` in `prefect-dbt` v0.2.7 ### Experimental -- Fix read worker pool queue endpoint — https://github.com/PrefectHQ/prefect/pull/7995 -- Fix error in worker pool queue endpoint — https://github.com/PrefectHQ/prefect/pull/7997 -- Add filtering to flow runs by worker pool and worker pool queue attributes — https://github.com/PrefectHQ/prefect/pull/8006 + +- Fix read worker pool queue endpoint — +- Fix error in worker pool queue endpoint — +- Add filtering to flow runs by worker pool and worker pool queue attributes — ### Contributors -* @ohadch made their first contribution in https://github.com/PrefectHQ/prefect/pull/7982 -* @mohitsaxenaknoldus made their first contribution in https://github.com/PrefectHQ/prefect/pull/7980 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.4...2.7.5 +- @ohadch made their first contribution in +- @mohitsaxenaknoldus made their first contribution in + +**All changes**: ## Release 2.7.4 @@ -3565,41 +3943,49 @@ def flaky_function(): return 42 ``` -See https://github.com/PrefectHQ/prefect/pull/7961 for implementation details. +See for implementation details. ### Enhancements -- Add task run names to the `/graph` API route — https://github.com/PrefectHQ/prefect/pull/7951 -- Add vcs directories `.git` and `.hg` (mercurial) to default `.prefectignore` — https://github.com/PrefectHQ/prefect/pull/7919 -- Increase the default thread limit from 40 to 250 — https://github.com/PrefectHQ/prefect/pull/7961 + +- Add task run names to the `/graph` API route — +- Add vcs directories `.git` and `.hg` (mercurial) to default `.prefectignore` — +- Increase the default thread limit from 40 to 250 — ### Deprecations -- Add removal date to tag-based work queue deprecation messages — https://github.com/PrefectHQ/prefect/pull/7930 + +- Add removal date to tag-based work queue deprecation messages — ### Documentation -- Fix `prefect deployment` command listing — https://github.com/PrefectHQ/prefect/pull/7949 -- Add workspace transfer documentation — https://github.com/PrefectHQ/prefect/pull/7941 -- Fix docstring examples in `PrefectFuture` — https://github.com/PrefectHQ/prefect/pull/7877 -- Update `setup.py` metadata to link to correct repo — https://github.com/PrefectHQ/prefect/pull/7933 + +- Fix `prefect deployment` command listing — +- Add workspace transfer documentation — +- Fix docstring examples in `PrefectFuture` — +- Update `setup.py` metadata to link to correct repo — ### Experimental -- Add experimental workers API routes — https://github.com/PrefectHQ/prefect/pull/7896 + +- Add experimental workers API routes — ### Collections + - New [`prefect-google-sheets` collection](https://stefanocascavilla.github.io/prefect-google-sheets/) ### Contributors -* @devanshdoshi9 made their first contribution in https://github.com/PrefectHQ/prefect/pull/7949 -* @stefanocascavilla made their first contribution in https://github.com/PrefectHQ/prefect/pull/7960 -* @quassy made their first contribution in https://github.com/PrefectHQ/prefect/pull/7919 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.3...2.7.4 +- @devanshdoshi9 made their first contribution in + +- @stefanocascavilla made their first contribution in +- @quassy made their first contribution in + +**All changes**: ## Release 2.7.3 ### Fixes -- Fix bug where flows with names that do not match the function name could not be loaded — https://github.com/PrefectHQ/prefect/pull/7920 -- Fix type annotation for `KubernetesJob.job_watch_timeout_seconds` — https://github.com/PrefectHQ/prefect/pull/7914 -- Keep data from being lost when assigning a generator to `State.data` — https://github.com/PrefectHQ/prefect/pull/7714 + +- Fix bug where flows with names that do not match the function name could not be loaded — +- Fix type annotation for `KubernetesJob.job_watch_timeout_seconds` — +- Keep data from being lost when assigning a generator to `State.data` — ## Release 2.7.2 @@ -3649,72 +4035,82 @@ When viewing task run concurrency in the UI, each limit has its own page. Includ image - ### Enhancements -- Improve Prefect import time by deferring imports — https://github.com/PrefectHQ/prefect/pull/7836 -- Add Opsgenie notification block — https://github.com/PrefectHQ/prefect/pull/7778 -- Add individual concurrency limit page with active runs list — https://github.com/PrefectHQ/prefect/pull/7848 -- Add `PREFECT_KUBERNETES_CLUSTER_UID` to allow bypass of `kube-system` namespace read — https://github.com/PrefectHQ/prefect/pull/7864 -- Refactor `pause_flow_run` for consistency with engine state handling — https://github.com/PrefectHQ/prefect/pull/7857 -- API: Allow `reject_transition` to return current state — https://github.com/PrefectHQ/prefect/pull/7830 -- Add `SecretDict` block field that obfuscates nested values in a dictionary — https://github.com/PrefectHQ/prefect/pull/7885 + +- Improve Prefect import time by deferring imports — +- Add Opsgenie notification block — +- Add individual concurrency limit page with active runs list — +- Add `PREFECT_KUBERNETES_CLUSTER_UID` to allow bypass of `kube-system` namespace read — +- Refactor `pause_flow_run` for consistency with engine state handling — +- API: Allow `reject_transition` to return current state — +- Add `SecretDict` block field that obfuscates nested values in a dictionary — ### Fixes -- Fix bug where agent concurrency slots may not be released — https://github.com/PrefectHQ/prefect/pull/7845 -- Fix circular imports in the `orchestration` module — https://github.com/PrefectHQ/prefect/pull/7883 -- Fix deployment builds with scripts that contain flow calls — https://github.com/PrefectHQ/prefect/pull/7817 -- Fix path argument behavior in `LocalFileSystem` block — https://github.com/PrefectHQ/prefect/pull/7891 -- Fix flow cancellation in `Process` block on Windows — https://github.com/PrefectHQ/prefect/pull/7799 + +- Fix bug where agent concurrency slots may not be released — +- Fix circular imports in the `orchestration` module — +- Fix deployment builds with scripts that contain flow calls — +- Fix path argument behavior in `LocalFileSystem` block — +- Fix flow cancellation in `Process` block on Windows — ### Documentation -- Add documentation for Automations UI — https://github.com/PrefectHQ/prefect/pull/7833 -- Mention recipes and tutorials under Recipes and Collections pages — https://github.com/PrefectHQ/prefect/pull/7876 -- Add documentation for Task Run Concurrency UI — https://github.com/PrefectHQ/prefect/pull/7840 -- Add `with_options` example to collections usage docs — https://github.com/PrefectHQ/prefect/pull/7894 -- Add a link to orion design and better title to UI readme — https://github.com/PrefectHQ/prefect/pull/7484 + +- Add documentation for Automations UI — +- Mention recipes and tutorials under Recipes and Collections pages — +- Add documentation for Task Run Concurrency UI — +- Add `with_options` example to collections usage docs — +- Add a link to orion design and better title to UI readme — ### Collections -- New [`prefect-kubernetes`](https://prefecthq.github.io/prefect-kubernetes/) collection for [Kubernetes](https://kubernetes.io/) — https://github.com/PrefectHQ/prefect/pull/7907 -- New [`prefect-bitbucket`](https://prefecthq.github.io/prefect-bitbucket/) collection for [Bitbucket](https://bitbucket.org/product) — https://github.com/PrefectHQ/prefect/pull/7907 + +- New [`prefect-kubernetes`](https://prefecthq.github.io/prefect-kubernetes/) collection for [Kubernetes](https://kubernetes.io/) — +- New [`prefect-bitbucket`](https://prefecthq.github.io/prefect-bitbucket/) collection for [Bitbucket](https://bitbucket.org/product) — ## Contributors + - @jlutran -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.7.1...2.7.2 +**All changes**: ## Release 2.7.1 ### Task concurrency limits page + You can now add task concurrency limits in the ui! ![image](https://user-images.githubusercontent.com/6200442/206586749-3f9fff36-5359-41a9-8727-60523cf89071.png) ### Enhancements -- Add extra entrypoints setting for user module injection; allows registration of custom blocks — https://github.com/PrefectHQ/prefect/pull/7179 -- Update orchestration rule to wait for scheduled time to only apply to transition to running — https://github.com/PrefectHQ/prefect/pull/7585 -- Use cluster UID and namespace instead of cluster "name" for `KubernetesJob` identifiers — https://github.com/PrefectHQ/prefect/pull/7747 -- Add a task run concurrency limits page — https://github.com/PrefectHQ/prefect/pull/7779 -- Add setting to toggle interpreting square brackets as style — https://github.com/PrefectHQ/prefect/pull/7810 -- Move `/health` API route to root router — https://github.com/PrefectHQ/prefect/pull/7765 -- Add `PREFECT_API_ENABLE_HTTP2` setting to allow HTTP/2 to be disabled — https://github.com/PrefectHQ/prefect/pull/7802 -- Monitor process after kill and return early when possible — https://github.com/PrefectHQ/prefect/pull/7746 -- Update `KubernetesJob` to watch jobs without timeout by default — https://github.com/PrefectHQ/prefect/pull/7786 -- Bulk deletion of flows, deployments, and work queues from the UI — https://github.com/PrefectHQ/prefect/pull/7824 + +- Add extra entrypoints setting for user module injection; allows registration of custom blocks — +- Update orchestration rule to wait for scheduled time to only apply to transition to running — +- Use cluster UID and namespace instead of cluster "name" for `KubernetesJob` identifiers — +- Add a task run concurrency limits page — +- Add setting to toggle interpreting square brackets as style — +- Move `/health` API route to root router — +- Add `PREFECT_API_ENABLE_HTTP2` setting to allow HTTP/2 to be disabled — +- Monitor process after kill and return early when possible — +- Update `KubernetesJob` to watch jobs without timeout by default — +- Bulk deletion of flows, deployments, and work queues from the UI — ### Fixes -- Add lock to ensure that alembic commands are not run concurrently — https://github.com/PrefectHQ/prefect/pull/7789 -- Release task concurrency slots when transition is rejected as long as the task is not in a running state — https://github.com/PrefectHQ/prefect/pull/7798 -- Fix issue with improperly parsed flow run notification URLs — https://github.com/PrefectHQ/prefect/pull/7173 -- Fix radar not updating without refreshing the page — https://github.com/PrefectHQ/prefect/pull/7824 -- UI: Fullscreen layouts on screens < `lg` should take up all the available space — https://github.com/PrefectHQ/prefect/pull/7792 + +- Add lock to ensure that alembic commands are not run concurrently — +- Release task concurrency slots when transition is rejected as long as the task is not in a running state — +- Fix issue with improperly parsed flow run notification URLs — +- Fix radar not updating without refreshing the page — +- UI: Fullscreen layouts on screens < `lg` should take up all the available space — ### Documentation -- Add documentation for creating a flow run from deployments — https://github.com/PrefectHQ/prefect/pull/7696 -- Move `wait_for` examples to the tasks documentation — https://github.com/PrefectHQ/prefect/pull/7788 + +- Add documentation for creating a flow run from deployments — +- Move `wait_for` examples to the tasks documentation — ## Contributors -* @t-yuki made their first contribution in https://github.com/PrefectHQ/prefect/pull/7741 -* @padbk made their first contribution in https://github.com/PrefectHQ/prefect/pull/7173 + +- @t-yuki made their first contribution in + +- @padbk made their first contribution in ## Release 2.7.0 @@ -3738,19 +4134,19 @@ Note: this process is robust to agent restarts, but does require that an agent i Support for cancellation has been added to all core library infrastructure types: -- Docker Containers (https://github.com/PrefectHQ/prefect/pull/7684) -- Kubernetes Jobs (https://github.com/PrefectHQ/prefect/pull/7701) -- Processes (https://github.com/PrefectHQ/prefect/pull/7635) +- Docker Containers () +- Kubernetes Jobs () +- Processes () Cancellation support is in progress for all collection infrastructure types: -- ECS Tasks (https://github.com/PrefectHQ/prefect-aws/pull/163) -- Google Cloud Run Jobs (https://github.com/PrefectHQ/prefect-gcp/pull/76) -- Azure Container Instances (https://github.com/PrefectHQ/prefect-azure/pull/58) +- ECS Tasks () +- Google Cloud Run Jobs () +- Azure Container Instances () At this time, this feature requires the flow run to be submitted by an agent — flow runs without deployments cannot be cancelled yet, but that feature is [coming soon](https://github.com/PrefectHQ/prefect/pull/7150). -See https://github.com/PrefectHQ/prefect/pull/7637 for more details +See for more details ### Flow run pause and resume @@ -3773,7 +4169,7 @@ A timeout can be supplied to the `pause_flow_run` utility — if the flow run is This blocking style of pause that keeps infrastructure running is supported for all flow runs, including subflow runs. -See https://github.com/PrefectHQ/prefect/pull/7637 for more details. +See for more details. ### Logging of prints in flows and tasks @@ -3810,8 +4206,7 @@ This print statement will appear locally as normal, but won't be sent to the Pre See [the logging documentation](https://docs.prefect.io/concepts/logs/#logging-print-statements) for more details. -See https://github.com/PrefectHQ/prefect/pull/7580 for implementation details. - +See for implementation details. ### Agent flow run concurrency limits @@ -3829,53 +4224,56 @@ This feature is especially useful for limiting resource consumption when running Thanks to @eudyptula for contributing! -See https://github.com/PrefectHQ/prefect/pull/7361 for more details. - +See for more details. ### Enhancements -- Add agent reporting of crashed flow run infrastructure — https://github.com/PrefectHQ/prefect/pull/7670 -- Add Twilio SMS notification block — https://github.com/PrefectHQ/prefect/pull/7685 -- Add PagerDuty Webhook notification block — https://github.com/PrefectHQ/prefect/pull/7534 -- Add jitter to the agent query loop — https://github.com/PrefectHQ/prefect/pull/7652 -- Include final state logs in logs sent to API — https://github.com/PrefectHQ/prefect/pull/7647 -- Add `tags` and `idempotency_key` to `run deployment` — https://github.com/PrefectHQ/prefect/pull/7641 -- The final state of a flow is now `Cancelled` when any task finishes in a `Cancelled` state — https://github.com/PrefectHQ/prefect/pull/7694 -- Update login to prompt for "API key" instead of "authentication key" — https://github.com/PrefectHQ/prefect/pull/7649 -- Disable cache on result retrieval if disabled on creation — https://github.com/PrefectHQ/prefect/pull/7627 -- Raise `CancelledRun` when retrieving a `Cancelled` state's result — https://github.com/PrefectHQ/prefect/pull/7699 -- Use new database session to send each flow run notification — https://github.com/PrefectHQ/prefect/pull/7644 -- Increase default agent query interval to 10s — https://github.com/PrefectHQ/prefect/pull/7703 -- Add default messages to state exceptions — https://github.com/PrefectHQ/prefect/pull/7705 -- Update `run_sync_in_interruptible_worker_thread` to use an event — https://github.com/PrefectHQ/prefect/pull/7704 -- Increase default database query timeout to 10s — https://github.com/PrefectHQ/prefect/pull/7717 + +- Add agent reporting of crashed flow run infrastructure — +- Add Twilio SMS notification block — +- Add PagerDuty Webhook notification block — +- Add jitter to the agent query loop — +- Include final state logs in logs sent to API — +- Add `tags` and `idempotency_key` to `run deployment` — +- The final state of a flow is now `Cancelled` when any task finishes in a `Cancelled` state — +- Update login to prompt for "API key" instead of "authentication key" — +- Disable cache on result retrieval if disabled on creation — +- Raise `CancelledRun` when retrieving a `Cancelled` state's result — +- Use new database session to send each flow run notification — +- Increase default agent query interval to 10s — +- Add default messages to state exceptions — +- Update `run_sync_in_interruptible_worker_thread` to use an event — +- Increase default database query timeout to 10s — ### Fixes -- Prompt workspace selection if API key is set, but API URL is not set — https://github.com/PrefectHQ/prefect/pull/7648 -- Use `PREFECT_UI_URL` for flow run notifications — https://github.com/PrefectHQ/prefect/pull/7698 -- Display all parameter values a flow run was triggered with in the UI (defaults and overrides) — https://github.com/PrefectHQ/prefect/pull/7697 -- Fix bug where result event is missing when wait is called before submission completes — https://github.com/PrefectHQ/prefect/pull/7571 -- Fix support for sync-compatible calls in `deployment build` — https://github.com/PrefectHQ/prefect/pull/7417 -- Fix bug in `StateGroup` that caused `all_final` to be wrong — https://github.com/PrefectHQ/prefect/pull/7678 -- Add retry on specified httpx network errors — https://github.com/PrefectHQ/prefect/pull/7593 -- Fix state display bug when state message is empty — https://github.com/PrefectHQ/prefect/pull/7706 + +- Prompt workspace selection if API key is set, but API URL is not set — +- Use `PREFECT_UI_URL` for flow run notifications — +- Display all parameter values a flow run was triggered with in the UI (defaults and overrides) — +- Fix bug where result event is missing when wait is called before submission completes — +- Fix support for sync-compatible calls in `deployment build` — +- Fix bug in `StateGroup` that caused `all_final` to be wrong — +- Add retry on specified httpx network errors — +- Fix state display bug when state message is empty — ### Documentation -- Fix heading links in docs — https://github.com/PrefectHQ/prefect/pull/7665 -- Update login and `PREFECT_API_URL` configuration notes — https://github.com/PrefectHQ/prefect/pull/7674 -- Add documentation about AWS retries configuration — https://github.com/PrefectHQ/prefect/pull/7691 -- Add GitLab storage block to deployment CLI docs — https://github.com/PrefectHQ/prefect/pull/7686 -- Add links to Cloud Run and Container Instance infrastructure — https://github.com/PrefectHQ/prefect/pull/7690 -- Update docs on final state determination to reflect `Cancelled` state changes — https://github.com/PrefectHQ/prefect/pull/7700 -- Fix link in 'Agents and Work Queues' documentation — https://github.com/PrefectHQ/prefect/pull/7659 + +- Fix heading links in docs — +- Update login and `PREFECT_API_URL` configuration notes — +- Add documentation about AWS retries configuration — +- Add GitLab storage block to deployment CLI docs — +- Add links to Cloud Run and Container Instance infrastructure — +- Update docs on final state determination to reflect `Cancelled` state changes — +- Fix link in 'Agents and Work Queues' documentation — ### Contributors -- @brian-pond made their first contribution in https://github.com/PrefectHQ/prefect/pull/7659 -- @YtKC made their first contribution in https://github.com/PrefectHQ/prefect/pull/7641 -- @eudyptula made their first contribution in https://github.com/PrefectHQ/prefect/pull/7361 + +- @brian-pond made their first contribution in +- @YtKC made their first contribution in +- @eudyptula made their first contribution in - @hateyouinfinity - @jmrobbins13 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.9...2.7.0 +**All changes**: ## Release 2.6.9 @@ -3901,235 +4299,268 @@ It also detects existing authentication: There's also a new `prefect cloud logout` command (contributed by @hallenmaia) to remove credentials from the current profile. ### Enhancements -- Add automatic upper-casing of string log level settings — https://github.com/PrefectHQ/prefect/pull/7592 -- Add `infrastructure_pid` to flow run — https://github.com/PrefectHQ/prefect/pull/7595 -- Add `PrefectFormatter` to reduce logging configuration duplication — https://github.com/PrefectHQ/prefect/pull/7588 -- Update `CloudClient.read_workspaces` to return a model — https://github.com/PrefectHQ/prefect/pull/7332 -- Update hashing utilities to allow execution in FIPS 140-2 environments — https://github.com/PrefectHQ/prefect/pull/7620 + +- Add automatic upper-casing of string log level settings — +- Add `infrastructure_pid` to flow run — +- Add `PrefectFormatter` to reduce logging configuration duplication — +- Update `CloudClient.read_workspaces` to return a model — +- Update hashing utilities to allow execution in FIPS 140-2 environments — ### Fixes -- Update logging setup to support incremental configuration — https://github.com/PrefectHQ/prefect/pull/7569 -- Update logging `JsonFormatter` to output valid JSON — https://github.com/PrefectHQ/prefect/pull/7567 -- Remove `inter` CSS import, which blocked UI loads in air-gapped environments — https://github.com/PrefectHQ/prefect/pull/7586 -- Return 404 when a flow run is missing during `set_task_run_state` — https://github.com/PrefectHQ/prefect/pull/7603 -- Fix directory copy errors with `LocalFileSystem` deployments on Python 3.7 — https://github.com/PrefectHQ/prefect/pull/7441 -- Add flush of task run logs when on remote workers — https://github.com/PrefectHQ/prefect/pull/7626 + +- Update logging setup to support incremental configuration — +- Update logging `JsonFormatter` to output valid JSON — +- Remove `inter` CSS import, which blocked UI loads in air-gapped environments — +- Return 404 when a flow run is missing during `set_task_run_state` — +- Fix directory copy errors with `LocalFileSystem` deployments on Python 3.7 — +- Add flush of task run logs when on remote workers — ### Documentation -- Add docs about CPU and memory allocation on agent deploying ECS infrastructure blocks — https://github.com/PrefectHQ/prefect/pull/7597 + +- Add docs about CPU and memory allocation on agent deploying ECS infrastructure blocks — ### Contributors + - @hallenmaia - @szelenka -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.8...2.6.9 +**All changes**: ## Release 2.6.8 ### Enhancements -- Add `--run-once` to `prefect agent start` CLI — https://github.com/PrefectHQ/prefect/pull/7505 -- Expose `prefetch-seconds` in `prefect agent start` CLI — https://github.com/PrefectHQ/prefect/pull/7498 -- Add start time sort for flow runs to the REST API — https://github.com/PrefectHQ/prefect/pull/7496 -- Add `merge_existing_data` flag to `update_block_document` — https://github.com/PrefectHQ/prefect/pull/7470 -- Add sanitization to enforce leading/trailing alphanumeric characters for Kubernetes job labels — https://github.com/PrefectHQ/prefect/pull/7528 + +- Add `--run-once` to `prefect agent start` CLI — +- Expose `prefetch-seconds` in `prefect agent start` CLI — +- Add start time sort for flow runs to the REST API — +- Add `merge_existing_data` flag to `update_block_document` — +- Add sanitization to enforce leading/trailing alphanumeric characters for Kubernetes job labels — ### Fixes -- Fix type checking for flow name and version arguments — https://github.com/PrefectHQ/prefect/pull/7549 -- Fix check for empty paths in `LocalFileSystem` — https://github.com/PrefectHQ/prefect/pull/7477 -- Fix `PrefectConsoleHandler` bug where log tracebacks were excluded — https://github.com/PrefectHQ/prefect/pull/7558 + +- Fix type checking for flow name and version arguments — +- Fix check for empty paths in `LocalFileSystem` — +- Fix `PrefectConsoleHandler` bug where log tracebacks were excluded — ### Documentation -- Add glow to Collection Catalog images in dark mode — https://github.com/PrefectHQ/prefect/pull/7535 + +- Add glow to Collection Catalog images in dark mode — - New [`prefect-vault`](https://github.com/pbchekin/prefect-vault) collection for integration with Hashicorp Vault ## Contributors -* @kielnino made their first contribution in https://github.com/PrefectHQ/prefect/pull/7517 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.7...2.6.8 +- @kielnino made their first contribution in + +**All changes**: ## Release 2.6.7 ### Enhancements -- Add timeout support to tasks — https://github.com/PrefectHQ/prefect/pull/7409 -- Add colored log levels — https://github.com/PrefectHQ/prefect/pull/6101 -- Update flow and task run page sidebar styling — https://github.com/PrefectHQ/prefect/pull/7426 -- Add redirect to logs tab when navigating to parent or child flow runs — https://github.com/PrefectHQ/prefect/pull/7439 -- Add `PREFECT_UI_URL` and `PREFECT_CLOUD_UI_URL` settings — https://github.com/PrefectHQ/prefect/pull/7411 -- Improve scheduler performance — https://github.com/PrefectHQ/prefect/pull/7450 https://github.com/PrefectHQ/prefect/pull/7433 -- Add link to parent flow from subflow details page — https://github.com/PrefectHQ/prefect/pull/7491 -- Improve visibility of deployment tags in the deployments page — https://github.com/PrefectHQ/prefect/pull/7491 -- Add deployment and flow metadata to infrastructure labels — https://github.com/PrefectHQ/prefect/pull/7479 -- Add obfuscation of secret settings — https://github.com/PrefectHQ/prefect/pull/7465 + +- Add timeout support to tasks — +- Add colored log levels — +- Update flow and task run page sidebar styling — +- Add redirect to logs tab when navigating to parent or child flow runs — +- Add `PREFECT_UI_URL` and `PREFECT_CLOUD_UI_URL` settings — +- Improve scheduler performance — +- Add link to parent flow from subflow details page — +- Improve visibility of deployment tags in the deployments page — +- Add deployment and flow metadata to infrastructure labels — +- Add obfuscation of secret settings — ### Fixes -- Fix missing import for `ObjectAlreadyExists` exception in deployments module — https://github.com/PrefectHQ/prefect/pull/7360 -- Fix export of `State` and `allow_failure` for type-checkers — https://github.com/PrefectHQ/prefect/pull/7447 -- Fix `--skip-upload` flag in `prefect deployment build` — https://github.com/PrefectHQ/prefect/pull/7437 -- Fix `visit_collection` handling of IO objects — https://github.com/PrefectHQ/prefect/pull/7482 -- Ensure that queries are sorted correctly when limits are used — https://github.com/PrefectHQ/prefect/pull/7457 + +- Fix missing import for `ObjectAlreadyExists` exception in deployments module — +- Fix export of `State` and `allow_failure` for type-checkers — +- Fix `--skip-upload` flag in `prefect deployment build` — +- Fix `visit_collection` handling of IO objects — +- Ensure that queries are sorted correctly when limits are used — ### Deprecations -- `PREFECT_CLOUD_URL` has been deprecated in favor of `PREFECT_CLOUD_API_URL` — https://github.com/PrefectHQ/prefect/pull/7411 -- `prefect.orion.utilities.names` has been deprecated in favor of `prefect.utilities.names` — https://github.com/PrefectHQ/prefect/pull/7465 + +- `PREFECT_CLOUD_URL` has been deprecated in favor of `PREFECT_CLOUD_API_URL` — +- `prefect.orion.utilities.names` has been deprecated in favor of `prefect.utilities.names` — ### Documentation -- Add support for dark mode — https://github.com/PrefectHQ/prefect/pull/7432 and https://github.com/PrefectHQ/prefect/pull/7462 -- Add [audit log documentation](https://docs.prefect.io/ui/audit-log/) for Prefect Cloud — https://github.com/PrefectHQ/prefect/pull/7404 -- Add [troubleshooting topics](https://docs.prefect.io/ui/troubleshooting/) for Prefect Cloud — https://github.com/PrefectHQ/prefect/pull/7446 + +- Add support for dark mode — and +- Add [audit log documentation](https://docs.prefect.io/ui/audit-log/) for Prefect Cloud — +- Add [troubleshooting topics](https://docs.prefect.io/ui/troubleshooting/) for Prefect Cloud — ### Collections -- Adds auto-registration of blocks from AWS, Azure, GCP, and Databricks collections — https://github.com/PrefectHQ/prefect/pull/7415 -- Add new [`prefect-hightouch`](https://prefecthq.github.io/prefect-hightouch/) collection for [Hightouch](https://hightouch.com/) — https://github.com/PrefectHQ/prefect/pull/7443 + +- Adds auto-registration of blocks from AWS, Azure, GCP, and Databricks collections — +- Add new [`prefect-hightouch`](https://prefecthq.github.io/prefect-hightouch/) collection for [Hightouch](https://hightouch.com/) — ### Contributors -- @tekumara -- @bcbernardo made their first contribution in https://github.com/PrefectHQ/prefect/pull/7360 -- @br3ndonland made their first contribution in https://github.com/PrefectHQ/prefect/pull/7432 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.6...2.6.7 +- @tekumara +- @bcbernardo made their first contribution in +- @br3ndonland made their first contribution in +**All changes**: ## Release 2.6.6 ### Enhancements + - Add work queue status and health display to UI — [#733](https://github.com/PrefectHQ/orion-design/pull/733), [#743](https://github.com/PrefectHQ/orion-design/pull/743), [#750](https://github.com/PrefectHQ/orion-design/pull/750) -- Add `wait_for` to flows; subflows can wait for upstream tasks — https://github.com/PrefectHQ/prefect/pull/7343 -- Add informative error if flow run is deleted while running — https://github.com/PrefectHQ/prefect/pull/7390 -- Add name filtering support to the `work_queues/filter` API route — https://github.com/PrefectHQ/prefect/pull/7394 -- Improve the stability of the scheduler service — https://github.com/PrefectHQ/prefect/pull/7412 +- Add `wait_for` to flows; subflows can wait for upstream tasks — +- Add informative error if flow run is deleted while running — +- Add name filtering support to the `work_queues/filter` API route — +- Improve the stability of the scheduler service — ### Fixes -- Fix GitHub storage error for Windows — https://github.com/PrefectHQ/prefect/pull/7372 -- Fix links to flow runs in notifications — https://github.com/PrefectHQ/prefect/pull/7249 -- Fix link to UI deployment page in CLI — https://github.com/PrefectHQ/prefect/pull/7376 -- Fix UI URL routing to be consistent with CLI — https://github.com/PrefectHQ/prefect/pull/7391 -- Assert that command is a list when passed to `open_process` — https://github.com/PrefectHQ/prefect/pull/7389 -- Fix JSON error when serializing certain flow run parameters such as dataframes — https://github.com/PrefectHQ/prefect/pull/7385 + +- Fix GitHub storage error for Windows — +- Fix links to flow runs in notifications — +- Fix link to UI deployment page in CLI — +- Fix UI URL routing to be consistent with CLI — +- Assert that command is a list when passed to `open_process` — +- Fix JSON error when serializing certain flow run parameters such as dataframes — ### Documentation -- Add versioning documentation — https://github.com/PrefectHQ/prefect/pull/7353 + +- Add versioning documentation — ### Collections + - New [`prefect-alert`](https://github.com/khuyentran1401/prefect-alert) collection for sending alerts on flow run fail - New [Fivetran](https://fivetran.github.io/prefect-fivetran/) collection - New [GitLab](https://prefecthq.github.io/prefect-gitlab/) collection ## Contributors + - @marwan116 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.5...2.6.6 +**All changes**: ## Release 2.6.5 ### Enhancements -- Add support for manual flow run retries — https://github.com/PrefectHQ/prefect/pull/7152 -- Improve server performance when retrying flow runs with many tasks — https://github.com/PrefectHQ/prefect/pull/7152 -- Add status checks to work queues — https://github.com/PrefectHQ/prefect/pull/7262 -- Add timezone parameter to `prefect deployment build` — https://github.com/PrefectHQ/prefect/pull/7282 -- UI: Add redirect to original block form after creating a nested block — https://github.com/PrefectHQ/prefect/pull/7284 -- Add support for multiple work queue prefixes — https://github.com/PrefectHQ/prefect/pull/7222 -- Include "-" before random suffix of Kubernetes job names — https://github.com/PrefectHQ/prefect/pull/7329 -- Allow a working directory to be specified for `Process` infrastructure — https://github.com/PrefectHQ/prefect/pull/7252 -- Add support for Python 3.11 — https://github.com/PrefectHQ/prefect/pull/7304 -- Add persistence of data when a state is returned from a task or flow — https://github.com/PrefectHQ/prefect/pull/7316 -- Add `ignore_file` to `Deployment.build_from_flow()` — https://github.com/PrefectHQ/prefect/pull/7012 + +- Add support for manual flow run retries — +- Improve server performance when retrying flow runs with many tasks — +- Add status checks to work queues — +- Add timezone parameter to `prefect deployment build` — +- UI: Add redirect to original block form after creating a nested block — +- Add support for multiple work queue prefixes — +- Include "-" before random suffix of Kubernetes job names — +- Allow a working directory to be specified for `Process` infrastructure — +- Add support for Python 3.11 — +- Add persistence of data when a state is returned from a task or flow — +- Add `ignore_file` to `Deployment.build_from_flow()` — ### Fixes -- Allow `with_options` to reset retries and retry delays — https://github.com/PrefectHQ/prefect/pull/7276 -- Fix proxy-awareness in the `OrionClient` — https://github.com/PrefectHQ/prefect/pull/7328 -- Fix block auto-registration when changing databases — https://github.com/PrefectHQ/prefect/pull/7350 -- Include hidden files when uploading directories to `RemoteFileSystem` storage — https://github.com/PrefectHQ/prefect/pull/7336 -- UI: added support for unsetting color-mode preference, `null` is now equivalent to "default" — https://github.com/PrefectHQ/prefect/pull/7321 + +- Allow `with_options` to reset retries and retry delays — +- Fix proxy-awareness in the `OrionClient` — +- Fix block auto-registration when changing databases — +- Include hidden files when uploading directories to `RemoteFileSystem` storage — +- UI: added support for unsetting color-mode preference, `null` is now equivalent to "default" — ### Documentation -- Add documentation for Prefect Cloud SSO — https://github.com/PrefectHQ/prefect/pull/7302 + +- Add documentation for Prefect Cloud SSO — ### Collections + - New [`prefect-docker`](https://prefecthq.github.io/prefect-docker/) collection for [Docker](https://www.docker.com/) - New [`prefect-census`](https://prefecthq.github.io/prefect-census/) collection for [Census](https://docs.getcensus.com/) ## Contributors -- @BallisticPain made their first contribution in https://github.com/PrefectHQ/prefect/pull/7252 + +- @BallisticPain made their first contribution in - @deepyaman - @hateyouinfinity - @jmg-duarte - @taljaards -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.4...2.6.5 +**All changes**: ## Release 2.6.4 ### Enhancements -- UI: Rename deployment "Overview" tab to "Description" — https://github.com/PrefectHQ/prefect/pull/7234 -- Add `Deployment.build_from_flow` toggle to disable loading of existing values from the API — https://github.com/PrefectHQ/prefect/pull/7218 -- Add `PREFECT_RESULTS_PERSIST_BY_DEFAULT` setting to globally toggle the result persistence default — https://github.com/PrefectHQ/prefect/pull/7228 -- Add support for using callable objects as tasks — https://github.com/PrefectHQ/prefect/pull/7217 -- Add authentication as service principal to the `Azure` storage block — https://github.com/PrefectHQ/prefect/pull/6844 -- Update default database timeout from 1 to 5 seconds — https://github.com/PrefectHQ/prefect/pull/7246 + +- UI: Rename deployment "Overview" tab to "Description" — +- Add `Deployment.build_from_flow` toggle to disable loading of existing values from the API — +- Add `PREFECT_RESULTS_PERSIST_BY_DEFAULT` setting to globally toggle the result persistence default — +- Add support for using callable objects as tasks — +- Add authentication as service principal to the `Azure` storage block — +- Update default database timeout from 1 to 5 seconds — ### Fixes -- Allow image/namespace fields to be loaded from Kubernetes job manifest — https://github.com/PrefectHQ/prefect/pull/7244 -- UI: Update settings API call to respect `ORION_UI_SERVE_BASE` environment variable — https://github.com/PrefectHQ/prefect/pull/7068 -- Fix entrypoint path error when deployment is created on Windows then run on Unix — https://github.com/PrefectHQ/prefect/pull/7261 + +- Allow image/namespace fields to be loaded from Kubernetes job manifest — +- UI: Update settings API call to respect `ORION_UI_SERVE_BASE` environment variable — +- Fix entrypoint path error when deployment is created on Windows then run on Unix — ### Collections + - New [`prefect-kv`](https://github.com/madkinsz/prefect-kv) collection for persisting key-value data -- `prefect-aws`: Update [`S3Bucket`](https://prefecthq.github.io/prefect-aws/s3/#prefect_aws.s3.S3Bucket) storage block to enable use with deployments — https://github.com/PrefectHQ/prefect-aws/pull/82 -- `prefect-aws`: Add support for arbitrary user customizations to [`ECSTask`](https://prefecthq.github.io/prefect-aws/ecs/) block — https://github.com/PrefectHQ/prefect-aws/pull/120 +- `prefect-aws`: Update [`S3Bucket`](https://prefecthq.github.io/prefect-aws/s3/#prefect_aws.s3.S3Bucket) storage block to enable use with deployments — +- `prefect-aws`: Add support for arbitrary user customizations to [`ECSTask`](https://prefecthq.github.io/prefect-aws/ecs/) block — - `prefect-aws`: Removed the experimental designation from the [`ECSTask`](https://prefecthq.github.io/prefect-aws/ecs/) block -- `prefect-azure`: New [`AzureContainerInstanceJob`](https://prefecthq.github.io/prefect-azure/container_instance/) infrastructure block to run flows or commands as containers on Azure — https://github.com/PrefectHQ/prefect-azure/pull/45 +- `prefect-azure`: New [`AzureContainerInstanceJob`](https://prefecthq.github.io/prefect-azure/container_instance/) infrastructure block to run flows or commands as containers on Azure — ### Contributors + - @Trymzet - @jmg-duarte -- @mthanded made their first contribution in https://github.com/PrefectHQ/prefect/pull/7068 +- @mthanded made their first contribution in -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.3...2.6.4 +**All changes**: ## Release 2.6.3 ### Fixes -- Fix handling of `cache_result_in_memory` in `Task.with_options` — https://github.com/PrefectHQ/prefect/pull/7227 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.2...2.6.3 +- Fix handling of `cache_result_in_memory` in `Task.with_options` — + +**All changes**: ## Release 2.6.2 ### Enhancements -- Add `CompressedSerializer` for compression of other result serializers — https://github.com/PrefectHQ/prefect/pull/7164 -- Add option to drop task or flow return values from memory — https://github.com/PrefectHQ/prefect/pull/7174 -- Add support for creating and reading notification policies from the client — https://github.com/PrefectHQ/prefect/pull/7154 -- Add API support for sorting deployments — https://github.com/PrefectHQ/prefect/pull/7187 -- Improve searching and sorting of flows and deployments in the UI — https://github.com/PrefectHQ/prefect/pull/7160 -- Improve recurrence rule schedule parsing with support for compound rules — https://github.com/PrefectHQ/prefect/pull/7165 -- Add support for private GitHub repositories — https://github.com/PrefectHQ/prefect/pull/7107 + +- Add `CompressedSerializer` for compression of other result serializers — +- Add option to drop task or flow return values from memory — +- Add support for creating and reading notification policies from the client — +- Add API support for sorting deployments — +- Improve searching and sorting of flows and deployments in the UI — +- Improve recurrence rule schedule parsing with support for compound rules — +- Add support for private GitHub repositories — ### Fixes -- Improve orchestration handling of `after_transition` when exception encountered — https://github.com/PrefectHQ/prefect/pull/7156 -- Prevent block name from being reused on the block creation form in the UI — https://github.com/PrefectHQ/prefect/pull/7096 -- Fix bug where `with_options` incorrectly updates result settings — https://github.com/PrefectHQ/prefect/pull/7186 -- Add backwards compatibility for return of server-states from flows and tasks — https://github.com/PrefectHQ/prefect/pull/7189 -- Fix naming of subflow runs tab on flow run page in the UI — https://github.com/PrefectHQ/prefect/pull/7192 -- Fix `prefect orion start` error on Windows when module path contains spaces — https://github.com/PrefectHQ/prefect/pull/7224 +- Improve orchestration handling of `after_transition` when exception encountered — +- Prevent block name from being reused on the block creation form in the UI — +- Fix bug where `with_options` incorrectly updates result settings — +- Add backwards compatibility for return of server-states from flows and tasks — +- Fix naming of subflow runs tab on flow run page in the UI — +- Fix `prefect orion start` error on Windows when module path contains spaces — ### Collections + - New [prefect-monte-carlo](https://prefecthq.github.io/prefect-monte-carlo/) collection for interaction with [Monte Carlo](https://www.montecarlodata.com/) ### Contributors + - @jmg-duarte -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.1...2.6.2 +**All changes**: ## Release 2.6.1 ### Fixes -- Fix bug where return values of `{}` or `[]` could be coerced to `None` — https://github.com/PrefectHQ/prefect/pull/7181 + +- Fix bug where return values of `{}` or `[]` could be coerced to `None` — ## Contributors -* @acookin made their first contribution in https://github.com/PrefectHQ/prefect/pull/7172 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.6.0...2.6.1 +- @acookin made their first contribution in + +**All changes**: ## Release 2.6.0 @@ -4187,7 +4618,7 @@ def four() ``` See the [documentation](https://docs.prefect.io/concepts/results/) for more details and examples. -See https://github.com/PrefectHQ/prefect/pull/6908 for implementation details. +See for implementation details. ### Waiting for tasks even if they fail @@ -4210,41 +4641,46 @@ def important_cleanup(): ... ``` -See https://github.com/PrefectHQ/prefect/pull/7120 for implementation details. +See for implementation details. ### Work queue match support for agents Agents can now match multiple work queues by providing a `--match` string instead of specifying all of the work queues. The agent will poll every work queue with a name that starts with the given string. Your agent will detect new work queues that match the option without requiring a restart! ``` -$ prefect agent start --match "foo-" +prefect agent start --match "foo-" ``` ### Enhancements -- Add `--param` / `--params` support `prefect deployment run` — https://github.com/PrefectHQ/prefect/pull/7018 -- Add 'Show Active Runs' button to work queue page — https://github.com/PrefectHQ/prefect/pull/7092 -- Update block protection to only prevent deletion — https://github.com/PrefectHQ/prefect/pull/7042 -- Improve stability by optimizing the HTTP client — https://github.com/PrefectHQ/prefect/pull/7090 -- Optimize flow run history queries — https://github.com/PrefectHQ/prefect/pull/7138 -- Optimize server handling by saving log batches in individual transactions — https://github.com/PrefectHQ/prefect/pull/7141 -- Optimize deletion of auto-scheduled runs — https://github.com/PrefectHQ/prefect/pull/7102 + +- Add `--param` / `--params` support `prefect deployment run` — +- Add 'Show Active Runs' button to work queue page — +- Update block protection to only prevent deletion — +- Improve stability by optimizing the HTTP client — +- Optimize flow run history queries — +- Optimize server handling by saving log batches in individual transactions — +- Optimize deletion of auto-scheduled runs — ### Fixes -- Fix `DockerContainer` log streaming crash due to "marked for removal" error — https://github.com/PrefectHQ/prefect/pull/6860 -- Improve RRule schedule string parsing — https://github.com/PrefectHQ/prefect/pull/7133 -- Improve handling of duplicate blocks, reducing errors in server logs — https://github.com/PrefectHQ/prefect/pull/7140 -- Fix flow run URLs in notifications and `prefect deployment run` output — https://github.com/PrefectHQ/prefect/pull/7153 + +- Fix `DockerContainer` log streaming crash due to "marked for removal" error — +- Improve RRule schedule string parsing — +- Improve handling of duplicate blocks, reducing errors in server logs — +- Fix flow run URLs in notifications and `prefect deployment run` output — ### Documentation -- Add documentation for support of proxies — https://github.com/PrefectHQ/prefect/pull/7087 -- Fix rendering of Prefect settings in API reference — https://github.com/PrefectHQ/prefect/pull/7067 + +- Add documentation for support of proxies — +- Fix rendering of Prefect settings in API reference — ### Contributors -* @jmg-duarte -* @kevin868 made their first contribution in https://github.com/PrefectHQ/prefect/pull/7109 -* @space-age-pete made their first contribution in https://github.com/PrefectHQ/prefect/pull/7122 -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.5.0...2.6.0 +- @jmg-duarte + +- @kevin868 made their first contribution in +- @space-age-pete made their first contribution in + +**All changes**: ## Release 2.5.0 @@ -4257,33 +4693,38 @@ $ prefect agent start --match "foo-" — Automatic linking of created flow run to the flow run it is created from. — Automatic tracking of upstream task results passed as parameters.
- See https://github.com/PrefectHQ/prefect/pull/7047, https://github.com/PrefectHQ/prefect/pull/7081, and https://github.com/PrefectHQ/prefect/pull/7084 + See , , and ### Enhancements -- Add ability to delete multiple objects on flow run, flow, deployment and work queue pages — https://github.com/PrefectHQ/prefect/pull/7086 -- Update `put_directory` to exclude directories from upload counts — https://github.com/PrefectHQ/prefect/pull/7054 -- Always suppress griffe logs — https://github.com/PrefectHQ/prefect/pull/7059 -- Add OOM warning to `Process` exit code log message — https://github.com/PrefectHQ/prefect/pull/7070 -- Add idempotency key support to `OrionClient.create_flow_run_from_deployment` — https://github.com/PrefectHQ/prefect/pull/7074 + +- Add ability to delete multiple objects on flow run, flow, deployment and work queue pages — +- Update `put_directory` to exclude directories from upload counts — +- Always suppress griffe logs — +- Add OOM warning to `Process` exit code log message — +- Add idempotency key support to `OrionClient.create_flow_run_from_deployment` — ### Fixes -- Fix default start date filter for deployments page in UI — https://github.com/PrefectHQ/prefect/pull/7025 -- Fix `sync_compatible` handling of wrapped async functions and generators — https://github.com/PrefectHQ/prefect/pull/7009 -- Fix bug where server could error due to an unexpected null in task caching logic — https://github.com/PrefectHQ/prefect/pull/7031 -- Add exception handling to block auto-registration — https://github.com/PrefectHQ/prefect/pull/6997 -- Remove the "sync caller" check from `sync_compatible` — https://github.com/PrefectHQ/prefect/pull/7073 + +- Fix default start date filter for deployments page in UI — +- Fix `sync_compatible` handling of wrapped async functions and generators — +- Fix bug where server could error due to an unexpected null in task caching logic — +- Add exception handling to block auto-registration — +- Remove the "sync caller" check from `sync_compatible` — ### Documentation -- Add `ECSTask` block tutorial to recipes — https://github.com/PrefectHQ/prefect/pull/7066 -- Update documentation for organizations for member management, roles, and permissions — https://github.com/PrefectHQ/prefect/pull/7058 + +- Add `ECSTask` block tutorial to recipes — +- Update documentation for organizations for member management, roles, and permissions — ## Collections + - New [prefect-soda-core](https://sodadata.github.io/prefect-soda-core/) collection for integration with [Soda](https://www.soda.io/). ### Contributors + - @taljaards -**All changes**: https://github.com/PrefectHQ/prefect/compare/2.4.5...2.5.0 +**All changes**: ## Release 2.4.5 @@ -4296,160 +4737,191 @@ Block protection was added in 2.4.1 to prevent users from deleting block types t **When running a server with this version, the client must be the same version. This does not apply to clients connecting to Prefect Cloud.** ### Enhancements -- Warn if user tries to login with API key from Cloud 1 — https://github.com/PrefectHQ/prefect/pull/6958 -- Improve concurrent task runner performance — https://github.com/PrefectHQ/prefect/pull/6948 -- Raise a `MissingContextError` when `get_run_logger` is called outside a run context — https://github.com/PrefectHQ/prefect/pull/6980 -- Adding caching to API configuration lookups to improve performance — https://github.com/PrefectHQ/prefect/pull/6959 -- Move `quote` to `prefect.utilities.annotations` — https://github.com/PrefectHQ/prefect/pull/6993 -- Add state filters and sort-by to the work-queue, flow and deployment pages — https://github.com/PrefectHQ/prefect/pull/6985 + +- Warn if user tries to login with API key from Cloud 1 — +- Improve concurrent task runner performance — +- Raise a `MissingContextError` when `get_run_logger` is called outside a run context — +- Adding caching to API configuration lookups to improve performance — +- Move `quote` to `prefect.utilities.annotations` — +- Add state filters and sort-by to the work-queue, flow and deployment pages — ### Fixes -- Fix login to private Docker registries — https://github.com/PrefectHQ/prefect/pull/6889 -- Update `Flow.with_options` to actually pass retry settings to new object — https://github.com/PrefectHQ/prefect/pull/6963 -- Fix compatibility for protected blocks when client/server versions are mismatched — https://github.com/PrefectHQ/prefect/pull/6986 -- Ensure `python-slugify` is always used even if [unicode-slugify](https://github.com/mozilla/unicode-slugify) is installed — https://github.com/PrefectHQ/prefect/pull/6955 + +- Fix login to private Docker registries — +- Update `Flow.with_options` to actually pass retry settings to new object — +- Fix compatibility for protected blocks when client/server versions are mismatched — +- Ensure `python-slugify` is always used even if [unicode-slugify](https://github.com/mozilla/unicode-slugify) is installed — ### Documentation -- Update documentation for specifying schedules from the CLI — https://github.com/PrefectHQ/prefect/pull/6968 -- Add results concept to documentation — https://github.com/PrefectHQ/prefect/pull/6992 + +- Update documentation for specifying schedules from the CLI — +- Add results concept to documentation — ### Collections -- New [`prefect-hex` collection](https://prefecthq.github.io/prefect-hex/) — https://github.com/PrefectHQ/prefect/pull/6974 -- New [`CloudRunJob` infrastructure block](https://prefecthq.github.io/prefect-gcp/cloud_run/) in `prefect-gcp` — https://github.com/PrefectHQ/prefect-gcp/pull/48 + +- New [`prefect-hex` collection](https://prefecthq.github.io/prefect-hex/) — +- New [`CloudRunJob` infrastructure block](https://prefecthq.github.io/prefect-gcp/cloud_run/) in `prefect-gcp` — ### Contributors -* @Hongbo-Miao made their first contribution in https://github.com/PrefectHQ/prefect/pull/6956 -* @hateyouinfinity made their first contribution in https://github.com/PrefectHQ/prefect/pull/6955 + +- @Hongbo-Miao made their first contribution in + +- @hateyouinfinity made their first contribution in ## Release 2.4.2 ### Fixes -- Remove types in blocks docstring attributes to avoid annotation parsing warnings — https://github.com/PrefectHQ/prefect/pull/6937 -- Fixes `inject_client` in scenarios where the `client` kwarg is passed `None` — https://github.com/PrefectHQ/prefect/pull/6942 + +- Remove types in blocks docstring attributes to avoid annotation parsing warnings — +- Fixes `inject_client` in scenarios where the `client` kwarg is passed `None` — ### Contributors -* @john-jam made their first contribution in https://github.com/PrefectHQ/prefect/pull/6937 + +- @john-jam made their first contribution in ## Release 2.4.1 ### Enhancements -- Add TTL to `KubernetesJob` for automated cleanup of finished jobs — https://github.com/PrefectHQ/prefect/pull/6785 -- Add `prefect kubernetes manifest agent` to generate an agent Kubernetes manifest — https://github.com/PrefectHQ/prefect/pull/6771 -- Add `prefect block type delete` to delete block types — https://github.com/PrefectHQ/prefect/pull/6849 -- Add dynamic titles to tabs in UI — https://github.com/PrefectHQ/prefect/pull/6914 -- Hide secret tails by default — https://github.com/PrefectHQ/prefect/pull/6846 -- Add runs tab to show flow runs on the flow, deployment, and work-queue pages in the UI — https://github.com/PrefectHQ/prefect/pull/6721 -- Add toggle to disable block registration on application start — https://github.com/PrefectHQ/prefect/pull/6858 -- Use injected client during block registration, save, and load — https://github.com/PrefectHQ/prefect/pull/6857 -- Refactor of `prefect.client` into `prefect.client.orion` and `prefect.client.cloud` — https://github.com/PrefectHQ/prefect/pull/6847 -- Improve breadcrumbs on radar page in UI — https://github.com/PrefectHQ/prefect/pull/6757 -- Reject redundant state transitions to prevent duplicate runs — https://github.com/PrefectHQ/prefect/pull/6852 -- Update block auto-registration to use a cache to improve performance — https://github.com/PrefectHQ/prefect/pull/6841 -- Add ability to define blocks from collections to be registered by default — https://github.com/PrefectHQ/prefect/pull/6890 -- Update file systems interfaces to be sync compatible — https://github.com/PrefectHQ/prefect/pull/6511 -- Add flow run URLs to notifications — https://github.com/PrefectHQ/prefect/pull/6798 -- Add client retries on 503 responses — https://github.com/PrefectHQ/prefect/pull/6927 -- Update injected client retrieval to use the flow and task run context client for reduced overhead — https://github.com/PrefectHQ/prefect/pull/6859 -- Add Microsoft Teams notification block — https://github.com/PrefectHQ/prefect/pull/6920 + +- Add TTL to `KubernetesJob` for automated cleanup of finished jobs — +- Add `prefect kubernetes manifest agent` to generate an agent Kubernetes manifest — +- Add `prefect block type delete` to delete block types — +- Add dynamic titles to tabs in UI — +- Hide secret tails by default — +- Add runs tab to show flow runs on the flow, deployment, and work-queue pages in the UI — +- Add toggle to disable block registration on application start — +- Use injected client during block registration, save, and load — +- Refactor of `prefect.client` into `prefect.client.orion` and `prefect.client.cloud` — +- Improve breadcrumbs on radar page in UI — +- Reject redundant state transitions to prevent duplicate runs — +- Update block auto-registration to use a cache to improve performance — +- Add ability to define blocks from collections to be registered by default — +- Update file systems interfaces to be sync compatible — +- Add flow run URLs to notifications — +- Add client retries on 503 responses — +- Update injected client retrieval to use the flow and task run context client for reduced overhead — +- Add Microsoft Teams notification block — ### Fixes -- Fix `LocalFileSystem.get_directory` when from and to paths match — https://github.com/PrefectHQ/prefect/pull/6824 -- Fix registration of block schema versions — https://github.com/PrefectHQ/prefect/pull/6803 -- Update agent to capture infrastructure errors and fail the flow run instead of crashing — https://github.com/PrefectHQ/prefect/pull/6903 -- Fix bug where `OrionClient.read_logs` filter was ignored — https://github.com/PrefectHQ/prefect/pull/6885 + +- Fix `LocalFileSystem.get_directory` when from and to paths match — +- Fix registration of block schema versions — +- Update agent to capture infrastructure errors and fail the flow run instead of crashing — +- Fix bug where `OrionClient.read_logs` filter was ignored — ### Documentation -- Add GitHub and Docker deployment recipe — https://github.com/PrefectHQ/prefect/pull/6825 -- Add parameter configuration examples — https://github.com/PrefectHQ/prefect/pull/6886 + +- Add GitHub and Docker deployment recipe — +- Add parameter configuration examples — ### Collections -- Add `prefect-firebolt` to collections catalog — https://github.com/PrefectHQ/prefect/pull/6917 + +- Add `prefect-firebolt` to collections catalog — ### Helm Charts + - Major overhaul in how helm charts in `prefect-helm` are structured and how we version and release them — [2022.09.21 release](https://github.com/PrefectHQ/prefect-helm/releases/tag/2022.09.21) ### Contributors + - @jmg-duarte - @taljaards - @yashlad681 -- @hallenmaia made their first contributions(!) in https://github.com/PrefectHQ/prefect/pull/6903, https://github.com/PrefectHQ/prefect/pull/6785, and https://github.com/PrefectHQ/prefect/pull/6771 -- @dobbersc made their first contribution in https://github.com/PrefectHQ/prefect/pull/6870 -- @jnovinger made their first contribution in https://github.com/PrefectHQ/prefect/pull/6916 -- @mathijscarlu made their first contribution in https://github.com/PrefectHQ/prefect/pull/6885 - +- @hallenmaia made their first contributions(!) in , , and +- @dobbersc made their first contribution in +- @jnovinger made their first contribution in +- @mathijscarlu made their first contribution in ## Release 2.4.0 ### Exciting New Features 🎉 + - Add `ECSTask` infrastructure block to run commands and flows on AWS ECS
See [the documentation](https://prefecthq.github.io/prefect-aws/ecs/) in the [prefect-aws collection](https://prefecthq.github.io/prefect-aws/) and usage notes in the [infrastructure guide](https://docs.prefect.io/concepts/infrastructure/#ecstask) ### Enhancements -- Update the deployments CLI to better support CI/CD use cases — https://github.com/PrefectHQ/prefect/pull/6697 -- Improve database query performance by removing unnecessary SQL transactions — https://github.com/PrefectHQ/prefect/pull/6714 -- Update blocks to dispatch instance creation using slugs — https://github.com/PrefectHQ/prefect/pull/6622 -- Add flow run start times to flow run metadata in UI — https://github.com/PrefectHQ/prefect/pull/6743 -- Update default infrastructure command to be set at runtime — https://github.com/PrefectHQ/prefect/pull/6610 -- Allow environment variables to be "unset" in infrastructure blocks — https://github.com/PrefectHQ/prefect/pull/6650 -- Add favicon switching feature for flow and task run pages — https://github.com/PrefectHQ/prefect/pull/6794 -- Update `Deployment.infrastructure` to accept types outside of the core library i.e. custom infrastructure or from collections — https://github.com/PrefectHQ/prefect/pull/6674 -- Update `deployment build --rrule` input to allow start date and timezones — https://github.com/PrefectHQ/prefect/pull/6761 + +- Update the deployments CLI to better support CI/CD use cases — +- Improve database query performance by removing unnecessary SQL transactions — +- Update blocks to dispatch instance creation using slugs — +- Add flow run start times to flow run metadata in UI — +- Update default infrastructure command to be set at runtime — +- Allow environment variables to be "unset" in infrastructure blocks — +- Add favicon switching feature for flow and task run pages — +- Update `Deployment.infrastructure` to accept types outside of the core library i.e. custom infrastructure or from collections — +- Update `deployment build --rrule` input to allow start date and timezones — ### Fixes -- Update crash detection to ignore abort signals — https://github.com/PrefectHQ/prefect/pull/6730 -- Protect against race condition with deployment schedules — https://github.com/PrefectHQ/prefect/pull/6673 -- Fix saving of block fields with aliases — https://github.com/PrefectHQ/prefect/pull/6758 -- Preserve task dependencies to futures passed as parameters in `.map` — https://github.com/PrefectHQ/prefect/pull/6701 -- Update task run orchestration to include latest metadata in context — https://github.com/PrefectHQ/prefect/pull/6791 + +- Update crash detection to ignore abort signals — +- Protect against race condition with deployment schedules — +- Fix saving of block fields with aliases — +- Preserve task dependencies to futures passed as parameters in `.map` — +- Update task run orchestration to include latest metadata in context — ### Documentation -- Task runner documentation fixes and clarifications — https://github.com/PrefectHQ/prefect/pull/6733 -- Add notes for Windows and Linux installation — https://github.com/PrefectHQ/prefect/pull/6750 -- Add a catalog of implementation recipes — https://github.com/PrefectHQ/prefect/pull/6408 -- Improve storage and file systems documentation — https://github.com/PrefectHQ/prefect/pull/6756 -- Add CSS for badges — https://github.com/PrefectHQ/prefect/pull/6655 + +- Task runner documentation fixes and clarifications — +- Add notes for Windows and Linux installation — +- Add a catalog of implementation recipes — +- Improve storage and file systems documentation — +- Add CSS for badges — ### Contributors -* @robalar made their first contribution in https://github.com/PrefectHQ/prefect/pull/6701 -* @shraddhafalane made their first contribution in https://github.com/PrefectHQ/prefect/pull/6784 + +- @robalar made their first contribution in + +- @shraddhafalane made their first contribution in ## 2.3.2 ### Enhancements -* UI displays an error message when backend is unreachable — https://github.com/PrefectHQ/prefect/pull/6670 + +- UI displays an error message when backend is unreachable — ### Fixes -* Fix issue where parameters weren't updated when a deployment was re-applied by @lennertvandevelde in https://github.com/PrefectHQ/prefect/pull/6668 -* Fix issues with stopping Orion on Windows machines — https://github.com/PrefectHQ/prefect/pull/6672 -* Fix issue with GitHub storage running in non-empty directories — https://github.com/PrefectHQ/prefect/pull/6693 -* Fix issue where some user-supplied values were ignored when creating new deployments — https://github.com/PrefectHQ/prefect/pull/6695 + +- Fix issue where parameters weren't updated when a deployment was re-applied by @lennertvandevelde in + +- Fix issues with stopping Orion on Windows machines — +- Fix issue with GitHub storage running in non-empty directories — +- Fix issue where some user-supplied values were ignored when creating new deployments — ### Collections -* Added [prefect-fugue](https://fugue-project.github.io/prefect-fugue/) + +- Added [prefect-fugue](https://fugue-project.github.io/prefect-fugue/) ### Contributors -* @lennertvandevelde made their first contribution! — [https://github.com/PrefectHQ/prefect/pull/6668](https://github.com/PrefectHQ/prefect/pull/6668) + +- @lennertvandevelde made their first contribution! — [https://github.com/PrefectHQ/prefect/pull/6668](https://github.com/PrefectHQ/prefect/pull/6668) ## 2.3.1 ### Enhancements -* Add sync compatibility to `run` for all infrastructure types — https://github.com/PrefectHQ/prefect/pull/6654 -* Update Docker container name collision log to `INFO` level for clarity — https://github.com/PrefectHQ/prefect/pull/6657 -* Refactor block documents queries for speed ⚡️ — https://github.com/PrefectHQ/prefect/pull/6645 -* Update block CLI to match standard styling — https://github.com/PrefectHQ/prefect/pull/6679 + +- Add sync compatibility to `run` for all infrastructure types — + +- Update Docker container name collision log to `INFO` level for clarity — +- Refactor block documents queries for speed ⚡️ — +- Update block CLI to match standard styling — ### Fixes -* Add `git` to the Prefect image — https://github.com/PrefectHQ/prefect/pull/6653 -* Update Docker container runs to be robust to container removal — https://github.com/PrefectHQ/prefect/pull/6656 -* Fix parsing of `PREFECT_TEST_MODE` in `PrefectBaseModel` — https://github.com/PrefectHQ/prefect/pull/6647 -* Fix handling of `.prefectignore` paths on Windows — https://github.com/PrefectHQ/prefect/pull/6680 + +- Add `git` to the Prefect image — + +- Update Docker container runs to be robust to container removal — +- Fix parsing of `PREFECT_TEST_MODE` in `PrefectBaseModel` — +- Fix handling of `.prefectignore` paths on Windows — ### Collections -* [prefect-juptyer](https://prefecthq.github.io/prefect-jupyter/) + +- [prefect-juptyer](https://prefecthq.github.io/prefect-jupyter/) ### Contributors -* @mars-f made their first contribution — https://github.com/PrefectHQ/prefect/pull/6639 -* @pdashk made their first contribution — https://github.com/PrefectHQ/prefect/pull/6640 + +- @mars-f made their first contribution — + +- @pdashk made their first contribution — ## 2.3.0 @@ -4484,6 +4956,7 @@ Block protection was added in 2.4.1 to prevent users from deleting block types t - Add the ability to specify relative sub-paths when working with remote storage for deployments — [#6518](https://github.com/PrefectHQ/prefect/pull/6518) - Prevent non-UUID slugs from raising errors on `/block_document` endpoints — [#6541](https://github.com/PrefectHQ/prefect/pull/6541) - Improve Docker image tag parsing to support the full Moby specification — [#6564](https://github.com/PrefectHQ/prefect/pull/6564) + ### Fixes - Set uvicorn `--app-dir` when starting Orion to avoid module collisions — [#6547](https://github.com/PrefectHQ/prefect/pull/6547) @@ -4507,7 +4980,6 @@ Block protection was added in 2.4.1 to prevent users from deleting block types t - Add details about flow run retention policies — [#6577](https://github.com/PrefectHQ/prefect/pull/6577) - Fix flow parameter name docstring in deployments — [#6599](https://github.com/PrefectHQ/prefect/pull/6599) - ### Contributors Thanks to our external contributors! @@ -4519,74 +4991,85 @@ Thanks to our external contributors! ## 2.2.0 ### Exciting New Features 🎉 -* Added automatic detection of static arguments to `Task.map` in https://github.com/PrefectHQ/prefect/pull/6513 + +- Added automatic detection of static arguments to `Task.map` in ### Fixes -* Updated deployment flow run retry settings with runtime values in https://github.com/PrefectHQ/prefect/pull/6489 -* Updated media queries for flow-run-filter in https://github.com/PrefectHQ/prefect/pull/6484 -* Added `empirical_policy` to flow run update route in https://github.com/PrefectHQ/prefect/pull/6486 -* Updated flow run policy retry settings to be nullable in https://github.com/PrefectHQ/prefect/pull/6488 -* Disallowed extra attribute initialization on `Deployment` objects in https://github.com/PrefectHQ/prefect/pull/6505 -* Updated `deployment build` to raise an informative error if two infrastructure configs are provided in https://github.com/PrefectHQ/prefect/pull/6504 -* Fixed calling async subflows from sync parents in https://github.com/PrefectHQ/prefect/pull/6514 + +- Updated deployment flow run retry settings with runtime values in + +- Updated media queries for flow-run-filter in +- Added `empirical_policy` to flow run update route in +- Updated flow run policy retry settings to be nullable in +- Disallowed extra attribute initialization on `Deployment` objects in +- Updated `deployment build` to raise an informative error if two infrastructure configs are provided in +- Fixed calling async subflows from sync parents in ## 2.1.1 ### Fixes -* Fixed log on abort when the flow run context is not available in https://github.com/PrefectHQ/prefect/pull/6402 -* Fixed error message in `submit_run` in https://github.com/PrefectHQ/prefect/pull/6453 -* Fixed error if default parameters are missing on a deployment flow run in https://github.com/PrefectHQ/prefect/pull/6465 -* Added error message if `get_run_logger` receives context of unknown type in https://github.com/PrefectHQ/prefect/pull/6401 +- Fixed log on abort when the flow run context is not available in +- Fixed error message in `submit_run` in +- Fixed error if default parameters are missing on a deployment flow run in +- Added error message if `get_run_logger` receives context of unknown type in ## 2.1.0 ### Build Deployments in Python + The new, YAML-based deployment definition provides a simple, extensible foundation for our new deployment creation experience. Now, by popular demand, we're extending that experience to enable you to define deployments and build them from within Python. You can do so by defining a `Deployment` Python object, specifying the deployment options as properties of the object, then building and applying the object using methods of `Deployment`. See the [documentation](https://docs.prefect.io/concepts/deployments/) to learn more. ### Simplified Agents & Work Queues + Agents and work queues give you control over where and how flow runs are executed. Now, creating an agent (and corresponding work queue) is even easier. Work queues now operate strictly by name, not by matching tags. Deployments, and the flow runs they generate, are explicitly linked to a single work queue, and the work queue is automatically created whenever a deployment references it. This means you no longer need to manually create a new work queue each time you want to want to route a deployment's flow runs separately. Agents can now pull from multiple work queues, and also automatically generate work queues that don't already exist. The result of these improvements is that most users will not have to interact directly with work queues at all, but advanced users can take advantage of them for increased control over how work is distributed to agents. These changes are fully backwards compatible. See the [documentation](https://docs.prefect.io/concepts/work-queues/) to learn more. ### Improvements and bug fixes -* Added three new exceptions to improve errors when parameters are incorrectly supplied to flow runs in https://github.com/PrefectHQ/prefect/pull/6091 -* Fixed a task dependency issue where unpacked values were not being correctly traced in https://github.com/PrefectHQ/prefect/pull/6348 -* Added the ability to embed `BaseModel` subclasses as fields within blocks, resolving an issue with the ImagePullPolicy field on the KubernetesJob block in https://github.com/PrefectHQ/prefect/pull/6389 -* Added comments support for deployment.yaml to enable inline help in https://github.com/PrefectHQ/prefect/pull/6339 -* Added support for specifying three schedule types — cron, interval and rrule — to the `deployment build` CLI in https://github.com/PrefectHQ/prefect/pull/6387 -* Added error handling for exceptions raised during the pre-transition hook fired by an OrchestrationRule during state transitions in https://github.com/PrefectHQ/prefect/pull/6315 -* Updated `visit_collection` to be a synchronous function in https://github.com/PrefectHQ/prefect/pull/6371 -* Revised loop service method names for clarity in https://github.com/PrefectHQ/prefect/pull/6131 -* Modified deployments to load flows in a worker thread in https://github.com/PrefectHQ/prefect/pull/6340 -* Resolved issues with capture of user-raised timeouts in https://github.com/PrefectHQ/prefect/pull/6357 -* Added base class and async compatibility to DockerRegistry in https://github.com/PrefectHQ/prefect/pull/6328 -* Added `max_depth` to `visit_collection`, allowing recursion to be limited in https://github.com/PrefectHQ/prefect/pull/6367 -* Added CLI commands for inspecting and deleting Blocks and Block Types in https://github.com/PrefectHQ/prefect/pull/6422 -* Added a Server Message Block (SMB) file system block in https://github.com/PrefectHQ/prefect/pull/6344 — Special thanks to @darrida for this contribution! -* Removed explicit type validation from some API routes in https://github.com/PrefectHQ/prefect/pull/6448 -* Improved robustness of streaming output from subprocesses in https://github.com/PrefectHQ/prefect/pull/6445 -* Added a default work queue ("default") when creating new deployments from the Python client or CLI in https://github.com/PrefectHQ/prefect/pull/6458 + +- Added three new exceptions to improve errors when parameters are incorrectly supplied to flow runs in + +- Fixed a task dependency issue where unpacked values were not being correctly traced in +- Added the ability to embed `BaseModel` subclasses as fields within blocks, resolving an issue with the ImagePullPolicy field on the KubernetesJob block in +- Added comments support for deployment.yaml to enable inline help in +- Added support for specifying three schedule types — cron, interval and rrule — to the `deployment build` CLI in +- Added error handling for exceptions raised during the pre-transition hook fired by an OrchestrationRule during state transitions in +- Updated `visit_collection` to be a synchronous function in +- Revised loop service method names for clarity in +- Modified deployments to load flows in a worker thread in +- Resolved issues with capture of user-raised timeouts in +- Added base class and async compatibility to DockerRegistry in +- Added `max_depth` to `visit_collection`, allowing recursion to be limited in +- Added CLI commands for inspecting and deleting Blocks and Block Types in +- Added a Server Message Block (SMB) file system block in — Special thanks to @darrida for this contribution! +- Removed explicit type validation from some API routes in +- Improved robustness of streaming output from subprocesses in +- Added a default work queue ("default") when creating new deployments from the Python client or CLI in ### New Collections + - [prefect-monday](https://prefecthq.github.io/prefect-monday/) - [prefect-databricks](https://prefecthq.github.io/prefect-databricks/) - [prefect-fugue](https://github.com/fugue-project/prefect-fugue/) -**Full Changelog**: https://github.com/PrefectHQ/prefect/compare/2.0.4...2.1.0 +**Full Changelog**: ## 2.0.4 ### Simplified deployments + The deployment experience has been refined to remove extraneous artifacts and make configuration even easier. In particular: -- `prefect deployment build` no longer generates a  `manifest.json` file. Instead, all of the relevant information is written to the `deployment.yaml` file. +- `prefect deployment build` no longer generates a  `manifest.json` file. Instead, all of the relevant information is written to the `deployment.yaml` file. - Values in the `deployment.yaml` file are more atomic and explicit -- Local file system blocks are no longer saved automatically -- Infrastructure block values can now be overwritten with the new `infra_overrides` field +- Local file system blocks are no longer saved automatically +- Infrastructure block values can now be overwritten with the new `infra_overrides` field ### Start custom flow runs from the UI + Now, from the deployment page, in addition to triggering an immediate flow run with default parameter arguments, you can also create a custom run. A custom run enables you to configure the run's parameter arguments, start time, name, and more, all while otherwise using the same deployment configuration. The deployment itself will be unchanged and continue to generate runs on its regular schedule. ### Improvements and bug fixes + - Made timeout errors messages on state changes more intuitive - Added debug level logs for task run rehydration - Added basic CLI functionality to inspect Blocks; more to come @@ -4602,6 +5085,7 @@ This release contains a number of bug fixes and documentation improvements. We've released `prefect-dbt` — a collection of Prefect integrations for working with dbt in your Prefect flows. This collection has been built as part of a partnership with dbt Labs to ensure that it follows best practices for working with dbt. ### Improvements and bug fixes + - Azure storage blocks can use `.prefectignore` - Resolved bugs and improved interface in the Orion client. - Resolved a bug in Azure storage blocks that would cause uploads to get stuck. @@ -4613,15 +5097,16 @@ We've released `prefect-dbt` — a collection of Prefect integrations for workin - Fixes `--manifest-only` flag of `prefect deployment build` command to ensure that using this flag, the manifest gets generated, but the upload to a storage location is skipped. - Added support for multiple YAML deployment paths to the `prefect deployment apply` command. - ## 2.0.2 This release implements a number of improvements and bug fixes in response to continued engagement by members of our community. Thanks, as always, to all who submitted ideas on how to make Prefect 2 even better. ### Introducing .prefectignore files - .prefectignore files allow users to omit certain files or directories from their deployments. Similar to other .ignore files, the syntax supports pattern matching, so an entry of `*.pyc` will ensure *all* .pyc files are ignored by the deployment call when uploading to remote storage. Prefect provides a default .prefectignore file, but users can customize it to their needs. + + .prefectignore files allow users to omit certain files or directories from their deployments. Similar to other .ignore files, the syntax supports pattern matching, so an entry of `*.pyc` will ensure _all_ .pyc files are ignored by the deployment call when uploading to remote storage. Prefect provides a default .prefectignore file, but users can customize it to their needs. ### Improvements and bug fixes + - Users can now leverage Azure storage blocks. - Users can now submit bug reports and feature enhancements using our issue templates. - Block deletion is now more performant. @@ -4634,6 +5119,7 @@ This release implements a number of improvements and bug fixes in response to co The response to Prefect 2 has been overwhelming in the best way possible. Thank you to the many community members who tried it out and gave us feedback! Thanks in particular to the students at this week's Prefect Associate Certification Course (PACC) in San Jose for their thoughtful recommendations. This release is a compilation of enhancements and fixes that make for a more resilient, performant, and refined Prefect experience. ### Improvements and bug fixes + - Schedules set via the API or UI are now preserved when building deployments from the CLI - JSON types are now coerced to none, following Javascript convention and supporting standards compatibility - The `prefect deployment execute` command has been removed to avoid confusion between running a flow locally from a Python script and running it by an agent using `prefect deployment run` @@ -4651,15 +5137,19 @@ We're thrilled to announce that, with this release, Prefect 2.0 has exited its p Prefect 2.0 documentation is now hosted at [docs.prefect.io](https://docs.prefect.io). Prefect 1.0 documentation is now hosted at [docs-v1.prefect.io](https://docs-v1.prefect.io). ### Upgrading from Prefect 1.0 + Flows written with Prefect 1.0 will require modifications to run with Prefect 2.0. If you're using Prefect 1.0, please see our [guidance on Discourse for explicitly pinning your Prefect version in your package manager and Docker](https://discourse.prefect.io/t/the-general-availability-release-of-prefect-2-0-going-live-on-wednesday-27th-of-july-may-break-your-flows-unless-you-take-action-as-soon-as-possible/1227), so that you can make the transition to Prefect 2.0 when the time is right for you. See our [migration page](https://upgrade.prefect.io/) to learn more about upgrading. ### Upgrading from earlier versions of Prefect 2.0 + We have shipped a lot of breaking changes to Prefect 2.0 over the past week. Most importantly, **recent changes to deployments required that schedules for all previously created deployments be turned off**. You can learn more about the changes via the [deployments concept documentation](https://docs.prefect.io/concepts/deployments/), the [tutorial](https://docs.prefect.io/tutorials/deployments/), or the [discourse guide](https://discourse.prefect.io/t/deployments-are-now-simpler-and-declarative/1255). ## 2.0b16 ### Simplified, declarative deployments + Prefect 2.0's deployments are a powerful way to encapsulate a flow, its required infrastructure, its schedule, its parameters, and more. Now, you can create deployments simply, with just two commands: + 1. `prefect deployment build ./path/to/flow/file.py:name_of_flow_obj --name "Deployment Name"` produces two files: — A manifest file, containing workflow-specific information such as the code location, the name of the entrypoint flow, and flow parameters — A `deployment.yaml` file — a complete specification of the metadata and configuration for the deployment such as the name, tags, and description @@ -4668,6 +5158,7 @@ Prefect 2.0's deployments are a powerful way to encapsulate a flow, its required Once the deployment is created with the Orion server, it can now be edited via the UI! See the [Deployments documentation to learn more](https://orion-docs.prefect.io/concepts/deployments/). ### Improvements and bug fixes + - The [Dask and Ray tutorials](https://orion-docs.prefect.io/tutorials/dask-ray-task-runners/) have been updated to reflect recent changes - The [Blocks concept doc](https://orion-docs.prefect.io/concepts/blocks/) has been updated to reflect recent enhancements and includes additional examples - The [Storage concept doc](https://orion-docs.prefect.io/concepts/storage/) has been updated to reflect recent enhancements @@ -4678,6 +5169,7 @@ Once the deployment is created with the Orion server, it can now be edited via t ## 2.0b15 ### Uniquely refer to blocks with slugs + Blocks are a convenient way to secure store and retrieve configuration. Now, retrieving configuration stored with blocks is even easier with slugs, both human and machine readable unique identifiers. By default, block type slugs are a lowercase, dash delimited version of the block type name, but can be customized via the `_block_type_slug` field on a custom Block subclass. Block document slugs are a concatenation of [block-type-slug]/[block-document-name] and can be used as an argument to the `Block.load` method. Slugs and block document names may only include alphanumeric characters and dashes. **Warning**: This breaking change makes this release incompatible with previous versions of the Orion server and Prefect Cloud 2.0 @@ -4687,14 +5179,17 @@ Blocks are a convenient way to secure store and retrieve configuration. Now, ret ## 2.0b14 ### Retrieve the state of your tasks or flows with the `return_state` kwarg + Beginning with 2.0b9, Prefect 2.0 began returning function results, instead of Prefect futures and states, by default. States are still an important concept in Prefect 2. They can be used to dictate and understand the behavior of your flows. Now, you can access the state for _any_ task or flow with the new `return_state` kwarg. Just set `return_state=True` in you flow or task call and you can access its state with the `.result()` method, even if it's been submitted to a task runner. ### `prefect cloud` commands are easier to use + The `prefect cloud login` command no longer overwrites your current profile with a new API URL and auth key. Instead, the command will prompt you to create a new profile when logging into Prefect Cloud 2.0. Subsequent calls to prefect cloud login using the same key will simply "log in" to prefect cloud by switching to the profile associated with that authentication key. The new `prefect cloud workspace ls` command lists available workspaces. ### Other improvements and bug fixes + - The anchor datetime (aka start datetime) for all newly created interval schedules will be the current date & time - The `prefect orion start` command now handles keyboard interrupts - CLI performance has been sped up 30-40% through improved import handling @@ -4704,6 +5199,7 @@ The new `prefect cloud workspace ls` command lists available workspaces. ## 2.0b13 ### Improvements and bug fixes + - RRule schedule strings are now validated on initialization to confirm that the provided RRule strings are valid - Concepts docs have been updated for clarity and consistency - `IntervalSchedule`'s now coerce naive datetimes to timezone-aware datetimes, so that interval schedules created with timezone-unaware datetimes will work @@ -4711,9 +5207,11 @@ The new `prefect cloud workspace ls` command lists available workspaces. ## 2.0b12 ### Work queue pages now display upcoming runs + A new "Upcoming runs" tab has been added to the work queue page, enabling you to see all of the runs that are eligible for that work queue before they are picked up by an agent. ### Other improvements and bug fixes + - You can now set a concurrency limit when creating a work queue via the CLI - In order to avoid unwittingly breaking references to shared blocks, block names are no longer editable - Getting started documentation has been updated and edited for clarity @@ -4724,9 +5222,11 @@ A new "Upcoming runs" tab has been added to the work queue page, enabling you to This release builds upon the collection of small enhancements made in the previous release. ### Default storage has been removed + For convenience, earlier versions of Prefect 2.0 allowed for a global storage setting. With forthcoming enhancements to blocks, this will no longer be necessary. ### Other improvements and bug fixes + - We have published a [guide for migrating workflows from Prefect 1.0 (and lower) to Prefect 2.0](https://orion-docs.prefect.io/migration_guide/) - The Flow run page now has a clearer empty state that is more consistent with other pages - Tutorial documentation has been further updated to reflect new result behavior @@ -4741,6 +5241,7 @@ For convenience, earlier versions of Prefect 2.0 allowed for a global storage se This release is the first of a series of smaller releases to be released daily. ### Improvements and bug fixes + - The Blocks selection page now includes more complete and consistent metadata about each block type, including block icons, descriptions, and examples - We've added a new [CLI style guide](https://github.com/PrefectHQ/prefect/blob/orion/docs/contributing/style.md#command-line-interface-cli-output-messages) for contributors - Work queues no longer filter on flow runner types, this capability will instead be achieved through tags @@ -4753,11 +5254,13 @@ Big things are in the works for Prefect 2! This release includes breaking change **With next week's release on July 27th, Prefect 2 will become the default package installed with `pip install prefect`. Flows written with Prefect 1 will require modifications to run with Prefect 2**. Please ensure that your package management process enables you to make the transition when the time is right for you. ### Code as workflows + As Prefect 2 usage has grown, we've observed a pattern among users, especially folks that were not previously users of Prefect 1. Working with Prefect was so much like working in native Python, users were often surprised that their tasks returned futures and states, Prefect objects, rather than results, the data that their Python functions were handling. This led to unfamiliar, potentially intimidating, errors in some cases. With this release, Prefect moves one step closer to code as workflows — tasks now return the results of their functions, rather than their states, by default. This means that you can truly take most native Python scripts, add the relevant @flow and @task decorators, and start running that script as a flow, benefitting from the observability and resilience that Prefect provides. States and futures are still important concepts in dictating and understanding the behavior of flows. You will still be able to easily access and use them with the `.submit()` method. You will need to modify tasks in existing Prefect 2 flows to use this method to continue working as before. ### Other improvements and bug fixes + - A new `Secret` block can store a string that is encrypted at rest as well as obfuscated in logs and the UI - Date filters on the flow run page in the UI now support filtering by date _and_ time - Each work queue page in the UI now includes a command to start a corresponding agent @@ -4774,11 +5277,13 @@ This is our biggest release yet! It's full of exciting new features and refineme This release removes the deprecated `DaskTaskRunner` and `RayTaskRunner` from the core library, breaking existing references to them. You can find them in their respective collections [prefect-ray](https://prefecthq.github.io/prefect-ray/) and [prefect-dask](https://prefecthq.github.io/prefect-dask). It also removes the previously deprecated restart policy for the `KubernetesFlowRunnner`. Most importantly, there are new **breaking changes** to the Deployments interface described below. ### Flow Run Retries + Flow run retries have been one of our most requested features, especially given how easy it is to run a flow as a "subflow" or "child flow" with Prefect 2.0. Flow run retries are configured just as task retries are — with the `retries` and `retry_delay_seconds` parameters. If both a task and its flow have retries configured, tasks within the flow will retry up to their specified task retry limit for each flow run. For example, if you have a **flow** configured with a limit of 2 retries (up to 3 total runs, including the initial attempt), and a **task** in the flow configured with 3 retries (up to 4 attempts per flow run, including the initial attempt). The task could run up to a total of 12 attempts, since task retry limits are reset after each flow run or flow run attempt. ### Notifications + At any time, you can visit the Prefect UI to get a comprehensive view of the state of all of your flows, but when something goes wrong with one of them, you need that information immediately. Prefect 2.0’s new notifications can alert you and your team when any flow enters any state you specify, with or without specific tags. To create a notification, go to the new Notifications page via the sidebar navigation and select “Create Notification.” Notifications are structured just as you would describe them to someone. For example, if I want to get a Slack message every time my daily-ETL flow fails, my notification will simply read: @@ -4792,7 +5297,9 @@ When the conditions of the notification are triggered, you’ll receive a simple Currently, notifications can only be sent to a [Slack webhook](https://api.slack.com/messaging/webhooks) (or email addresses if you are using [Prefect Cloud 2.0](https://app.prefect.cloud)). Over time, notifications will support additional messaging services. Let us know which messaging services you’d like to send your notifications to! ### Flow packaging and deployment + We've revisited our flow packaging and deployment UX, making it both more powerful and easier to use. `DeploymentSpec`s are now just `Deployment`s. Most of the fields are unchanged, but there are a few differences: + - The `flow_storage` field has been replaced with a `packager` field. - The `flow_location`, `flow_name`, and `flow` parameters are now just `flow`. @@ -4813,27 +5320,31 @@ Learn more in the [Deployment concept documentation](https://docs.prefect.io/con You can continue to use your existing `DeploymentSpec`s, but they are deprecated and will be removed in the coming weeks. ### Blocks + We've been working on Blocks behind the scenes for a while. Whether you know it or not, if you've used the past few releases, you've used them. Blocks enable you to securely store configuration with the Prefect Orion server and access it from your code later with just a simple reference. Think of Blocks as secure, UI-editable, type-checked environment variables. We're starting with just a few Blocks — mostly storage, but over time we’ll expand this pattern to include every tool and service in the growing modern data stack. You'll be able to set up access to your entire stack once in just a few minutes, then manage access forever without editing your code. In particular, we've made the following enhancements: + - Block document values can now be updated via the Python client with the `overwrite` flag. - Blocks now support secret fields. By default, fields identified as secret will be obfuscated when returned to the Prefect UI. The actual values can still be retrieved as necessary. -- `BlockSchema` objects have a new `secret_fields: List[str]` item in their schema's extra fields. This is a list of all fields that should be considered "secret". It also includes any secret fields from nested blocks referenced by the schema. +- `BlockSchema` objects have a new `secret_fields: List[str]` item in their schema's extra fields. This is a list of all fields that should be considered "secret". It also includes any secret fields from nested blocks referenced by the schema. - You can now browse your Blocks on the new "Blocks" page, create, and edit them right in the UI. ### Other Improvements + - Task keys, previously a concatenation of several pieces of metadata, are now only the qualified function name. While it is likely to be globally unique, the key can be used to easily identify every instance in which a function of the same name is utilized. - Tasks now have a `version` that you can set via the task decorator, like the flow version identifier on flow runs. - An Orion setting, `PREFECT_ORION_DATABASE_PASSWORD`, has been added to allow templating in the database connection URL - A link to API reference documentation has been added to the Orion startup message. - Where possible, Prefect 2.0 now exits processes earlier for synchronous flow or task runs that are cancelled. This reduces the range of conditions under which a task run would be marked failed, but continue to run. - All Prefect client models now allow extras, while the API continues to forbid them, such that older Prefect 2.0 clients can receive and load objects from the API that have additional fields, facilitating backwards compatibility. -- The _all_ attribute has been added to __init__.py for all public modules, declaring the public API for export. +- The _all_ attribute has been added to **init**.py for all public modules, declaring the public API for export. - A new endpoint, `/deployments/{id}/work_queue_check`, enables you to to check which work queues the scheduled runs of a deployment will be eligible for. - ### Bug fixes + - Attempting to create a schedule with a cron string that includes a "random" or "hashed" expression will now return an error. ### Contributors + - [Cole Murray](https://github.com/ColeMurray) - [Oliver Mannion](https://github.com/tekumara) - [Steve Flitcroft](https://github.com/redsquare) @@ -4844,6 +5355,7 @@ We've been working on Blocks behind the scenes for a while. Whether you know it This release includes a number of important improvements and bug fixes in response to continued feedback from the community. Note that this release makes a **breaking change** to the Blocks API, making the `2.0b7` Orion server incompatible with previous Orion client versions.``` ### Improvements + - Added the color select to the Orion UI in OSS (enabling users to change their state color scheme) for the UI. - Added anonymous blocks, allowing Prefect to dynamically store blocks for you without cluttering your workspace. - Performance improvements to the service that marks flows runs as late. @@ -4854,12 +5366,14 @@ This release includes a number of important improvements and bug fixes in respon - Improved task naming conventions for tasks defined using lambda functions ### Documentation improvements + - Updated screenshots and description of workflows to reflect new UI - Revised and extended Prefect Cloud quickstart tutorial - Added deployments page - Added documentation for `prefect cloud workspace set` command ### Collections + - [prefect-sqlalchemy](https://prefecthq.github.io/prefect-sqlalchemy/) - [prefect-dask](https://prefecthq.github.io/prefect-dask/) - [prefect-ray](https://prefecthq.github.io/prefect-ray/) @@ -4870,6 +5384,7 @@ This release includes a number of important improvements and bug fixes in respon Note that the Dask and Ray task runners have been moved out of the Prefect core library to reduce the number of dependencies we require for most use cases. Install from the command line with `pip install prefect-dask` and import with `from prefect_dask.task_runners import DaskTaskRunner`. ### Bug fixes + - [Allow Orion UI to run on Windows](https://github.com/PrefectHQ/prefect/pull/5802) - Fixed a bug in terminal state data handling that caused timeouts - Disabled flow execution during deployment creation to prevent accidental execution. @@ -4897,10 +5412,10 @@ Note that this release makes a **breaking change** to the Blocks API, making the After the upgrade your data will remain intact, but you will need to upgrade to `2.0b6` to continue using the Cloud 2.0 API. You can upgrade in just a few simple steps: -- Install the latest Prefect 2.0 python package: `pip install -U "prefect>=2.0b6"` -- Restart any existing agent processes - - If you are using an agent running on Kubernetes, update the Prefect image version to `2.0b6` in your Kubernetes manifest and re-apply the deployment. - - You don't need to recreate any deployments or pause any schedules — stopping your agent process to perform an upgrade may result in some Late Runs, but those will be picked up once you restart your agent. +- Install the latest Prefect 2.0 python package: `pip install -U "prefect>=2.0b6"` +- Restart any existing agent processes + - If you are using an agent running on Kubernetes, update the Prefect image version to `2.0b6` in your Kubernetes manifest and re-apply the deployment. + - You don't need to recreate any deployments or pause any schedules — stopping your agent process to perform an upgrade may result in some Late Runs, but those will be picked up once you restart your agent. ## 2.0b5 @@ -4922,7 +5437,6 @@ While most of the development of Prefect 2.0 is still happening internally, we'r - @mkarbo - @AlessandroLollo - ### Flow and task runners - Flow runners now pass all altered settings to their jobs instead of just the API key and URL @@ -5022,7 +5536,6 @@ You might not see these fixes in your day-to-day, but we're dedicated to improvi ## 2.0b2 - ### Improvements - Docker flow runners can connect to local API applications on Linux without binding to `0.0.0.0`. @@ -5041,6 +5554,7 @@ We are excited to introduce this branch as [Prefect 2.0](https://www.prefect.io/ This is the first release that's compatible with Prefect Cloud 2.0's beta API — more exciting news to come on that soon! ### Expanded UI + Through our technical preview phase, our focus has been on establishing the right [concepts](https://docs.prefect.io/concepts/overview/) and making them accessible through the CLI and API. Now that some of those concepts have matured, we've made them more accessible and tangible through UI representations. This release adds some very important concepts to the UI: **Flows and deployments** @@ -5052,6 +5566,7 @@ If you've ever created a deployment without a schedule, you know it can be diffi With the [2.0a13 release](https://github.com/PrefectHQ/prefect/blob/orion/RELEASE-NOTES.md#work-queues), we introduced [work queues](https://docs.prefect.io/concepts/work-queues/), which could only be created through the CLI. Now, you can create and edit work queues directly from the UI, then copy, paste, and run a command that starts an agent that pulls work from that queue. ### Collections + Prefect Collections are groupings of pre-built tasks and flows used to quickly build data flows with Prefect. Collections are grouped around the services with which they interact. For example, to download data from an S3 bucket, you could use the `s3_download` task from the [prefect-aws collection](https://github.com/PrefectHQ/prefect-aws), or if you want to send a Slack message as part of your flow you could use the `send_message` task from the [prefect-slack collection](https://github.com/PrefectHQ/prefect-slack). @@ -5079,6 +5594,7 @@ SETTING = "VALUE" ``` ### Other enhancements + - It's now much easier to explore Prefect 2.0's major entities, including flows, deployments, flow runs, etc. through the CLI with the `ls` command, which produces consistent, beautifully stylized tables for each entity. - Improved error handling for issues that the client commonly encounters, such as network errors, slow API requests, etc. - The UI has been polished throughout to be sleeker, faster, and even more intuitive. @@ -5127,11 +5643,13 @@ We've also rehauled our [settings reference](https://docs.prefect.io/api-ref/pre ## 2.0a12 ### Filters + Orion captures valuable metadata about your flows, deployments, and their runs. We want it to be just as simple to retrieve this information as it is to record it. This release exposes a powerful set of filter operations to cut through this body of information with ease and precision. Want to see all of the runs of your Daily ETL flow? Now it's as easy as typing `flow:"Daily ETL"` into the filter bar. This update also includes a query builder UI, so you can utilize and learn these operators quickly and easily. ## 2.0a11 ### Run Orion on Kubernetes + You can now can run the Orion API, UI, and agent on Kubernetes. We've included a new Prefect CLI command, `prefect kubernetes manifest orion`, that you can use to automatically generate a manifest that runs Orion as a Kubernetes deployment. Note: Prefect 2.0 beta versions prior to 2.0b6 used the CLI command `prefect orion kubernetes-manifest`. @@ -5218,13 +5736,16 @@ This release adds pull policies to the `DockerFlowRunner` allowing full control ### Flow Runners On the heels of the recent rename of Onion's `Executor` to `TaskRunner`, this release introduces `FlowRunner`, an analogous concept that specifies the infrastructure that a flow runs on. Just as a task runner can be specified for a flow, which encapsulates tasks, a flow runner can be specified for a deployment, which encapsulates a flow. This release includes two flow runners, which we expect to be the most commonly used: + - **SubprocessFlowRunner** — The subprocess flow runner is the default flow runner. It allows for specification of a runtime Python environment with `virtualenv` and `conda` support. - **DockerFlowRunner** — Executes the flow run in a Docker container. The image, volumes, labels, and networks can be customized. From this release on, Docker images for use with this flow runner will be published with each release. Future releases will introduce runners for executing flows on Kubernetes and major cloud platform's container compute services (e.g. AWS ECS, Google Cloud Run). ### Other enhancements + In addition to flow runners, we added several other enhancements and resolved a few issues, including: + - Corrected git installation command in docs - Refined UI through color, spacing, and alignment updates - Resolved memory leak issues associated with the cache of session factories @@ -5233,19 +5754,24 @@ In addition to flow runners, we added several other enhancements and resolved a ## 2.0a6 ### Subflows and Radar follow up + With the 2.0a5 release, we introduced the ability to navigate seamlessly between subflows and parent flows via Radar. In this release, we further enabled that ability by: + - Enabling the dedicated subflow runs tab on the Flow Run page - Tracking of upstream inputs to subflow runs - Adding a flow and task run count to all subflow run cards in the Radar view - Adding a mini Radar view on the Flow run page ### Task Runners + Previous versions of Prefect could only trigger execution of code defined within tasks. Orion can trigger execution of significant code that can be run _outside of tasks_. In order to make the role previously played by Prefect's `Executor` more explicit, we have renamed `Executor` to `TaskRunner`. A related `FlowRunner` component is forthcoming. ### Other enhancements + In addition to task runners and subflow UI enhancements, we added several other enhancements and resolved a few issues, including: + - Introduced dependency injection pathways so that Orion's database access can be modified after import time - Enabled the ability to copy the run ID from the flow run page - Added additional metadata to the flow run page details panel @@ -5256,12 +5782,15 @@ In addition to task runners and subflow UI enhancements, we added several other ## 2.0a5 ### Radar: A new way of visualizing workflows + Orion can orchestrate dynamic, DAG-free workflows. Task execution paths may not be known to Orion prior to a run—the graph “unfolds” as execution proceeds. Radar embraces this dynamism, giving users the clearest possible view of their workflows. Orion’s Radar is based on a structured, radial canvas upon which tasks are rendered as they are orchestrated. The algorithm optimizes readability through consistent node placement and minimal edge crossings. Users can zoom and pan across the canvas to discover and inspect tasks of interest. The mini-map, edge tracing, and node selection tools make workflow inspection a breeze. Radar also supports direct click-through to a subflow from its parent, enabling users to move seamlessly between task execution graphs. ### Other enhancements + While our focus was on Radar, we also made several other material improvements to Orion, including: + - Added popovers to dashboard charts, so you can see the specific data that comprises each visualization - Refactored the `OrionAgent` as a fully client side construct - Enabled custom policies through dependency injection at runtime into Orion via context managers @@ -5286,7 +5815,6 @@ In this release of Orion, we've reached feature parity with the existing Dask ex You can [create customizable temporary clusters](https://docs.prefect.io/tutorials/dask-task-runner/) and [connect to existing Dask clusters](https://docs.prefect.io/tutorials/dask-task-runner/). Additionally, because flows are not statically registered, we're able to easily expose Dask annotations, which allow you to [specify fine-grained controls over the scheduling of your tasks](https://docs.prefect.io/tutorials/dask-task-runner/) within Dask. - ### Subflow executors [Subflow runs](https://docs.prefect.io/concepts/flows/#composing-flows) are a first-class concept in Orion and this enables new execution patterns. @@ -5297,7 +5825,6 @@ This pattern can be nested or reused multiple times, enabling groups of tasks to Check out our [multiple executor documentation](https://docs.prefect.io/concepts/executors/#using-multiple-task-runners) for an example. - ### Other enhancements While we're excited to talk about these new features, we're always hard at work fixing bugs and improving performance. This release also includes: