Parallelization Questions

[ProfessorManhattan]

Hey, I have a few questions:

1. In a for or while loop, how should we be calling other tasks? Calling tasks with the task taskname starts another process and all of the run: once are reset. Also, how can we run all the tasks in parallel? Is it recommended to run task taskname &?
2. In the cmds block, how can we run, say, 3 different tasks in parallel?
3. If a task has 3 deps and those 3 deps all have a couple - task: taskname lines in their cmds block, will all 3 deps still run in parallel (including their secondary task calls)?
4. If 3 tasks running in parallel call the same task, is there a way of queueing the "same task" so that it only run consecutively? This is needed in cases like an apt-get where only one instance can run at a time.

[andreynering]

Hi @ProfessorManhattan,

There's no ideal answer at the moment. Calling task binary inside the Taskfile is not ideal, but will work for simple use cases.

We're a bit overdue on implementing builtin support for looping files, etc. There are some old issues about it like #82 and #140, so it may eventually happen. We're planning v4 on #694, but I can't promise this will be implemented.

[ghostsquad]

@ProfessorManhattan thank you for sharing! I love seeing implementations by power users. It will really help inform me how task is used out in the wild, and determine what features are most needed.

[ghostsquad]

@ProfessorManhattan do you have a specific task/taskfile you can point me at that calls out the task binary? So I can see what you are trying to do with it there?

[ghostsquad]

Bodega takes go-task to the next level by improving the TUI experience and including features that allow you to use the project as a CLI-generator.

🤣 oh boy... I'm doing the same thing with v4 of task.

This certainly is quite the Taskfile.yml you have.

[ProfessorManhattan]

@ghostsquad -- let's collaborate

[ProfessorManhattan]

Hey @ghostsquad -- I'd say the gnarliest circumstance I can think of is where I'm running task with a routine that includes committing code with git but then the Husky pre-commit git hooks trigger and also call a task. There are also maybe a dozen or two loops that include task... you can find them by searching for "task " with a space after it. Here's an example of one: https://gitlab.com/megabyte-labs/common/shared/-/blob/master/.config/taskfiles/docker/Taskfile-test.yml#L93

[ghostsquad]

I can try to answer some of your questions, as well as give you some insight into how I would like to address some of these things in v4.

❓ In a for or while loop, how should we be calling other tasks? Calling tasks with the task taskname starts another process and all of the run: once are reset. Also, how can we run all the tasks in parallel? Is it recommended to run task taskname &?

It seems you want to run some sub-task or a step an indeterminate number of times, of which you would know only at runtime?
As of task v3, there's no support for loops in this way. In v4 I am looking to introduce several features that would likely solve this problem:

1. The introduction of matrix functionality, similar to that in github actions - I'm currently considering 2 implementations of this: task matrix and step matrix

2. Dynamic Fields - This is not yet documented, as I'm still mulling over how I want this to behave, but basically, anywhere in the Taskfile.yml where you can provide a list or map, I'll be including an option for you to provide a template, which needs to return json, which is then interpreted as if that value was the value of the original field. Example

tasks:
   a:
     do:
     - echo "a"
   b:
     do:
     - echo "b"
   c:
     do:
     - echo "c"
   ham:
     needs:
       tpl: '{{ list "a" "b" "c" | mustToJson }}'

If indeed you don't know exactly what dependencies you need to run until runtime, this may be able to solve that.

3. In v4, Each task will have a field called serialGroups which allows you to more liberally put all your dependent tasks inside deps (renamed to needs in v4) and still control parallelization. more info • inspired by

---

❓ In the cmds block, how can we run, say, 3 different tasks in parallel?

In both v3 and v4, you can define a task to run within cmds (renamed to do in v4), and that task can have deps (renamed to needs in v4). Those needs (deps) run in parallel.

---

❓ If a task has 3 deps and those 3 deps all have a couple - task: taskname lines in their cmds block, will all 3 deps still run in parallel (including their secondary task calls)?

@andreynering may know the answer off hand, I'm still getting intimate with the code base. But according to the documentation

If a task executed by multiple cmds or multiple deps you can control when it is executed using run. run can also be set at the root of the Taskfile to change the behavior of all the tasks unless explicitly overridden.

Supported values for run:

• always (default) always attempt to invoke the task regardless of the number of previous executions
• once only invoke this task once regardless of the number of references
• when_changed only invokes the task once for each unique set of variables passed into the task

---

❓ If 3 tasks running in parallel call the same task, is there a way of queueing the "same task" so that it only run consecutively? This is needed in cases like an apt-get where only one instance can run at a time.

This would be solved with the v4 serialGroups feature mentioned above. Task uniqueness is determined by it's name and vars. serialGroups only cares about the task by name when putting it into a "serial group". Thus, multiple instances of the same task, but with different vars, when they have the serialGroups field set will be serialized (queued).

[ghostsquad]

@ProfessorManhattan Sorry, task has taken a back seat for me. I've been focusing on other things recently. Additionally, @andreynering has changed some settings, so I'm not really even a maintainer anymore. I've been punished for merging a PR that broke something because no regression tests existed.

[ProfessorManhattan]

Hey @ghostsquad .. ok why not do quarterly releases or something like that? My company does biweekly releases so all the developers are walking on ice.

[ghostsquad]

It's not a release cadence that really matters. Stuff happens, bugs show up in the code. There simply needs to be a process and understanding for how to deal with it when it does happen instead expecting perfection on every merge.

[ProfessorManhattan]

Yeah.. I'm totally with you.. I personally think the master branch should be looked at like a beta branch and the release channels like Homebrew should be powered by a secondary branch.. not sure how your CI works. Still forming an opinion about it though, personally.

[ghostsquad]

The way I treat continuous deployment/delivery is that main/master is "what you want to go to production", but it must first pass all manual/automated tests/gates before that happens. The best way in git to then track things is with tags. It's more or less "GithubFlow" with a mix of "OneFlow". The mix happens when a hotfix is needed in the live running production environment. Only when a hotfix is needed, would you branch off of the current tagged commit, commit the hotfix and retag the new commit, followed by merging that back into main.

Here's a brief article for further reading.
https://betterprogramming.pub/a-simple-git-repository-strategy-93a0c7450f23

Maybe I should write a blog post about this. This makes the most sense for deployable applications, but generally speaking, it can work with tools & libraries too. Creating a tag indicates a release in some form. That release could be a beta, prerelease, or release candidate. Which could then be tagged again if no additional work is needed to indicate that it's been tested/vetted.

The automated CI/CD process could then be watching for tags, and acting on the tag based on semver semantics.

---

A rollback can be handled in various ways. For something like this repo (i.e. tool/library), either "rolling forward" via a new commit that reverts a change, or a commit that fixes the problem are all that really needs to be done.