Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable root remote taskfiles #1347

Merged
merged 6 commits into from
Mar 25, 2024
Merged

feat: enable root remote taskfiles #1347

merged 6 commits into from
Mar 25, 2024

Conversation

pd93
Copy link
Member

@pd93 pd93 commented Sep 23, 2023

Fixes #1531

This PR enables the ability to pass a remote Taskfile URI to the --taskfile flag. This means that your root Taskfile no longer needs to exist locally.

When specifying the --taskfile flag, Task usually assumes that the execution directory is the directory that the root Taskfile resides in. Since this isn't possible with a remote Taskfile, the execution directory will default to the current working directory instead when the Task command is executed.

However, I have also added the ability to change this default directory using the existing --dir flag. This now means that you are able to specify BOTH --taskfile and --dir flags simultaneously - Something that was not previously allowed. The following semantics are now applied:

  • If --taskfile is specified without --dir, the specified Taskfile is used and the directory will default to the directory containing the specified Taskfile (no change).
  • If --dir is specified without --taskfile, Task will search the specified directory for a Taskfile and use that as the entrypoint. The directory will be the one specified (no change).
  • If BOTH --dir and --taskfile are specified, the entrypoint will be the specified taskfile and the directory will be the specified directory.
  • If NEITHER are specified, Task will recursively search up the filesystem for a Taskfile starting in the current working directory. If one is found, the directory is set to the directory containing the found Taskfile.

Finally, this PR also fixes issues related to resolving the location of included Taskfiles inside remote Taskfiles. For example, if you are using a remote Taskfile that includes a "local" file. The included file will now be resolved as a remote file too and the include will work as expected.

@pd93 pd93 mentioned this pull request Sep 22, 2023
15 tasks
@pd93 pd93 marked this pull request as draft September 24, 2023 01:58
@wburningham
Copy link

Hi @pd93, I would love to use this feature with the other changes that are in v3.30.1. Could you merge this or rebase off of v3.30.1?

@wburningham
Copy link

@pd93 just following up again on this PR. Is there feedback (workflow usage details) or coding support I could provide to help get this work across the line?

@pd93 pd93 changed the base branch from main to stdin-node February 13, 2024 01:07
@pd93 pd93 mentioned this pull request Feb 19, 2024
Base automatically changed from stdin-node to main February 22, 2024 20:59
@pd93 pd93 added the state: wip A work in progress. label Feb 29, 2024
@pd93 pd93 marked this pull request as ready for review March 4, 2024 18:05
@pd93 pd93 removed the state: wip A work in progress. label Mar 4, 2024
@pd93
Copy link
Member Author

pd93 commented Mar 4, 2024

This PR has taken a long time to complete 😆 I did not initially realise the complexity of the changes that would be required to complete this work. Over the last couple of months or so I have spent many hours reworking this from the ground up. The functionality described in the PR description required several major changes to some core code and I wanted to make sure I'm not breaking anything for existing users.

I've spent the last couple of days testing. I suspect there will still be some issues, but the main changes are now complete and ready to be reviewed 🚀

@vmaerten @wburningham Since you've both been trying out remote Taskfiles recently, would you mind kicking the tyres on this for me? I'm pretty sure it's ready to go now. This should resolve both remote root Taskfiles and includes in remote Taskfiles.

@pd93
Copy link
Member Author

pd93 commented Mar 4, 2024

@brianbraunstein Since this PR also implements the ability to use both --taskfile and --dir flags as previously requested by you in #655 (comment), it would be great if you are also able to give this a go with regard to the new stdin reader.

@vmaerten
Copy link
Member

vmaerten commented Mar 4, 2024

@pd93 For the includes in remote Taskfiles, I've tested it with two differents setup and it works as expected
Thanks a lot 🔥

@wburningham
Copy link

I got around to testing this. I ran

task --dir . -t https://gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628/taskfile.yaml```

Where the remote taskfile looks like this:

version: 3

tasks:
default:
cmds:
- cmd: echo "CLI_ARGS = '{{.CLI_ARGS}}'"
silent: true
- cmd: echo "CLI_FORCE = '{{.CLI_FORCE}}'"
silent: true
- cmd: echo "TASK = '{{.TASK}}'"
silent: true
- cmd: echo "ROOT_TASKFILE = '{{.ROOT_TASKFILE}}'"
silent: true
- cmd: echo "ROOT_DIR = '{{.ROOT_DIR}}'"
silent: true
- cmd: echo "TASKFILE = '{{.TASKFILE}}'"
silent: true
- cmd: echo "TASKFILE_DIR = '{{.TASKFILE_DIR}}'"
silent: true
- cmd: echo "USER_WORKING_DIR = '{{.USER_WORKING_DIR}}'"
silent: true
- cmd: echo "TASK_VERSION = '{{.TASK_VERSION}}'"
silent: true


Using the command above, should `TASKFILE_DIR` be the value from `--dir` flag?

@pd93
Copy link
Member Author

pd93 commented Mar 15, 2024

Hey @wburningham. Special variables for remote Taskfiles are something I haven't worked on yet. It seems they're pretty broken right now. I will add this to the todo list in #1317 and take a look at these in a future PR.

As long as the ability to access root files remotely and to include other remote files in a remote file is working correctly then I'm happy to merge this as-is.

Using the command above, should TASKFILE_DIR be the value from --dir flag?

To answer your question directly. TASKFILE_DIR is supposed to contain the path to the directory containing the Taskfile that holds the current task. Since a remote file is, by definition, not on the same filesystem, I'd expect it to contain a URL to the remote directory containing the file.

@wburningham
Copy link

I think I found a bug related to the special variables. I can see TASKFILE_DIR being set to the remote taskfile when the taskfile is specified with the --taskfile flag. If the taskfile is references with an includes directive in a local taskfile then TASKFILE_DIR should be the local taskfile dir (I think, or at least that is how it works today).

Steps to reproduce:

❯ cat taskfile.yaml 
version: 3

includes:
  remote: https://gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628/taskfile.yaml

tasks:
  default:
    cmds: 
      - task: remote:default

❯ go install github.com/go-task/task/v3/cmd/task@v3.34.1

❯ task --yes
CLI_ARGS = ''
CLI_FORCE = 'false'
TASK = 'remote:default'
ROOT_TASKFILE = '/private/tmp/test/Taskfile.yaml'
ROOT_DIR = '/private/tmp/test'
TASKFILE = ''
TASKFILE_DIR = '/private/tmp/test'
USER_WORKING_DIR = '/private/tmp/test'
task: [remote:default] echo "TASK_VERSION = 'v3.34.1 (h1:yAAxUM54zoaHv+OtDnGgkWSVeiRuaOCn1lPUXPQQA0o=)'"
TASK_VERSION = 'v3.34.1 (h1:yAAxUM54zoaHv+OtDnGgkWSVeiRuaOCn1lPUXPQQA0o=)'

❯ go install github.com/go-task/task/v3/cmd/task@46470038b134034edf37c0926072fc094822384d

❯ task --yes
CLI_ARGS = ''
CLI_FORCE = 'false'
TASK = 'remote:default'
ROOT_TASKFILE = '/private/tmp/test'
ROOT_DIR = '/private/tmp/test'
TASKFILE = 'https://gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628/taskfile.yaml'
TASKFILE_DIR = 'https:/gist.githubusercontent.com/wburningham/86586099785bcfd4c0289dba09ae2829/raw/b3fa6add37c947b77c435cb07d7257f8f64c6628'
USER_WORKING_DIR = '/private/tmp/test'
task: [remote:default] echo "TASK_VERSION = 'v3.35.2-0.20240310214338-46470038b134 (h1:tHuecP/JTRy2ehr+s0Wo+GilpuUqtFLEHb79pCDVias=)'"
TASK_VERSION = 'v3.35.2-0.20240310214338-46470038b134 (h1:tHuecP/JTRy2ehr+s0Wo+GilpuUqtFLEHb79pCDVias=)'

Notice how TASKFILE and TASKFILE_DIR change between the two versions.

Copy link
Member

@andreynering andreynering left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

locally at all. For example:

```shell
task --taskfile https://taskfile.dev hello
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could consider having a proper Taskfile being served in the website, so copying and pasting would work 🙂

Could be a GitHub or GitHub Gist URL as well...

Suggested change
task --taskfile https://taskfile.dev hello
task --taskfile https://taskfile.dev/taskfiles/hello.yml hello

Just an idea, does not need to be done on this PR...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not entirely clear what you mean by this

We could consider having a proper Taskfile being served in the website

Is this not already complete in this PR? see https://github.com/go-task/task/pull/1347/files#diff-eea9e58f9afa47c7cdf1393b2d383a4713f91171f6603ced3937cacc09fe9376

I will merge for now and we can adjust later if need be.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, got it, it's already there in the root 🙂

It just surprised me that it automatically fetches the /Taskfile.yml path. I thought the whole path would have to be explicit...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what the RemoteExists function is doing. It's similar to what we do with local files where only a directory is given. Task will look around for a file that makes sense. I thought this would be nice for projects that want to use Taskfiles to do some setup work. e.g.

task --taskfile https://myproject.org init or something similar is a really nice, clean way of asking your devs to do run some setup task. Whereas task --taskfile https://myproject.org/Taskfile.yml init is a bit less tidy.

The same applies to the new taskfile.dev/Taskfile.yml file. If a user wants to check that their installation of Task works, they can now run task --taskfile https://taskfile.dev and it will reply with "Hello Task!". This is a bit less clunky than task --taskfile https://taskfile.dev/Taskfile.yml.

Happy to revisit this if you don't like the behaviour though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually an interesting feature, we can keep it 👍

I did quick review at the time, I so end up not paying attention to that function 🙂

I just suggest to improve the documentation a bit to explain that this is a possibility, just like for local files.

}

// Request the given URL
resp, err := http.DefaultClient.Do(req)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does these HTTP calls have a timeout by default? It would be desired IMO.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good shout! I've added the existing timeout functionality in the HTTO node.Read(...) to taskfile.RemoteExists(...)

@pd93 pd93 merged commit 64b7d34 into main Mar 25, 2024
13 checks passed
@pd93 pd93 deleted the remote-root-taskfiles branch March 25, 2024 19:13
pd93 added a commit that referenced this pull request Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Include a Taskfile in a remote Taskfile does not work
4 participants