chore(file domain): add network.DownloadFile and use go-getter to download files #760
+253
−94
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new method,
network.DownloadFile
, which users go-getter to download files. It's currently in use by the file domain. Go-getter's Get supports local files, network downloads (network.Fetch
also already handles that!) and also a number of protocols such as ftp, s3, git, etc.I limited the functionality to downloading INDIVIDUAL FILES and not full directories and repositories for now, since it would take more testing and a slightly larger refactor of the file domain. (I have an idea for that, but wanted to get this PR dealt with first). A future PR will add
network.Download
for directories, git repos, etc. (This will require an additional field in the file spec,source
or something similar. Users will still need to list the files that the validations run against).I added one test that pulls a file from our git repo to the e2e tests. There is a comment in the file warning future-us of that, but it's still worth mentioning that will break and annoy us some day (and be easily fixed, but still. Annoying).
There are some other changes in the e2e test files - It was getting hard to find the actual test output, so I removed all the calls to message.Info. I also noticed I had randomly started using assert where I wanted require in the files domain e2e tests, so that's fixed too 🙄
...
Related Issue
Relates to #693
Type of change
more of a precursor to a new feature, but close enough?