How to proxy a detached commit version in a Github public repo?

[sean822]

Hi, i recently used Athens to proxy Go dependencies in my CI & CD pipelines, and i met a problem that Athens can not proxy a detached commit version in a Github public repo. After a lot of searching and reading the document, I think i need to ask for help here.

The goal i use Athens

I have a lot of pipeline to run , and i want to speed up the build, so i want Athens to proxy all of my go dependencies, both public and private repo.

The error log while execute go build

go: github.com/my-github-tem/my-private-repo-name/v3@v3.9.8 requires
	github.com/hairyhenderson/toml@v0.3.1-0.20191004034452-2a4f3b6160f2: reading http://localhost:3000/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.mod: 404 Not Found

My Athens config and go command i execute

Athens config

- ATHENS_STORAGE_TYPE=disk
- ATHENS_DISK_STORAGE_ROOT=/var/lib/athens
- ATHENS_GONOSUM_PATTERNS=github.com/my-github-team/*
- ATHENS_GITHUB_TOKEN=xxx

Go command i execte

GOPROXY=http://localhost:3000 GONOSUMDB=github.com/my-github-team/* CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build 

What i have tried to solve this

Confirm if version v0.3.1-0.20191004034452-2a4f3b6160f2 is vaild

After a lot of investigations, i found that the version v0.3.1-0.20191004034452-2a4f3b6160f2 above is a detached commit . Github tell me 'This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository' When i visit https://github.com/hairyhenderson/toml/tree/2a4f3b6160f2, but it really exist.
I tried to clone the whole repo hairyhenderson/toml and git checkout 2a4f3b6160f2, Git tell me [2a4f3b6160f2](error: pathspec '2a4f3b6160f2' did not match any file(s) known to git)

Confirm if no proxy is ok

Just execute go build -x and it works fine.

...
go: downloading github.com/hairyhenderson/toml v0.3.1-0.20191004034452-2a4f3b6160f2
# get https://proxy.golang.org/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.zip
# get https://proxy.golang.org/github.com/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.info: 404 Not Found (0.112s)
# get https://proxy.golang.org/github.com/hairyhenderson/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.info: 404 Not Found (0.112s)
# get https://proxy.golang.org/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.zip: 200 OK (0.068s)
...

Append https://proxy.golang.org as another proxy after Athens

Execute GOPROXY=http://localhost:3000,https://proxy.golang.org GONOSUMDB=github.com/my-github-team/* CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build and it works fine

Read the document

I read the page https://docs.gomods.io/configuration/download/, it said

sync: Synchronously download the module from VCS via go mod download, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior.

I think it should behave like i execute go mod download locally, but actually, it didn't

Search for potential answer

I tried to search same problem by google, but found no same

My Question

Did i misunderstand something? what should i do to solve this problem ?

[matt0x6F]

The simple answer here is that Athens does whatever go mod download -json <url> does. So whatever the Go tooling is capable of, Athens is capable of.

[sean822]

1. if i run go mod download -json github.com/hairyhenderson/toml@v0.3.1-0.20191004034452-2a4f3b6160f2 with default go env , i get output like below:

{
	"Path": "github.com/hairyhenderson/toml",
	"Version": "v0.3.1-0.20191004034452-2a4f3b6160f2",
	"Info": "/Users/lollipop/go/pkg/mod/cache/download/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.info",
	"GoMod": "/Users/lollipop/go/pkg/mod/cache/download/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.mod",
	"Zip": "/Users/lollipop/go/pkg/mod/cache/download/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.zip",
	"Dir": "/Users/lollipop/go/pkg/mod/github.com/hairyhenderson/toml@v0.3.1-0.20191004034452-2a4f3b6160f2",
	"Sum": "h1:Dc4YWWuY02jqhCnErAH++juCTwEPLstAOOVhyPXeE7Q=",
	"GoModSum": "h1:Qzp4VYCUWsSh/aiz4pJGa+R7aXrGdepyDeZP4+9kToA="
}

2. but if i run GOPROXY=http://localhost:3000 go mod download -json github.com/hairyhenderson/toml@v0.3.1-0.20191004034452-2a4f3b6160f2, i get output like this:

{
	"Path": "github.com/hairyhenderson/toml",
	"Version": "v0.3.1-0.20191004034452-2a4f3b6160f2",
	"Error": "github.com/hairyhenderson/toml@v0.3.1-0.20191004034452-2a4f3b6160f2: reading http://localhost:3000/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.info: 404 Not Found"
}

and i check the athens container log ,i find this:

INFO[9:44AM]: github.com/hairyhenderson/toml@v0.3.1-0.20191004034452-2a4f3b6160f2: invalid version: unknown revision 2a4f3b6160f2	http-method=GET http-path=/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.info kind=Not Found module=github.com/hairyhenderson/toml operation=download.InfoHandler ops=[download.InfoHandler pool.Info protocol.Info protocol.processDownload stash.Pool stasher.Stash stasher.fetchModule goGetFetcher.Fetch module.downloadModule] request-id=1734f091-6568-4003-b46c-778a4c3ddb36 version=v0.3.1-0.20191004034452-2a4f3b6160f2 
INFO[9:44AM]: incoming request	http-method=GET http-path=/github.com/hairyhenderson/toml/@v/v0.3.1-0.20191004034452-2a4f3b6160f2.info http-status=404 request-id=1734f091-6568-4003-b46c-778a4c3ddb36

3. do i misunderstand your meaning? should't step 1 and step 2 has same output ? by the way, i clean the $(go env GOPATH)/pkg/ before each step to avoid the cache affect the result .

[sean822]

I read the config.dev.toml again, and i found the environment GoBinaryEnvVars and i know why.I should override the default value GoBinaryEnvVars = ["GOPROXY=direct"] with ATHENS_GO_BINARY_ENV_VARS='GOPROXY=proxy.golang.org,direct;
Thanks a lot for your help !