OICD login stopped working (at least for Auth0)

[NickLarsenNZ]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I have ArgoCD configured for OICD login via Auth0. This has been working fine, but today it stopped working. After a lot of troubleshooting, I noticed the image version had updated. I have a cluster that is working and using an older image:

• a17d6af8dc8461e1dc637d99c651b07e083e7df5f92bc2994d0b0cf74929b961 never redirects to Auth0.
• 3115fcc3a25357c05185802568f15fdc556dc3eab38a1fea11aae8cbd5e763ba works, although the image is not listed.

To Reproduce

A list of the steps required to reproduce the issue. Best of all, give us the URL to a repository that exhibits this issue.

kustomization.yml

resources:
- https://raw.githubusercontent.com/argoproj/argo-cd/8b57bc990cd72c6475e2ae88126a1ed57e2460a3/manifests/ha/install.yaml # was v2.2.1,  but excluded the notifications controller
- ./ingress.yml
- ./grpc-service.yml

patches:
- https://raw.githubusercontent.com/argoproj/argo-cd/8b57bc990cd72c6475e2ae88126a1ed57e2460a3/notifications_catalog/install.yaml # replace hash with version once it's included

oidc.config

name: Auth0
issuer: https://my.auth0.com/
clientID: ABCD....EFG
clientSecret: $argocd-oidc:auth0.clientSecret
requestedScopes:
- openid
- profile
- email

Then open the UI, and open the browser's Network Inspector, click LOG IN VIA AUTH0. You'll notice you are never redirected away from the ArgoCD URL, and end up straight back at the login page.

{"error":"no session information","code":16,"message":"no session information"}

Expected behavior

Clicking the LOG IN VIA AUTH0 button should redirect to Auth0, and on successful consent/token issue, redirect back to /auth/callback.

Screenshots

Version

argocd: v2.3.0+988d760
  BuildDate: 2022-01-19T07:28:56Z
  GitCommit: 988d7604743339da02983317b789a6cc9072ae75
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified    

Logs

argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.claims=null grpc.request.content= grpc.service=version.VersionService grpc.start_time="2022-01-19T15:12:25Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2022-01-19T15:12:25Z" grpc.time_ms=35.927 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:25Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="Ignore status for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:25Z" grpc.time_ms=10.82 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:25Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:25Z" grpc.time_ms=7.56 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:25Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2022-01-19T15:12:25Z" grpc.time_ms=4.83 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:26Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:26Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:26Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:26Z" grpc.time_ms=7.152 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:26Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:26Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:26Z" level=info msg="Ignore status for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:26Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:26Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:26Z" grpc.time_ms=12.217 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:41Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.claims=null grpc.request.content= grpc.service=version.VersionService grpc.start_time="2022-01-19T15:12:41Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:41Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:41Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:41Z" level=info msg="Ignore status for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:41Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:41Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2022-01-19T15:12:41Z" grpc.time_ms=25.147 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:41Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:41Z" grpc.time_ms=17.455 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:42Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2022-01-19T15:12:42Z" grpc.time_ms=7.739 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:42Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:42Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:42Z" grpc.time_ms=6.441 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:42Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:42Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:42Z" grpc.time_ms=5.676 span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:43Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:43Z" span.kind=server system=grpc
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:43Z" level=info msg="Ignore status for CustomResourceDefinitions"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:43Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
argocd-server-8b499f766-prh6k time="2022-01-19T15:11:56Z" level=info msg="Starting configmap/secret informers"
argocd-server-8b499f766-z62gk time="2022-01-19T15:12:43Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:43Z" grpc.time_ms=14.273 span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:11:56Z" level=info msg="Configmap/secret informer synced"
argocd-server-8b499f766-prh6k time="2022-01-19T15:11:56Z" level=info msg="argocd v2.3.0+988d760 serving on port 8080 (url: https://argocd.dev.sydney.mu, tls: true, namespace: argocd, sso: true)"
argocd-server-8b499f766-z62gk time="2022-01-19T15:13:14Z" level=info msg="Alloc=15005 TotalAlloc=43533 Sys=73809 NumGC=11 Goroutines=154"
argocd-server-8b499f766-prh6k time="2022-01-19T15:11:56Z" level=info msg="0xc000c89a40 subscribed to settings updates"
argocd-server-8b499f766-prh6k time="2022-01-19T15:11:56Z" level=info msg="Starting rbac config informer"
argocd-server-8b499f766-prh6k time="2022-01-19T15:11:56Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:25Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:25Z" span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:25Z" level=info msg="Ignore status for CustomResourceDefinitions"
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:25Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:25Z" grpc.time_ms=20.463 span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:25Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2022-01-19T15:12:25Z" grpc.time_ms=8.368 span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:26Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:26Z" span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:26Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:26Z" grpc.time_ms=6.332 span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:41Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:41Z" span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:41Z" level=info msg="Ignore status for CustomResourceDefinitions"
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:41Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:41Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2022-01-19T15:12:41Z" grpc.time_ms=54.333 span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:42Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2022-01-19T15:12:42Z" grpc.time_ms=8.063 span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:42Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:42Z" span.kind=server system=grpc
argocd-server-8b499f766-prh6k time="2022-01-19T15:12:42Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2022-01-19T15:12:42Z" grpc.time_ms=5.133 span.kind=server system=grpc

I have pinned to a specific version, but the K8s manifests use latest. I think these should be pinned (at least to a minor version).

My workaround is to add the following to my kustomization.yml

# Began having problems with oidc login, noticed the image hash was: a17d6af8dc8461e1dc637d99c651b07e083e7df5f92bc2994d0b0cf74929b961
# This one is working: 3115fcc3a25357c05185802568f15fdc556dc3eab38a1fea11aae8cbd5e763ba but I don't see it in quay.io
images:
- name: quay.io/argoproj/argocd
  newTag: v2.2.2

[TheGeka]

Thanks, was searching like crazy to see if I did the setup wrong but reverting to 2.2.2 fixed it for me.

[NickLarsenNZ]

@TheGeka, do you also use Auth0 or another OIDC provider?

[TheGeka]

I use Keycloak as OIDC provider.

[NickLarsenNZ]

I'm thinking this is a UI bug, as I can still do SSO login via the CLI:

I spoke too soon

---

Working cluster

1. Click Login via Auth0
2. GET https://argocd.nonprod.sydney.mu/auth/login?return_url=https://argocd.nonprod.sydney.mu/applications

   303
   Location: https://my.auth0.com/authorize?client_id=xxxxxxxxxxxxxxxxxxxx&redirect_uri=https://argocd.nonprod.sydney.mu/auth/callback&response_type=code&scope=openid+profile+email&state=xxxxxxxxxx
   

Not Working cluster:

1. Click Login via Auth0
2. GET https://argocd.dev.sydney.mu/auth/login?return_url=https://argocd.dev.sydney.mu/applications

   200 OK
   

[NickLarsenNZ]

Accidentally closed via description text containing "fixed"

[alexmt]

The https://raw.githubusercontent.com/argoproj/argo-cd/8b57bc990cd72c6475e2ae88126a1ed57e2460a3/manifests/ha/install.yaml uses the latest image tag. So it pulls the most recent version of the Argo CD. We've recently migrated to golang-jwt/jwt (#8136). It might be related