Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provider socket does not exist if provider pod starts before driver pod #345

Open
mf-jhellman opened this issue Apr 26, 2024 · 2 comments · May be fixed by #346 or #384
Open

Provider socket does not exist if provider pod starts before driver pod #345

mf-jhellman opened this issue Apr 26, 2024 · 2 comments · May be fixed by #346 or #384
Labels
bug Something isn't working

Comments

@mf-jhellman
Copy link

Describe the bug
It appears that sometimes on our nodes the secrets-store-csi-driver-provider-aws pod will start before the secrets-store-csi-driver pod. When that happens, our application pods that mount AWS secrets on such a node will get stuck in ContainerCreating status and the secrets-store-csi-driver pod generates logs such as:

"failed to mount secrets store object content" err="rpc error: code = Canceled desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial unix /etc/kubernetes/secrets-store-csi-providers/aws.sock: connect: no such file or directory\"" pod="mynamespace/mypod"

The secrets-store-csi-driver-provider-aws does not generate any logs which indicate an issue.

To Reproduce

Steps to reproduce the behavior:
Start the secrets-store-csi-driver-provider-aws pod will before the secrets-store-csi-driver pod and create an application pod that mounts AWS secrets.

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Yes/No
Haven't tried

If yes, the issue is likely with the k8s Secrets Store CSI driver, not the AWS provider. Open an issue in that repo.

Expected behavior
The application pod mounts the secrets

Environment:
OS, Go version, etc.
EKS 1.29.1, secrets-store-csi-driver 1.4.2, secrets-store-csi-driver-provider-aws 1.0.r2-68-gab548b3-2024.03.20.21.58

Additional context
We noticed the secrets-store-csi-driver-provider-aws chart does not include type: DirectoryOrCreate in the providervol volume definition. Is it possible when this hostpath doesn't exist on startup that the provider won't be able to use the socket but not generate any errors?

@mf-jhellman mf-jhellman added the bug Something isn't working label Apr 26, 2024
@lobanov
Copy link

lobanov commented Jun 8, 2024

Encountered this issue too. I was migrating from YAML-based deployment of ASCP as described here to Helm Chart, and secret mounts stopped working. As a workaround switched back to using the manifest.

@simonmarty
Copy link
Contributor

One way to fix this would be to add a dependencies block to the Helm chart to ensure the CSI driver always deploys before the AWS provider. This is something we originally elected against in order to reduce the number of manual version bumps we need to conduct on this project. I think it's time we revisit this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
3 participants