Pod Identity Association not recognised by secrets store CSI driver #300

ecole-startupcraft · 2023-12-13T10:33:04Z

Describe the bug
Attempting to mount secrets using Pod Identity EKS add-on results in the following error:

Warning FailedMount 103s (x29 over 44m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/secrets-store-inline, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account demo-app-pod-sa (namespace: default)
To Reproduce

Steps to reproduce the behavior:

Create EKS Cluster
Install Pod Identity add-on
Create service account
Associate service account with IAM role using EKS Pod Identity Association
Deploy pod:

      - kind: Pod
        apiVersion: v1
        metadata:
          name: secrets-store-inline
        spec:
          serviceAccountName: demo-pod-sa
          containers:
          - image: registry.k8s.io/e2e-test-images/busybox:1.29-4
            name: busybox
            imagePullPolicy: IfNotPresent
            command:
            - "/bin/sleep"
            - "10000"
            volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          volumes:
            - name: secrets-store-inline
              csi:
                driver: secrets-store.csi.k8s.io
                readOnly: true
                volumeAttributes:
                  secretProviderClass: "aws-secrets"

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Yes/No
AWS EKS specific

Expected behavior
Secrets Volume properly mounted

Environment:
Helm chart secrets-store-csi-driver-provider-aws v0.3.5

The text was updated successfully, but these errors were encountered:

gustavclausen · 2023-12-14T09:20:08Z

This is must likely due to the provider using an older version of the AWS SDK (1.47.10) which doesn't support the container credential provider (supporting the EKS Pod Identity functionality). The minimum required SDK version is v1.47.11 (see https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html). Thus, this is also a thing to consider.

ran2806 · 2023-12-20T13:22:45Z

I am also facing the same issue. Is there any solution on it?

egorksv · 2023-12-21T13:21:33Z

I am also facing the same issue. Is there any solution on it?

As a workaround I'm using good old OIDC-mapped IAM role, but would definitely want to move on to the new way.

steven-so · 2024-01-05T22:38:27Z

I am also facing the same issue. Is there any solution on it?

As a workaround I'm using good old OIDC-mapped IAM role, but would definitely want to move on to the new way.

What role does the service account need to "MountVolume.SetUp failed for volume 'secrets-store-inline'"?

steven-so · 2024-01-05T23:20:16Z

I'm also getting:
Warning FailedMount 6s (x6 over 22s) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/user-service-depl-858474494c-5nr6w, err: rpc error: code = Unknown desc = us-west-2: An IAM role must be associated with service account default (namespace: default)

however I'm not using Pod Identity add-on.

I'm trying to spin up a private docker hub image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
    spec:
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"
      containers:
        - name: my-service
          image: myprivate/dockerimage
          env:
            - name: JWT_KEY
              valueFrom:
                secretKeyRef:
                  name: jwt-secret
                  key: JWT_KEY
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      imagePullSecrets:
        - name: "docker-hub"

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  secretObjects:
    - secretName: jwt-secret
      type: Opaque
      data:
        - objectName: jwt-secret
          key: JWT_KEY
    - secretName: docker-hub
      type: kubernetes.io/dockerconfigjson
      data:
        - objectName: "docker-configjson"
          key: ".dockerconfigjson"
  parameters:
    region: us-west-2
    objects: |
      - objectName: "jwt-secret"
        objectType: "secretsmanager"
      - objectName: "docker-configjson"
        objectType: "secretsmanager"

jbct · 2024-02-05T19:35:54Z

Thank you for the report, we will look into this issue.

kevarr · 2024-02-09T21:18:26Z

To offer a potential use-case: I'd like to use a cross-account IAM role to centralize permission management to a SecretsManager secret, use EKS Pod Identities to allow a Pod running in one account (Account A) to assume the centralized role in another account (Account B), and mount the secret into the Pod using secrets-store-csi-driver-provider-aws.

With EKS Pod Identities you can only associate roles that are in the same AWS account as the cluster. The documentation states that to achieve cross-account access you should use role chaining[1]. This is in contrast to IRSA which allows you to directly assume a role in another account [2].

If I were to use the provider to mount the secret into my Pod it would need to assume the role in Account B using the credentials of Account A before making the GetSecretValue API request.

[1] https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
[2] https://aws.amazon.com/blogs/containers/cross-account-iam-roles-for-kubernetes-service-accounts/

dprangnell · 2024-02-14T21:06:28Z

I haven't tested this, but it's almost definitely resolved in the latest release 0.3.6 which deploys v1.49.19 of the SDK.

dprangnell · 2024-02-17T21:04:04Z

Well I spoke too soon, I tested it and it doesn't work with release 0.3.6:

  Warning  FailedMount  40s (x11 over 6m52s)  kubelet MountVolume.SetUp failed for volume "secret-from-secret-manager" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/demo-app, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account pod-identity (namespace: default)

helm list -n kube-system      
NAME                        	NAMESPACE  	REVISION	UPDATED                             	STATUS  	CHART                                      	APP VERSION
csi-secrets-store           	kube-system	1       	2024-02-17 10:52:16.835931 -0800 PST	deployed	secrets-store-csi-driver-1.4.1             	1.4.1      
secrets-provider-aws        	kube-system	1       	2024-02-17 10:52:34.524537 -0800 PST	deployed	secrets-store-csi-driver-provider-aws-0.3.6

    spec:
      serviceAccountName: pod-identity
      volumes:
        - name: secret-from-secret-manager
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"
      containers:
      - name: demo-app
        image: "example/demo-app:latest"
        volumeMounts:
        - name: secrets-from-secret-manager
          mountPath: "/mnt/secrets-store"
          readOnly: true
        imagePullPolicy: Always

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:{{.Values.region}}:{{.Values.accountid}}:secret:{{.Values.secretname}}"

Fatma-J · 2024-02-29T11:15:33Z

@dprangnell Same for me.
Have you succeeded in fixing the issue?

giedriuskilcauskas · 2024-03-08T07:49:31Z

That specific check for annotation is here: https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/auth/auth.go#L110-L126
And I think there is no way to workaround it with parameters without code change.

DanielMcAssey · 2024-03-13T15:09:25Z

Hitting this issue too, unfortunately OIDC is a pain with its limitations on roles

jbct · 2024-03-14T20:47:07Z

Thank you for the request. The EKS POD Identities page calls out incompatibility with other CSI storage drivers and we are working to get documentation updated to include the Secrets Manager and Config Provider for Secret Store. We have this in our backlog and have marked it as a future enhancement.

DanielMcAssey · 2024-03-14T22:40:27Z

I have secrets manager manually installed, that page indicates that it should work, but I am hitting the same issue as the above

yambottle · 2024-03-19T22:28:51Z

System Info

k8s Version: v1.28.6, created by kOps with EC2 instances(not EKS)
cilium: quay.io/cilium/operator:v1.13.10
pod-identity-webhook: amazon/amazon-eks-pod-identity-webhook:v0.4.0
csi-secrets-store: 1.4.2
secrets-store-csi-driver-provider-aws: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-58-g4ddce6a-2024.01.31.21.42 from doc

Setup Steps

kOps created cluster already existis before this test, it has been created with Service Account Issuer Discovery and AWS IAM Roles for Service Accounts (IRSA)
Installed csi-secrets-store and secrets-store-csi-driver-provider-aws following the doc
Use kOps to provision IAM role and policy for service account:

spec:
  iam:
    serviceAccountExternalPermissions:
      - name: default-secret-manager
        namespace: default
        aws:
          inlinePolicy: |-
            [
              {
                "Effect": "Allow",
                "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
                "Resource": ["arn:*:secretsmanager:*:*:secret:*"]
              }
            ]

Create SA default-secret-manager
Create SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    region: xxxx
    objects:  |
      - objectName: "xxxxx"
        objectType: "secretsmanager"

Create Test Pod:

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  serviceAccount: default-secret-manager
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command: ["/bin/bash", "-c", "tail -f /dev/null"]
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: test-secret
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: test-secret
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"
  restartPolicy: Always

Got pod stuck on ContainerCreating with error: MountVolume.SetUp failed for volume "test-secret" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/aws-cli, err: rpc error: code = Unknown desc = xxxx: An IAM role must be associated with service account default-secret-manager (namespace: default)
Tried to attach SecretsManagerReadWrite policy, got the same error
If I manually attach AWSS3ReadOnly policy to the kOps provisioned role, and comment out the volume mount

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  serviceAccount: default-secret-manager
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command: ["/bin/bash", "-c", "tail -f /dev/null"]
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

I'm able to access the pre-created secret with the same service account

bash-4.2# aws secretsmanager get-secret-value --secret-id xxxxx
{
    "ARN": "arn:aws:secretsmanager:xxx:xxx:secret:xxx",
    "Name": "xxxx",
    "VersionId": "xxxxx",
    "SecretString": "xxxxxx,
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-03-19T21:00:25.681000+00:00"
}

Hope this info could help to resolve this issue sooner!

giedriuskilcauskas · 2024-03-27T07:58:48Z

Create SA default-secret-manager

@yambottle you need to put annotation on SA with the role created in previous step.

AnhQKatalon · 2024-05-10T07:36:10Z

I am currently having the exact same issue. This prevents us from migrating to use Pod Identity because all of our services are using Secret CSI to pull the secrets from Secrets Manager

swt-yoromero · 2024-06-14T13:52:42Z

Hola a todos!!!. Me paso lo mismo, tengo EKS 1.30, y lo solucione agregandole permisos al rol ya que faltaban permisos para los recursos secrets, este era el error que me daba encontrado en los logs del pod:
kubectl logs -n kube-system secrets-store-csi-driver-45w8a

secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I0613 20:17:17.679272       1 reflector.go:424] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
E0613 20:17:17.679334       1 reflector.go:140] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I0613 20:18:00.577364       1 reflector.go:424] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"

luego de agregar estos permisos [list, watch, get] me dio este error:

I0613 20:46:34.691739       1 secretproviderclasspodstatus_controller.go:342] "The secret operation failed with forbidden error. If you installed the CSI driver using helm, ensure syncSecret.enabled=true is set.\n"
E0613 20:46:34.694549       1 secretproviderclasspodstatus_controller.go:338] "failed to create Kubernetes secret" err="secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot create resource \"secrets\" in API group \"\" in the namespace \"fincuo\"" spc="fincuo/api-funded-natives-secrets" pod="fincuo/api-funded-natives-5df4c46dc7-zgm74" secret="fincuo/api-funded-natives-secrets" spcps="fincuo/api-funded-natives-5df4c46dc7-zgm74-fincuo-api-funded-natives-secrets"

y lo que agregue al ClusterRole quedo fue:

- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

quedando asi el recurso:

# Source: secrets-store-csi-driver/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: secretproviderclasses-role
 labels:
   app.kubernetes.io/instance: "secrets-store-csi-driver"
   app.kubernetes.io/managed-by: "Helm"
   app.kubernetes.io/name: "secrets-store-csi-driver"
   app.kubernetes.io/version: "1.4.3"
   app: secrets-store-csi-driver
   helm.sh/chart: "secrets-store-csi-driver-1.4.3"
rules:
- apiGroups:
 - ""
 resources:
 - events
 verbs:
 - create
 - patch
- apiGroups:
 - ""
 resources:
 - pods
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - secrets-store.csi.x-k8s.io
 resources:
 - secretproviderclasses
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - secrets-store.csi.x-k8s.io
 resources:
 - secretproviderclasspodstatuses
 verbs:
 - create
 - delete
 - get
 - list
 - patch
 - update
 - watch
- apiGroups:
 - secrets-store.csi.x-k8s.io
 resources:
 - secretproviderclasspodstatuses/status
 verbs:
 - get
 - patch
 - update
- apiGroups:
 - storage.k8s.io
 resourceNames:
 - secrets-store.csi.k8s.io
 resources:
 - csidrivers
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - ""
 resources:
 - secrets
 verbs:
 - create
 - delete
 - get
 - list
 - patch
 - update
 - watch

mohamed-jawad-etg · 2024-06-19T13:15:33Z

facing the same issue, can't find away around and I don't want to use OIDC

jgrigg · 2024-06-24T04:46:54Z

I am also very interested in this. We have adopted Pod Identities for general cross-account AWS access but are having to resort to IRSA for mounted secrets (via ESO) which is yuk.

Worth noting that we are also using role chaining (via profile/ini config). Ideally the CSI driver would allow configuration of a role to assume/chain in such an 'external' account.

psivananda · 2024-07-12T04:04:46Z

modified code and it's working for me for both IRSA and PIA

To make it work for PIA

we need to update helm chart to add cluster name as ENV
build helm chart, docker image and push to your registries
Remove eks.amazonaws.com/role-arn annotations from serviceaccount

psivananda@8eb8b55

Mohammad9227 · 2024-09-08T05:52:48Z

@psivananda Can you please share an update on this. The PR is still open
#376

VarunNagesh · 2024-09-12T10:59:39Z

@yambottle , was the issue fixed.
If so, what annotation did you add in service-account to mitigate the issue?
Can I get the service-account's manifest?
cc: @giedriuskilcauskas

yambottle · 2024-09-12T15:46:45Z

@yambottle , was the issue fixed. If so, what annotation did you add in service-account to mitigate the issue? Can I get the service-account's manifest? cc: @giedriuskilcauskas

Sorry, I moved on to External Secret Operator since that error. It's pretty straight forward to setup, not sure if it's an alternative to you, but I do prefer secrets-store-csi-driver-provider-aws.

Correct me if I'm wrong, I guess @giedriuskilcauskas is suggesting to add an annotation at this end of doc which I guess it is EKS specific(I have mentioned I'm not using EKS, I have a kOps+EC2 setup)

Name:                my-service-account
Namespace:           default
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role

but it's just my guess, this annotation might not be EKS specific, I just haven't test it out yet. Feel free to try it out!

maati55 · 2024-09-24T15:38:39Z

We are facing the same issue, any update on this?

secrets-store-csi-driver: 1.4.5
secrets-store-csi-driver-provider-aws: 0.3.9

simonmarty · 2024-09-24T23:11:09Z

We're actively working on supporting Pod Identity Association in the AWS provider.

ecole-startupcraft added the bug Something isn't working label Dec 13, 2023

simonmarty linked a pull request Jul 16, 2024 that will close this issue

Add support for Pod Identity Association #376

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pod Identity Association not recognised by secrets store CSI driver #300

Pod Identity Association not recognised by secrets store CSI driver #300

ecole-startupcraft commented Dec 13, 2023

gustavclausen commented Dec 14, 2023 •

edited

Loading

ran2806 commented Dec 20, 2023

egorksv commented Dec 21, 2023

steven-so commented Jan 5, 2024

steven-so commented Jan 5, 2024

jbct commented Feb 5, 2024

kevarr commented Feb 9, 2024 •

edited

Loading

dprangnell commented Feb 14, 2024

dprangnell commented Feb 17, 2024

Fatma-J commented Feb 29, 2024

giedriuskilcauskas commented Mar 8, 2024

DanielMcAssey commented Mar 13, 2024

jbct commented Mar 14, 2024

DanielMcAssey commented Mar 14, 2024

yambottle commented Mar 19, 2024 •

edited

Loading

giedriuskilcauskas commented Mar 27, 2024

AnhQKatalon commented May 10, 2024

swt-yoromero commented Jun 14, 2024

mohamed-jawad-etg commented Jun 19, 2024

jgrigg commented Jun 24, 2024 •

edited

Loading

psivananda commented Jul 12, 2024 •

edited

Loading

Mohammad9227 commented Sep 8, 2024

VarunNagesh commented Sep 12, 2024 •

edited

Loading

yambottle commented Sep 12, 2024 •

edited

Loading

maati55 commented Sep 24, 2024

simonmarty commented Sep 24, 2024

Pod Identity Association not recognised by secrets store CSI driver #300

Pod Identity Association not recognised by secrets store CSI driver #300

Comments

ecole-startupcraft commented Dec 13, 2023

gustavclausen commented Dec 14, 2023 • edited Loading

ran2806 commented Dec 20, 2023

egorksv commented Dec 21, 2023

steven-so commented Jan 5, 2024

steven-so commented Jan 5, 2024

jbct commented Feb 5, 2024

kevarr commented Feb 9, 2024 • edited Loading

dprangnell commented Feb 14, 2024

dprangnell commented Feb 17, 2024

Fatma-J commented Feb 29, 2024

giedriuskilcauskas commented Mar 8, 2024

DanielMcAssey commented Mar 13, 2024

jbct commented Mar 14, 2024

DanielMcAssey commented Mar 14, 2024

yambottle commented Mar 19, 2024 • edited Loading

System Info

Setup Steps

giedriuskilcauskas commented Mar 27, 2024

AnhQKatalon commented May 10, 2024

swt-yoromero commented Jun 14, 2024

mohamed-jawad-etg commented Jun 19, 2024

jgrigg commented Jun 24, 2024 • edited Loading

psivananda commented Jul 12, 2024 • edited Loading

Mohammad9227 commented Sep 8, 2024

VarunNagesh commented Sep 12, 2024 • edited Loading

yambottle commented Sep 12, 2024 • edited Loading

maati55 commented Sep 24, 2024

simonmarty commented Sep 24, 2024

gustavclausen commented Dec 14, 2023 •

edited

Loading

kevarr commented Feb 9, 2024 •

edited

Loading

yambottle commented Mar 19, 2024 •

edited

Loading

jgrigg commented Jun 24, 2024 •

edited

Loading

psivananda commented Jul 12, 2024 •

edited

Loading

VarunNagesh commented Sep 12, 2024 •

edited

Loading

yambottle commented Sep 12, 2024 •

edited

Loading