Duplicated vault namespace - vault_auth_backend v3.16.0

[arcezd]

Hi there,

We started facing an issue using the vault terraform provider 3.16.0, where the vault namespace seems to be duplicated at the vault API call, maybe it's related to concatenating the vault namespace used by the login token and the namespace specified at the vault resource.

Terraform Version

v1.5.0

Affected Resource(s)

Please list the resources as a list, for example:

• vault_auth_backend

This may be affecting other resources.

Terraform Configuration Files

variable "vault_namespace" {
  type        = string
  description = "Vault namespace"
}

variable "k8s_api_server" {
  type        = string
  description = "Kubernetes api server url"
}

variable "k8s_ca_cert" {
  type        = string
  description = "Kubernetes api server ca cert"
}

variable "k8s_vault_injector_token" {
  type        = string
  description = "Kubernetes vault injector token"
}

variable "k8s_issuer" {
  default     = "https://kubernetes.default.svc.cluster.local"
  type        = string
  description = "Kubernetes issuer"
}

variable "k8s_disable_iss_validation" {
  default     = "true"
  type        = string
  description = "Kubernetes disable issuer validation"
}

resource "vault_kubernetes_auth_backend_config" "k8s" {
  namespace = var.vault_namespace

  backend                = vault_auth_backend.k8s.path
  kubernetes_host        = var.k8s_api_server
  kubernetes_ca_cert     = var.k8s_ca_cert
  token_reviewer_jwt     = var.k8s_vault_injector_token
  issuer                 = var.k8s_issuer
  disable_iss_validation = var.k8s_disable_iss_validation
}

Debug Output

╷
│ Error: error reading from Vault: Error making API request.
│ 
│ Namespace: acc1/aws/namespace-01/acc1/aws/namespace-01
│ URL: GET https://vault-endpoint/v1/sys/auth
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│ 	* permission denied
│ 
│ 
│ 
│   with module.vault_k8s_auth.vault_auth_backend.k8s,
│   on ../../modules/vault-k8s-auth/main.tf line 1, in resource "vault_auth_backend" "k8s":
│    1: resource "vault_auth_backend" "k8s" {
│ 
╵

Panic Output

Expected Behavior

What should have happened?
The GET https://vault-endpoint/v1/sys/auth call should succeed with the proper namespace: acc1/aws/namespace-01.

Actual Behavior

What actually happened?
The GET https://vault-endpoint/v1/sys/auth call fails showing a duplicated namespace: acc1/aws/namespace-01/acc1/aws/namespace-01.

Steps to Reproduce

We're executing this using a GitHub actions workflow, but it seems like:

1. Get the Vault token using the GitHub JWT.

github-workflow.yaml
    - id: github-oidc-token
        name: Get GitHub OIDC Token
        uses: actions/github-script@v6
        with:
          script: |
            const coredemo = require('@actions/core')
            let id_token = await coredemo.getIDToken()
            coredemo.setOutput('gh_id_token', id_token)
      - id: vault-login
        name: Login to Vault
        run: |
          vault --version
          export VAULT_TOKEN=$(vault write -field=token ${{ vars.VAULT_AUTH_PATH }} role=${{ vars.VAULT_AUTH_ROLE }} jwt=${GITHUB_OIDC_JWT_TOKEN})
          vault login -no-store -field=policies $VAULT_TOKEN
          vault status
          echo "TOKEN=$(echo $VAULT_TOKEN)" >> $GITHUB_OUTPUT
        env:
          GITHUB_OIDC_JWT_TOKEN: ${{ steps.github-oidc-token.outputs.gh_id_token }}
          VAULT_NAMESPACE: ${{ vars.VAULT_NAMESPACE }}

2. terraform plan

Important Factoids

Running in GitHub Workflow

References

[VitaliyZasada]

Confirmed. We started to receive permission denied error for vault_database_secret_backend_connection resource in 3.16.0 version, so downgraded to 3.15.2

[Xavier59]

Yes, same problem here.

We have the following configuration:

provider "vault" {
  address = var.vault_addr
  auth_login {
    path      = "auth/approle/login"
    namespace = "admin"

    parameters = {
      role_id   = var.role_id
      secret_id = var.secret_id
    }
  }
}

Request should have header X-Vault-Namespace: admin but has header X-Vault-Namespace: admin/admin
We decided to downgrade for now.

Could it be coming from #1841 ? @benashz

[Xavier59]

Investigated a bit ...
This is present in the logs:

[INFO] provider.terraform-provider-vault_v3.16.0_x5: 2023/06/21 14:58:46 [WARN] The provider namespace should be set whenever using namespaced auth tokens. You may want to update your provider configuration's namespace to be "admin", before executing terraform. Future releases may not support this type of configuration

So up to

terraform-provider-vault/internal/provider/meta.go

Lines 295 to 311 in 31207b8

	if namespace == "" && tokenNamespace != "" {
	// set the provider namespace to the token's namespace
	// this is here to ensure that do not break any configurations that are relying on the
	// token's namespace being used during resource provisioning.
	// In the future we should drop support for this behaviour.
	log.Printf("[WARN] The provider namespace should be set whenever "+
	"using namespaced auth tokens. You may want to update your provider "+
	"configuration's namespace to be %q, before executing terraform. "+
	"Future releases may not support this type of configuration.", tokenNamespace)
	namespace = tokenNamespace
	// set the namespace on the provider to ensure that all child
	// namespace paths are properly honoured.
	if err := d.Set(consts.FieldNamespace, namespace); err != nil {
	return nil, err
	}
	}

The namespace seems to be correct (i.e not admin/admin).
I'm not sure however why the warning pop up since the namespace is specified in my configuration.

So this my not work as expected ?

terraform-provider-vault/internal/provider/meta.go

Line 227 in 31207b8

namespace := d.Get(consts.FieldNamespace).(string)

[iniinikoski]

Yes, same problem here.

We have the following configuration:

provider "vault" {
  address = var.vault_addr
  auth_login {
    path      = "auth/approle/login"
    namespace = "admin"

    parameters = {
      role_id   = var.role_id
      secret_id = var.secret_id
    }
  }
}

Request should have header X-Vault-Namespace: admin but has header X-Vault-Namespace: admin/admin We decided to downgrade for now.

Could it be coming from #1841 ? @benashz

Same issue here. Following-up, decided to stay in 3.15.0 for now.

[Sartigan]

While upgrading from 3.15.2 to 3.16.0... I suffer from a similar issue but not exactly in the same way.

Our app_role is located at the root level and we define the namespace in the provider configurations only (not in the terraform resources) And It looks like with the new version of the provider, it adds the namespace in the login request; probably due to the concatenation issue described by the others above. Removing the namespace from the configuration seems to resolve the invalid role ID issue... but that's not the intended way.

│ Error: Error making API request.
│ 
│ Namespace: mynamespace
│ URL: PUT https://vault-endpoint/v1/auth/approle/login
│ Code: 400. Errors:
│ 
│ * invalid role ID
│ 
│   with provider["registry.terraform.io/hashicorp/vault"].cluster,
│   on providers.tf line 73, in provider "vault":
│   73: provider "vault" {

With the following provider configurations:

provider "vault" {
  address = var.vault_endpoint
  namespace = var.vault_namespace
  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id   = var.approle_id
      secret_id = var.approle_secret
    }
  }
}

But by removing the namespace from the provider configurations and adding it to the terraform resources instead, the plan won't work as it will try to check in the root and not in the newly defined namespace in the resource which was previously in the provider configuration.

│ Error: error reading from Vault: Error making API request.
│ 
│ URL: GET https://my-vault.com/v1/sys/auth
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│ 	* permission denied
│ 
╵
╷
│ Error: error reading from Vault: Error making API request.
│ 
│ URL: GET https://my-vault.com/v1/sys/policies/acl/my-policy
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│ 	* permission denied
│

Something is definitely wrong with the namespace starting from 3.16.0 and most likely due to #1841

We can't upgrade, we have to stay on version 3.15.2 for now...

[Callek]

Casual bump to @benashz (who might pull in others) - This has blocked me updating for a while now, and I start to get concerned when we near a year with no plausible update paths for critical infra like this.

[iniinikoski]

Thanks for raising this @Callek. We'll also open a support ticket on this one soon if the comments stay quiet as this is a true blocker.

[iniinikoski]

Has anyone found any new information on this...? I bumped into this when rewriting our Vault modules (again) and this is somewhat very annoying. I'll probably raise this to Hashicorp's support next week (as we're are customers)... It's especially nasty when they fixed the lazy-loading of provider configs in version 3.23.0 but we cannot bump this provider version into production as it'd create the namespaces on root-level :/

[iniinikoski]

Replying to myself here: we've discovered the issue, probably.

Version 3.22.0 brough this option into the provider: https://github.com/hashicorp/terraform-provider-vault/blob/main/CHANGELOG.md#3220-nov-1-2023 - and - Hashicorp decided to make this to default to true. I believe this has broken many setups trying to use newer versions. We had to set this to false everywhere and then we were able to bump the provider to all the way to latest version 4 (4.4.0 if I remember correctly atm).

Maybe things got broken in https://github.com/hashicorp/terraform-provider-vault/blob/main/CHANGELOG.md#3210-oct-9-2023 when another bug was fixed (see "Fix failure when auth_login is specified and vault token is picked up from the runtime/execution environment"). Soo, you fix a bug and break things elsewhere...