consul_certificate_authority Vault Connect-CA #331

const-tmp · 2023-01-19T18:00:02Z

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

 terraform version
Terraform v1.3.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/consul v2.17.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/vault v3.12.0
+ provider registry.terraform.io/vultr/vultr v2.12.0

Affected Resource(s)

consul_certificate_authority

Terraform Configuration Files

# main.tf

module "connect_pki" {
  source = "../vault/pki"
  prefix = "connect"
}

resource "vault_policy" "connect-ca" {
  name   = "connect-ca"
  policy = templatefile("${path.module}/../../config/vault/policies/connect-ca.hcl", {
    root_path  = module.connect_pki.root.path
    inter_path = module.connect_pki.inter.path
  })
}

resource "vault_token" "connect-ca" {
  policies         = [vault_policy.connect-ca.name]
  renewable        = true
  explicit_max_ttl = 0
}

variable "vault_addr" {
  type = string
}

resource "consul_certificate_authority" "connect" {
  connect_provider = "vault"

  config = {
    Address             = "http://vault.service.consul:8200/"
    Token               = vault_token.connect-ca.client_token
    RootPKIPath         = module.connect_pki.root.path
    IntermediatePKIPath = module.connect_pki.inter.path
  }
}


# ../vault/pki/main.tf

variable "prefix" {
  type = string
}

resource "vault_mount" "root" {
  path                  = "${var.prefix}-root"
  type                  = "pki"
  max_lease_ttl_seconds = 87600 * 60 * 60
}

resource "vault_pki_secret_backend_config_urls" "urls" {
  backend                 = vault_mount.root.path
  issuing_certificates    = ["http://127.0.0.1:8200/v1/${var.prefix}-root/ca"]
  crl_distribution_points = ["http://127.0.0.1:8200/v1/${var.prefix}-root/crl"]
}

resource "vault_mount" "inter" {
  path                  = "${var.prefix}-inter"
  type                  = "pki"
  max_lease_ttl_seconds = 43600 * 60 * 60
}

output "root" {
  value = vault_mount.root
}

output "inter" {
  value = vault_mount.root
}


# connect-ca policy
path "/sys/mounts/${root_path}" {
  capabilities = [ "read" ]
}

path "/sys/mounts/${inter_path}" {
  capabilities = [ "read" ]
}

path "/sys/mounts/${inter_path}/tune" {
  capabilities = [ "update" ]
}

path "/${root_path}/" {
  capabilities = [ "read" ]
}

path "/${root_path}/root/sign-intermediate" {
  capabilities = [ "update" ]
}

path "/${inter_path}/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "auth/token/renew-self" {
  capabilities = [ "update" ]
}

path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}

path "/${root_path}/root/sign-self-issued" {
  capabilities = [ "sudo", "update" ]
}

Debug Output

module.vault-consul-connect-ca.consul_certificate_authority.connect: Creating...
2023-01-19T19:51:27.318+0200 [INFO]  Starting apply for module.vault-consul-connect-ca.consul_certificate_authority.connect
2023-01-19T19:51:27.318+0200 [DEBUG] module.vault-consul-connect-ca.consul_certificate_authority.connect: applying the planned Create change
2023-01-19T19:51:27.318+0200 [TRACE] GRPCProvider: ApplyResourceChange
2023-01-19T19:51:27.574+0200 [TRACE] maybeTainted: module.vault-consul-connect-ca.consul_certificate_authority.connect encountered an error during creation, so it is now marked as tainted
2023-01-19T19:51:27.575+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.vault-consul-connect-ca.consul_certificate_authority.connect
2023-01-19T19:51:27.575+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.vault-consul-connect-ca.consul_certificate_authority.connect
2023-01-19T19:51:27.575+0200 [TRACE] evalApplyProvisioners: module.vault-consul-connect-ca.consul_certificate_authority.connect is tainted, so skipping provisioning
2023-01-19T19:51:27.575+0200 [TRACE] maybeTainted: module.vault-consul-connect-ca.consul_certificate_authority.connect was already tainted, so nothing to do
2023-01-19T19:51:27.575+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.vault-consul-connect-ca.consul_certificate_authority.connect
2023-01-19T19:51:27.575+0200 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.vault-consul-connect-ca.consul_certificate_authority.connect
2023-01-19T19:51:27.575+0200 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2023-01-19T19:51:27.583+0200 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 345
2023-01-19T19:51:27.583+0200 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2023-01-19T19:51:27.592+0200 [ERROR] vertex "module.vault-consul-connect-ca.consul_certificate_authority.connect" error: failed to set CA configuration: Unexpected response code: 500 (error having Vault cross-sign cert: Error making API request.

URL: PUT http://10.27.96.5:8200/v1/connect-root/root/sign-self-issued
Code: 403. Errors:

* permission denied)
2023-01-19T19:51:27.592+0200 [TRACE] vertex "module.vault-consul-connect-ca.consul_certificate_authority.connect": visit complete, with errors
2023-01-19T19:51:27.592+0200 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/consul\"] (close)" errored, so skipping
2023-01-19T19:51:27.592+0200 [TRACE] dag/walk: upstream of "module.vault-consul-connect-ca (close)" errored, so skipping
2023-01-19T19:51:27.592+0200 [TRACE] dag/walk: upstream of "root" errored, so skipping
2023-01-19T19:51:27.592+0200 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2023-01-19T19:51:27.597+0200 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 346
2023-01-19T19:51:27.597+0200 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
╷
│ Error: failed to set CA configuration: Unexpected response code: 500 (error having Vault cross-sign cert: Error making API request.
│ 
│ URL: PUT http://10.27.96.5:8200/v1/connect-root/root/sign-self-issued
│ Code: 403. Errors:
│ 
│ * permission denied)
│ 
│   with module.vault-consul-connect-ca.consul_certificate_authority.connect,
│   on modules/vault-consul-connect/main.tf line 24, in resource "consul_certificate_authority" "connect":
│   24: resource "consul_certificate_authority" "connect" {
│ 
╵
2023-01-19T19:51:27.604+0200 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2023-01-19T19:51:27.604+0200 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2023-01-19T19:51:27.605+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-01-19T19:51:27.606+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/consul/2.17.0/darwin_arm64/terraform-provider-consul_v2.17.0_x4 pid=10681
2023-01-19T19:51:27.606+0200 [DEBUG] provider: plugin exited

Expected Behavior

Consul set up Vault as Connect CA as mentioned here and here

Actual Behavior

Got error

╷
│ Error: failed to set CA configuration: Unexpected response code: 500 (error having Vault cross-sign cert: Error making API request.
│ 
│ URL: PUT http://10.27.96.5:8200/v1/connect-root/root/sign-self-issued
│ Code: 403. Errors:
│ 
│ * permission denied)

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

terraform apply

Important Factoids

ACL disabled

The text was updated successfully, but these errors were encountered:

const-tmp · 2023-01-19T21:57:48Z

Tried to do the same with HTTP request like this:

data "http" "example" {
  provider        = http-full
  url             = "http://${var.consul_addr}:8500/v1/connect/ca/configuration"
  method          = "PUT"
  request_headers = {
    content-type = "application/json"
  }
  request_body = jsonencode({
    provider : "vault",
    config : {
      address : "http://${var.vault_addr}:8200",
      auth_method = {
        type       = "approle"
        mount_path = "approle"
        params     = {
          role_id   = vault_approle_auth_backend_role.consul-connect-ca.role_id
          secret_id = vault_approle_auth_backend_role_secret_id.consul-connect-ca.secret_id
        }
      }
      RootPKIPath : module.connect_pki.root.path,
      IntermediatePKIPath : module.connect_pki.inter.path,
      private_key_type : "ec",
      private_key_bits : 256
    },
    ForceWithoutCrossSigning : false
  })
}

and the error was the same:

╷
│ Error: HTTP request error. Response code: 500,  Error Response body: error having Vault cross-sign cert: Error making API request.
│ 
│ URL: PUT http://10.27.96.5:8200/v1/connect-root/root/sign-self-issued
│ Code: 403. Errors:
│ 
│ * permission denied
│

Also I tried setting all capabilities to policy:

path "/${root_path}/root/sign-self-issued" {
  capabilities = ["sudo", "create", "read", "update", "delete", "list"]
}

with the same error.

So, I guess the issue is not about provider, but about Consul itself or documentation.

const-tmp · 2023-01-19T22:46:53Z

I also tried this method with consul connect ca set-config:

module "consul-set-ca-config" {
  source     = "../utils/auto-update-file"
  connection = {
    host        = var.consul_public_ip
    port        = 22
    user        = "root"
    agent       = true
    password    = null
    private_key = null
  }
  remote_exec = {
    inline=[
      "consul connect ca set-config -config-file /tmp/connect-ca-vault.json",
      "rm /tmp/connect-ca-vault.json"
    ]
  }
  template    = {
    path = "${path.module}/../../config/consul/config-connect-ca-provider-vault.json"
    data = {
      vault_addr  = var.vault_addr
      vault_token = vault_token.consul-connect-ca.client_token
      root_path   = "connect-root"
      inter_path  = "connect-inter"
    }
    destination = "/tmp/connect-ca-vault.json"
  }
}

Error:

module.vault-consul-connect-ca.module.consul-set-ca-config.null_resource.config_vault_server (remote-exec): Error setting CA configuration: Unexpected response code: 500 (error having Vault cross-sign cert: Error making API request.

module.vault-consul-connect-ca.module.consul-set-ca-config.null_resource.config_vault_server (remote-exec): URL: PUT http://10.27.96.5:8200/v1/connect-root/root/sign-self-issued
module.vault-consul-connect-ca.module.consul-set-ca-config.null_resource.config_vault_server (remote-exec): Code: 403. Errors:

module.vault-consul-connect-ca.module.consul-set-ca-config.null_resource.config_vault_server (remote-exec): * permission denied)
╷
│

Again, token policies are:

path "/sys/mounts/connect-root" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect-inter" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect-inter/tune" {
  capabilities = [ "update" ]
}

path "/connect-root/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect-inter/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "auth/token/renew-self" {
  capabilities = [ "update" ]
}

path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}

path "connect-root/root/sign-self-issued" {
  capabilities = [ "sudo", "update" ]
}

const-tmp · 2023-01-19T22:53:17Z

Verify than token has documented policies:

root@consul-0:~# cat /tmp/connect-ca-vault.json
{
  "Provider": "vault",
  "Config": {
    "Address": "http://vault.service.consul:8200",
    "Token": "hvs.CAESIEGK5gWhg4fzbcGskSwqCKNIExrN-VR2yGGAbLGv_8vBGh4KHGh2cy5IY2k4dVNsV1NkdW5IajZWVFl0ZTdMbUU",
    "RootPKIPath": "connect-root",
    "IntermediatePKIPath": "connect-inter",
    "PrivateKeyType": "ec",
    "PrivateKeyBits": 256
  },
  "ForceWithoutCrossSigning": false
}

➜  ~ vault token lookup hvs.CAESIEGK5gWhg4fzbcGskSwqCKNIExrN-VR2yGGAbLGv_8vBGh4KHGh2cy5IY2k4dVNsV1NkdW5IajZWVFl0ZTdMbUU
Key                  Value
---                  -----
accessor             RXbM1nx2E8o2rAID87XJUA5h
creation_time        1674167964
creation_ttl         768h
display_name         token-token
entity_id            n/a
expire_time          2023-02-20T22:39:24.483537562Z
explicit_max_ttl     0s
id                   hvs.CAESIEGK5gWhg4fzbcGskSwqCKNIExrN-VR2yGGAbLGv_8vBGh4KHGh2cy5IY2k4dVNsV1NkdW5IajZWVFl0ZTdMbUU
issue_time           2023-01-19T22:39:24.295257026Z
last_renewal         2023-01-19T22:39:26.483537658Z
last_renewal_time    1674167966
meta                 <nil>
num_uses             0
orphan               false
path                 auth/token/create
policies             [connect-ca default]
renewable            true
ttl                  767h48m10s
type                 service

➜  ~ vault policy read connect-ca
path "/sys/mounts/connect-root" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect-inter" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect-inter/tune" {
  capabilities = [ "update" ]
}

path "/connect-root/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect-inter/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "auth/token/renew-self" {
  capabilities = [ "update" ]
}

path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}

path "connect-root/root/sign-self-issued" {
  capabilities = [ "sudo", "update" ]
}

remilapeyre · 2023-05-01T13:25:40Z

Hi @nullc4t, I think there is an issue with the way you have setup permissions for Consul to access the Vault PKI mount as is reported in Consul's log.

I had no issues to use the Vault CA with the following configuration:

locals {
  vault_token = "iY1KK0A7MygzZUwz3WoIkSvq"
}

terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "3.15.0"
    }
    consul = {
      source  = "hashicorp/consul"
      version = "2.17.0"
    }
  }
}

provider "vault" {
  address = "http://localhost:8200"
  token   = local.vault_token
}

provider "consul" {}

resource "vault_mount" "pki" {
  path                      = "pki"
  type                      = "pki"
  default_lease_ttl_seconds = 315360000
  max_lease_ttl_seconds     = 315360000
}

resource "consul_certificate_authority" "connect" {
  connect_provider = "vault"

  config = {
    Address             = "http://localhost:8200/"
    Token               = local.vault_token
    RootPKIPath         = vault_mount.pki.path
    IntermediatePKIPath = vault_mount.pki.path
  }
}

Can you please check that the ACL in Vault are properly configured for your usage? The audit log should have a lot of information to help you debug the issue.

const-tmp changed the title ~~Vault Connect-CA~~ consul_certificate_authority Vault Connect-CA Jan 19, 2023

const-tmp changed the title ~~consul_certificate_authority Vault Connect-CA~~ consul_certificate_authority Vault Connect-CA Jan 19, 2023

remilapeyre added this to the next milestone Feb 20, 2023

remilapeyre removed this from the next milestone May 1, 2023

remilapeyre added the waiting-response label May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

consul_certificate_authority Vault Connect-CA #331

consul_certificate_authority Vault Connect-CA #331

const-tmp commented Jan 19, 2023

const-tmp commented Jan 19, 2023

const-tmp commented Jan 19, 2023

const-tmp commented Jan 19, 2023

remilapeyre commented May 1, 2023

consul_certificate_authority Vault Connect-CA #331

consul_certificate_authority Vault Connect-CA #331

Comments

const-tmp commented Jan 19, 2023

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

const-tmp commented Jan 19, 2023

const-tmp commented Jan 19, 2023

const-tmp commented Jan 19, 2023

remilapeyre commented May 1, 2023