Skip to content
This repository has been archived by the owner on Jan 25, 2023. It is now read-only.

add one prod ready example #166

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

add one prod ready example #166

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b9b9772
add certs for consul in ami and agent.hcl
Oct 11, 2019
cc93b14
fix string length base64 gossip enc key
Oct 11, 2019
223bf5b
remove terragrunt deps
Oct 15, 2019
79a4ec2
remove duplicated var unseal
Oct 15, 2019
d39adb5
Update examples/vault-prod-ready/README.md
Jan 10, 2020
a88afd8
remove stuff from readme from copy paste
Jan 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions examples/vault-consul-ami/tls/agent.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
acl = {
enabled = true,
default_policy = "deny",
enable_token_persistence = true
}
16 changes: 13 additions & 3 deletions examples/vault-consul-ami/vault-consul.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
"min_packer_version": "0.12.0",
"variables": {
"aws_region": "us-east-1",
"vault_version": "1.1.0",
"vault_version": "1.2.3",
"consul_module_version": "v0.7.3",
"consul_version": "1.5.3",
"consul_version": "1.6.1",
"consul_download_url": "{{env `CONSUL_DOWNLOAD_URL`}}",
"vault_download_url": "{{env `VAULT_DOWNLOAD_URL`}}",
"install_auth_signing_script": "true",
Expand Down Expand Up @@ -87,6 +87,10 @@
" /tmp/terraform-aws-vault/modules/install-vault/install-vault --version {{user `vault_version`}};",
"fi"
]
},{
"type": "file",
"source": "{{template_dir}}/tls/agent.hcl",
"destination": "/tmp/agent.hcl"
},{
"type": "file",
"source": "{{template_dir}}/auth/sign-request.py",
Expand Down Expand Up @@ -149,7 +153,13 @@
" /tmp/terraform-aws-consul/modules/install-consul/install-consul --download-url {{user `consul_download_url`}};",
"else",
" /tmp/terraform-aws-consul/modules/install-consul/install-consul --version {{user `consul_version`}};",
"fi"
"fi",
"sudo cp /opt/vault/tls/ca.crt.pem /opt/consul/tls/ca/ca.crt.pem",
"sudo cp /opt/vault/tls/vault.crt.pem /opt/consul/tls/vault.crt.pem",
"sudo cp /opt/vault/tls/vault.key.pem /opt/consul/tls/vault.key.pem",
"sudo chown -R consul:consul /opt/consul/tls/",
"sudo chmod -R 600 /opt/consul/tls",
"sudo chmod 700 /opt/consul/tls"
],
"pause_before": "30s"
},{
Expand Down
180 changes: 180 additions & 0 deletions examples/vault-prod-ready/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
# Vault auto unseal cluster private with kms and acl for consul with encryption. PROD READY

This example icombines the followings: Vault auto unseal - Vault private cluster - Consul with encryption and also adds some stuff that we expect in a prod ready working state, mainly:
pgold30 marked this conversation as resolved.
Show resolved Hide resolved
* Secure instances using security groups and remove [0.0.0.0/0]
* Enable ACL on consul and pass the token in a secure way using AWS Secrets manager
* Enable encryption on Consul (based on https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/example-with-encryption)
* Auto generate gossip_encryption_key and store it in AWS Secrets manager
* Only allow ssh from Vpn server
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not everyone who uses Vault will have a VPN server. Perhaps this should be a more generic list of security group IDs to whitelist? You could put the security group ID of a VPN server in that list, or a bastion host, or whatever else your organization is using.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree

* Creates an ELB in front of Consul instances
* Creates an ELB in front of Vault instances
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the ELBs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be any load balancer , to balance request and still have one record entry when having multiple instances?

* Removed the block for deploying cluster in default VPC and availability zones, this needs to be set explicitly to enforce security

This example use Terragrunt to solve some dependencies but can by used with terraform also.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably should keep these examples to Terraform-only. Anything that works with Terraform works with Terragrunt, but not necessarily vice versa.


Thanks @mkrull for the ACL token provision

# HOWTO
* 1 - Run terragrunt
* 2 - run `vault operator init` on one of the vault servers
* 3 - To login in consul look for master token under aws secrets manager
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand what this is saying. Could you try to clarify the language a bit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will change it to make it more clear. But basically it says that you need to ssh vault and run that command to get the token and that the consul token will be stored under aws secret manager


# Vault auto unseal example
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You already have a # (an h1) up above, so this should be ##... And perhaps it needs a more specific title, as you already mentioned vault auto unseal example at the top.


This folder shows an example of Terraform code that deploys a [Vault][vault] cluster
in AWS with [auto unseal][auto_unseal]. Auto unseal is a Vault feature
that automatically [unseals][seal] each node in the cluster at boot using [Amazon KMS][kms].
Without auto unseal, Vault operators are expected to manually unseal each Vault node
after it boots, a cumbersome process that typically requires multiple Vault operators
to each enter a Vault master key shard.

This example creates a private Vault cluster that is accessible only from within
the VPC within the AWS account in which it resides, or other VPCs that are peered
with the Vault VPC. The Vault cluster uses [Consul][consul] as the storage backend,
so this example also deploys a separate Consul server cluster using the
[consul-cluster module][consul_cluster] from the Consul AWS Module. Each of the
servers in this example has [Dnsmasq][dnsmasq] installed (via the [install-dnsmasq module][dnsmasq_module])
or [setup-systemd-resolved][setup_systemd_resolved] (in the case of Ubuntu 18.04)
which allows them to use the Consul server cluster for service discovery and thereby
access Vault via DNS using the domain name `vault.service.consul`.

For more info on how the Vault cluster works, check out the [vault-cluster][vault_cluster]
documentation.

**Note**: To keep this example as simple to deploy and test as possible, it deploys
the Vault cluster into your default VPC and default subnets, all of which are publicly
accessible. This is OK for learning and experimenting, but for production usage,
we strongly recommend deploying the Vault cluster into the private subnets of a custom VPC.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not accurate, right? Is this a copy/paste from the other auto unseal example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, removed.


**Billing Warning**: Every time you create a KMS key, you're charged $1 for the month,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of this, all of our KMS tests take in a KMS key ID as an input variable. That way, we can create a single KMS key we use for testing, rather than a new one for each test run.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, looks like you may be doing just that. You may want to explain that here.

even if you immediately delete it.

Private Vault Cluster Example
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another heading? More copy/paste below?


This folder shows an example of Terraform code to deploy a [Vault](https://www.vaultproject.io/) cluster in
[AWS](https://aws.amazon.com/) using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). The Vault cluster uses
[Consul](https://www.consul.io/) as a storage backend, so this example also deploys a separate Consul server cluster
using the [consul-cluster module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster)
from the Consul AWS Module.

This example creates a private Vault cluster, which is private in the sense that the EC2 Instances are not fronted by a
load balancer, as is the case in the [Vault Public Example](/examples/root-example). Keep in mind that if the Vault
nodes are deployed to public subnets (i.e. subnets that have a route to the public Internet), this "private" cluster will
still be accessible from the public Internet.

Each of the servers in this example has [Dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) installed (via the
[install-dnsmasq module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq)) or
[setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved)
(in the case Ubuntu of 18.04)
which allows it to use the Consul server cluster for service discovery and thereby access Vault via DNS using the
domain name `vault.service.consul`. For an example of a Vault cluster
that is publicly accessible, see [the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example).

![Vault architecture](https://github.com/hashicorp/terraform-aws-vault/blob/master/_docs/architecture.png?raw=true)

You will need to create an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
that has Vault and Consul installed, which you can do using the [vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami)).

For more info on how the Vault cluster works, check out the [vault-cluster](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) documentation.

**Note**: To keep this example as simple to deploy and test as possible, it deploys the Vault cluster into your default
VPC and default subnets, all of which are publicly accessible. This is OK for learning and experimenting, but for
production usage, we strongly recommend deploying the Vault cluster into the private subnets of a custom VPC.

### Quick start

1. `git clone` this repo to your computer.
1. Build a Vault and Consul AMI. See the [vault-consul-ami example][vault_consul_ami]
documentation for instructions. Don't forget to set the variable `vault_download_url`
with the url of the enterprise version of Vault if you wish to use Vault Enterprise.
Make sure to note down the ID of the AMI.
1. Install [Terraform][terraform].
1. [Create an AWS KMS key][key_creation]. Take note of the key alias.
1. Open `vars.tf`, set the environment variables specified at the top of the file,
and fill in any other variables that don't have a default. Put the AMI ID you
previously took note into the `ami_id` variable and the KMS key alias into
`auto_unseal_kms_key_alias`.
1. Run `terraform init`.
1. Run `terraform apply`.
1. Run the [vault-examples-helper.sh script][examples_helper] to
print out the IP addresses of the Vault server and some example commands you
can run to interact with the cluster: `../vault-examples-helper/vault-examples-helper.sh`.
1. Ssh to an instance in the vault cluster and run `vault operator init` to initialize
the cluster, then `vault status` to check that it is unsealed. If you ssh to a
different node in the cluster, you might have to restart Vault first with
`sudo systemctl restart vault.service` so it will rejoin the cluster and unseal.
To avoid doing that, you can start your cluster with initially just one node and
start the server, then change the `vault_cluster_size` variable back to 3 and and
run `terraform apply again`. The new nodes will join the cluster already unsealed
in this case.

### Seal

All data stored by Vault is encrypted with a Master Key which is not stored anywhere
and Vault only ever keeps in memory. When Vault first boots, it does not have the
Master Key in memory, and therefore it can access its storage, but it cannot decrypt
its own data. So you can't really do anything apart from unsealing it or checking
the server status. While Vault is at this state, we say it is "sealed".

Since vault uses [Shamir's Secret Sharing][shamir], which splits the master key into
pieces, running `vault operator unseal <unseal key>` adds piece by piece until there
are enough parts to reconstruct the master key. This is done on different machines in the
vault cluster for better security. When Vault is unsealed and it has the recreated
master key in memory, it can then be used to read the stored decryption keys, which
can decrypt the data, and then you can start performing other operations on Vault.
Vault remains unsealed until it reboots or until someone manually reseals it.

### Auto-unseal

Vault has a feature that allows automatic unsealing via Amazon KMS. It
allows operators to delegate the unsealing process to AWS, which is useful for failure
situations where the server has to restart and then it will be already unsealed or
for the creation of ephemeral clusters. This process uses an AWS KMS key as
a [seal wrap][seal_wrap] mechanism: it encrypts and decrypts Vault's master key
(and it does so with the whole key, replacing the Shamir's Secret Sharing method).

This feature is enabled by adding a `awskms` stanza at Vault's configuration. This
module takes this into consideration on the [`run-vault`][run_vault] binary, allowing
you to pass the following flags to it:
* `--enable-auto-unseal`: Enables the AWS KMS Auto-unseal feature and adds the `awskms`
stanza to the configuration
* `--auto-unseal-kms-key-id`: The key id of the AWS KMS key to be used
* `--auto-unseal-region`: The AWS region where the KMS key lives

In this example, like in other examples, we execute `run-vault` at the [`user-data`
script][user_data], which runs on boot for every node in the Vault cluster. The
`key-id` is passed to this script by Terraform, after Terraform reads this value from a
data source through the key alias. This means that the AWS key has to be previously
manually created and we are using Terraform just to find this resource, not to
create it. It is important to notice that AWS KMS keys have a [cost][kms_pricing]
per month per key, as well as an API usage cost.

```
data "aws_kms_alias" "vault-example" {
name = "alias/${var.auto_unseal_kms_key_alias}"
}
```

If you wish to use Vault Enterprise, you still need to apply your Vault
Enterprise License to the cluster with `vault write /sys/license "text=$LICENSE_KEY_TEXT"`.

[ami]: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html
[auto_unseal]: https://www.vaultproject.io/docs/enterprise/auto-unseal/index.html
[consul_cluster]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster
[consul]: https://www.consul.io/
[dnsmasq_module]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq
[dnsmasq]: http://www.thekelleys.org.uk/dnsmasq/doc.html
[setup_systemd_resolved]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved
[examples_helper]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh
[key_creation]: https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html
[kms]: https://aws.amazon.com/kms/
[kms_pricing]: https://aws.amazon.com/kms/pricing/
[run_vault]: https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault
[seal_wrap]: https://www.vaultproject.io/docs/enterprise/sealwrap/index.html
[seal]: https://www.vaultproject.io/docs/concepts/seal.html
[shamir]: https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
[terraform]: https://www.terraform.io/
[user_data]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-auto-unseal/user-data-vault.sh
[vault_cluster]: https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster
[vault_consul_ami]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami
[vault]: https://www.vaultproject.io/
Loading