add one prod ready example #166

pgold30 · 2019-10-11T12:41:36Z

Vault auto unseal cluster private with kms and acl for consul with encryption

Tested with latest version of Vault 1.2.3 and consul 1.6.1

This example combines the following examples: Vaul auto unseal - Vault private cluster - Consul with encryption and also adds some stuff that we expect in a prod ready working state, mainly:

Secure instances using security groups and remove [0.0.0.0/0]
Enable ACL on consul and pass the token in a secure way using AWS Secrets manager
Enable encryption on Consul (based on https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/example-with-encryption)
Auto generate gossip_encryption_key and store it in AWS Secrets manager
Only allow ssh from Vpn server
Creates an ELB in front of Consul instances
Creates an ELB in front of Vault instances
Removed the block for deploying cluster in default VPC and availability zones, this needs to be set explicitly to enforce security

This example use Terragrunt to solve some dependencies but can by used with terraform also.

Thanks @mkrull for the ACL token provision

brikis98 · 2019-10-15T09:17:13Z

@pgold30 Thanks for the PR! This is awesome work. However, a few gotchas:

This code includes references to a bunch of proprietary Gruntwork code. That means the examples won't work for many open source users, leading to lots of 404's, confusion, etc. We should only use open source code in these open source repos.
It looks like this code is meant to be used in the Gruntwork Reference Architecture: e.g., it pulls in VPC data via terraform_remote_state. While very handy, it again won't work for many open source users who aren't using the Reference Architecture, Terragrunt, etc.
Each example in the /examples folder needs a corresponding test in the /test folder. Note that creating a KMS master key for each test run will be a bit too expensive ($1 per run!), so we'll need not only new test code, but also some tweaks to the module to make it more testable.
Side note: there's something odd/ironic about using AWS Secrets Manager to bootstrap Vault 😬

If you want to simplify this example to something any open source user can use and make it testable, we can put it in this repo. Alternatively, we are planning to create an alternative repo later this quarter for Gruntwork customers where we can put such production-ready examples that include proprietary code, so another option is to wait until that's ready, and move your PR that to that repo. Let me know what you'd prefer!

pgold30 · 2019-10-15T09:31:31Z

Hi @brikis98 thanks for your feedback!
1- Indeed we use gruntwork , terragrunt and other stuff, didn't wanted to remove this because i assume that will be nice for people using the same stack to have a prod ready example, will remove this stuff then in a new commit.
2- Will remove those refs
3- I dont know how many test we run but having a prod ready example seems appropiate , the issue here seems to be in the test code more than in this approach , since the auto-unseal example makes use of this also.
4- Indeed, but wanted to find a solution for bootstraping vault, and use acl and token in consul whitout hardcoding so this looks like a nice and simple solution to do this, let me know if you think there is a better way.

Will apply the corrections to make it open source here and will be happy to help when you make the specific Gruntwork prod-ready example

pgold30 · 2019-11-07T12:45:10Z

@brikis98 Please let me know if the comments make sense or if i need to modify some stuff

brikis98 · 2019-11-11T20:21:41Z

Apologies, am currently traveling. Will check this out as soon as I can!

brikis98

Apologies again for the long delay. This is a large PR that required a fair amount of thought, and I just was not able to carve out time to get to it until now. I finally went through and left a bunch of comments, especially around a lot of copy/paste.

A candid piece of feedback: I'm a bit worried about this example being the "prod-ready" one. One issue is there are no automated tests; all of the code in our repos must be tested, or we can't maintain it or feel confident it works as expected. But the even bigger issue is that this example code is still missing a number of items that you would need for going to prod, especially related to server-hardening, monitoring, and logging. Adding all of these items is out of scope for this repo, but without those items, I feel like we'd be misleading users by saying this is production-ready code. What do you think of, as an alternative, writing up a set of docs that go through a (relatively comprehensive) list of the considerations to keep in mind when going to prod, rather than partial example code? It could be a step-by-step guide that includes many of the items you have in this example (e.g., enabling gossip encryption, passing secrets securely, etc), plus any items that are missing.