Running Vault and Consul on Kubernetes

# Running Vault + Consul on Kubernetes

<div>
 Presented by Michael Herman at <a href="https://www.meetup.com/DenverMicroservices">Denver Microservices</a>
</div>

---

### Agenda

##### (1) Intro

1. About Me
1. Objectives

##### (2) Theory

1. What is Vault?
1. Why Vault?
1. Using Vault

##### (3) Practice

1. Project Overview
1. Minikube
1. Setup (TLS, Consul, Vault)
1. Init, Unseal, and Authenticate
1. Static Secrets
1. Dynamic Secrets
1. Encryption as a Service

##### (4) Next Steps

---

## Intro

---

### About Michael

```
$ whoami
michael.herman
```

#### Day Job

Senior Software Engineer at [ClickFox](https://www.clickfox.com/).
 
<img src="images/clickfox.png" style="max-width: 4%; border:0; box-shadow: none;" alt="clickfox logo">

#### Also

1. Co-founder/author of [Real Python](https://realpython.com)
1. 😍 - [tech writing/education](http://mherman.org), [open source](http://github.com/mjhea0), [financial models](http://www.starterfinancialmodel.com/), [radiohead](http://radiohead.com/)

---

### Objectives

By the end of this talk, you should be able to...

1. Explain what **Vault** is and why you may want to use it

1. Describe the basic **Vault architecture** along with dynamic and static secrets, the various backends (storage, secret, auth, audit), and how Vault can be used as an "encryption as a service"

1. Spin up a Vault + Consul cluster on **Kubernetes**

1. **Init** and **unseal** Vault

1. **Authenticate** against Vault

1. Work with **static** and **dynamic** secrets

1. Use the **Transit** backend as an "encryption as a service"

---

## Theory

---

### What is Vault?

[Vault](https://www.vaultproject.io/) is an open-source tool used for securely storing and managing secrets.

#### Secrets?

Secrets are securely-sensitive or personally identifiable info (PII) like database credentials, SSH keys, usernames/passwords, AWS IAM credentials, API tokens, SSNs, credit card numbers.
--

#### Distributing secrets

1. Who has access to them?
1. Who manages them?
1. How do you control who has access to them?
1. How do your apps get them?
1. How are they updated?
1. How are they revoked?

---

### Why Vault?

Vault provides answers to the questions on the previous slide and helps to solve the following problems with secret management:

| Current problems                                   | Vault's Goals                                            |
|----------------------------------------------------|----------------------------------------------------------|
| Secrets are everywhere.                            | Vault is the single source of truth for all secrets.     |
| They are generally unencrypted.                    | Vault manages encryption (during transit and at rest) out of the box. |
| It's difficult to dynamically generate them.       | Secrets can be dynamically generated.                       |
| It's even more difficult to lease and revoke them. | Secrets can be leased and revoked.                       |
| There's no audit trail.                            | There's an audit trail for generating and using secrets. |

https://www.vaultproject.io/docs/what-is-vault/index.html

---

### Using Vault (1)

#### Secret Management

Vault has two types of secrets:

1. **[Static](https://www.vaultproject.io/guides/secret-mgmt/static-secrets.html)** secrets (think encrypted Redis or Memcached) have refresh intervals but they do not expire unless explicitly revoked. They are defined ahead of time and then shared.

1. **[Dynamic](https://www.vaultproject.io/intro/getting-started/dynamic-secrets.html)** secrets have enforced leases and generally expire after a short period of time. Since they don’t exist until they are accessed, there’s less exposure - so dynamic secrets are much more secure.

#### Encryption as a Service

The [Transit](https://www.vaultproject.io/docs/secrets/transit/index.html) backend can be used as an "encryption as a service":

1. Encrypt and decrypt data "in-transit" without storing it
1. Easily integrate encryption into your application workflow

---

### Using Vault (2)

Backends are everywhere!

| Backend                                                                      | Use                                      | Examples                                                       |
|------------------------------------------------------------------------------|------------------------------------------|----------------------------------------------------------------|
| [Storage](https://www.vaultproject.io/docs/configuration/storage/index.html) | Where secrets are stored                 | Consul`*`, Filesystem, In-Memory, PostgreSQL, S3, Google Cloud Storage                  |
| [Secret](https://www.vaultproject.io/docs/secrets/index.html)                | Handles static or dynamic secrets        | AWS`*`, Databases, Key/Value`*`, RabbitMQ, SSH, Google KMS                       |
| [Auth](https://www.vaultproject.io/docs/auth/index.html)                     | Handles authentication and authorization | AWS, Azure, Google Cloud, GitHub, Tokens`*`, Username & Password |
| [Audit](https://www.vaultproject.io/docs/auth/index.html)                    | Logs all requests and responses                     | File, Syslog, Socket                                           |

`*` used in this presentation

<a href="images/vault-architecture.png">vault-architecture.png</a>

https://www.vaultproject.io/docs/internals/architecture.html

---

## Practice

---

### Project Overview

[https://github.com/testdrivenio/vault-consul-kubernetes](https://github.com/testdrivenio/vault-consul-kubernetes)

```sh
├── certs
│   ├── config
│   │   ├── ca-config.json
│   │   ├── ca-csr.json
│   │   ├── consul-csr.json
│   │   └── vault-csr.json
├── consul
│   ├── config.json
│   ├── service.yaml
│   └── statefulset.yaml
├── create.sh
└── vault
    ├── config.json
    ├── deployment.yaml
    └── service.yaml
```

Prerequisites:

- Go, [cfssl](https://github.com/cloudflare/cfssl) and cfssljson, Consul CLI, Vault CLI

---

### Minikube

[Minikube](https://kubernetes.io/docs/setup/minikube/) is a tool which allows developers to use and run a Kubernetes cluster locally.

#### Getting Started

First, install the following tools:

1. **Install a  [Hypervisor](https://kubernetes.io/docs/tasks/tools/install-minikube/#install-a-hypervisor)** (like [VirtualBox](https://www.virtualbox.org/wiki/Downloads) or [HyperKit](https://github.com/moby/hyperkit)) to manage virtual machines
1. **Install and Set Up [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)** to deploy and manage apps on Kubernetes
1. **Install [Minikube](https://github.com/kubernetes/minikube/releases)**

Then, start the cluster:

```sh
$ minikube start --vm-driver=virtualbox
$ minikube dashboard
```

---

### Project Setup

#### Steps

- Create the TLS Certificates:

See https://testdriven.io/running-vault-and-consul-on-kubernetes

- Run *create.sh* to deploy a Vault and Consul cluster:

```sh
    $ sh create.sh
    ```

- In a new terminal window, set the following environment variables:

```sh
    $ export VAULT_ADDR=https://127.0.0.1:8200
    $ export VAULT_CACERT="certs/ca.pem"
    ```

- Verify:

```sh
    $ kubectl get pods
    $ vault status
    ```

---

### Init, Unseal, and Authenticate

Initialize Vault:

```sh
$ vault operator init
```

Take note of the unseal keys and the initial root token. You will need to provide 3 of the unseal keys every time the Vault server is re-sealed.

(Why 3 keys? https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing)

Now you can unseal the Vault using 3 of the keys:

```sh
$ vault operator unseal
```

Authenticate:

```sh
$ vault login
Token (will be hidden):
```

This uses the root policy. In production you'll want to set up policies with different levels of access.

https://www.vaultproject.io/docs/concepts/policies.html

---

### Static Secrets

Vault can be managed through the [CLI](https://www.vaultproject.io/docs/commands/index.html), [HTTP API](https://www.vaultproject.io/api/index.html), or [UI](https://www.vaultproject.io/docs/configuration/ui/index.html).

#### CLI Examples

https://www.vaultproject.io/docs/secrets/kv/index.html

Create a secret:

```sh
$ vault kv put secret/foo bar=precious
Success! Data written to: secret/foo
```

Read a secret:

```sh
$ vault kv get secret/foo
```

Delete a secret:

```sh
$ vault kv delete secret/foo
```

---

### Dynamic Secrets (1)

Vault supports a number of dynamic secret backends for generating secrets dynamically when needed.

For example, with the [AWS](https://www.vaultproject.io/docs/secrets/aws/index.html) and [Google Cloud](https://www.vaultproject.io/docs/secrets/gcp/index.html) backends, you can create access credentials based on IAM policies. The [databases](https://www.vaultproject.io/docs/secrets/databases/index.html) backend, meanwhile, generates database credentials based on configured roles.

*Dynamic Secrets:*

- are generated on demand
- have limited access based on role
- are "leased"
- can be revoked
- come with an audit trail

<a href="https://www.hashicorp.com/blog/why-we-need-dynamic-secrets">https://www.hashicorp.com/blog/why-we-need-dynamic-secrets</a>

---

### Dynamic Secrets (2)

#### EXAMPLE - Generate dynamic, on-demand AWS access credentials

Enable the AWS Secrets backend:

```sh
$ vault secrets enable -path=aws aws
Success! Enabled the aws secrets engine at: aws/
```

Authenticate against AWS:

```sh
$ vault write aws/config/root access_key=foo secret_key=bar
Success! Data written to: aws/config/root
```

Create role and credentials:

```sh
$ vault write aws/roles/ec2-read \
  arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess
$ vault read aws/creds/ec2-read
```

*Secrets are generated when they are requested (i.e., a web app requests access to S3). They are not available in the store before this.*

https://www.vaultproject.io/intro/getting-started/dynamic-secrets.html

---

### Encryption as a Service

Enable transit:

```sh
$ vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/
```

Configure named encryption key:

```sh
$ vault write -f transit/keys/foo
Success! Data written to: transit/keys/foo
```

Encrypt and decrypt:

```sh
$ vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")
Key Value
--- -----
ciphertext vault:v1:/Tun95IT+dVTvDfYiCHdI5rGPSAxgvPcFaDDtneRorQCyBOgv9mPKw==

$ vault write transit/decrypt/foo ciphertext=vault:v1:/Tun95IT+dVTvDfYiCHdI5rGPSAxgvPcFaDDtneRorQCyBOgv9mPKw==
Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

$ base64 -D <<< "bXkgcHJlY2lvdXMK"
my precious
```

---

### That's it!

What's next?

##### What has not been covered?

1. Leases and revocation
1. Setting up new policies
1. Deployment
1. Audit backend

##### Resources

1. Full blog post - https://testdriven.io/running-vault-and-consul-on-kubernetes
1. Slides - http://mherman.org/presentations/vault-kubernetes
1. Repo - https://github.com/testdrivenio/vault-consul-kubernetes
1. Why We Need Dynamic Secrets - https://www.hashicorp.com/blog/why-we-need-dynamic-secrets (advantages of using dynamic secrets)

##### Questions?

✌️