Going Serverless with OpenFaaS, Kubernetes, and Python

# Going Serverless with OpenFaaS,<br>Kubernetes, and Python

<br>

<div>
  <span style="vertical-align:60%;">Presented by <em>Michael Herman</em> at</span> 
  <img src="images/pycon2018.png" style="max-width: 15%; border:0; box-shadow: none;" alt="PyCon 2018 logo">
</div>

---

### Agenda (part 1)

⏱ ~ 30 minutes

##### (1) Intro

1. About Me
1. Objectives
1. What are we building?
1. Architecture
1. Project Setup
1. Why Microservices?

#### (2) Theory

1. IaaS and PaaS
1. Serverless and FaaS
1. OpenFaaS

---

### Agenda (part 2)

⏱ ~ 30 minutes

##### (3) Web App

1. Python/Flask
1. Postgres
1. Nginx
1. Redis
1. Worker

##### (4) OpenFaaS

1. Health Check
1. Eval

---

### Agenda (part 3)

⏱ ~ 2 hours

##### (5) Deployment

1. Container Orchestration
1. Digital Ocean Setup
1. Deploy Web App
1. Deploy OpenFaaS Stack
    - Docker Swarm mode
    - Kubernetes

##### (6) Challenges

1. Webhook
1. Build History
1. Monitoring
1. More

##### (7) Goodbye

1. Next Steps
1. Questions

---

## Intro

---

### About Michael

```
$ whoami
michael.herman
```

#### Day Job:

Software Engineer at [ClickFox](https://www.clickfox.com/).
 
<img src="images/clickfox.png" style="max-width: 4%; border:0; box-shadow: none;" alt="clickfox logo">

#### Docker:

1. Avid Docker user since 2014.
1. Last year I architected and set up [On-Demand Environments With Docker and AWS ECS](http://mherman.org/blog/2017/09/18/on-demand-test-environments-with-docker-and-aws-ecs/).

#### Also:

1. Co-founder/author of [Real Python](https://realpython.com)
1. 😍 - [tech writing/education](http://mherman.org), [open source](http://github.com/mjhea0), [financial models](http://www.starterfinancialmodel.com/), [radiohead](http://radiohead.com/)

---

### Objectives

By the end of this talk, you should be able to...

1. Configure and run **microservices** locally with Docker

1. Integrate **Redis Queue** into a Flask app and create tasks

1. Run tasks in the background with a separate **worker** process

1. Run unit and integration **tests** inside a Docker container

1. Deploy **Flask**, **Postgres**, and **Redis** to Digital Ocean via Docker Compose and Machine

1. Explain what **container orchestration** is and why you may need to incorporate an orchestration tool into your deployment process

1. Deploy **OpenFaaS** to Digital Ocean using both **Docker Swarm Mode** and **Kubernetes**

1. Monitor your OpenFaaS functions with **Prometheus** and **Grafana**

---

### TestDriven.io

Much of this tutorial comes from the following course I wrote at [Testdriven.io](http://testdriven.io/)...

<br>

<div class="grid">
  <div style="float: left; width: 45%;">
    <div text-align="center">
      <img src="images/cover_small.png" style="max-width: 80%; border:2px solid; border-color:grey; box-shadow: none;" alt="microservice tech">
    </div>
  </div>
  <div style="float: left; width: 55%;">
    <h4 style="margin-top: 0;"><em>Microservices with Docker, Flask, and React</em></h4>
    <p>Along with learning how to build a number of Python-based microservices, you'll also be introduced to more advanced topics like:</p>
    <ul>
      <li>Test-driven Development (TDD)</li>
      <li>AWS ECS, RDS, and Lambda</li>
      <li>React</li>
      <li>Blue/Green Deploys</li>
      <li>CI/CD</li>
    </ul>
  </div>
</div>

---

### What are we building?

<div class="grid">
  <div style="float: left; width: 65%;">
    <div>
      <img src="images/tech.png" style="max-width:85%; border:0; box-shadow: none; padding-top:20px;" alt="tech">
    </div>
  </div>
  <div style="float: left; width: 35%;">
    <h4 style="margin-top: 0;"><em>OpenCI</em></h4>
    <p>A serverless continuous integration system powered by Python, Flask, and OpenFaaS.</p>
    <ul>
      <li><em>What is it?</em><br>A Travis CI clone that you have full control over.</li>
      <br>
      <li><em>Why are we building it?</em><br>It's more interesting than your typical CRUD app.</li>
    </ul>
  </div>
</div>

https://github.com/testdrivenio/openci

---

### Architecture

**Web App:**

| Name      | Container    | Tech                    |
|-----------|--------------|-------------------------|
| Web       | `web`        | Python, Flask, gunicorn |
| Web DB    | `web-db`     | Postgres                |
| Redis     | `web-redis`  | Redis                   |
| Worker    | `web-worker` | Python                  |
| Nginx     | `web-nginx`  | Nginx                   |

<br>

**OpenFaas Back-end:**

| Name         | Container   | Tech          |
|--------------|-------------|---------------|
| API Gateway  | `gateway`   | Golang        |
| Code Eval    | `eval`      | Python        |
| Health Check | `ping`      | Python        |

---

### Project Setup

Fire up the app locally:

```sh
$ git clone https://github.com/testdrivenio/openci \
  --branch pycon --single-branch
$ cd openci
$ docker-compose -f docker-compose-web.yml up -d --build
```

<small><em><strong>NOTE:</strong> Using Docker Machine? Replace `localhost` above with `DOCKER_MACHINE_IP`.</em></small>

#### Docker Compose

An orchestration tool for running multi-container apps.

*Often, when developing applications with a microservice architecture, you cannot fully test out all services until you deploy to a staging server. This takes much too long to get feedback. Docker helps to speed up this process by making it easier to link together small, independent services locally.*

Docker 101: http://mherman.org/docker-workshop

---

### Why Microservices?

**Pros**

1. *Separation of Concerns*: different apps can have different code bases and dependencies, localized errors, less coupling can lead to easier scaling
1. *Smaller Code Bases*: without having to grasp the entire system, it's often easier to understand the code base

**Cons**

1. *Design Complexity*: microservices are complex, start with modules
1. *Network Complexity*: many helper functions become AJAX calls, ping-pong-like effect in terms of network requests, network lag
1. *Data Persistence*: managing state (DBs, queues, service discovery) is hard
1. *Security*: more surface area = more areas prone to attack
1. *Integration Tests*: testing is hard, especially if you try to test like you'd test a monolith

You need - *Strong communication + docs, mature devops, lots of planning*

<small class="center">!!! More on microservices: https://testdriven.io/part-one-microservices !!!</small>

---

## Theory

---

### IaaS and PaaS

**Infrastructure-as-a-Service (IaaS)**:

IaaS models provide the base infrastructure (compute resources, networking, cloud storage) that end users then have to provision and manage on their own. *Examples*: DigitalOcean, Rackspace, parts of AWS

**Platform-as-a-Service (PaaS)**

PaaS models provide a platform for end users to manage applications without the overhead of having to set up and maintain the underlying infrastructure. *Examples*: Heroku, Elastic Beanstalk, Google App Engine

<p style="margin-top:0;"><small>(<a href="http://robertgreiner.com/2014/03/windows-azure-iaas-paas-saas-overview/">Source</a>)</small></p>

---

### Serverless and FaaS

**Serverless**:

Serverless is a cloud computing execution model that abstracts away much of the complexity associated with server management (implementing, provisioning, tweaking, scaling) from developers. Pricing models are generally based on actual resources consumed.

*On-demand* (boots in milliseconds), *elastic* (auto-scaling), *pay-as-you-go* (spin down when not used!)

**Functions as a Service (FaaS)**:

FaaS is a serverless model that enables developers to run modular, stateless pieces of code in response to events (like HTTP requests).

<div class="grid">
  <div style="float: left; width: 45%;">
    <div text-align="center">
      <div>
        <img src="images/mono-micro-functions.png" style="max-width: 90%; padding-top:10px; padding-bottom:0; box-shadow: none;" alt="monolith vs microservice vs faas">
      </div>
      <p style="margin-top:0;"><small>(<a href="http://lukeangel.co/cross-platform/docker-servless-faas-functions-as-a-service/">Source</a>)</small></p>
    </div>
  </div>
  <div style="float: left; width: 55%;">
    <p><em>What's it good for?</em></p>
    <ul>
      <li>Handling HTTP requests</li>
      <li>Processing tasks from a queue</li>
      <li>Running cron-like jobs</li>
    </ul>
  </div>
</div>

---

### Serverless and FaaS (continued...)

**Question: How does FaaS differ from the PaaS model?**

1. *PaaS* - Applications typically run on the server at all times.
1. *FaaS* - Functions run as needed. When needed, the functions take a few milliseconds to start and then they are shut down after execution.

**Examples of FaaS:**

1. [AWS Lambda](https://aws.amazon.com/lambda/)
1. [Google Cloud Functions](https://cloud.google.com/functions/)
1. [Microsoft Azure Functions](https://azure.microsoft.com/en-us/services/functions/)

**FaaS Advantages**:

1. *Increased velocity* (infrastructure is managed by someone else, so devs can focus on writing business-valuable code)
1. *Scalability, availability, and fault tolerance comes out of the box*
1. *Decreased costs* (no idle resources)

---

### Serverless and FaaS (continued...)

**FaaS Disadvantages**:

1. *Decreased transparency* (infrastructure is managed by someone else)
1. *Debugging* (remote debugging sucks)
1. *Confusion* (lots of moving pieces)
1. *Restrictions* (i.e., AWS Lambda functions only support certain languages and functions can only run for 5 minutes)
1. *Vendor lock-in*

---

### OpenFaaS

[OpenFaaS](https://www.openfaas.com/) is a framework for building serverless functions on top of containers. It helps to mitigate many of the disadvantages with FaaS. You can run serverless functions:

1. In whatever language you want
1. For however long you need
1. Wherever you want

https://blog.alexellis.io/introducing-functions-as-a-service/

<br>

**Bottom-line:**

1. Think about how much control you need to have when choosing your cloud model.
1. What are you team's strengths and weaknesses?
1. What do you want to focus your time and resources on?

---

## Web App

---

### Web DB

---

### Web DB

https://github.com/testdrivenio/openci/tree/pycon/services/db

#### Steps:

1. Review *[Dockerfile](https://github.com/testdrivenio/openci/blob/pycon/services/db/Dockerfile)* and *[docker-compose-web.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-web.yml)*
1. Build the image and run the container:

```sh
    $ docker-compose -f docker-compose-web.yml up -d --build web-db
    ```

1. Test/Sanity Check:

```sh
    $ docker exec -ti web-db psql -U postgres -W
    ```

---

### Web

---

### Web

https://github.com/testdrivenio/openci/tree/pycon/services/web

#### Steps:

1. Review the [code](https://github.com/testdrivenio/openci/tree/pycon/services/web), *[Dockerfile](https://github.com/testdrivenio/openci/blob/pycon/services/web/Dockerfile)*, and *[docker-compose-web.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-web.yml)*
1. Build the image and run the container:

```sh
    $ docker-compose -f docker-compose-web.yml up -d --build web
    ```

1. Test/Sanity Check:

```sh
    # create and seed the db
    $ docker-compose -f docker-compose-web.yml \
      run web python manage.py create_db
    $ docker-compose -f docker-compose-web.yml \
      run web python manage.py create_data
    # run unit and integration tests
    $ docker-compose -f docker-compose-web.yml run web python manage.py test
    ```

Navigate to [http://localhost:5002](http://localhost:5002) in your browser.

---

### Nginx

---

### Nginx

https://github.com/testdrivenio/openci/tree/pycon/services/nginx

#### Steps:

1. Review the [code](https://github.com/testdrivenio/openci/tree/pycon/services/nginx), *[Dockerfile](https://github.com/testdrivenio/openci/blob/pycon/services/nginx/Dockerfile)*, and *[docker-compose-web.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-web.yml)*

1. Build the image and run the container:

```sh
    $ docker-compose -f docker-compose-web.yml up -d --build web-nginx
    ```

1. Test/Sanity Check: navigate to [http://localhost](http://localhost)

<br>

---

### Redis

---

### Redis

#### Steps:

1. Review the  *[docker-compose-web.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-web.yml)*

1. Build the image and run the container:

```sh
    $ docker-compose -f docker-compose-web.yml up -d --build web-redis
    ```

<br>

---

### Worker

---

### Worker

#### Steps:

1. Review the  *[docker-compose-web.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-web.yml)*

1. Build the image and run the container:

```sh
    $ docker-compose -f docker-compose-web.yml up -d --build web-worker
    ```

<br>

---

## OpenFaaS

---

### API Gateway

---

### API Gateway

1. Routes requests to the appropriate function
1. Collects metrics via [Prometheus](https://prometheus.io/)
1. Handles scaling
1. Provides a nice UI

---

### Health Check

---

### Health Check

https://github.com/testdrivenio/openci/tree/pycon/services/functions/ping

#### Steps:

1. Review *[Dockerfile](https://github.com/testdrivenio/openci/blob/pycon/services/functions/ping/Dockerfile)* and *[docker-compose-faas.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-faas.yml)*

1. Build the image:

```sh
    $ docker build -t python-ping ./services/functions/ping
    ```

1. Test/Sanity Check:

```sh
    $ docker run -i python-ping ls
    handler.py
    requirements.txt

$ docker run -i python-ping python3 handler.py
    pong
    ```

---

### Eval

---

### Eval (v1)

https://github.com/testdrivenio/openci/tree/pycon/services/functions/eval

#### Steps:

1. Review *[Dockerfile01](https://github.com/testdrivenio/openci/blob/pycon/services/functions/eval/Dockerfile01)* and *[docker-compose-faas.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-faas.yml)*

1. Build the image:

```sh
    $ docker build -t python-eval:v1 \
        -f ./services/functions/eval/Dockerfile01 ./services/functions/eval
    ```

1. Test/Sanity Check:

```sh
    $ docker run -i python-eval:v1 ls
    handler.py
    requirements.txt
    tmp

$ docker run -i python-eval:v1 python3 handler.py
    .
    Ran 1 test in 0.000s
    ```

---

### Eval (v2)

https://github.com/testdrivenio/openci/tree/pycon/services/functions/eval

#### Steps:

1. Review *[Dockerfile02](https://github.com/testdrivenio/openci/blob/pycon/services/functions/eval/Dockerfile02)* and *[docker-compose-faas.yml](https://github.com/testdrivenio/openci/blob/pycon/docker-compose-faas.yml)*

([Watchdog](https://github.com/openfaas/faas/tree/master/watchdog) is a tiny HTTP server, bundled with each function, that receives the HTTP request from the API Gateway, triggers the function, and handles `stdin` and `stdout`)

1. Build the image:

```sh
    $ docker build -t python-eval:v2 \
        -f ./services/functions/eval/Dockerfile02 ./services/functions/eval
    ```

1. Test/Sanity Check:

```sh
    $ docker run -i python-eval:v2 ls
    handler.py
    requirements.txt
    tmp
    ```

---

## Deployment

---

### What is Container Orchestration?

As you move from deploying containers on a single machine to deploying them across a number of machines, you need an orchestration tool to manage (and automate) the arrangement, coordination, and availability of the containers across the entire system.

This is where [Kubernetes](https://kubernetes.io/) and [Docker Swarm](https://docs.docker.com/engine/swarm/) fit in along with a number of other orchestration tools - like [ECS](https://aws.amazon.com/ecs/), [Mesos](http://mesos.apache.org/), and [Nomad](https://www.nomadproject.io/).

<br>

**Which one?**

1. *Kubernetes* - large community, flexible, most features, complex setup
1. *Docker Swarm* - easier to set up, limited by the Docker API
1. *ECS* - easiest to set up, integrated with AWS

<br>

<small><em>Awesome comparison resource -> https://blog.kublr.com/choosing-the-right-containerization-and-cluster-management-tool-fdfcec5700df</em></small>

---

### Orchestration Feature Wish-list

<table style="width:100%;">
<thead>
<tr>
<th>Feature</th>
<th>Info</th>
</tr>
</thead>
<tbody>
<tr>
<td>Health checks</td>
<td>Verify when a task is ready to accept traffic</td>
</tr>
<tr>
<td>Dynamic port-mapping</td>
<td>Ports are assigned dynamically when a new container is spun up</td>
</tr>
<tr>
<td>Zero-downtime deployments</td>
<td>Deployments do not disrupt end users</td>
</tr>
<tr>
<td>Service discovery</td>
<td>Automatic detection of new containers and services</td>
</tr>
<tr>
<td>Auto scaling</td>
<td>Automatically scale resources up or down based on the load</td>
</tr>
<tr>
<td>Provisioning</td>
<td>New containers should select hosts based on resources and configuration</td>
</tr>
</tbody>
</table>

**What else?**

Load balancing, Logging across containers, monitoring...

---

### Digital Ocean Setup

1. Don't have an account? Use https://m.do.co/c/d8f211a4b4c2 to get $10 in Digital Ocean credit! 😊

1. Generate a [Personal Access Token](https://www.digitalocean.com/community/tutorials/how-to-use-the-digitalocean-api-v2)

1. Add the token to your environment:

```sh
    $ export DIGITAL_OCEAN_ACCESS_TOKEN=YOUR_TOKEN
    ```

<br>

---

### Deploy Web App

---

### Deploy Web App

#### Steps:

1. Spin up a new Docker Machine backend on Digital Ocean:

```sh
    $ docker-machine create --driver digitalocean \
      --digitalocean-access-token $DIGITAL_OCEAN_ACCESS_TOKEN web;
    ```

1. Point the Docker client at it:

```sh
    $ docker-machine env web
    $ eval $(docker-machine env web)
    ```

1. Build the images and spin up the containers:

```sh
    $ docker-compose -f docker-compose-web-prod.yml up -d --build
    ```

1. Get the IP:

```sh
    $ docker-machine ip web
    ```

Ensure you can view the app in the browser.

---

### Deploy Web App (continued...)

#### Steps:

1. Create the database:

```sh
    $ docker-compose -f docker-compose-web-prod.yml \
      run web python manage.py create_db

$ docker-compose -f docker-compose-web-prod.yml \
      run web python manage.py create_admin

$ docker-compose -f docker-compose-web-prod.yml \
      run web python manage.py create_data
    ```

1. Make sure the tests pass:

```sh
    $ docker-compose -f docker-compose-web-prod.yml \
      run web python manage.py test
    ```

1. Need to debug? Start with the logs:

```sh
    $ docker-compose -f docker-compose-web-prod.yml logs
    ```

---

### Deploy OpenFaaS

---

### Docker Swarm Mode

Docker [Swarm mode](https://docs.docker.com/engine/swarm/) provides support for container orchestration.

#### Steps:

1. Create a new Docker Machine and point the Docker client at it:

```sh
    $ docker-machine create --driver digitalocean \
        --digitalocean-access-token $DIGITAL_OCEAN_ACCESS_TOKEN openfaas;

$ docker-machine env openfaas
    $ eval $(docker-machine env openfaas)
    ```

1. Initialize Swarm mode:

```sh
    $ docker-machine ssh openfaas -- docker swarm init \
        --advertise-addr $(docker-machine ip openfaas)
    ```

---

### Docker Swarm Mode (continued...)

#### Steps:

1. Build the images:

```sh
    $ docker build -t python-eval:latest \
        -f ./services/functions/eval/Dockerfile01 ./services/functions/eval

$ docker build -t python-ping ./services/functions/ping
    ```

1. Deploy:

```sh
    $ docker stack deploy func --compose-file docker-compose-faas.yml --prune
    ```

1. Grab the IP:

```sh
    $ docker-machine ip openfaas
    ```

1. Test:

```sh
    $ curl $(echo http://$(docker-machine ip openfaas):8080)/function/ping
      pong
    ```

---

### Docker Swarm Mode (continued...)

#### Steps:

1. Test in the browser - http://DOCKER_MACHINE_IP:8080:

---

### Docker Swarm Mode (continued...)

#### Steps:

1. Build again:

```sh
    $ docker build -t python-eval:latest \
    -f ./services/functions/eval/Dockerfile02 ./services/functions/eval
    ```

1. Re-deploy:

```sh
    $ docker stack rm func
    $ docker stack deploy func --compose-file docker-compose-faas.yml --prune
    ```

1. Test:

```sh
    $ curl -X POST \
      $(echo http://$(docker-machine ip openfaas):8080)/function/func_eval -d \
      '{"namespace": "testdrivenio", "repo_name": "pycon-sample"}'

'{"namespace": "testdrivenio", "repo_name": "pycon-sample"}'
    {"status": true, "data": ".\n-----------------------------------------------------------\n
    Ran 1 test in 0.000s\n\nOK\n\n"}
    ```

---

### Docker Swarm Mode (continued...)

#### Steps:

1. Update the `OPENFAAS_URL` environment variable in *docker-compose-web-prod.yml* and update the web app.

1. Test!

<br>

#### Need to debug OpenFaaS?

Start [here](http://docs.openfaas.com/deployment/troubleshooting/).

---

### Kubernetes

#### Steps:

1. Create a new *Ubuntu 16.04.4 x64* droplet on [Digital Ocean](https://m.do.co/c/d8f211a4b4c2).

1. SSH into the box and then run the following commands to install Docker along with [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) (bootstraps a Kubernetes cluster), [kubelet](https://kubernetes.io/docs/reference/generated/kubelet/) (configures containers to run on a host) and [kubernetes-cni](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#cni) (handles the networking):

```sh
    $ apt-get update && apt-get install -y apt-transport-https
    $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
        apt-key add -
    $ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
    $ apt-get update && apt-get install -qy docker.io
    $ apt-get update && apt-get install -y kubelet kubeadm kubernetes-cni
    ```

1. Create a new user:

```sh
    $ adduser michael
    $ usermod -aG sudo michael
    $ su - michael
    ```

---

### Kubernetes (continued...)

#### Steps:

1. Deploy Kubernetes:

```sh
    $ sudo kubeadm init
    ```

1. Then run:

```sh
    $ mkdir -p $HOME/.kube
    $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
    ```

1. Install overlay network:

```sh
    $ kubectl apply -f \
        "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
    ```

1. Allow containers to be placed on the master:

```sh
    $ kubectl taint nodes --all node-role.kubernetes.io/master-
    ```

---

### Kubernetes (continued...)

#### Steps:

1. Deploy OpenFaaS:

```sh
    $ git clone https://github.com/openfaas/faas-netes
    $ cd faas-netes/
    $ kubectl apply -f ./namespaces.yml,./yaml
    ```

1. Install the [CLI](https://github.com/openfaas/faas-cli):

```sh
    $ cd ~
    $ curl -sL https://cli.openfaas.com | sudo sh
    ```

1. Clone down the project:

```sh
    $ git clone https://github.com/testdrivenio/openci \
      --branch pycon
    $ cd openci
    ```

---

### Kubernetes (continued...)

#### Steps:

1. Build the images:

```sh
    $ sudo faas-cli build -f template.yml
    ```

1. Log in to Docker Hub:

```sh
    $ sudo docker login
    ```

1. Push the images to Docker Hub:

```sh
    $ sudo faas-cli push -f template.yml
    ```

1. Deploy:

```sh
    $ sudo faas-cli deploy -f template.yml \
      --gateway=http://45.55.137.69:31112
    ```

1. Update the `OPENFAAS_URL` environment variable in *docker-compose-web-prod.yml* and update the web app.

---

### Kubernetes (continued...)

#### Steps:

1. Test!

1. Remove the droplet.

---

## Challenges

---

### Webhook

Add a [webhook](https://developer.github.com/webhooks/) to GitHub so that OpenCI is triggered on `push` events.

<div class="grid">
  <div style="float: left; width: 50%;">
    <div>
      <img src="images/webhook.png" style="max-width:85%; border:0; box-shadow: none; padding-top:20px;" alt="webhook">
    </div>
  </div>
  <div style="float: left; width: 50%;">
    <br>
    <h4 style="margin-top: 0;">Steps:</h4>
    <ul>
      <li>Deploy OpenFaaS</li>
      <li>Create a new function (good use case for an <a href="https://github.com/openfaas/faas/blob/master/guide/asynchronous.md">async</a> function?)</li>
      <li>Add the function to <em>docker-compose-faas.yml</em></li>
      <li>Set up the GitHub Webhook in the project repo</li>
      <li>Test</li>
      <li>Update the function</li>
      <li>Test again</li>
      <li>Deploy the web app stack</li>
      <li>Add route handler</li>
      <li>Update the function</li>
      <li>Final test</li>
    </ul>
  </div>
</div>

---

### Build History

Save and show the build history on the web app.

<div class="grid">
  <div style="float: left; width: 65%;">
    <div>
      <img src="images/build_history.png" style="max-width:90%; border:0; box-shadow: none; padding-top:20px;" alt="webhook">
    </div>
  </div>
  <div style="float: left; width: 35%;">
    <br>
    <h4 style="margin-top: 0;">Steps:</h4>
    <ul>
      <li>Add model</li>
      <li>Add a View history button to the UI</li>
      <li>Update <em>create_data</em> script in <em>services/web/manage.py</em></li>
      <li>Add route handler</li>
      <li>Create a new template</li>
      <li>Update the <em>get_status()</em> and <em>update_project()</em> views</li>
    </ul>
  </div>
</div>

---

### Monitoring

Create a dashboard from the Prometheus metrics with [Grafana](https://grafana.com/).

#### Steps

1. Re-deploy:

```sh
    $ docker stack rm func
    $ docker stack deploy func \
      --compose-file docker-compose-faas-full.yml --prune
    ```

1. Deploy [OpenFaaS Grafana](https://github.com/stefanprodan/faas-grafana):

```sh
    $ docker service create -d --name=func_grafana \
    --publish=3000:3000 --network=func_functions stefanprodan/faas-grafana:4.6.3
    ```

1. Navigate to [http://YOUR_IP:3000/dashboard/db/openfaas](http://YOUR_IP:3000/dashboard/db/openfaas) to view Grafana and [http://YOUR_IP:9090/graph](http://YOUR_IP:9090/graph) to view Prometheus

1. Run the following to see some action on Grafana:

```sh
    $ while [ true ]; do curl $(echo http://$(docker-machine ip openfaas):8080)/function/ping; done;
    ```

---

### More

1. *Handling Dependencies* - Can the `eval` function currently handle any dependency in the *requirements.txt* file? How about any library from the Python standard library?
1. *Private repos* - Add environment variables so that the `eval` function can clone down private repos.
1. *Language agnostic* - Refactor the `eval` function so that it can handle any language, not just Python.
1. *Production Dockerfiles* - Using [multistage](https://docs.docker.com/develop/develop-images/multistage-build/) builds, optimize the Dockerfiles for production.
1. *Scaling* - Implement [auto-scaling](https://docs.openfaas.com/architecture/autoscaling/).
1. *Branches* - Add the ability to support different branches.

---

### That's it!

What's next?

##### Check your understanding

1. [Secure](https://blog.alexellis.io/lock-down-openfaas/) the OpenFaaS functions
1. Review the [OpenFaaS Workshop](https://github.com/openfaas/workshop)
1. Go through the remaining challenges

##### Resources

1. Slides - http://mherman.org/presentations/pycon-2018
1. Repo - https://github.com/testdrivenio/openci
1. ***[Testdriven.io](http://testdriven.io/) - full course!*** ❤️
1. [How to Build 12 Factor Microservices on Docker](https://www.packtpub.com/books/content/how-to-build-12-factor-design-microservices-on-docker-part-1)
1. [Docker Cheat Sheet](https://github.com/wsargent/docker-cheat-sheet)

##### Questions?

✌️