One day I'll implement Prometheus (a more free and open software) in place of InfluxDB but I haven't had the opportunity yet, so today we talk about the TIG stack:

• Telegraf records data about the state of a machine, and sends it to
• InfluxDB which stores the data as time-series data, to be read by
• Grafana which displays real-time graphs in a dashboard

Let's start with a dashboard screenshot:

It never hurts to be able to have metrics for your various machines available at a glance, but there's so much more you can do than just imitate Task Manager. See this post and the followup for a great writeup.

Some software issues are just hardware or system issues in disguise, so it pays to have easy access to information about system health across an organisation. Whether you need to find a bottleneck or just check that all machines are running or have network connectivity, when the issue arises, you'll be glad you set this up.

Deploy It

Without further ado, some configs:

version: '2'

services:

  influxdb:
    image: influxdb:alpine
    restart: unless-stopped
    ports:
      - '12102:8086'
    volumes:
      - ./influxdb/data:/var/lib/influxdb2:rw
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=${INFLUXDB_USERNAME}
      - DOCKER_INFLUXDB_INIT_PASSWORD=${INFLUXDB_PASSWORD}
      - DOCKER_INFLUXDB_INIT_ORG=${INFLUXDB_DEFAULT_ORG}
      - DOCKER_INFLUXDB_INIT_BUCKET=${INFLUXDB_PRIMARY_BUCKET}
      - DOCKER_INFLUXDB_INIT_RETENTION=${INFLUXDB_PRIMARY_BUCKET_RETENTION}
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${INFLUXDB_ADMIN_TOKEN}

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    ports:
      - '12101:3000'
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning
    depends_on:
      - influxdb
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USERNAME}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}

INFLUXDB_USERNAME=admin
INFLUXDB_PASSWORD=correct-horse-battery-staple
INFLUXDB_DEFAULT_ORG=lchapman.dev
INFLUXDB_PRIMARY_BUCKET=weekly
INFLUXDB_PRIMARY_BUCKET_RETENTION=7d
INFLUXDB_ADMIN_TOKEN=right-brumby-capacitor-fastener

GRAFANA_USERNAME=admin
GRAFANA_PASSWORD=running-out-of-ideas

...

# Grafana
http://grafana.lchapman.dev {
        reverse_proxy localhost:12101
}

# Influx
http://influx.lchapman.dev {
        reverse_proxy localhost:12102
}

...

Some notes:

1. See my post about Caddy for more details about the Caddyfile.
2. I use the default org for my personal infrastructure, then any specific infrastructure (for a client, for example) will use a separate org (i.e. client1.lchapman.dev).
3. The choice of 7 days of retention is arbitrary, choose a time interval that suits your needs.
4. The InfluxDB volume needs to point to the influxdb2 folder. The compose file I worked from originally had been adapted from InfluxDB v1 (where the folder was just influxdb) and therefore my first couple of trial runs had no persistence.

So that's the database and the dashboard set up in containers, but where does the data come from? Initially I had Telegraf running in the compose stack but the system stats that were coming from it were related to the container rather than the system, so it becomes one of the rare softwares I install using more traditional means. See here for details. After that, deploy your config and off you go.

# For InfluxDB OSS 2:
[[outputs.influxdb_v2]]
  urls = ["https://influx.lchapman.dev"]
  token = "your-token-here-but-don't-forget-to-make-it-first"
  organization = "lchapman.dev"
  bucket = "weekly"

[agent]
  interval = "10s"
  hostname = "gateway"

# Inputs

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
  interfaces = ["eth*", "tun0", "lo"]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

More notes:

1. The URL represents the route to my InfluxDB. If you're running multiple databases you can specify multiple destinations. Neat.
2. The token doesn't exist yet. Go to Influx > Load Data > API Tokens > Generate API Token (Custom API Token). Name it something descriptive like weekly_Telegraf_gateway, give it write access to the weekly bucket and click Generate.
3. The outputs.influxdb_v2 section will be consistent across all of your machines, but the agent section will not. Set the hostname per machine.
4. The inputs section here is populated with a template I found useful, but do check out the Telegraf docs to get the data you care about. There's a whole ecosystem of plugins. Seriously, check it out.

Display It

We have a database that is being populated: great! Let's see it then. InfluxDBv2 uses its own language to query data called Flux. Flux lang is flexible and readable and while not everybody is happy that it replaced the SQL-like language used in InfluxDBv1, it's fine. Having to learn yet another language is fine. It's fine.

Use the Data Explorer within your Influx web interface to start crafting your queries, then click Script Editor to access the actual Flux query.

from(bucket: "weekly")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "mem")
  |> filter(fn: (r) => r["_field"] == "used")
  |> filter(fn: (r) => r["host"] == "gateway")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

It's all pretty easy to follow, except maybe the v variable which is supplied by the viewer, in this case the Data Explorer but later it will be Grafana. The highest level of data organisation is in a "measurement" which contains some "field". Past that there are tags, such as the host tag here which is set by Telegraf as per our config.

The aggregateWindow function is worth explaining. Without it, the request could return an arbitrarily large number of datapoints, and while I do love high levels of precision, there's a tradeoff to be made regarding the load placed on the database, especially if your dashboard is set to update every 10 seconds. The value passed to every could be a constant, like 1m (1 minute), or it can be dynamic, like v.windowPeriod: a 24 hour window produces 2m period, 7 days is 15m, 2 hours is 10s and anything less is actually doing nothing because my data collection interval is 10 seconds anyway. Set createEmpty to true if your machine ever powers off; there's not much reason to leave it false regardless. For my graphs of CPU usage I set the aggregate function to max rather than mean because I'm looking for bottlenecks and mean hides them.

A new Grafana instance may require you to head to the Connections menu and configure the InfluxDB connection. Use the Flux query language rather than InfluxQL, it's the one in active development. I found that the URL field works fine as either of http://influxdb:8086 or https://influx.lchapman.dev/, with the former being preferable because traffic need not exit the container stack in this case. Turn off the basic auth toggle. Set the organization and default bucket as per your environmental variables. Min time interval of 10s is correct here because that's my minimum data write period.

The token I supply to Grafana doesn't exist yet either. I create a read-only token for the weekly bucket called weekly_Grafana. Paste it in, click save & test and you should see a green tick and the message "datasource is working. 1 buckets found".

Now you'll want to go to Dashboards > New Dashboard > Add vizualization, paste the query from Influx into the A query, then click somewhere outside the query textbox. Grafana will see your changes and load the data. If it looks good, click Save. In my case, I'll want to fiddle with the options a bit. Set the unit to bytes (SI) in Standard Options and while I'm there I'll set a min and max. A proper title helps too.

One last lesson before I wrap this up: computed series names. If you're doing a per-core CPU graph, you'll immediately see an issue:

Note the map function to apply a transform to every data point, converting a measurement of core idleness to core busy-ness. Also note the legend being awful due to series names being just daft. I suppose it's a sensible default behaviour but it falls apart in this particular use case. Here's our silver bullet:

  |> map(fn: (r) => ({ _value:r._value, _time:r._time, _field:r.cpu }))

Time and value are passed through but the field name is overwritten by just the value of the cpu tag. Handy! I suppose I could write a function to strip the "cpu" part from "cpu7" but... I'm not going to. The result:

The eagle-eyed amongst you will notice a lack of optimisation: why not move the first map function after the aggregateWindow, reducing the number of rows the map operates on? Doing so reduces the query time from 500ms to 350ms. Moving the minus operation into the second map and removing the first altogether brings it down to 280ms, for a total reduction of processing time of 45% - not bad. My sample sizes for this data were ~2, hardly rigorous, but there really is a measurable change. Combining the filter calls from 4 to 1 (a series of and statements) however does not elicit any performance gains worth noting, and reduces readability to miserable lows.

There's plenty more I'll learn about Flux-lang as I go, but for now, I don't mind it. It seems to have the tools I need. I'll try Prometheus one day, but not today.

A quick refresher on the gateway model I employ: one machine is accessible from the internet, minimising the exposed surface area of my network. This machine is the gateway and serves all requests. From a DNS perspective, all of my subdomains are CNAME records pointing to the gateway.

Nginx and Apache are the main names in the web server game, and while I've used them both, my eye was drawn to Caddy. It's sleek and modern and addresses most of my gripes with the older players. I quickly fell in love with Caddy and one day it will be my downfall, but until then, I'll use it as my hammer.

I began with a simple configuration and have since found myself running multiple layers of Caddy, but I'll get to that further down.

As I run Debian-based operating systems most of the time, I followed these instructions to install the stable release. If you're not familiar with systemd or just want a refresher, there's a page in the docs that explains how to manage it.

Configuring Caddy is done by loading a "Caddyfile" into the running service, or at process launch with caddy run --config /path/to/Caddyfile. It's easy enough to make a formatting mistake, so I operate a "working document" that resides in my home directory. Caddy provides a function to format the document into a valid Caddyfile which is invoked as caddy fmt working > Caddyfile. Copy that to /etc/caddy and reload and you're good to go.

#!/bin/bash

caddy fmt working > Caddyfile && \
sudo cp Caddyfile /etc/caddy/ && \
rm Caddyfile && \
caddy reload --config /etc/caddy/Caddyfile

So what does my working config look like?

www.lchapman.dev, lchapman.dev {
        # Not doing a website yet, redirect to the blog
        redir https://blog.lchapman.dev
}

blog.lchapman.dev {
        reverse_proxy 10.8.0.2
}

# Other services in similar formats
# ...

You might find you don't need to use fmt as part of your workflow if you don't make mistakes, but I move fast and make plenty.

Now we get to the part where Caddy is brought into line with the likes of Nginx and Apache: the Caddyfile is just a shorthand for the real config. Use adapt function to parse your Caddyfile and return the result, and you'll be greeted with this:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":443"],
          "routes": [{
            "match": [{
              "host": ["blog.lchapman.dev"]
            }],
            "handle": [{
              "handler": "subroute",
              "routes": [{
                "handle": [{
                  "handler": "reverse_proxy",
                  "upstreams": [{
                    "dial": "10.8.0.2:80"
                  }]
                }]
              }]
            }],
            "terminal": true
          }, {
            "match": [{
              "host": ["www.lchapman.dev", "lchapman.dev"]
            }],
            "handle": [{
              "handler": "subroute",
              "routes": [{
                "handle": [{
                  "handler": "static_response",
                  "headers": {
                    "Location": ["https://blog.lchapman.dev"]
                  },
                  "status_code": 302
                }]
              }]
            }],
            "terminal": true
          }]
        }
      }
    }
  }
}

It is verbose but that's standard for web server configuration files. Caddy allows us to avoid maintaining this JSON behemoth by hand, and I am very grateful for it.

Note how line 6 shows that Caddy is listening to port 443. By failing to specify http or https in the Caddyfile, Caddy defaults to TLS via LetsEncrypt, handling the process of requesting, serving and updating HTTPS certificates behind the scenes. TLS is terminated at the gateway and all traffic in the lchapman.dev private network is passed around via HTTP, with the eventual response returned and encrypted by the gateway on the way out into the WAN.

This is a great configuration - it's simple, easy, and no-maintenance. But what if I don't want to terminate TLS at the gateway? Say, perhaps, some software running downstream expects encrypted traffic. Say, perhaps, the correct course of action is to modify an Nginx config in a container to expect plaintext traffic; but I don't feel like doing that, so what to do?

Caddy does offer a solution! Alas, it's not so simple. I'll do it anyway though because Caddy is my hammer and I spy a sghiny new nail.

If you refer above to the JSON config you'll see the "apps" key at the top level and "http" as its first and only child. The http app is great but it has limits and unfortunately we're looking for a feature that is out of scope. Enter layer4. Layer4, or caddy-l4, or Project Conncept, is an app written by Matt Holt (who is the major contributor to and sponsor of Caddy) to extend Caddy's capabilities to layer 4, where Caddy's base functionality mainly deals with layer 7. I expect layer4 will one day be shipped with Caddy proper, but for now we must build it ourselves.

Enter xcaddy - Custom Caddy Builder. They really have thought of everything. Install it with go or grab it with apt. Build your own caddy with it. It's too easy.

$ xcaddy build latest \ --output caddy-l4 \ --with github.com/mholt/caddy-l4

I opted to name the binary differently to the binary installed with apt and copied it to /usr/bin alongside apt-caddy. I also copied the service files in /lib/systemd/system (caddy.service and caddy-api.service) and made new ones for caddy-l4.

[Unit]
Description=Caddy Layer4
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy-l4 run --environ --config /etc/caddy/config.json
ExecReload=/usr/bin/caddy-l4 reload --config /etc/caddy/config.json --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

The idea is that caddy-l4 listens to port 443, follows a minimal amount of rules, and delegates most of the work back to regular caddy which now listens to port 8443. Layer4 does not yet have Caddyfile support so we'll have to implement the config in JSON. I built on the examples on the Github repo and had help from the community server on Discord and came up with this:

{
  "apps": {
    "layer4": {
      "servers": {
        "first_layer": {
          "listen": [
            ":443",
            ":80"
          ],
          "routes": [{
              "match": [{
                  "tls": {
                    "sni": [
                      "nginx.service.lchapman.dev"
              ]}}],
              "handle": [{
                  "handler": "proxy",
                  "upstreams": [{
                      "dial": [
                        "10.8.0.2:443"
              ]}]}]},
            {
              "match": [{
                  "tls": {}
              }],
              "handle": [{
                  "handler": "proxy",
                  "upstreams": [{
                      "dial": [
                        "localhost:8443"
            ]}]}]},
            {
              "match": [{
                  "http": []
              }],
              "handle": [{
                  "handler": "proxy",
                  "upstreams": [{
                      "dial": [
                        "localhost:8080"
  ]}]}]}]}}}}
}

Please excuse the formatting atrocities committed here, I wanted to save you from endless scrolling and keep the meat of the content on one screen.

If the request is HTTPS with a particular SNI (FQDN), we match it and forward it to the next layer of Caddy with TLS intact. For everything else, if it's on port 80 then forward to port 8080 and 443 to 8443. Port 80 is interesting here because I won't receive web traffic on it via my domain because .dev domains implement HSTS at the domain level, meaning web browsers refuse to send an insecure HTTP request to lchapman.dev at all. This is good, because developers should be expected to lead the way with security. Why open port 80 at all then, and why bother forwarding it? Caddy uses ACME to manage certificates and the process to obtain or renew a certificate requires port 80 - it's difficult to bootstrap TLS if TLS is required to do so. Our outer caddy-l4 instance doesn't manage any certificates so we forward all port 80 traffic to the inner caddy, which does manage the certificates.

A slight modification to the original Caddyfile is required to make this work. Add the following to the top:

{
	admin 0.0.0.0:2020
	http_port 8080
	https_port 8443
}

The admin line prevents the two caddy processes from trying to bind to the same port (2019) to run their API services. The other two lines have hopefully been explained ahead of time.

That's the full explanation of the gateway's multi-Caddy configuration. To provide the full picture, I'll give a brief rundown of the third and final instance of Caddy which resides at 10.8.0.2. It runs a Caddyfile but it specifies http or https rather than omitting them.

# ______           __     __         ___ __ __        
#|      |.---.-.--|  |.--|  |.--.--.'  _|__|  |.-----.
#|   ---||  _  |  _  ||  _  ||  |  |   _|  |  ||  -__|
#|______||___._|_____||_____||___  |__| |__|__||_____|
#                            |_____|

#	port	subdomain	service
#	2368	blog		Ghost
#	3000	nginx.service	TLS-only service
#	...	...		...

# Blog
http://blog.lchapman.dev {
	reverse_proxy localhost:2368 {
		trusted_proxies private_ranges
	}
}

# TLS-only service
https://nginx.service.lchapman.dev {
	reverse_proxy https://localhost:3000 {
		trusted_proxies private_ranges
		transport http {
			tls_server_name nginx.service.lchapman.dev
		}
	}
}

# ...

Some services work without specifying trusted proxies, some don't. It never hurts to include the directive though, so go nuts with it. Ghost requires it. The TLS-only service also requires us to specify the tls_server_name otherwise it will send a request to localhost which will be received by localhost but the whole problem we're solving is that this specific Nginx setup is expecting all traffic to be addressed to nginx.service.lchapman.dev and be encrypted with the proper certificate. We overwrite it here and life is good.

That's all there is to it. Let's say I want to spin up a new container with a new service: hello-world. (Dockerhub link)

It's pretty easy:

1. Set up a compose file with a hello-world service using the hello-world image and expose it on port 3001
2. docker compose up -d
3. Set up a DNS entry with type CNAME, hostname hello.lchapman.dev, destination gateway.lchapman.dev.
4. Add an entry to the gateway working config with matcher hello.lchapman.dev and reverse_proxy to 10.8.0.2. Reload with reload.sh.
5. Add an entry to the 10.8.0.2 working config with matcher http://hello.lchapman.dev and reverse_proxy to http://localhost:3001. Reload with reload.sh.
6. Send a request to https://hello.lchapman.dev/ and receive an error because Caddy doesn't have a certificate for that domain yet
7. Come back a minute later, refresh and see hello world

The thought of going back to Nginx for this sends shivers down my spine.

BONUS CONTENT: Caddy in Docker

Yes. Do it. I almost did do it, especially when I was in the process of deploying caddy-l4, but I ultimately didn't because my gateway is a tiny VM with minimum specs (for minimum pricetag) and I didn't want to install Docker on it at all.

The way I see it, anything Nginx/Apache can do, Caddy can probably do in a way that is nicer to work with. Better? I'll let somebody with a lot more knowledge decide that. Caddy is nicer though, I'll die on that hill.

I've always had a passion for free and open source software. Naturally this has evolved into a self-hosting obsession. Some softwares that would otherwise be gated behind large upfront costs or subscription fees have perfectly serviceable free alternatives, at the cost of having to administer them yourself.

I have a couple of machines running Ubuntu that function as servers. They live in a home network, and while one could forward ports and expose them to the WAN, I choose not to do that out of an abundance of caution. Instead I use what I call a "gateway machine".

The gateway machine is actually a VM run on the cloud, specifically the cheapest one I could find that I liked. I've used DigitalOcean as a cloud provider for work and found them reliable and reasonably priced, and their wide range of tutorials is also fantastic and has guided me through many processes. The gateway is a single core "droplet" instance with minimal memory and costs $4 USD per month.

The main function of the gateway is to have a static, public IP address that my domain (lchapman.dev) is pointed to. Ports are opened as needed, but I try to avoid exposing any more ports than are strictly necessary, so 443 is the main one. ".DEV" domains are HSTS domains (HTTP Strict Transport Security) which means the browser will reject any connections that aren't utilising TLS (Transport Layer Security), so all traffic must be properly encrypted. Encrypted web traffic conventially uses port 443.

Your best friend for encrypting web traffic and facilitating HTTPS is Let'sEncrypt, the free and open certificate authority. The method I learned first for achieving HTTPS traffic was using certbot to generate certificates, then using Nginx to encrypt and serve content with those certificates. Nginx is as old as the hills and all powerful but there are modern alternatives that I find much more comfortable - Nginx is still the car you take to the track to race, but not my preference for a daily driver.

Caddy Server (specifically Caddy 2) does everything I need and does it in a human-friendly way, so I'm very happy to have discovered it.

Each VM or machine that I run that serves any kind of content runs Caddy which listens to incoming traffic and handles them based on rules. My gateway machine receives all incoming traffic and Caddy handles it all, routing it wherever it needs to go.

Here's a snippet from the gateway Caddyfile (config):

www.lchapman.dev, lchapman.dev {
        redir https://blog.lchapman.dev
}

blog.lchapman.dev {
        reverse_proxy 10.8.0.2
}

It's short, simple and readable. All traffic to www.lchapman.dev or lchapman.dev is redirected to blog.lchapman.dev. Requests to blog.lchapman.dev are passed to 10.8.0.2, a machine on a local network. Any further domains that need to be routed are done so with similar directives.

Using a reverse proxy to send traffic to 10.8.0.2 means that clients only know they are interacting with blog.lchapman.dev and nothing more, but the actual blog is hosted elsewhere. Ghost is the blog software and is hosted on a machine connected to a VPN.

The only software running on the gateway machine besides Caddy is an OpenVPN server. All machines that I host softwares on are connected as clients to that VPN. One of the benefits of this configuration are that I can pick up my machines and move them to a new location without having to reconfigure my network - they reconnect to the VPN and everything works as before. This flexibility is very handy for self-hosted machines.

All that remains to discuss here are the servers that host my softwares. Currently I don't run them as a cluster but I plan to down the track. Each machine receives requests that are handled by Caddy. Here's an example:

http://blog.lchapman.dev {
        reverse_proxy localhost:2368 {
                trusted_proxies private_ranges
        }
}

Note how HTTP is specified: HTTPS is terminated at the gateway and all of my self-hosted services don't need to worry about TLS or certificates. I do need to turn off TLS for some softwares, e.g. GitLab, but that's usually provided as a config option.

The unencrypted request is received and passed on to Ghost on port 2368. Without specifying "trusted_proxies private_ranges", Ghost will try to redirect requests to https and there will be a request loop, but that's a story for another post.

Check out Traffic Management with Caddy for a full writeup about how I use Caddy!