... it's better to have good questions

OpenNMS Horizon, Docker, Traefik and Let's Encrypt

2021-08-11 3 min read technology Ronny Trommer

I work from home for over 6 years now and especially when you like networking, want to get stuff up and running and breaking it - you start looking around :) You’ve heard about k8s, k{0,3}s or Microk8s but you don’t want to use it to run your private blog and you find yourself in a spot where the benefits running stuff in containers justify the pain - this article might be something for you :)

My use case is, providing public facing web sites reasonabily quick and simple. The setting for me was pretty straight forward and always the same. You want to make XYZ web application publicly available and you want a Let’s Encrypt TLS certificate.

Setting up TLS in XYZ web application is always differently weird and painful. My receipe for it, keep the web app as it is - lock it down and use a TLS reverse proxy instead.

In my lab environment I’ve named TLS ingress and used mostly nginx for it. I’ve found it cumbersome with the certbot TLS Let’s Encrypt dance, I got it running but never liked the setup nor the maintenance of it. I’ve ditched nginx for this job a while ago and use Traefik instead in my container environments and so far I’m happy with it.

The Traefik killer features for me:

  • Let’s Encrypt built-in, I just don’t have to care anymore
  • Dynamic event driven service discovery using the Docker daemon when you run on the same box
  • Simple configuration by just annotating my app containers with labels

If you want to give it a try, here an example running OpenNMS Horizon behind Traefik. In this example here you run both on the same box.

I create a dedicated network bridge on my Linux box which has all the external traffic between Traefik an the app containers.

docker create network public-ingress

Here is my ingress service described in a docker-compose.yaml which I have in a ingress folder. I’ve created systemd service unit to start it up on system boot.

Replace the Let’s Encrypt mail address YOUR-MAIL-HERE in the following configuration:

version: '3'

    external: true

    image: "traefik:v2.4"
    container_name: ingress
      - "--log.filePath=/logs/traefik.log"
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.addServicesLabels=true"
      - "--accesslog=true"
      - "--accesslog.filepath=/logs/access.log"
      - "--entryPoints.metrics.address=:8082"
      - "--metrics.prometheus.entryPoint=metrics"
      - "--log.level=INFO"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - ""
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.httpchallenge=true"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
      # - "--certificatesresolvers.myresolver.acme.caserver="
      - ""
      - ""
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
      - './letsencrypt:/letsencrypt'
      - './logs:/logs'
      - public-ingress
      - "80:80/tcp"
      - "8082:8082/tcp"
      - "443:443/tcp"

If you just want to fiddle around and get yourself familiar how Traefik obtains TLS certificates and hand shakes, you can uncomment the CA server line above. You will use now the staging environment and you don’t get temporarily locked out for hitting the API endpoints too often. I monitor Traefik with Prometheus, you can remove this parts if you want.

We have some OpenNMS stacks preconfigured and explain it on a minimal example. Set your OpenNMS web UI to use https instead of http by creating a properties file. When you use the stack-play example you can do it like this:

cd minimal-horizon
mkdir ./container-fs/opt/opennms-overlay/etc/
echo "opennms.web.base-url = https://%x%c/" > ./container-fs/opt/opennms-overlay/etc/

Annotate OpenNMS Horizon container with the following labels. You have to replace YOUR-FQDN-HERE with the public DNS name you want to have your OpenNMS reachable. There is no reason for static port publishing anymore more, traefik routes it for you based on the traefik.port label.

      ## DELETE ME!!!
      ## - "8980:8980/tcp"
      - "traefik.enable=true"
      - "traefik.http.routers.horizon.rule=Host(`YOUR-FQDN-HERE`)"
      - "traefik.http.routers.horizon.entrypoints=websecure"
      - "traefik.http.routers.horizon.tls.certresolver=myresolver"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.port=8980"

As soon you start OpenNMS Horizon, Traefik gets notified and does all the Let’s Encrypt certificate handling for your Horizon host.

So long and thanks for all the fish …