How Docker Broke the Internet for Me

We have seen two major outages in the last weeks caused by AWS and Azure networking issues. Both outages had a significant impact on services running in the cloud and affected a large number of users. The self-hosting people were laughing about it, but some got struck with the recent Docker 29.0.0 release. It simply increased the minimal API version from 1.24 to 1.44.

This change broke Traefik and some tooling, like the testcontainers-java. The maintainers of Traefik already released a fix, and probably created some stress for them. A lot of issues poped up in their GitHub repository, like this one https://github.com/traefik/traefik/pull/12256. I felt sorry for them, many users use their stuff - including me - to drive HTTP/HTTPS traffic to their websites. It was great to see how quickly shared workarounds. The Traefik maintainers were able to provide and ship a fix in a new release very quickly. All you need to do is upgrade to 3.6.1, and you are good to go again.

macOS with Apple Silicon and x86-64-v3 support

I’m maintaining container images in the OpenNMS ecosystem. My local machine is a Mac with Apple Silicon. While I was upgrading some base images using RHEL UBI 10 / CentOS 10 images, I noticed the following error message.

0.112 Fatal glibc error: CPU does not support x86-64-v3

That happens if you try to run on an emulated x86 architecture running on ARM. After some investigation, I’ve found an article in the Red Hat developer forum talking about upgrading the Microarchitecture level from v2 to v3. The problem introduced is that v3 isn’t supported. I’m using Orbstack and you can reproduce this problem simply running the following docker command.

Streaming telemetry with gNMI

This document describes how you can set up a lab environment for testing gNMI using a virtual HPE Juniper router running Junos 24.2 R1-S2.5. It shows how you can collect metrics for interfaces and CPU utilization collected with OpenNMS. This isn’t something I do every day, so I just preserve this here for my future self and the ones who want to get up to speed a bit quicker.

What you will learn

• Enable gNMI over gRPC with sensors on a HPE Juniper router
• Verify the gRPC connections and configuration
• Enable OpenNMS Horizon to subscribe to gRPC streams for sensor data
• Set up TLS on gRPC using a self-signed certificate authority
• Visualize the collected data in Grafana

Limitations

• Streaming telemetry can only be configured from the OpenNMS Core instance, see enhancement NMS-18064
• Metric labels for network interfaces are right now just resource id strings

Before you begin

You need the following environment to get a lab running:

Hello Containerlab with Orbstack

I still remember signing up on DockerHub 11 years ago. Learning how to build container images on real world projects is definitely a plus. Having a software and some use cases in the back of your mind, helps you to achieve things quicker and with some purpose. I’ve started to work with containerlab to build some network test environments mainly for three use cases:

• Layer 2 network topologies for network monitoring tests using LLDP, CDP and Bridge-MIB
• Routing topologies with protocols like BGP, OSPF or IS-IS in general
• Some vendor specific network gear for monitoring with SNMP and streaming telemetry

Playing with OpenNMS from this perspective opens some use cases around Netflow, IPFIX, BGP monitoring, and SNMP in general. I was using Docker4Mac for a very long time. At some point in time I have switched to colima which was slim and easy to use. With switching to ARM on my Mac it got a bit more complicated.

SNMP vs. Prometheus – On The Wire

I’ve been working with network monitoring tools for a long time. Working with network devices, there is still today a very high probability you have to deal with SNMP. If you work with modern applications or infrastructure, especially in dynamic environments with containers, you will inevitably run into Prometheus and its ecosystem when you need to design and build monitoring solutions.

By design, both agents have different goals in mind. The world was very different when SNMP aimed to be “simple” in 1980 as it was for Prometheus in 2012. To give you some idea, this was the world when people worked with SNMP RFCs in 1983:

A cookbook for a K8s playground

In my last weeks, I had to work with deployments of OpenNMS with Kubernetes. Instead of spending dollars on cloud providers for my lab, I’ve bought a beefy cheap box for my home network for less than 1.500,-€ about a year ago. It saved me probably already more than I would have spent on similar resources in the cloud for my playgrounds. It has an Intel(R) Core(TM) i9-10880H CPU, 64 GB RAM, and 2 TB SSD which has enough steam to run VMware ESXi on it. The main goal for this lab is, to have something you can quickly ditch into the bin and rebuild from scratch without worrying, and at the beginning of something new, you’ll break it a lot for sure :)

ioquake3 in a container

Once upon a time, people had no internet access, or it was very expensive and slow. To have some fun, they spent weekends with their friends and hung out playing games over a local area network. It was so much fun it gained some interest and the space from your friends house was just too small for all the people. Parents and families went crazy and electric bills went through the roof. Locations got bigger and peoples needed a bit more advanced networks and dedicated servers. You started writing your first programs managing tournaments and automating dedicated servers. … it was long before we talked about something like Ansible or Salt Stack :)

This website is not available in your country

Note to my future self. You might find yourself in a situation where you are blocked from an internet service because of your geographic location. A way to get around is using a VPN from a friend in a non-evil geographic location or using a 3rd party VPN service. Sometimes you don’t have that option right away and you need something quickly - Torbrowser for the rescue.

Download and install the latest Torbrowser. Edit the torrc file and set a strict exit node from a country of your choice.

Setting up a VMware Test environment

To test functions like importing OVA files in VMware ESXi and with vCenter the trial phase and a local deployment can be used. You need the following requirements:

• VMware Workstation on Windows or VMware Fusion on Mac OSX
• VMware Hypervisor ISO image to install the ESXi host system
• VMware vCenter ISO image for local deployment

If you just want to learn how it works you can get a 60 days trial license registering an account on VMware. I had to manually select a Bridged network for the ESXi server. The vCenter appliance used a Bridge network by default. connected them to a bridged network. You can use DHCP if you want, but you need to make sure they get the same IP addresses afterwards.

VMware multipathd log spam

While I was deploying Loki with Promtail I’ve seen a lot of log spam from Ubuntu virtual machines in my VMware environment. As a note to myself and for some others who want cleaner system logs – here is what I’ve found to get rid of it.

The log entries look like these here:

2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sgio uid: No such file or directory
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sysfs uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get udev uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: add missing path

The best article I’ve found was from SUSE describing the problems source. In a nutshell, VMware doesn’t provide information needed by udev to generate the /dev/disk/by-id. To solve the problem you have to set in the Virtual Machine the attribute disk.EnableUUID=true. For the reason I have a few VM’s it’s pretty tidious to do all these things manually.