SNMP vs. Prometheus – On The Wire

I’ve been working with network monitoring tools for a long time. Working with network devices, there is still today a very high probability you have to deal with SNMP. If you work with modern applications or infrastructure, especially in dynamic environments with containers, you will inevitably run into Prometheus and its ecosystem when you need to design and build monitoring solutions.

By design, both agents have different goals in mind. The world was very different when SNMP aimed to be “simple” in 1980 as it was for Prometheus in 2012. To give you some idea, this was the world when people worked with SNMP RFCs in 1983:

A cookbook for a K8s playground

In my last weeks, I had to work with deployments of OpenNMS with Kubernetes. Instead of spending dollars on cloud providers for my lab, I’ve bought a beefy cheap box for my home network for less than 1.500,-€ about a year ago. It saved me probably already more than I would have spent on similar resources in the cloud for my playgrounds. It has an Intel(R) Core(TM) i9-10880H CPU, 64 GB RAM, and 2 TB SSD which has enough steam to run VMware ESXi on it. The main goal for this lab is, to have something you can quickly ditch into the bin and rebuild from scratch without worrying, and at the beginning of something new, you’ll break it a lot for sure :)

ioquake3 in a container

Once upon a time, people had no internet access, or it was very expensive and slow. To have some fun, they spent weekends with their friends and hung out playing games over a local area network. It was so much fun it gained some interest and the space from your friends house was just too small for all the people. Parents and families went crazy and electric bills went through the roof. Locations got bigger and peoples needed a bit more advanced networks and dedicated servers. You started writing your first programs managing tournaments and automating dedicated servers. … it was long before we talked about something like Ansible or Salt Stack :)

This website is not available in your country

Note to my future self. You might find yourself in a situation where you are blocked from an internet service because of your geographic location. A way to get around is using a VPN from a friend in a non-evil geographic location or using a 3rd party VPN service. Sometimes you don’t have that option right away and you need something quickly - Torbrowser for the rescue.

Download and install the latest Torbrowser. Edit the torrc file and set a strict exit node from a country of your choice.

Setting up a VMware Test environment

To test functions like importing OVA files in VMware ESXi and with vCenter the trial phase and a local deployment can be used. You need the following requirements:

• VMware Workstation on Windows or VMware Fusion on Mac OSX
• VMware Hypervisor ISO image to install the ESXi host system
• VMware vCenter ISO image for local deployment

If you just want to learn how it works you can get a 60 days trial license registering an account on VMware. I had to manually select a Bridged network for the ESXi server. The vCenter appliance used a Bridge network by default. connected them to a bridged network. You can use DHCP if you want, but you need to make sure they get the same IP addresses afterwards.

VMware multipathd log spam

While I was deploying Loki with Promtail I’ve seen a lot of log spam from Ubuntu virtual machines in my VMware environment. As a note to myself and for some others who want cleaner system logs – here is what I’ve found to get rid of it.

The log entries look like these here:

2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sgio uid: No such file or directory
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sysfs uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get udev uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: add missing path

The best article I’ve found was from SUSE describing the problems source. In a nutshell, VMware doesn’t provide information needed by udev to generate the /dev/disk/by-id. To solve the problem you have to set in the Virtual Machine the attribute disk.EnableUUID=true. For the reason I have a few VM’s it’s pretty tidious to do all these things manually.

Installing Node Exporter on Linux

In OpenNMS Horizon 28+ is now a PrometheusCollector available. It scrapes the metrics from the provided exporter pages and allows to add data collections. As of speaking today it is not 100% feature complete, scraping data types like histograms is not implemented yet. If you want to play around here is a quick way to get the Linux Node_Exporter installed.

The following steps are executed in a root shell with sudo -i.

Hardening SSH for audit

Running a server in the public requires some additional work. Especially if you want management access via SSH for Ansible or if you want break stuff manually with fiddeling around :)

You can run an SSH audit of your public server using https://www.sshaudit.com. This section here is a very condensed way to get an A rating.

Just use strong host key for authentication of the host

# file: /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Delete existing keys and re-generate the RSA and ED25519 keys

UCARP and High Availability

If you have ever played with BSD you probably ran into CARP. It allows you to build a high available service which is provided by two physical servers behind a virtual shared IP address. The CARP nodes define a master and a backup system. A master serves the content and if the master crashes, the backup system takes over automatically the virtual IP (VIP) and the client won’t notice.

Disclaimer: You should be aware this setup will not share load and increase your network throughput. It just used to increase availabilty and room to do maintenance without bringing your service down.

Continue reading

Docker build and cache invalidation

Right now I’m working with my work mates @opennms integrating the docker image building in our CI/CD environment. We build our container image based on CentOS and we noticed the caching doesn’t work for ${reasons}.

Running a docker build -t myimage . ended up always in installing packages from the official yum repositories even we haven’t changed anything in the Dockerfile.

To understand things better, I went back to drawing board and started with a simple example and rebuilding things step by step to understand when gets the docker build cache unnecessarily invalidated.