VMware multipathd log spam

While I was deploying Loki with Promtail I’ve seen a lot of log spam from Ubuntu virtual machines in my VMware environment. As a note to myself and for some others who want cleaner system logs – here is what I’ve found to get rid of it.

The log entries look like these here:

2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sgio uid: No such file or directory
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sysfs uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get udev uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: add missing path

The best article I’ve found was from SUSE describing the problems source. In a nutshell, VMware doesn’t provide information needed by udev to generate the /dev/disk/by-id. To solve the problem you have to set in the Virtual Machine the attribute disk.EnableUUID=true. For the reason I have a few VM’s it’s pretty tidious to do all these things manually.

Installing Node Exporter on Linux

In OpenNMS Horizon 28+ is now a PrometheusCollector available. It scrapes the metrics from the provided exporter pages and allows to add data collections. As of speaking today it is not 100% feature complete, scraping data types like histograms is not implemented yet. If you want to play around here is a quick way to get the Linux Node_Exporter installed.

The following steps are executed in a root shell with sudo -i.

Hardening SSH for audit

Running a server in the public requires some additional work. Especially if you want management access via SSH for Ansible or if you want break stuff manually with fiddeling around :)

You can run an SSH audit of your public server using https://www.sshaudit.com. This section here is a very condensed way to get an A rating.

Just use strong host key for authentication of the host

# file: /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Delete existing keys and re-generate the RSA and ED25519 keys

UCARP and High Availability

If you have ever played with BSD you probably ran into CARP. It allows you to build a high available service which is provided by two physical servers behind a virtual shared IP address. The CARP nodes define a master and a backup system. A master serves the content and if the master crashes, the backup system takes over automatically the virtual IP (VIP) and the client won’t notice.

Disclaimer: You should be aware this setup will not share load and increase your network throughput. It just used to increase availabilty and room to do maintenance without bringing your service down.

Continue reading

Docker build and cache invalidation

Right now I’m working with my work mates @opennms integrating the docker image building in our CI/CD environment. We build our container image based on CentOS and we noticed the caching doesn’t work for ${reasons}.

Running a docker build -t myimage . ended up always in installing packages from the official yum repositories even we haven’t changed anything in the Dockerfile.

To understand things better, I went back to drawing board and started with a simple example and rebuilding things step by step to understand when gets the docker build cache unnecessarily invalidated.

Docker, Java, Signals and Pid 1

Running a Java application in a container seems to be very easy. The devil is in the details and I want to shed some light on the PID 1 problem when you run Java applications in containers. Theoretical, processes in containers should not have state so you just don’t care, but reality is different.

Signals are used to message running process to behave in certain ways. A common case is to terminate a process nicely sending a SIGTERM to the process identified by the process id using the tool ps. People can implement logic shutting down an application gracefully. When you issue the command kill <pid> you send a SIGTERM to a process. We have other signals you can use, especially when application provide terminals. Hitting CTRL + C sends SIGINT to the terminal process. Some signals are quite out of time, the signal to hangup SIGHUP comes from terminal applications. It is often used nowadays telling a running process to re-read configurations.

SSL and Java

Running applications with a current Java is not a big deal thanks Let’s Encrypt. This article describes what happens if you want to authenticate your OpenNMS against LDAP using SSL with a self-certified certificate.

First of all I assume you have confiured verything so you can authenticate against LDAP in plaintext and you got a role mapping as you wanted it. If not you can have a look here.

So the naive approach would be, just changing the line in your activeDirectory.xml from

Send notifications with Signal

In some cases it is nice to have notifications from OpenNMS in a separate channel on a smartphone and you don’t want to pay for SMS. Here is a tutorial where I use Signal using the signal-cli.

This Howto will describe how to download the latest signal-cli tool, link it to your existing Signal account and how to configure OpenNMS to use it as a notification target. You should have already an OpenNMS Horizon or Meridian running and you need a Signal account with the Signal app installed and configured on your smartphone.

Everyone can change it and why you shouldn't

Open-source software is great. If there is something you don’t like, you can at least - try to change it. A lot of open-source software out there is not primarily used by private people. There are many companies who provide professional services around open-source software. Some of them try to enhance the appearance with custom User Interfaces, their company logo to fit their own Corporate Identity. There are several motivations, mostly they want to be distinguished on the marked or need some easy to maintain little customization which allows their sales guys easier to sell a project and not using the software from the community projects website. How hard could it be to make something pretty, right?

Guidance to Survive Monitoring

While working in the monitoring field for a long time, here are some rules I try to follow when requirements go awry.

Rule #1: Only create an alert when human interaction is required

When you setup a monitoring, it tends to get noisy very quickly. The problem is, people want to know everything and want to monitor everything. You tend to build a system which sends you a lot of alarms and you will get alarm fatique. To get most out of your monitoring solution, you have to always keep in mind Rule #1. When you alert for something, ask yourself is it really necessary to wake some one up in the middle of the night. There is nothing more horrible than waking someone up and it is a false alert.