Quick manual Kafka OpenNMS stack

We have gathered a few ready-to-run Docker stacks in our public stack-play GitHub repository. But sometimes you need Kafka, Zookeeper and OpenNMS quickly on a baremetal deployment without Docker. Here a few quick notes how to get the bare minimum up and running.

Zookeeper

Install OpenJDK 11 JRE

sudo apt install -y openjdk-11-jre

Create a user for zookeeper

sudo adduser --system --home /opt/zookeeper --disabled-login zookeeper

Create a logs directory

sudo mkdir /var/log/zookeeper
sudo chown zookeeper:nogroup /var/log/zookeeper -R

Install zookeeper

Setting up a VMware Test environment

To test functions like importing OVA files in VMware ESXi and with vCenter the trial phase and a local deployment can be used. You need the following requirements:

• VMware Workstation on Windows or VMware Fusion on Mac OSX
• VMware Hypervisor ISO image to install the ESXi host system
• VMware vCenter ISO image for local deployment

If you just want to learn how it works you can get a 60 days trial license registering an account on VMware. I had to manually select a Bridged network for the ESXi server. The vCenter appliance used a Bridge network by default. connected them to a bridged network. You can use DHCP if you want, but you need to make sure they get the same IP addresses afterwards.

VMware multipathd log spam

While I was deploying Loki with Promtail I’ve seen a lot of log spam from Ubuntu virtual machines in my VMware environment. As a note to myself and for some others who want cleaner system logs – here is what I’ve found to get rid of it.

The log entries look like these here:

2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sgio uid: No such file or directory
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get sysfs uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: failed to get udev uid: Invalid argument
2021-04-07 20:14:21 opennms-bgp multipathd[693]: sda: add missing path

The best article I’ve found was from SUSE describing the problems source. In a nutshell, VMware doesn’t provide information needed by udev to generate the /dev/disk/by-id. To solve the problem you have to set in the Virtual Machine the attribute disk.EnableUUID=true. For the reason I have a few VM’s it’s pretty tidious to do all these things manually.

Installing Node Exporter on Linux

In OpenNMS Horizon 28+ is now a PrometheusCollector available. It scrapes the metrics from the provided exporter pages and allows to add data collections. As of speaking today it is not 100% feature complete, scraping data types like histograms is not implemented yet. If you want to play around here is a quick way to get the Linux Node_Exporter installed.

The following steps are executed in a root shell with sudo -i.

Hardening SSH for audit

Running a server in the public requires some additional work. Especially if you want management access via SSH for Ansible or if you want break stuff manually with fiddeling around :)

You can run an SSH audit of your public server using https://www.sshaudit.com. This section here is a very condensed way to get an A rating.

Just use strong host key for authentication of the host

# file: /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

Delete existing keys and re-generate the RSA and ED25519 keys

UCARP and High Availability

If you have ever played with BSD you probably ran into CARP. It allows you to build a high available service which is provided by two physical servers behind a virtual shared IP address. The CARP nodes define a master and a backup system. A master serves the content and if the master crashes, the backup system takes over automatically the virtual IP (VIP) and the client won’t notice.

Disclaimer: You should be aware this setup will not share load and increase your network throughput. It just used to increase availabilty and room to do maintenance without bringing your service down.

Continue reading

Docker build and cache invalidation

Right now I’m working with my work mates @opennms integrating the docker image building in our CI/CD environment. We build our container image based on CentOS and we noticed the caching doesn’t work for ${reasons}.

Running a docker build -t myimage . ended up always in installing packages from the official yum repositories even we haven’t changed anything in the Dockerfile.

To understand things better, I went back to drawing board and started with a simple example and rebuilding things step by step to understand when gets the docker build cache unnecessarily invalidated.

Docker, Java, Signals and Pid 1

Running a Java application in a container seems to be very easy. The devil is in the details and I want to shed some light on the PID 1 problem when you run Java applications in containers. Theoretical, processes in containers should not have state so you just don’t care, but reality is different.

Signals are used to message running process to behave in certain ways. A common case is to terminate a process nicely sending a SIGTERM to the process identified by the process id using the tool ps. People can implement logic shutting down an application gracefully. When you issue the command kill <pid> you send a SIGTERM to a process. We have other signals you can use, especially when application provide terminals. Hitting CTRL + C sends SIGINT to the terminal process. Some signals are quite out of time, the signal to hangup SIGHUP comes from terminal applications. It is often used nowadays telling a running process to re-read configurations.

SSL and Java

Running applications with a current Java is not a big deal thanks Let’s Encrypt. This article describes what happens if you want to authenticate your OpenNMS against LDAP using SSL with a self-certified certificate.

First of all I assume you have confiured verything so you can authenticate against LDAP in plaintext and you got a role mapping as you wanted it. If not you can have a look here.

So the naive approach would be, just changing the line in your activeDirectory.xml from