OpenNMS Horizon with RRDtool

As described in the previous article we have built and installed an OpenNMS Horizon Core component from the source. It comes with a Java implementation of RRDTool called JRobin. The portability of Java applications allowed users to run OpenNMS platforms where RRDTool wasn’t easily available. It was threadsafe and allowed more threads writing data. RRDTool implemented that functionality and surpassed JRobin performance and feature wise.

💁‍♀️ If you just don’t care in a dev or testing environment, you can use JRobin for simplicity, because it’s just there and works out of the box. For any production environment, I highly recommend to use RRDTool. It gives you much better support for tools, performance and features. Migrating later is doable but painful.

Demystifying iplike in OpenNMS Horizon

As described in the previous article we have built and installed an OpenNMS Horizon Core component from the source.

With setting up the database schema with ${OPENNMS_HOME}/bin/install -dis a function IPLIKE is created for the OpenNMS database.

It allows us to get IP address matches for IPv4 and IPv6 addresses with filters used in all IP filters in the tool, e.g. IPADDR IPLIKE 192.168.0-3.0-255.

By default, the function is implemented in a SQL procedural language (PL/pgSQL). As OpenNMS had to deal with larger IP address inventories, an optimized version in C was created which is available as the IPLIKE package. The C version of this stored procedure has to be built against header files from specific PostgreSQL major versions. This is the reason you see iplike-pgsql{12,13,14,15} packages in the OpenNMS repositories.

JniPing vs. JnaPing

As described in the previous article we have build an OpenNMS Horizon Core component from source. If you don’t do anything else, it will uses an ICMP implementation using Java Native Access (JNA). The big benefit here, it’s all Java and supports IPv4 and IPv6. You also don’t need additional permissions on your Linux system such as net.ipv4.ping_group_range and SELinux. It makes it perfect for local development and also if you want to run OpenNMS on exotic architectures where you can’t easily compile or build the JNI equivalent written in C from the source code. The downside it comes with some overhead for each ICMP service test. You can see the effect on the latency measurements, especially on very fast responding IP addresses, such as the local loopack interface.

OpenNMS - Auf die harte Tour

I asked many questions in 2004 on IRC when I tried to get my first OpenNMS instance up and running. People in the community held my hand when I was struggling. They helped me to get to my personal “Aha!” moments. If you have time and patience, this is great, because this is a great learning opportunity. In the world of User Experience Design, this is called “friction”. How can you determine friction? My background is that of someone who cut his hands on sharp metal changing network equipment and operating IT gear for others – I have empathy for people running OpenNMS. I like to run user empathy sessions with someone in your target group and figure out where and how they struggle. If you have no one, the next best option is to put yourself in the shoes and get your hands dirty.

Kickstart your homelab with netboot.xyz

I use a home lab for daily work and for testing purposes, I need to hop between Linux distros often. I’ve replaced my homegrown PXE boot environment with netboot.xyz. I built it for a few operating systems and all was good. I ran into netboot.xyz, and it was quickly pretty clear, this is what you want in a homelab when you have to hop between operating systems using virtual machines. Reducing the need for building bootable thumb drives or shuffling ISOs around is great. It has batteries included with boot menu configs, and it works out of the box. It allows also a lot of customization if needed. As a reminder to my future self and for others here is my config to get it working quickly.

UDP tuning and performance testing

Problem statement

• Ingesting UDP traffic is complicated to measure
• Packet drops, connectionless and unreliable
• Measuring on ingest on the network interface card
• How can you make sure you measure reasonably?
• You want a method to create some confidence how many UDP packets your system drops

Create a lab environment to reproduce the problem

• Make the problem visible using with overloading a small device Raspberry Pi 3
• Use sysctl default settings
• Use something like hping3 or iperf to create a overload situation

You can’t improve what you don’t measure

• Show tools like dropwatch or ss -lump or SNMP udp metrics to visualize packet drops
• Compare packets received with tcpdump vs. iperf
• Theory should show who be tcpdump should have more but not all then the sender

Increase buffers size?

• What happens if you increase the buffer size?

Use PF_RING

• How does the behavior change when you use PF_RING with TCPDUMP

Conclusion

Stop using PowerPoint as a working document

I’m spending more time working with people with management titles, and I find a few things disturbing. I have attended working sessions where the collaborative document was a PowerPoint slide deck. That was super confusing to me because a slide deck is something I have used when I had to attend a conference to give a talk about a specific topic. A PowerPoint slide deck as a work deliverable blew my mind.

Mirroring a container registry

I was working on an article How to run an air gap installation of OpenNMS Horizon on Rocky Linux. I ran into a similar use case and it was not about RPMs or DEB packages, it was all about container images and registries. My question was, how can I get “all” container images into a private registry from DockerHub? Getting your hands dirty with a private registry is something I’ve described in Running a private container registry for testing. Here is a short how-to on how I did it for my future self or anyone else with a similar question.

Make your network traffic visible using flows

Getting metrics about the utilization of network interfaces isn’t very complicated these days. Most of the time your network gear is instrumented with an SNMP agent which you can use to measure it or if you get modern, they have telemetry streaming, and they send you measurements with protocols like gNMI. I’m not explaining fancy new tech here, everything mentioned here is old, but your devices might have these capabilities but aren’t exploited :)

Net-SNMP version 3 and OpenNMS

To monitor your systems you rely heavily on SNMP, it gives out of the box a lot of possibilities getting important performance and status information.

The main topic security is often not considered. SNMP version 1 and 2c transmit everything in plain text over the wire. There is also no user, password authentication method, just a shared community string which gives access to the information. To address these problems SNMP v3 was introduced.