Dealing with secrets in OpenNMS Horizon

Dealing with secrets in a monitoring platform is a tedious task. The nature of monitoring systems is providing integrations in as many systems. Here is my mental model looking at the various ways for integrating OpenNMS. Especially having a 20 year old platform gives you some very own challenges. Talking about integration of monitoring systems, In OpenNMS Horizon/Meridian (OpenNMS for short), are some places where you have just one option, and some with various options. This article should help you to get an idea where you need to deal with secrets and credentials. The north/south areas serve monitoring purpose whereas the west/east directions address scaling topics in the context of volume, geographic or feature sets.

Hello Containerlab with Orbstack

I still remember signing up on DockerHub 11 years ago. Learning how to build container images on real world projects is definitely a plus. Having a software and some use cases in the back of your mind, helps you to achieve things quicker and with some purpose. I’ve started to work with containerlab to build some network test environments mainly for three use cases:

• Layer 2 network topologies for network monitoring tests using LLDP, CDP and Bridge-MIB
• Routing topologies with protocols like BGP, OSPF or IS-IS in general
• Some vendor specific network gear for monitoring with SNMP and streaming telemetry

Playing with OpenNMS from this perspective opens some use cases around Netflow, IPFIX, BGP monitoring, and SNMP in general. I was using Docker4Mac for a very long time. At some point in time I have switched to colima which was slim and easy to use. With switching to ARM on my Mac it got a bit more complicated.

OpenNMS Horizon with RRDtool

As described in the previous article we have built and installed an OpenNMS Horizon Core component from the source. It comes with a Java implementation of RRDTool called JRobin. The portability of Java applications allowed users to run OpenNMS platforms where RRDTool wasn’t easily available. It was threadsafe and allowed more threads writing data. RRDTool implemented that functionality and surpassed JRobin performance and feature wise.

💁‍♀️ If you just don’t care in a dev or testing environment, you can use JRobin for simplicity, because it’s just there and works out of the box. For any production environment, I highly recommend to use RRDTool. It gives you much better support for tools, performance and features. Migrating later is doable but painful.

Demystifying iplike in OpenNMS Horizon

As described in the previous article we have built and installed an OpenNMS Horizon Core component from the source.

With setting up the database schema with ${OPENNMS_HOME}/bin/install -dis a function IPLIKE is created for the OpenNMS database.

It allows us to get IP address matches for IPv4 and IPv6 addresses with filters used in all IP filters in the tool, e.g. IPADDR IPLIKE 192.168.0-3.0-255.

By default, the function is implemented in a SQL procedural language (PL/pgSQL). As OpenNMS had to deal with larger IP address inventories, an optimized version in C was created which is available as the IPLIKE package. The C version of this stored procedure has to be built against header files from specific PostgreSQL major versions. This is the reason you see iplike-pgsql{12,13,14,15} packages in the OpenNMS repositories.

JniPing vs. JnaPing

As described in the previous article we have build an OpenNMS Horizon Core component from source. If you don’t do anything else, it will uses an ICMP implementation using Java Native Access (JNA). The big benefit here, it’s all Java and supports IPv4 and IPv6. You also don’t need additional permissions on your Linux system such as net.ipv4.ping_group_range and SELinux. It makes it perfect for local development and also if you want to run OpenNMS on exotic architectures where you can’t easily compile or build the JNI equivalent written in C from the source code. The downside it comes with some overhead for each ICMP service test. You can see the effect on the latency measurements, especially on very fast responding IP addresses, such as the local loopack interface.

OpenNMS - Auf die harte Tour

I asked many questions in 2004 on IRC when I tried to get my first OpenNMS instance up and running. People in the community held my hand when I was struggling. They helped me to get to my personal “Aha!” moments. If you have time and patience, this is great, because this is a great learning opportunity. In the world of User Experience Design, this is called “friction”. How can you determine friction? My background is that of someone who cut his hands on sharp metal changing network equipment and operating IT gear for others – I have empathy for people running OpenNMS. I like to run user empathy sessions with someone in your target group and figure out where and how they struggle. If you have no one, the next best option is to put yourself in the shoes and get your hands dirty.

Kickstart your homelab with netboot.xyz

I use a home lab for daily work and for testing purposes, I need to hop between Linux distros often. I’ve replaced my homegrown PXE boot environment with netboot.xyz. I built it for a few operating systems and all was good. I ran into netboot.xyz, and it was quickly pretty clear, this is what you want in a homelab when you have to hop between operating systems using virtual machines. Reducing the need for building bootable thumb drives or shuffling ISOs around is great. It has batteries included with boot menu configs, and it works out of the box. It allows also a lot of customization if needed. As a reminder to my future self and for others here is my config to get it working quickly.

UDP tuning and performance testing

Problem statement

• Ingesting UDP traffic is complicated to measure
• Packet drops, connectionless and unreliable
• Measuring on ingest on the network interface card
• How can you make sure you measure reasonably?
• You want a method to create some confidence how many UDP packets your system drops

Create a lab environment to reproduce the problem

• Make the problem visible using with overloading a small device Raspberry Pi 3
• Use sysctl default settings
• Use something like hping3 or iperf to create a overload situation

You can’t improve what you don’t measure

• Show tools like dropwatch or ss -lump or SNMP udp metrics to visualize packet drops
• Compare packets received with tcpdump vs. iperf
• Theory should show who be tcpdump should have more but not all then the sender

Increase buffers size?

• What happens if you increase the buffer size?

Use PF_RING

• How does the behavior change when you use PF_RING with TCPDUMP

Conclusion

Stop using PowerPoint as a working document

I’m spending more time working with people with management titles, and I find a few things disturbing. I have attended working sessions where the collaborative document was a PowerPoint slide deck. That was super confusing to me because a slide deck is something I have used when I had to attend a conference to give a talk about a specific topic. A PowerPoint slide deck as a work deliverable blew my mind.

Mirroring a container registry

I was working on an article How to run an air gap installation of OpenNMS Horizon on Rocky Linux. I ran into a similar use case and it was not about RPMs or DEB packages, it was all about container images and registries. My question was, how can I get “all” container images into a private registry from DockerHub? Getting your hands dirty with a private registry is something I’ve described in Running a private container registry for testing. Here is a short how-to on how I did it for my future self or anyone else with a similar question.