UDP tuning and performance testing

Problem statement

• Ingesting UDP traffic is complicated to measure
• Packet drops, connectionless and unreliable
• Measuring on ingest on the network interface card
• How can you make sure you measure reasonably?
• You want a method to create some confidence how many UDP packets your system drops

Create a lab environment to reproduce the problem

• Make the problem visible using with overloading a small device Raspberry Pi 3
• Use sysctl default settings
• Use something like hping3 or iperf to create a overload situation

You can’t improve what you don’t measure

• Show tools like dropwatch or ss -lump or SNMP udp metrics to visualize packet drops
• Compare packets received with tcpdump vs. iperf
• Theory should show who be tcpdump should have more but not all then the sender

Increase buffers size?

• What happens if you increase the buffer size?

Use PF_RING

• How does the behavior change when you use PF_RING with TCPDUMP

Conclusion

Stop using PowerPoint as a working document

I’m spending more time working with people with management titles, and I find a few things disturbing. I have attended working sessions where the collaborative document was a PowerPoint slide deck. That was super confusing to me because a slide deck is something I have used when I had to attend a conference to give a talk about a specific topic. A PowerPoint slide deck as a work deliverable blew my mind.

Mirroring a container registry

I was working on an article How to run an air gap installation of OpenNMS Horizon on Rocky Linux. I ran into a similar use case and it was not about RPMs or DEB packages, it was all about container images and registries. My question was, how can I get “all” container images into a private registry from DockerHub? Getting your hands dirty with a private registry is something I’ve described in Running a private container registry for testing. Here is a short how-to on how I did it for my future self or anyone else with a similar question.

Make your network traffic visible using flows

Getting metrics about the utilization of network interfaces isn’t very complicated these days. Most of the time your network gear is instrumented with an SNMP agent which you can use to measure it or if you get modern, they have telemetry streaming, and they send you measurements with protocols like gNMI. I’m not explaining fancy new tech here, everything mentioned here is old, but your devices might have these capabilities but aren’t exploited :)

Net-SNMP version 3 and OpenNMS

To monitor your systems you rely heavily on SNMP, it gives out of the box a lot of possibilities getting important performance and status information.

The main topic security is often not considered. SNMP version 1 and 2c transmit everything in plain text over the wire. There is also no user, password authentication method, just a shared community string which gives access to the information. To address these problems SNMP v3 was introduced.

IPv6 prefix delegation with FRITZ!OS 7.50 and Ubiquiti ER-4

I started working remotely in 2010 for OpenNMS as an open-source network monitoring advocate. I have a little home lab with some real hardware that allows me to play on various things without giving me a big surprise bill from a cloud provider at the end of the month. I have a FRITZ!Box 7530 connected to my ISP 1&1. I get native IPv6 and IPv4 connectivity over an IPv4-in-IPv6-Tunnel. As my main router, I have a Ubiquiti ER-4. It gives me enhanced firewall capabilities and flexibility, especially with routing protocols like OSPF and BGP. I have two networks, one for my work-related stuff and another with all the fun crap you have for fun at home.

Running a private container registry for testing

When I signed up for my DockerHub account in 2013, I never thought sooner than later everything ends up in a container image as it is today. DockerHub was the first public free as in free beer registry to distribute your container images. Containers are now everywhere, and DockerHub, a corporate entity running and funding DockerHub, introduced usage limits for the free tier and started commercializing its registry service. I need to play with software in a micro-service architecture on a platform like Kubernetes, and these limits can be daunting.

Build pmacct on Mac OSX from source

I need a way to generate real-world NetFlow data, and pmacct is a great tool that helped me with various things. I couldn’t find a precompiled package so I have compiled it from the source myself. To get it running on my MacBook Pro with Apple Silicon, I had to jump through a few hoops to get it running. Here is a quick guide to making it work for someone else, including my future self :)

APT Monitoring with Prometheus

As soon you run a Linux server, you need to make sure your systems are up-to-date. It would also be handy to know if you had Kernel upgrades which require a reboot of the system to get applied.

I came across a blog post from Tom Henderson which provided a solution using the Node_Exporter with Prometheus. The way he solved this is pretty slick. I had to tweak the scripts a bit to get it running. He integrated an APT hook which runs a command on apt update and after installing packages. The hooks create a Prometheus file which can be easily shipped with the Node_Exporter.

SNMP vs. Prometheus – On The Wire

I’ve been working with network monitoring tools for a long time. Working with network devices, there is still today a very high probability you have to deal with SNMP. If you work with modern applications or infrastructure, especially in dynamic environments with containers, you will inevitably run into Prometheus and its ecosystem when you need to design and build monitoring solutions.

By design, both agents have different goals in mind. The world was very different when SNMP aimed to be “simple” in 1980 as it was for Prometheus in 2012. To give you some idea, this was the world when people worked with SNMP RFCs in 1983: