Breaking OpenNMS metric collection with SNMP

The most common question I get about OpenNMS at scale is some version of this: I have a lot of network equipment, I want SNMP performance data from all of it, how big does the box need to be? The honest answer used to be “it depends”, followed by a shrug about agent behavior.

What I reached for in those situations was the instrumentation log reader. It tells you two things about a running system: how long it takes to collect metrics from your agents, and how long it takes to persist them. The node inventory matters just as much. It tells you the node-to-interface ratio and how the metric collection is composed in your environment.

To shed some light on this and remove variation, the approach for these benchmarks was to use a uniform fleet of “Cisco CRS-X” devices, each with 144 interfaces, yielding exactly 1 738 metrics per node. For a single interface, this includes standard tabular metrics like ifHCInOctets, ifHCOutOctets, ifInErrors, and ifOutDiscards. At the node level, it includes scalar metrics like sysUpTime alongside processor and memory pool statistics. For SNMP load simulation I use nl6. Using this uniform profile in the simulator gave us a reference to (a) scale the metric collection up or down simply by changing the device inventory, and (b) add network latency to simulate real-world agent behaviour more realistically. The fleet was measured in two conditions. In the cleanroom, nl6 answered every SNMP request in about 0.1 ms, which is far faster than any real agent and isolates what OpenNMS itself costs. For the realistic runs I used tc with netem on the simulator to delay every SNMP response by 50 to 100 ms, which is where real equipment tends to sit.

That first number—how long it takes to collect—is the one this post is built on, and you can measure it on your own deployment today. The persistence time I deliberately took off the table. Over six days in September I measured the collection path properly, on a single Core, across eight capacity searches. This is what came out: how to tell if your system is genuinely overloaded, and which knobs to tune to size it correctly.

Hardware and Virtual Machine Architecture

Every benchmark was executed on a single Dell PowerEdge R630 host equipped with two Intel Xeon E5-2650 v4 processors (24 physical cores, 48 threads) and 252 GiB of memory. The host runs Proxmox VE 9.2.2 with NUMA enabled, ensuring that memory allocations stay local to the processor socket executing the virtual machine to avoid cross-node latency penalties.

The environment itself consists of six Ubuntu 24.04 LTS virtual machines consuming a total of 28 vCPUs. The collection path is driven by an OpenNMS Core (8 vCPU / 16 GiB), a Minion (4 vCPU / 8 GiB), and the nl6 network simulator (4 vCPU / 8 GiB). A PostgreSQL database (8 vCPU / 16 GiB) and a Kafka broker (2 vCPU / 4 GiB) complete the topology, wired together as follows:

Network layout of the six VMs. Everything between the Minion and Core is Kafka. The database sits off the sample path entirely.

The overall load on the underlying hardware was comfortably within its limits: the VMs provisioned only 52 GiB of the host’s 252 GiB of memory and 28 of its 48 available processor threads. This left abundant headroom on the hypervisor, ensuring that resource contention at the host level did not interfere with the benchmark results. To ensure these results can be verified and the environment is entirely reproducible, the complete infrastructure is provisioned using Terraform and configured with Ansible. All of the tooling and deployment code used for these experiments is publicly available in the opennms-benchmark repository.

Prerequisites: What these numbers cover

Read this first, because it decides whether the numbers transfer to your deployment.

The benchmark measures the collection path only. Collected metrics are forwarded to Kafka and nothing else. There is no time series database write, no Newts, no RRD, no Cortex. Thresholding is disabled on every service. Thirteen daemons that do not participate in the SNMP collection are switched off.

That is deliberate: it isolates what Collectd, the Minion and the Core’s decode-and-forward path cost, without a storage backend’s write amplification sitting on top. It also means these figures are an upper bound. Turn thresholding on, or point the metrics at a time series database, and you are adding work this benchmark does not count. The shape of the rule below will still hold. The constants will move.

Part 1: Identifying Overload

If you take one operational habit from this post, take this one: load average will lie to you. The charts below are the cleanroom sweep: 0.1 ms agents, a 100-thread pool, and the fleet held at 10 055, 12 055 and 13 555 devices in turn. At 12 055 devices the Core kept 5.6 of its eight cores busy, the collection queue drained to zero in 94 % of the samples, and the one-minute load average sat at 32, peaking near 80. That is a busy but healthy machine. The load average says otherwise.

One-minute load average over the cleanroom sweep. The Core averages 17 at 10 055 devices and 50 at 13 555, while the next chart shows 4.5 and 6.5 busy cores at the same moments. Nothing unusual is happening.

The reason is the shape of the workload. Collectd threads spend their lives blocked on SNMP round trips, and they wake in bursts as responses arrive. Hundreds of threads become runnable within the same instant, each for a few milliseconds. Load average counts that queue depth. It does not tell you whether the processor is keeping up. To know if your system is overloaded or running suboptimally, watch these indicators instead:

Cores busy, not CPU percentage. Percentages move when you resize; cores do not.

The Collectd pending queue returning to zero within each interval. This is the real pass criterion. A pool pinned at its ceiling is fine if the queue still drains.

Completed collections against the fleet’s demand. If you need 36.7 a second and you are getting 36.7, you are fine regardless of what the load average says.

Old-generation GC. If G1 is collecting the old generation under steady load, your heap is tight and it is being charged to your processor.

On a 10 GiB heap the old generation is quiet at 10 055 devices, collected 0.3 times a minute at 12 055 and 1.4 times a minute at 13 555. That is the tightness signal, and it shows up a full rung before the processor runs out.

Part 2: Sizing and Tuning Guidance

To design and size a system properly, we need to understand what one collection actually costs and how resources relate.

The lab, and why latency matters

The fleet is simulated with nl6, which stands up SNMP agents that behave like real equipment rather than like a loopback. As mentioned, every device shares the same profile: a Cisco CRS-X with 144 interfaces over SNMP v2c. One collection yields the 1 738 numeric samples, collected every 300 seconds.

The important part is what sits between the Minion and the agents: a uniform 50 to 100 ms delay on every response packet (median 75.6 ms measured on the wire). Without that, a simulator answers in about 0.1 ms, and you measure OpenNMS talking to itself. Real agents answer in tens to hundreds of milliseconds, and that single difference costs this hardware a factor of five when nothing else is tuned.

The anatomy of a collection

Collectd walks SNMP tables with GETBULK. Each request asks for a slice of the table, and the collection thread waits for the answer before sending the next one. A collection is a chain of sequential round trips, and the thread is blocked, not computing, for almost all of it.

That gives the first constant:

thread time per collection    T = P × R + 0.75 s

P is the number of sequential GETBULK requests, R is the agent’s response time, and 0.75 s is the Core’s own work per collection.

The second constant is what a collection costs the processor, and it does not depend on latency at all. Measured across four captures: 66 µs of Core CPU per sample, or about 15 000 samples a second per core.

What an 8 vCPU / 16 GiB Core carries

The Core in these runs is 8 vCPU, 16 GiB of memory with a 10 GiB heap on G1, paired with a 4 vCPU Minion. Here is what it reached under different conditions:

Compared like for like at a 100-thread pool, realistic agent latency costs a factor of 5.6. Doubling the pool wins some of that back. Once the SNMP configuration is tuned as well, latency stops being the constraint and the machine itself becomes one. This is that tuned 200-thread search filling up, one fleet size at a time:

Pool occupancy across ten fleet sizes, from the 200-thread search at 76 ms with max-repetitions=5. The peak touches 200 from the first rung; that alone is not a problem. Watch the mean: it climbs a rung with every fleet step, and the knee is where it stops having anywhere left to go.

How resources relate

Each resource is driven by a different input.

Threads are driven by latency. By Little’s law, busy threads is the collection rate times the thread time: N = C × T. Latency enters here and nowhere else. A slow fleet needs a big pool because each unit of work occupies a thread for longer while it waits.

Cores are driven by the metric rate. K = S / 15 000, where S is samples per second. Latency does not appear. Doubling agent response time does not cost you a single extra core.

Heap is driven by the fleet size, plus one decoded response in flight per busy thread:

live heap ≈ 2 GiB + 0.7 MiB × devices + 2 MiB × threads

Latency reaches memory only indirectly, through the thread count.

Spotting the limits: Scheduler vs Memory

When network latency increases (or agents become slow), your thread pool will eventually run out of capacity. As thread time per collection climbs, it will hit the hard ceiling of your maximum thread pool size. This is what it looks like when the Collectd scheduler goes from a healthy state to a problematic state under increasing latency:

Thread time per collection over the latency sweep. Each step is one netem increment; the x-axis is wall clock, not delay. The ceiling is the 100-thread pool spread over 3 803 devices every 300 s, or 7.9 s per collection. The step at 50 ms reaches it, and from there the scheduler is completely blocked.

When the scheduler hits this wall, the temptation is to simply increase the thread pool size to compensate for the slow network. However, you cannot infinitely scale the thread pool because you will hit the JVM memory wall first.

Every thread you add requires memory in two places:

1. Native OS Memory: Threads waiting on the network cost virtually zero CPU, but they do consume native OS memory for their stack (~1 MiB per thread). The kernel handles the “thundering herd” of waking threads efficiently, but the basic footprint remains.
2. In-flight JVM Heap (The real killer): Every busy thread holds about 2 MiB of decoded response data in the heap while processing.

If you blindly increase the thread pool to bypass network latency, you will inevitably exhaust the heap before the OS struggles to schedule the threads. Adding threads to a heap-bound Core makes it worse, not better.

How to read the memory indicators

If you want to know whether you have too many threads for your JVM heap, do not look at CPU context switching; look at the Old-generation GC and the Heap Valleys.

Heap and old-generation collections across the 300-thread search, 12 250 to 14 250 devices. Lines represent: Heap used (blue), Heap ceiling (orange), and Old-gen collections per minute (green).

Two indicators in this chart show the system running out of heap under too many active threads:

1. The floor of the heap used. The peaks of the blue line are normal allocation. The valleys show what survives a minor collection, and they are the number to watch. Here the floor rises from about 7.5 to 8.5 GiB over ten hours under a 10 GiB ceiling. That is live data held by busy threads, and it does not come back.
2. Old-generation collections. Under healthy load, transient SNMP data dies young and minor collections clear it. Here the green line steps from about one to 2.4 collections a minute as the fleet grows. G1 is reclaiming the old generation more and more often, and each pass finds less to free because the threads are still holding it.

This is what a heap-bound collector looks like before it stops: a steadily larger share of the processor spent collecting garbage instead of metrics.

The single biggest lever: max-repetitions

max-repetitions tells the agent how many rows of each table column to return per GETBULK. The compiled-in default is 2. I measured 5.

A factor of 2.3 in fleet size, for one line of configuration, and it costs the Core nothing. Before you buy cores, change this.

A wrinkle to watch out for: on Horizon 36 the Config Manager owns snmp-config, not the file. Writing etc/snmp-config.xml changed nothing, on a reload event or a restart. One PUT to rest/cm/snmp-config/default applied it immediately.

With a well-behaved agent, you cannot break collection by setting this too high. You just stop gaining once responses hit the path MTU. The agent truncates to fit, and the walk simply takes another round trip. Set it to 5 or 6 for this device shape and move on.

The MTU trap

The risk is not the attribute; it is a path MTU below your response size (e.g., from a VPN or tunnel) combined with blocked ICMP. Every SNMP datagram I captured had the don’t-fragment bit set. An oversized response is never reassembled; it is dropped.

If a collection needs 61 requests and the 61st response is one byte too big, you lose all 61. The Minion retries that exact request, it fails again, and the walk is abandoned. Nothing is persisted. Worst of all, the node looks healthy: it answers reachability pings and the first few SNMP scalar requests, but produces no data.

A sizing recipe

For a fleet of D devices producing m samples each per interval I, at an agent response time R:

First, measure your own P and R. Both come off the wire in one capture on the Minion:

tcpdump -ni <iface> -s 0 'udp port 161 and host <one-device>'

Count the requests between the start of one collection and the start of the next: that is P. Take the median gap between a request and its response: that is R. (The instrumentation log reader will give you the collection time directly if you prefer, which is T).

Then:

1. Set max-repetitions first, sized to the device’s tables, and verify responses stay clear of the path MTU.
2. Threads: 1.2 × (D / I) × T, where T = P × R + 0.75 s. The 1.2 absorbs the scheduling wave.
3. Cores: (D × m / I) / 15 000, then divide by 0.85 for headroom.
4. Heap: 2 GiB + 0.7 MiB × D + 2 MiB × threads, times 1.4, on a VM of 1.5 times that. Then watch old-gen GC.
5. Re-check the interval. Doubling the interval halves the rate, the threads, and the cores at once.

Do not go below four cores, whatever step 3 says. A restart makes every service in the fleet fall due at once, and working that backlog off costs about 1.5 times steady-state CPU. A machine sized tightly for the steady state has nowhere to put that, so the queue never drains.

Where this stops applying

Every constant here is a property of one device profile. A different device moves P, the samples per collection and the 66 µs together. The form of the rule survives; the constants must be measured again.

The agents in this lab fail cleanly, by timing out. An agent that answers late holds a collection thread far longer than one that is simply gone. That is the outage that loads a collector, and it is the experiment I have not run yet.

And the storage caveat from the top still stands. These are collection-path numbers with Kafka forwarding and nothing else. Your mileage will vary in exactly the direction you would expect.