Excessive Jitter on RouterOS

Updated for RouterOS 6.47.3 on 5-Sep-2020

Updated for RouterOS 6.46.7 on 14-Sep-2020

Abstract

Introduction

Starting in the spring of 2020, RMHAM NetOps started to notice excessive jitter on the network when the network was upgraded RouterOS 4.46. The jitter was particularly noticable on the P25 repeaters. Reverting some of the microwave links to RouterOS 6.45.9 significantly reduced jitter.

Further investigation shows that in addition to increased jitter on the microwave links, there is also significantly increased jitter on routers connected by ethernet. The exact source of the jitter remains unknown but it appears to have been introduced in RouterOS 6.46.2 and persists to various degrees until 6.47.1. RouterOS 6.47.3 results appear similar to 6.47.1.

Routeros 6.46.7 replaced Routeros 6.45.9 as the new Long Term Stable release on September 14, 2020. The jitter observed in this version is similar to RouterOS 6.47.3 and except for VPN connections is lower than that of 6.45.9.

Materials and Methods

Figure 1. RMHAM Netlab

Figure 1 is a photograph of the NetLab. Purple cables represent direct connections between routers. Red cables represent indirect connections between routers via the RB2011 which represents commercial internet links. The NetLab is normally used for training, demonstration and testing.

Two Raspberry Pi 3B computers are connected part of the NetLab. One is used to provide NTP, DNS and similar services. The other is connected to the RB2011 and is used for penetration testing of firewalls and similar functions.

Every router in the NetLab was configured with the same version of RouterOS. The Routerboard firmware was updated to match to RouterOS version. All routing was performed using OSPF. A separate subnet was created for every link between routers. Routers connected via the RB2011 runs a PPTP VPN with a high OSPF weight. These VPN connections were not selected because the direct connections provided a lower cost. During the experiments, no Dijkstras were run on any of the routers because the networks was stable and no packets wer lost.

Timing was generated using the cping program. The cping program sends one ICMP packet per second to each router in the network and measures the round trip time. Measurements were made from a Raspberry Pi 3B computer connected near the center of the network. The identical hardware configuration was used for all measurements.

For each experiment, the appropriate NPK was downloaded to each router. All routers were then rebooted twice, once to install the appropriate packages, and once to update the Routerboard firmware. The identical configuration was used on each router for all experiments.

The last experiment attempted to use RouterOS 7.1beta2, but the system failed to route appropriately with the existing configuration.

Ping times were recorded over a six hour period commencing withing a few minutes of rebooting all NetLab routers. Individual ping times were recorded to a file for 21600 pings (one ping per second for 6 hours). No packets were lost during any of the experiments.

Experiments were run for RouterOS 6.45.9, the last long term stable verion of the RouterOS 6.45 series, all stable verions of the RouterOS 6.46 and 6.47 series. The NPK for each version of RouterOS was downloaded from the https://download.mikrotik.com/routeros/XXX/routeros-mipsbe-XXX.npk where XXX represents the version of RouterOS.

Results

Figure 2a. ICMP ping times RouterOS 6.45.9

Figure 2b. ICMP ping times RouterOS 6.46

Figure 2c. ICMP ping times RouterOS 6.46.1

Figure 2d. ICMP ping times RouterOS 6.46.2

Figure 2e. ICMP ping times RouterOS 6.46.3

Figure 2f. ICMP ping times RouterOS 6.46.4

Figure 2g. ICMP ping times RouterOS 6.46.5

Figure 2h. ICMP ping times RouterOS 6.46.6

Figure 2h. ICMP ping times RouterOS 6.46.7

Figure 2i. ICMP ping times RouterOS 6.47

Figure 2j. ICMP ping times RouterOS 6.47.1

Figure 2k. ICMP ping times RouterOS 6.47.2

Figure 2l. ICMP ping times RouterOS 6.47.3

The key result observed in Figure 2 is that starting with RouterOS 6.46.2, the jitter increased significantly. The median ping time remains about 1ms, but the at seemingly random intervals, ping times spike dramatically. No pings were lost, but times periodically spiked by one or two orders of magnitude.

To illustrate this behavior, Figure 3 shows a subset of the data from second 10000 for 10500.

Figure 3a. ICMP ping times RouterOS 6.45.9

Figure 3b. ICMP ping times RouterOS 6.46

Figure 3c. ICMP ping times RouterOS 6.46.1

Figure 3d. ICMP ping times RouterOS 6.46.2

Figure 3e. ICMP ping times RouterOS 6.46.3

Figure 3f. ICMP ping times RouterOS 6.46.4

Figure 3g. ICMP ping times RouterOS 6.46.5

Figure 3h. ICMP ping times RouterOS 6.46.6

Figure 3h. ICMP ping times RouterOS 6.46.7

Figure 3i. ICMP ping times RouterOS 6.47

Figure 3j. ICMP ping times RouterOS 6.47.1

Figure 3k. ICMP ping times RouterOS 6.47.2

Figure 3l. ICMP ping times RouterOS 6.47.3

Another interesting observation is that this behavior does not start immediately on rebooting the router. Figure 4 shows the first 30 minutes of each experiment.

Figure 4a. ICMP ping times RouterOS 6.45.9

Figure 4b. ICMP ping times RouterOS 6.46

Figure 4c. ICMP ping times RouterOS 6.46.1

Figure 4d. ICMP ping times RouterOS 6.46.2

Figure 4e. ICMP ping times RouterOS 6.46.3

Figure 4f. ICMP ping times RouterOS 6.46.4

Figure 4g. ICMP ping times RouterOS 6.46.5

Figure 4h. ICMP ping times RouterOS 6.46.6

Figure 4h. ICMP ping times RouterOS 6.46.7

Figure 4i. ICMP ping times RouterOS 6.47

Figure 4j. ICMP ping times RouterOS 6.47.1

Figure 4k. ICMP ping times RouterOS 6.47.2

Figure 4l. ICMP ping times RouterOS 6.47.3

The jitter was quantified by calculating the standard deviation of the ping times to each of the 32 routers. The results are shown in Figure 5.

Figure 5a. Standard Deviation of ICMP ping times for all versions.

Figure 5b. Standard Deviation of ICMP ping times for all LTS and latest verions.

Figure 5a shows that the dramatic increase in jutter for versions 6.46.2, 6.46.3 and 6.46.4 by router. Figure 5b shows just the two Long Term Stable versions 6.45.9 and 6.46.7 and the latest version 6.47.3. Figure 5b densonstartes that for many routers, teh standard deviation is significantly lower in version 6.46.7 than in 6.45.9. The Genoa router is connected over an SSTP VPN instead of a direct connection. This connection showed significantly more jitter than directly connected links, although the average ping time is essentially the same.

Detailed statistics are shown in Appendix 1.

Discussion and Conclusions

A search of the Mikrotik forums showed no reports of the problem as detailed above. The source of the problem is not understood, but appears to have been resolved by RouterOS 6.47.2. The problem does appear to have resurfaced to a lesser extent in RouterOS 6.47.3 as it seems similar to 6.47.1.

The problem was readily apparrent when the production network was upgraded to RouterOS 6.46.6, but only by testing all versions in a controlled environment could this problem be isolated and evaluated.

Based on this experience, we recommend that the RMHAM network should only deploy the long term stable branch of RouterOS on the production network. The Long Term Stable branch receives security updates and therefore is as secure as later versions from the stable branch. There does not appear to be any advantages to using more recent versions of RouterOS 6.

The updated long term stable (LTS) RouterOS 6.46.7 appears to be well behaved and is an appropriate candidate for overwinter conditions 2020-2021.

Initial evaluation of RouterOS 7 did not go well. These problems may well be resolved by the time RouterOS 7 reaches Stable status, but significant testing should be performed before transitioning the network to RouterOS 7.

Appendix 1. Detailed Statistics