Performing Network Diagnostics with MTR

Introduction

Today, we’re looking at how to diagnose network issues. We’ll use a specialized tool called MTR, which is basically a combination of two common network tools: ping, and traceroute. MTR lets you identify if and how a breakdown occurs between your server and a server you’re trying to reach.

Prerequisites

MTR works on every Linux distribution we tested, including Ubuntu 16 & 18 as well as Debian 8 & 9. There are versions for Windows and Mac as well. You’ll need admin access on the machine to install the program, but after that it can run without root permissions

Step 1: Install MTR

The latest instructions for installing MTR are available on the project GitHub page. However, in most cases you can install MTR just by using the package manager. For example, on Ubuntu the command is just:

sudo apt-get install mtr

Likewise, on Macs you can just use Homebrew to install:

brew install mtr

Finally, for Windows users you can use WinMTR, which is built for Windows machines. Visit the site, download the executable and you’re good to go.

Step 2: Generate an MTR Report

MTR works by trying to reach the desired site multiple times and summarizing the results in a report. To test your connection to Google, for example, you can run:

mtr --report google.com

And check the output. MTR will try 10 times to reach the site, and collect summary information on the steps it took to get there.

Step 3: Read the Report

The output from MTR contains a few different pieces of information. It should look something like this:

root@debian-server:~# mtr --report google.com 
Start: Tue Nov  6 02:05:45 2018 
HOST: debian-s-1vcpu-1gb-nyc3-01  
Loss% Snt Last Avg Best Wrst StDev  
1.|-- 209.97.144.254             0.0% 10 2.3 0.8 0.4 2.3   0.5  
2.|-- 138.197.248.40             0.0% 10 0.4 0.4 0.3 0.6   0.0
...
11.|-- 108.170.227.211            0.0% 10 2.1 2.1 2.0 2.1   0.0 
12.|-- lga34s19-in-f14.1e100.net  0.0% 10 1.9 2.0 1.9 2.3   0.0

Let’s unpack this a little. Each line tells you a single hop your network traffic took between your server and the destination server. The “Loss%” line tells you how many packets were lost in between steps.

The “Snt” column tells you how many packets were sent. By default, MTR will send 10 packets, which is usually enough to identify issues. If you’re seeing only intermittent network issues, try bumping the number of packets up by passing the -c flag with the command. For example, to run with 100 packets run:

mtr --report -c 100 google.com

Finally, the last few columns tell you summary statistics for the latency of network traffic. The most informative are the average and the worst, which should give you an idea of what typical traffic is like.

Step 4: Diagnose the Issue

If you’re having network issues, there are a few ways that MTR can help you see what’s going on. One common pitfall is when one particular link in the network is experiencing high packet loss. This appears in the report as a single row that stands out in the “Loss%” column, like below:

root@debian-server:~# mtr --report google.com 
Start: Tue Nov  6 02:15:45 2018 
HOST: debian-s-1vcpu-1gb-nyc3-01  
Loss% Snt Last Avg Best Wrst StDev  
1.|-- 209.97.144.254             0.0% 10 2.3 0.8 0.4 2.3   0.5  
2.|-- 138.197.248.40             0.0% 10 0.4 0.4 0.3 0.6   0.0  
3.|-- 108.170.227.211            3.8% 10 2.1 2.1 2.0 2.1   0.0 
4.|-- lga34s19-in-f14.1e100.net  3.8% 10 1.9 2.0 1.9 2.3   0.0

Here, it’s pretty easy to identify the server that’s causing the issue, which is a lossy hop along the chain. Once you’ve identified the issue, you’ll need to talk to your network administrator and supply them with the IP address of the lossy link to fix the problem.

One important note: packet loss does not always indicate that there’s a problem. Many switches and routers will rate limit requests coming from MTR to prevent network slowdowns. You can identify this type of rate limiting packet loss when the Loss% is only in one row of the report, but later rows have 0%

Another common issue is when outbound traffic is fine, but the final line has a 100% loss. This type of report looks like this:

root@debian-server:~# mtr --report google.com 
Start: Tue Nov  6 02:13:45 2018 
HOST: debian-s-1vcpu-1gb-nyc3-01  
Loss% Snt Last Avg Best Wrst StDev  
1.|-- 209.97.144.254             0.0% 10 2.3 0.8 0.4 2.3   0.5  
2.|-- 138.197.248.40             0.0% 10 0.4 0.4 0.3 0.6   0.0  
3.|-- 108.170.227.211            0.0% 10 2.1 2.1 2.0 2.1   0.0 
4.|-- lga34s19-in-f14.1e100.net  100.0 10 1.9 2.0 1.9 2.3   0.0

Typically, this is indicative of a firewall or IP table configuration issue. The traffic is successfully reaching the destination, but the response coming back from the destination server is getting lost. To fix this, you’ll need to change your firewall settings to allow the response to be received.

Advanced MTR Techniques

For advanced users, MTR includes several additional diagnostic tools that may come in handy. First, as we mentioned above, you can vary the number of packets MTR sends when generating a report with the -c flag. You can also vary the rate at which MTR sends the packets using the -i flag. The default is 1 second.

For example, to send 100 packets at 10 packets per second, use the command:

mtr --report -i 0.1 -c 100 google.com

If you’re being rate limited, you may also want to send packets more slowly. For example, once every two seconds:

mtr --report -i 2 -c 100 google.com

By default, MTR uses DNS to look up the names of servers if possible, before putting them in the report. You can include the long hostname using the -w flag. On the other hand, you can exclude hostnames altogether and just get IP addresses using the –no-DNS flag.

Finally, depending on your version you may be able to use the -p flag to specify the port MTR uses. By default, MTR uses ICMP which doesn’t have ports. If you specify the -p flag, MTR switches to using TCP along the specified port. This may be helpful when diagnosing issues related to specific services. For example, if you’re unable to SSH to your server, you can test it on port 22:

mtr -rwc 10 -i 2 --no-dns -p 22 myserver.ip

Conclusion

By now, you’ve learned how to install MTR and generate a network diagnostic report. Once generated, you can read the report and identify the cause of common network issues. Many issues have a common pattern, which MTR can easily identify.