Reading packet loss in mtr and traceroute

Distinguish lost probe replies at one hop from loss that continues to the destination.

An isolated red hop in mtr can be misleading. Compare its loss with every later hop and with the destination before escalating.

How mtr probes a path

mtr sends probes with increasing TTL values. Routers can return ICMP Time Exceeded from their control plane when a probe expires. A router can rate-limit those replies while still forwarding traffic normally in its data plane.

Read the whole report

Compare later hops

If hop 7 reports 42% loss while the destination and later hops report none, the router probably limits probe replies. Loss that persists through later hops to the destination calls for investigation.

$ mtr -rwbzc 100 198.51.100.10
HOST                         Loss%  Snt  Last   Avg  Best  Wrst StDev
 1. AS64512 10.0.0.1          0.0%  100   0.4   0.5   0.3   1.2   0.1
 5. AS64500 203.0.113.17      0.0%  100   8.1   8.4   7.9  12.0   0.6
 7. AS64501 192.0.2.33       42.0%  100  31.2  30.8  30.1  44.9   2.1   # 58 replies to 100 probes
 9. AS64501 192.0.2.61        0.0%  100  31.0  31.1  30.6  33.0   0.4
12. AS64502 198.51.100.10     0.0%  100  32.4  32.2  31.8  35.1   0.5

Patterns worth investigating

  • Loss begins at a hop and persists to the destination.
  • Latency increases at one point and stays elevated farther along the path.
  • Measurements vary considerably at the destination. This may show jitter that affects calls or video.

Useful options

  • -T -P 443 switches to TCP probes on port 443.
  • -b -z includes addresses, names and AS numbers.
  • -c 100 -r -w creates a wide report after 100 probes.

When sending a report to the provider, capture both directions if possible. Return traffic can take a different route.

What to put in the ticket

Record the time and time zone, source, destination, service port and the symptom seen by the application. Include the final hop in the report. If the target normally ignores ICMP, compare the report with a connection to the service itself.

Each hop gets its own series of probes. Router ICMP rate limits are described in RFC 1812, § 4.3.2.8.