Traceroute: More Complex Than You Think

Hello, my name is Lowell Vanderpool and this channel is dedicated to IT students, IT professionals, and anyone who enjoys learning technical subjects. Traceroute is a common, very popular tool by network administrators to analyze packet switch networks. If I want to analyze the latency between one point and the other, and I want to see the router hops in between, I use Traceroute. ISPs will tell you, as well as NOCs, that one of the dirty secrets is, is they get hundreds of help desk tickets with Traceroute results. They delete them. And the reason they delete them is because too many people do not understand that the results from Traceroute are more complex to interpret than you might think. Let's learn how to look at Traceroute with a different view so that your help desk ticket doesn't get deleted. Now, Traceroute's been around for a long time. It was released in 1987 when, boy, WAN circuits were awful and routers were not much better. And Traceroute was a wonderful tool to help understand packet switch networks. So, Mr. Vanderpool, what can be complex about Traceroute? We send out three packets. We measure router hops. We review the time. It's done. We can see what's wrong. Not so fast. So, exactly how does Traceroute work? Well, you can see I'm sending a Traceroute to squarespace.com and you can see the IP address. Now, the Traceroute utility will start out with sending three ICMP packets, each sabotaged. Those packets know where they want to go. They want to go to squarespace.com. But they're sent out sabotaged with a time to live of one. So, when they get to that first router, router one in the picture, the router looks at their time to live, drops the packet, and sends an ICMP type 11 message back to my Traceroute software. And it creates a timestamp. And that's where I get the one millisecond, the one millisecond, the one millisecond that you see in this center graphic. You see here we're going to continue using that sabotaged packet. Traceroute is going to continue to sabotage that time to live so we can determine each router in our packet switch network that we are taking the router path from source to destination. Time is calculated as we send our packets out of Traceroute to each router. As it gets a ICMP type 11 packet back, it calculates the time. And that's where we see the latency measurements that we get for each probe. Traceroute implementation on various operating systems is different. Windows uses the protocol ICMP to implement Traceroute. Cisco uses UDP. And Linux uses all three protocols, UDP, TCP, and ICMP. TCP is used often to avoid firewalls. By default, Ubuntu's Traceroute implementation uses UDP port 33434. You can use ICMP with switches and arguments. And you also can use TCP with Ubuntu's version of Traceroute. It doesn't matter which implementation or protocol that you use to use in your Traceroute, whether you use TCP, UDP, or ICMP. They all depend on each router sending back to Traceroute an ICMP type 11 time exceeded packet. So what's the goal of Traceroute? One, to determine the path your packets are taking to the destination. So that's important. Determining the time, the latency it takes through each router to the destination. Sending three packets to give you a better understanding of what's happening as we're moving through the network. And giving insight to possible problems in your network. Here's some of the problems right away. Packet paths via the and can change in a fraction of a millisecond. Traceroute sends out three packets per router. It's possible for every packet to take a different route. So let's look at an important fact about Traceroute. One, every single probe packet sent from Traceroute is completely independent of another. Each packet may be forwarded down a complete different path. And it may or may not be visible to you in the results. We must understand Traceroute latency. The figures that you see here in the graphic, they're made up of three basic things. The time taken to forward the packet. The time the router takes to generate an ICMP response. And the time taken for the ICMP packet to reach the Traceroute software. That is what is shown in these latency values. Let's look at the typical Traceroute results and let's understand what we're seeing in this Traceroute results. On the far left, you see in purple, the hop count. In the center, orange, you see the latency per router. In the center, we're seeing in the green, we're seeing router DNS names. This is very, very important. And at the end, we see in red, the router IP. So if we're going to do a good job in understanding our Traceroute results, the more we know about that router that it goes through to get to our destination, the better we understand our Traceroute results. That DNS name of the router can tell us, number one, who owns the router. Is it AT&T? Is it Brighthouse Network? Is it XO? Is it Alchemy? It can also, the DNS name can tell us where the router is. Is it in Miami? Is it in Orlando? It can also give us a hint as to what type of router. Is it a residential router? Is it a core router? Is it an edge router? Is it a customer premise router? All of that gives us very important information as to what's going on at each router hop. The router's DNS name can give us important information about the role of the router. We must understand that because that can give us important clues as to why we're seeing latency at that particular router. So is it a core router? In the DNS name, you'll see abbreviations like CR, Core, GBR, BB, CCR, and EBR. Peering routers tend to use acronyms like BR, Border, Edge, IR, IGR, and Peer. Customer routers typically use acronyms like AR, AGGR, and things of this nature. I'll have detailed information about how to determine router names, router roles, and some of the acronyms and abbreviations used so that you can help better understand the role of the router. Those will be in the video notes as a free download in the video description. Another important bit of information that we must understand as we're looking at our results is identifying network boundaries. Routing policies change as you go from Spectrum's network to XO or XO to Alchemy. Look at my trace route result. There in the red, you see I'm going from Spectrum to XO. Those are two different networks run by two very different policies. Those policies can impact right away some of your latency. And here you can see in the green, I'm going from XO to Alchemy. So you know right there, there's where you got the edge of two networks and we kind of expect some differences between how Spectrum runs their routers and how XO runs theirs. Let's take a look at a problem. Here I've got the trace route results between Global Network and then Sprint Link. Look at hop three and you'll notice, ouch, we're going from a 0.5 millisecond latency to over a 100 millisecond latency when we hit sprintlink.net. Now what's the problem? Well, it's complex. It could be congestion between those two network boundaries. It could be a congested reverse path as Sprint returns that ICM packet. It could be in a slow network and getting a artificial delay. It could be a policy change between Global Network and Sprint Link. We don't know, but we do know that there's a problem here. So let's look at our three hop analysis of trace route as we go from Washington DC to San Jose. Notice asymmetrical paths can happen anywhere and these asymmetrical paths are normal in a packet switch network. They add more complexity in looking and analyzing our results. Let's look at hop at one. Well, it's pretty simple. Washington DC to Washington DC. One way, one hop, easy analysis. But look at hop two. As we go from red to Chicago, it goes one router direction but returns a totally different path. This makes our results more complex to interpret. Look at from Washington DC to San Jose. Again, we go in one path, but we return a totally different path. Another impact in trace route results is load balancing. Let's look at hop six, seven, and eight and see how load balancing impacts two of the three probes. In hop six, it has two packets taking 80.91.251.14 and one packet taking 80.91.249.77. That makes this results more complex to analyze. So how does MPLS networks impact trace route results? So many configured MPLS networks, if you'll look in our graphics, force router one, router two, router three, which are in that MPLS to travel all the way to the end of the LSP, the label switch path, before they're allowed to go back to the trace route software. So that you really can't see what's going on in the MPLS network. As this graphic very well puts the problem of trace route through an MPLS network, it calls it dumb as a stump. And if you look at the trace route PEA router, PEB router, everything in between trace route has no clue. The results is not giving you any information as to what's happening between the beginning of the MPLS network and the end of the MPLS network. So here's an MPLS trace route. And you can see as we get into AT&T's network, where the red line is, everything shown from green and the green arrow down, notice the latency values. They're all pretty much the same. So we really can't see what's going on in this MPLS network. So three primary causes of network-induced latency are serialization delay, router queuing delay, and propagation delay. It's well understood when we look at optic transceivers, they have to come from electrical to light and light to electrical. That induces a certain amount of latency. Layer two switches and routers add lots of RAM so that they will buffer their inputs so they don't have to drop packets and that's known as buffer bloat. We'll talk about that. And then, of course, electricity can travel down copper only so fast. Those are bounded by the laws of physics as well as light traveling down fiber optics, about 125 miles per every millisecond. So let's look at normal latency. New York to London is about 4,200 miles, should be about 67.6 milliseconds. That's normal and you should understand what normal latency is. Abnormal latency, Orlando to Miami is about 235 miles and I'm looking at latency at about 220 milliseconds. That's not normal. That's abnormal. Very important that you have a sense of what is normal latency due to propagation delay and what is immediately abnormal latency. That will help you understand your trace route results with more accuracy. Routers and layer three switches can introduce network latency. This is called buffer bloat. Many layer three switches and routers have first-in, first-out queuing large enough to add, listen, 250 milliseconds of latency in every packet. One way you can test buffer bloat in your network is go to this link and they've got great tools to help you determine, do you have buffer bloat in the path between your source and destination? The router itself can introduce delay to your trace route results. So if you look at this Juniper, it has a control plane and a forwarding plane. 90% of the packets that are coming into this router are in and out blazingly fast. But if you're a packet that needs attention from the router, like a trace route, you may have to go up into the CPU of the router and be processed before it can respond to what you need. That is typically handled at a slower pace, not priority. So you can often see router delay that is artificial. It's just the way that routers handle exceptional packets like trace route packets. This can show up in your trace route results. And if you're not careful, you're thinking it's network latency when it's really just router delay. Now, the results that you're seeing here are actually showing us a router that is set up for ICMP rate limiting. In other words, a router is deliberately looking at ICMP packets and maybe even in occasionally just discarding them. Now, we're looking at an MTR result. MTR is a tool that combines trace route and ping. Most IT administrators will be very comfortable with an MTR utility, but the tech was having a problem with this particular router and it was throwing his trace route results way off. And the problem was the router had ICMP rate limiting. This is another reason why many implementations of trace route uses UDP rather than ICMP. Here's an example why. So what is MTR? MTR is a two-in-one tool. It combines some of the idea of a trace route with ping in one utility. There is a Windows version. It's called MyTraceRoute. I'll give you that information in the video notes. You can see a free download link in the video description. MTR builds a handy table so you can look at your trace route and your ping trace route combination tool. It also does it continuously until you stop. So it gives you this nice table format. Here's a legend of all the columns so you can understand the results. Very similar to a trace route. It's a very handy tool. There are both Linux and Windows versions of this. So let's look at some rules for using trace route. Rule number one, perform bi-directional traces. So if you do a trace route to some destination host, if it's possible, pick up the phone and have someone at the other end do a trace route back to you. Because routes are often asymmetric. They may take one path in one direction and a different path. In the return, tracing the route in both ways can help you pinpoint the problem more accurately. The number one, bi-directional traces. Rule number two, trace route is blind to the return path and latency. Trace route shows you the path to the destination, but you have no idea in the result how it got back. That often is the problem. Another effective concept for trace route is using looking glasses. Looking glasses are basically a dedicated server installed by ISPs all across their network. And you can use these dedicated servers to do a trace. Hurricane Electric Internet Services is my favorite place to get looking glass servers. They have a huge list of looking glass servers all over the world. You can do HINGs, trace routes, BGP routes, BGP summaries. There's a URL. Check that place out. So let's look at this results. We got 11 hops in this trace route, and I've got in red hop six and seven. Look at the last hop. You've got 30 milliseconds, 16 milliseconds, and 23 milliseconds to your destination from you. No problem there. So what is wrong with hop six and seven? We have no idea. We see that it jumps up. It's higher than we want, but we have no idea what the problem is at hop six and seven. The bottom line is the last hop is the most important. We see only 30, 16, and 23 milliseconds between you and the destination. That's decent latency. And you're What's the mystery with six and seven? We have no clue. Router delay, congestion, the return path. We don't know. We don't care. You're connected. You have good latency between you and the destination. Now they do produce visual trace route tools. I don't know that they're much better than your basic command line version. I think MTR and trace route are fine. You can get the what's up visual trace. You can see the results of it. Pretty nice. Let's look at another false alarm. Here we see the results of a trace route. This is actually an MTR. And we see as we get into the bottom hops, we see no response from the host. We see all the packets drop. But as we look at the last hop, we're getting decent latency. We connected the last, the destination host from our location. There's no problem here. Make sure you understand that last hop is what's critical. What's happening in between? We really don't know, but we're connected and we have good latency. This is an MTR result, but you'll notice it's very similar to a trace route. Notice as we get to the Google network in hop four, we see latency jump and it stays high all the way to the destination. This is a problem. This is one of the giveaways. When you see your latency become high and stay consistently high all the way to the destination, you most likely have a problem at hop four. We don't know exactly what the problem is, but we definitely know we have an issue at hop four. Here's another interesting results. And here we can see the very last destination host is unreachable. But it is showing up in our trace route results. Even though this is an MTR, it's very similar to trace route. The very last destination shows up as a DNS name, but it can't be reached. Almost always that's a misconfigured host. So here's some results that show us a routing loop. Here we have a host unreachable with a ping, and then we do a trace route to that destination. And notice our results as we go from router one to router two to router one to router two. That's basically corrupted routing tables that are bouncing you back from one to the other. In summary, trace route results are more complex than you think. Every probe you send out is independent. Packet switch networks provide asymmetrical paths. It makes it more complex. Trace route results are blind to the return path. Trace route is blind to many MPLS networks. And trace route cannot show router delay or limiting or buffering. So what are our best practices? Try to do a trace route in both directions. Study the router DNS names for clues, type of router, location of router. Is this latency reasonable? The owner of the router. Observe when a network boundary changes in your trace route results. That last hop, important. Use a looking glass server to investigate network delays. And when you send a help desk ticket or a heat ticket to an ISP, include everything that I have shared with you in that heat ticket. Instead of getting deleted, you actually might get a response and an answer.