The Magic Behind Optical WAN Networking: OSI's Layer 2

This lecture is covering Layer 2 of the OSI Network Stack, and this is Part 2 of Layer 2. This lecture will be covering telecommunications carriers, Tier 1, 2, and 3 ISP providers, cable companies as it relates to delivering network, I won't touch video, and DSL, and looking at the underlying technologies that these players implement at the OSI Layer 1 and 2. This lecture is number 4 in a series exploring and understanding the 7-Layer Network Stack. You can find all the other videos by going to my channel, Lowell Vanderpool. I will put a link in the video description below of each of the videos that I've done already, and I'll put them in order. I'll also put a download link for a PDF file that represents my presentation. In Part 1 of the video covering the OSI Layer 2, we went into some detail in the Layer 2 services some data link protocols offer. Remember, not all Layer 2 protocols offer these services. This is a function of software engineers responsible for a particular protocol. Let's do a quick review of Video 1. Organization of the data to be sent in frames, frame synchronization, detection and recovery of transfer errors, flow control, the avoiding of overloading the receiver, buffer management, and media control. We covered all of this in Part 1 of this video series. Let's quickly review the purpose of Layer 2. Preparing the data from Layer 3 for signal processing at Layer 1. From digital data to, we send it to Layer 2 in frames and then on to Layer 1. Whether it's fiber, we're going to turn it into light pulses. If it's twisted pair, we're going to turn it into voltages. If it's coax cable or antennas, we're going to turn it into electromagnetic energy that's radiated. At the destination host, we're going to receive that digital data from Layer 1. We're going to, if needed, synchronize that digital data with the host clock. We're going to read frame content and check for errors. If that frame has errors, we'll try to correct them on the fly if the protocol supports forward error correction. If the frame has errors, we may drop the frame and send a negative acknowledgement so we can get a retransmission. In some protocols, Layer 2 will simply drop the frame and say, forget it, I'm done, and leave upper layers to decide like TCP. The wide area network space for Layer 2 has evolved from the early 1990s with leased lines, but they were expensive and they really slowed the WAN growth. Frame Relay came, it was cheaper and simpler to implement, but carriers often overbooked their capacity and made enterprise customers really unhappy. ATM was next and it was fast. It was great for voice video, but it was complex to set up and it fell out of favor with the next one, MPLS. It was a hit and WAN use grew exponentially. Without MPLS, I don't know that we would have the data center cloud environment that we enjoy today. VPN was great, connecting a dispersed company at many locations back to a central office. Now the rage is software-defined WANs and cloud-based software-defined WANs, which neither of them will I talk about in this lecture because they deserve their own coverage by themselves. One last point concerning this WAN technology we just talked about. Carriers spend millions of dollars putting this technology in their infrastructure and they slowly move from one technology to another. You can still get a T1 circuit almost anywhere in the U.S. and Canada, still today, and it's 20 years old. Carriers and ISPs do not replace equipment like we replace our cell phone. All of the successful Layer 2 WAN protocols I just mentioned would have not happened without two fiber optic technologies at Layer 1. First, SONNET SDH. It's a Layer 1 fiber backbone for global telecommunication companies, data centers, cloud providers, ISP cable companies, and mobile networks. But it is growing old and it's giving way to the next technology, OTN, optical transport network. This is the new global fiber transport network that is doubling its speed every two years. So what can you transport on a SONNET circuit? You can transport Ethernet, you can do ATM, you can do digital circuits like DS1, DS3, you can do the traditional telephone. SONNET is the acronym for Synchronous Optical Network. SONNET is offered as a point-to-point connection, so it's a really widely offered WAN-to-LAN connection. It is synchronous. All terminals in a SONNET network are normally tied from the same clock source. There are many line rates offered, OC1, 3, 12. You can look at them. One of the sweetest is there's a 10-gigabit Ethernet standard that allows direct connection to an OC192. Plug your WAN right into your LAN at a 10-gig rate. Pretty sweet. Now, you don't want to pay the bill, but it's still a sweet connection. All right, Mr. Vanderpool, what can we transport on an OTN fiber backbone? Well, we can do Ethernet at all flavors, 10-gigabit, 25, 40, 100, 400-gigabit, fiber channel, all flavors and speeds, InfiniBand, 5G, 10G, and notice a lot of storage protocols here, because that allows us to put our storage and compute anywhere in the world. It will transport OC3, 12, 48, 192. In other words, it will transport SONNET. It will do uncompressed video. CPRI up to 10-gigabits is the new 5G long-haul transport. Here is a pretty recent chart showing the line rate that OTN provides to its customers. In the description are some of the Layer 2 protocols using that line rate that's offered. If you come back to these line rates that are up to date, about every two years, you'll see there's many more listed, and at the very end, it's double the speed the last time you saw it. Moving optical speeds higher has required constant innovation. Coherent Optics is allowing vendors to push the next generation of equipment. So, Coherent Optics is a technique that uses modulation of the amplitude and phase of the light, much like we do in RF using Quadrature Amplitude Modulation, or QAM, for Wi-Fi. 802.11ac uses it and AX standards, as well as transmission across two polarizations, as shown in the picture below, to enable to transport considerably more information through a single fiber optic cable. So, just how fast is Coherent Optics? A record 50.8 terabytes per second using multiple wavelengths, each with a net information rate of about 1.3 terabytes per second. Over a 93-kilometer fiber optic route was done by Nokia in 2019. Let's slow down just a minute and go back to SONNET and look at this incredibly important Layer 1 WAN technology. Our friends outside the United States will know SONNET by its ITU standard name called SDH, or Synchronous Digital Hierarchy. The American Institute Standards developed the SONNET standard, and it was adopted in the United States and Canada. The ITU standards, the UN standards body, renamed it Synchronous Digital Hierarchy, or SDH. They are the same technology. SONNET and SDH are clocked, or master clocked, at Layer 1. Carriers provide a master clock signal to every node. This technology is excellent for voice, video, Ethernet, the SD1, SD3 circuits, and fiber channel. The basic SONNET frame is 810 bytes. It's exactly 125 microseconds. It has a dedicated 9 bytes of transport overhead. In that transport overhead, as you can see in the blue, is dedicated in the transport overhead, in that SONNET frame, are fields that allow it to talk to equipment that are involved in the SONNET system. If you'll look at the top right, you'll see a series of pieces of equipment for the SONNET transport system. You'll see on the left, you'll see the terminal SDS multiplexer demultiplexer. This is where we're going to bring in voice and data and ATM and Ethernet and combine it into a fiber optic signal. Now the regenerator in that line is going to be put in there. If it's over 40 kilometers long, it has to amplify that optic signal. Then say an add drop multiplexer. Then again, we have another regenerator. That's if we have 40 kilometers or more where we have to amplify that light. Then say to a destination terminal. There has to be a way to communicate between all of those pieces of equipment. That is done by the transport overhead. In there are fields that allow the multiplexer to communicate to the regenerator. That's called a section. Another field is dedicated from the multiplexer to the add drop multiplexer. That's called a line. Then from point to point, from the terminal to the terminal, is considered the path. In that transport overhead are fields that allow all that equipment to communicate back and forth so that the SONNET line can know exactly where it's at. If SONNET is brought to your local area network, they will bring a carrier fiber termination box into your telecommunication closet, which is seen on the right-hand side, and then a carrier interface box on the left-hand side. Out of that carrier interface box will be an Ethernet output to your router. On the YouTube channel Fiber Ninja shown on the slide, he takes you through an install of SONNET in a customer's local area network. He walks you through each piece of that equipment that AT&T puts at the DMARC wall and explains their purpose. I'll put a link to the video in the description. Go check it out. We'll look at the critical pieces of equipment found in a SONNET network. The good news is the same terminology that I'm going to use for SONNET is also used for OTN, so that saves us from having to relearn new equipment names. The STS multiplexer and demultiplexer is critical in bringing in all kinds of Layer 2 signals, Ethernet, voice, video, ATM, and combining them to a single optic out or a SONNET aggregate. And this piece of equipment will typically be in a data center or a carrier central office. The add drop multiplexer is an incredible piece of electronic optic equipment. It allows a SONNET signal to go right through if necessary, and it can add and remove traffic depending on the need for the carrier and the data center. The third piece of equipment that we're going to look at is called the digital cross-connect, and I'm going to show you a picture of the equipment in just a minute. First, let's look at its function. On the left-hand side is a picture that really shows everything. It shows you the traffic coming in, the arrows, and it shows you how it can flexibly route different SONNET and electrical signals coming in and flexibly route them in any direction to any circuit. On the top right, you can see it's actually performing the function of an STS multiplexer demultiplexer, which in this picture is called traffic consolidation. In the middle is the traffic segregation, where we can take traffic and separate it. And at the bottom, the cross-connect can also act as an add drop multiplexer. This is a less expensive digital cross-connect device. You can spend a half a million dollars on one of these pieces of equipment. Remember, SONNET is a point-to-point technology, but when they connect one node to another, they use multiple fiber connections, generally four, to create a ring from node to node. That allows SONNET to be self-healing end-to-end. As we look at SONNET and we move from OC1 line rate to OC3, OC12, we look at OC192 and OC768. At 40 gigabits per second, what takes place is we have to replace an ST multiplexer and ST demultiplexer at either end, because as those speeds go up, you have to put in better or more high-speed electronics, light sensors, and lasers. This becomes very expensive. OTN, or optical transport network, was established by the ITU. The standard was G.872. The fundamental principle behind OTN is WDM, wavelength division multiplexing. As you see in these two prisms, with light coming and splitting out into colors, or colors coming into the prism and combining into light, that splitting and combining is exactly what's taking place with wavelength division multiplexing. In the case of when we split light at the bottom, you see the prism that takes light and then splits it into colors, it's actually splitting that into wavelengths. Our eyes perceive that as different colors. This is an example of wavelength division multiplexing. If you notice, we have a site A and a site B, and on site A, we would take some electronic signal, say an Ethernet signal, layer 2, and convert it into optics, some form of wavelength of a visible spectrum. In this case, we're showing that we're representing it by that red waveform, and below it is a green and a blue and an orange. Those are all being taken into an optics filter and combined into a single beam of light. It goes across and goes into the site B, which has a optics filter that demultiplex that signal back out into its appropriate waveforms, and then converted back into Ethernet, fiber channel, whatever signal that we are passing through. Notice between site A and B, we have duplex action, so we have transmission both ways with wavelength division multiplexing. Let's compare SONET using time domain multiplexing and OTN using wavelength division multiplexing. At the top, we see two pieces of equipment and a fiber optic line, and let's say that's a 10 gig. Down below, we see two pieces of equipment and a fiber optic line between them, and that's at 10 gig. So, they're using the same speed between equipment. Up at the top, TDM takes a slice of each of the four inputs and puts it on the fiber optic line. There's no way to bump up that speed of 10 gigs without replacing the multiplexer equipment and the demultiplexer equipment. So, if your boss came to you and said, I want you to get 40 gigs out of this, you would have to go buy an OC768 multiplexer, an OC768 demultiplexer, and run a 40 gig line across to get 40 gigs throughput. Let's go down to the OTN circuit. Remember, I still have a 10 gig down there. I've got red light coming in, and I'm going to connect a 10 gigabit Ethernet to a red laser and begin to pulse that red light at 10 gigs. I'm going to connect a channel to a 10 gig Ethernet circuit on a yellow laser and send that pulse or modulate that signal at 10 gigs. And the same for the green, the same for the blue. The optic filter will combine all three wavelengths or pieces of light spectrum into one colored beam across. Remember, I still have a 10 gig connection between these two pieces of equipment, but I'm getting a total of 40 gig from this side to this side. You can see why WD wavelength division multiplexing far surpasses anything that TDM can produce. WDM is considered one of the best technologies to increase bandwidth over an existing fiber install or plant. It's important to understand there are two major flavors of WDM. One, coarse wavelength division multiplexing, CWDM. This supports up to 18 wavelength channels transmitted through a single fiber. Maximum distance up to about 160 kilometers, and that's only when you give up some of those channels we just talked about. Used in data centers and high-speed metro ring services. The second one is dense wavelength division multiplexing, DWDM. It can carry 40, 80, up to 160 wavelength channels per single fiber. Typically used in long-haul fiber. Expensive. Uses high wattage lasers that are much more difficult to maintain a consistent temperature. This is a great diagram of dense wave division multiplexing where we can see both the equipment involved as well as a block diagram of the light flow. Notice the abundance of boosters and preamps. That's to give us that long-haul transport. Coarse wave division multiplexing bandwidth grows each year. On the left-hand side we see a 40 gigabit coarse wave division multiplexing circuit, and each lane, lane 0, lane 1, lane 2, and 3 are each 10 gigabit ethernet converted into a appropriate light signal. And then it's out the optical multiplexer to a beam. On the right-hand side we have 100 gigabit CWDM, and again you can see the 10 gig circuits being brought in, optically filtered, and sent out. Right now 100 gigabit CWDM, very popular in today's data centers and cloud providers. Another important advantage that OTN offers relative to SONET and SDH is a stronger forward error correction. OTN uses a Reed-Solomon coding calculator across the payload. This improves the error correction significantly. Here we're going to compare OTN versus SONET again. At the top we have the conventional time domain multiplexing transmission system of 10 gigs. That's SONET above, and we're bringing in four OC48 circuits. Down below we're doing the same thing with DWDM. Back to the top, every 40 kilometers we have to add a repeater. So you can see each circuit needs those 40 kilometer repeaters as we go the entire distance to our destination. Down below the DWDM transmission takes four OC48 circuits, combines them into one single optic run through an optic amplifier, and this time we can use a repeater every 120 kilometers. You can look at the DWDM circuit versus the SONET circuit above and look at the reduction of equipment, maintenance, management, and you can see why DWDM is replacing SONET as we move down the road. Better scalability, higher bandwidth for lower cost. If the state of Florida came to Orange County Public Schools, which is located in Orlando, Florida, and said to it, you have to have a redundant data center for school data in the state of Georgia because of hurricane potential, we would have to, as you can see in the map of Florida, we would have to negotiate with many different operators in the state of Florida to get an OTN circuit up into Georgia. Not even talking about all the different operators in Georgia, but we would have to negotiate a circuit through all those different operators, CenturyLink, AT&T, Windstream, and probably more as we get into Georgia. The beauty of OTN is it provides monitoring even when you have many operators involved. OTN allows tandem connection monitoring with an OTN circuit. So on the left, I've got Orange County Public Schools data center. On the right, I've got my redundant data center in Georgia, and I've got all these operators trying to get that circuit from one to the other. The beauty of the monitoring system in OTN is it allows each operator to monitor the circuit between operators, operator A and B. I've got a monitoring circuit between both operators, and the nice thing is it's very from Orange County Public Schools to its redundant data center. I've got the PM end-to-end path monitoring that allows me to monitor the school system to monitor their data center. OTN provides monitoring in a far superior way than SONNET does. Here's an example of some OTN equipment. This is Sienna's product line. You can see they come from Massive to 2U Rack. Here is Cisco's OTN solutions, and you can see again from Massive enterprise carrier type equipment all the way down to a single U piece of equipment. So OTN can be very flexible and affordable. Comlink is a WAN provider for the Michigan, Wisconsin, Ohio area, as you can see in the fiber optic map on the left. They went to Inferia, which provides OTN equipment, and they were able to in just a couple days go from a 10 gigabit backbone to a 100 gigabit backbone with the flexibility of adding different modules and even bumping that higher. ADVA offers a wide variety of OTN equipment from the basic 1U Rack to Massive carrier type equipment, as you can see. Some of their equipment can produce up to 38.8 terabytes per second duplex capacity per fiber pair. They're pretty serious players in the OTN market. Nokia is a major player in the OTN product line. They have metro long-haul transports, back broadband, backhaul, data center interconnect, and much more. Even small local data providers such as Crown Castle Fiber, which you see their fiber layout over the city of Orlando, everyone can take advantage of OTN. They can double or triple their bandwidth by just simply replacing the equipment on the other end. This fiber plant can be up to 20 to 25 years old and still get that kind of performance. Lots of competition for OTN equipment. This is why it is so important for a network administrator to renegotiate their WAN contract every year or every two years, because you're going to miss out on significant WAN savings for your company. Here's a quick look at the OTN frame. If you'll notice towards the bottom on the left, you'll see those TCM fields, TCM1, TCM2, TCM3, TCM4. Those are all those tandem connection monitoring fields built into the frame that allows all those operators to still manage that OTN circuit. All right, let's get back to layer two in the WAN. Let's look at cable operators. And yes, I know the back of your hair is beginning to stand up as you think about your cable provider, but let's calm down and let's look at layer two from our cable provider. Here we have a good diagram of how most cable provider networks look today. You'll see you'll head in distribution hubs. Most of that is in a data center. It is all distributed via fiber optics and most of that is OTN. They may call it different equipment, but it's still OTN. It goes into optical nodes. Then the optical nodes are then transmitted into coax cable. You'll see those trunk RF amplifiers. That's all coax as it's sent out to all those happy, whining, complaining homeowners out there. Cable operators deliver Ethernet to your house based on the DOCSIS standards. Here are the different versions. Presently, I would say most cable operators have at least DOCSIS 3.0. Some of them are moving to 3.1. If your cable operator has DOCSIS 3.1, try to get a 3.1 modem installed in your home. It does make a difference. Remember, CableLabs is the organization that writes all of the DOCSIS standards and this is only Ethernet to your home. It has nothing to do with video. DOCSIS is an acronym for Data Over Cable Service Interface Specifications. It's how cable modems connect to the CMTS equipment and transmit data back and forth. Cable systems actually transmit RF signals in two directions. What's amazing as you study this technology is your Wi-Fi access point transmits RF energy to an antenna. Your cable modem does something very similar, except it transmits RF signals on a coax cable. The two have more things in common than you imagine. Here's a here's a breakdown of the DOCSIS architecture. Here we can see the cable modem termination system connected sometimes via fiber to the home, but most of us it's coax to each modem in the neighborhood. It all comes back to the head end or the cable LAN system. That's pretty much the DOCSIS architecture. Here we can see all the OSI layers between the cable modem protocol stack and the CTMS protocol stack. You can see Layer 2, 802.3 MAC on the cable modem. Layer 802.2 LC, which is a logical link. Then over to the CMTS stack and you can see Layer 2 over there. Because of the nature of cable systems, Layer 2 is very complex. This is a picture, a diagram, from DOCSIS 3.1. I downloaded the Layer 2 specification document just for Layer 2. It was 860 pages. CMTS cable equipment is going virtual. You can see this dedicated piece of equipment by AERIS and this is very, very expensive. But what's happened is a number of software vendors have actually emulated all the functionality of CMTS equipment and now you can port it onto a general platform like an HP Edgeline 8000 PC and run the entire functionality of that dedicated equipment on a general purpose computer. That's happening big time in the cable world. DSL, US carriers trying to stay in the last mile internet space. This is a basic diagram for DSL. If you're going to get phone service with your DSL internet, you're going to get, as shown in the diagram, you're going to get this residential gateway so they can split off your POTS or your plain old telephone system for your phone service and then, of course, your Ethernet. Everything depends on your home and the DSLAM. If you look back in the central office, you'll see the DSLAM unit. That's the key pieces of DSL. The big issue with DSL is it's delivered with copper pair. The same wire that used to deliver your home phone is now delivering high-speed internet. It does work, but it never will give you the bandwidth that other methods will. Now, if you're too far away, if your home is too far away from a DSLAM, say 35,000 feet or more, you're going to have a significant problem in getting high-speed internet. So, a lot of times, you'll see in the bottom of the picture, we see a fiber optic cable and a DSLAM unit move closer to the residence. That's going to improve their ability to get high-speed internet. Here's a DSL and its protocol stack for the various types of equipment from your DSL modem, your PC to your DSLAM unit. You can see it's pretty complex. They like to use point-to-point protocol and point-to-point over Ethernet. They even use ATM in case you're using voice. When you're connecting your home router up to DSL system, you're going to use, for your internet connection, a PPPoE or point-to-point over Ethernet. You're also going to use point-to-point protocol. That's because wired Ethernet networks lack basic security features to protect users against IP and MAC conflicts and rogue DHCP servers. It also allows the DSL providers to separate you from other users on the network. Ethernet, without a doubt, is the most popular Layer 2 protocol used in the local area network. What's happening, though, is it's growing in popularity in the WAN. New to Ethernet in the WAN are the new multi-speed switches that are coming in response to the new gigabit Ethernet. We now have 2.5 gigabit Ethernet and 5 gig Ethernet. Keep in mind, in the chart below, shows you your category cable that you must use for each of those new technologies. Ethernet in the WAN is a growing technology. If you look at the graphic on the right, you'll see a list of the U.S. carriers that are providing Ethernet services to the WAN. This is as of 2019, and you can see the top 7. So they're providing Ethernet private lines, virtual private lines, Metro LAN, and virtual private LANs. Popularity of Ethernet in the WAN is not just a U.S. or North America phenomenon. It's also global. Look at the graphic on your right. You'll see the 2019 top 7 global providers of Ethernet WAN services. Another important technology that's happening in Layer 2 is called link aggregation. You've all seen it. Ethernet bonding, Ethernet teaming, link bonding, link bundling, link teaming, and I could go on. Even retail motherboards are now being able to take two network cards on a retail board and bond them together. This is all Layer 2. Link aggregation is all about taking two circuits and making them act as one. So who are some famous users of Layer 2 channel bonding? Well, 802.11n, 802.11ac, 802.11ax, DOCSIS 3.0, DOCSIS 3.1, your mobile LTE, and 5G all use Layer 2 channel bonding.