Ethernet switches make their forwarding decision based on what field of the ethernet frame?

Switch

Ethernet switches that are used in HPC applications use cut-through operation in order to reduce latency. In this mode of operation, incoming frame headers are inspected and forwarding decisions can be made in a few hundred nanoseconds. Transmission of the frame can then start before the entire frame is received. Although this eliminates the possibility of checking the frame for errors using the frame check sequence before transmission, most fabrics of this type expect that the fabric interface adapters on the receiving compute nodes will perform this check when the packet is received and flag any errors.

Infiniband switches have less functional overhead than Ethernet switches in that they simply forward packets based on header address information. Forwarding tables simply consist of destination address—output port pairs that are populated during initialization and may be modified if something changes in the fabric. Unlike Ethernet, no routing or address learning features are required, which simplifies the design and allows Infiniband switches to achieve very low cut-through latencies on the order of 100-200 nS. Infiniband switches support multiple virtual channels per port which allow traffic types to be treated differently based on class of service indicators. Both Ethernet and Infiniband switches support link-level flow control with Infiniband using a credit based mechanism compared to Ethernet’s priority based flow control described earlier. Today, Infiniband switches are available with up to 32 54Gbps ports allowing the construction of large, low-latency fat-tree fabrics for massively parallel computing installation.

26.11 Switches and switching hubs

A switch is a very fast, low-latency, multiport bridge that is used to segment LANs. They are typically also used to increase communication rates between segments with multiple parallel conversations and also communication between technologies (such as between FDDI and 100BASE-TX).

A 4-port switching hub is a repeater that contains four distinct network segments (as if there were four hubs in one device). Through software, any of the ports on the hub can directly connect to any of the four segments at any time. This allows for a maximum capacity of 40 Mbps in a single hub.

Ethernet switches overcome the contention problem on normal CSMA/CD networks. They segment traffic by giving each connect a guaranteed bandwidth allocation. Figure 26.14 and Figure 26.15 show the two types of switches; their main features are:

Desktop switch (or workgroup switch) – These connect directly to nodes. They are economical with fixed configurations for end-node connections and are designed for standalone networks or distributed workgroups in a larger network.

Segment switch – These connect both 10 Mbps workgroup switches and 100 Mbps interconnect (backbone) switches that are used to interconnect hubs and desktop switches. They are modular, high-performance switches for interconnecting workgroups in mid- to large-size networks.

Ethernet Switches

An Ethernet switch is a multiport bridge that selectively forwards packets from one LAN port to another port. The bit rate on different ports may be different. Like hubs and bridges, switches may be interconnected to form larger networks. A switch's forwarding decision is based solely on layer 2 information. Switches do not modify the received packet. (Routers, by contrast, base their forwarding decision on layer 3 or network layer information, and also modify the received packets.)

Packets destined for different ports may be simultaneously forwarded by the switch, so a switch can increase the overall bit rate many times compared with a single shared LAN. But packets destined for the same port must be buffered by the switch. Thus a switch consists of a switching fabric, buffers, and the forwarding control mechanism. The switch fabric may be blocking or nonblocking, and the buffers may be segregated by either input port or output port or ports may share buffers. The fabric design and buffer management affect switch performance as discussed in Chapter 12.

The forwarding mechanism builds a table relating the MAC address of a computer on a LAN to the port number to which the LAN is connected. Such a table is built by associating the MAC source address of an incoming packet with the number of the incoming port. If a destination address cannot be resolved by the table, the switch, similarly to a bridge, sends the packet out on all the other ports. This can happen if the destination computer is beyond the LANs directly connected to the switch. Broadcast packets are sent to all the ports, except in the case of VLANs (see next page).

Performance depends on other features of switch design. A switch may forward a packet after it has been fully received (store-and-forward), or it may start forwarding as soon as the output port has been determined (cut-through). Although cut-through forwarding can clearly reduce latency, the switch cannot carry out a CRC check and so corrupted packets are forwarded.

Switches are deployed to improve LAN performance because they can increase the bit rate available to alleviate congestion and to match the bit rate to the traffic on different LANs. Thus, for example, in client/server networks, the clients would be connected to a switch's 10 Mbps ports and the servers would be connected to its 100 Mbps ports. However, if the traffic on switched LANs does not match port speeds, congestion can occur and degrade performance, as we discuss next.

The switch buffers can temporarily store packets contending for the same port. But if contention persists, the buffers will fill up and the switch will drop packets unless there is a flow control mechanism that sends a signal stopping the source from sending additional packets. (In the OSI model flow control is a function of the transport layer, layer 4, and not the link layer.) However, link-based flow control stops traffic on an entire link rather than stopping only the particular source responsible for the congestion. Link-based flow control can thereby interfere with an uncongested path. In the switched LAN of Figure 3.18, A is sending packets to X and B to Y. Both paths share link L. If the port at Y is congested (possibly because another source is also sending packets to Y), switch 2 will send a flow control signal to switch 1, disrupting the flow on the uncongested path from A to X. Virtual LANs provide a more flexible way to manage switched LANs.

Frame overhead

High-bandwidth Ethernet switch fabrics are now the leading networking solution used to interconnect servers and storage within large data center networks. With the movement toward modular rack scale architectures, some limitations with Ethernet are becoming exposed. As we discussed in Chapter 9, there are various methods for sending storage traffic across Ethernet networks such as iSCSI and FCoE. But for applications such as processor-to-processor transactions in high-performance compute clustering or for communication between the CPU and modular rack scale memory resources, Ethernet is not very efficient.

Ethernet has a fairly large frame overhead when transporting small segments of data. Consider transporting 64-bits of data between two processors in a cluster or between a processor and memory. Ethernet has a minimum frame size of 64-bytes. When you include the frame preamble and the minimum interframe gap, it can take 80-bytes to transport 64-bits of data which is an efficiency of 10%. In other words, a 10GbE link is carrying only 1Gbps of data. One could argue that data can be combined from multiple transactions to improve the frame payload utilization, but this just adds to the communication latency. In contrast, a CPU communication protocol, such as CPI, can transport 64-bits of data within 10-bytes which is an efficiency of 80%.

One way to solve this problem is to develop a more efficient communication protocol within the data center rack. This protocol should have low frame overhead to improve link bandwidth utilization along with high bandwidth and low latency. In addition, it could take advantage of link technologies used in high-volume products such as 100GbE in order to reduce cable costs. It is expected that, as rack scale architectures evolve, new fabric technologies like this will be employed in order to allow memory disaggregation from the CPUs and improve overall rack performance.

10.2 Ethernet virtual LANs

A LAN constructed with simple Ethernet switches is limited in terms of its bandwidth capacity and congestion tendency as the number of stations increases within a single network. To address this, LANs may be partitioned from each other and, in many cases, switched through overlapping physical switches. This partitioning creates virtual LANs (VLANs). Early implementations of VLANs broadcast certain packets to assigned ports, with dedicated cabling constructed to support connection to simple switch or router devices. Also, achieving high reliability in a heavily switched network requires the use of redundant cabling paths among switches in a spanning tree configuration. This becomes very difficult to manage as network size increases. To address these constraints, IEEE developed the 802.1Q standard, which defines a VLAN tag that is added to the basic Ethernet frame between the source address and user protocol type information. This tag includes a 2-byte VLAN protocol type identifier (or TPID) followed by a 2-byte set of tag control information that includes a priority level, canonical form indicator, and a unique VLAN identifier set between 0 and 4095. This tag allows a user to set up various VLAN configurations such as Port, MAC, ATM, or protocol-based VLANs as determined by the needs of the end network operator and the equipment used.

Once established, a VLAN allows a relatively large number of devices to communicate as if they were connected to the same physical network at an optimum speed with better management of wiring and bandwidth resources. However, as the number of devices present on any one network increased and the aggregate bandwidth needs also increased, basic 802.1Q VLANs became challenged in their ability to scale.

Spanning Tree

If we have a network of Ethernet switches, the forwarding mechanisms of the Ethernet star topology can still be used if the network topology is a spanning tree. A spanning tree is a connected network topology that does not have any loops or cycles; that is, it is acyclic. An acyclic topology has the property that between any pair of switches X and Y there is a unique path. A consequence of this property is that a switch X will forward and receive frames to and from switch Y through only one port. This will let the Ethernet switches maintain their forwarding tables.

If the physical topology of the network is an arbitrary mesh and not a tree, then links are blocked so that the unblocked network forms a spanning tree. Links that are blocked do not forward data frames. Figure 6.16(a) shows a spanning tree of switches in a mesh topology network.

The spanning tree protocol (STP) is a distributed algorithm run by the switches to form the spanning tree. The Ethernet links have weights assigned to them. The protocol creates the tree by first having the switches elect one of the switches as the root switch (see Figure 6.16(b)). If there are multiple candidate root switches, then ties are broken by comparing the candidates' Ethernet addresses. After the election, each of the other switches determines a shortest path toward the root based on link weights. For each nonroot switch, its port that leads to the shortest path toward the root is its root port as shown in Figure 6.16(b). The corresponding link is part of the tree, and the port at the other end of the link is called a designated port. All other ports are blocked. The unblocked links form a spanning tree as shown in Figure 6.16(b). Note that root ports are used to forward packets toward the root switch, while designated ports are used to forward packets away from the root switch to the outlying switches.

To determine a root and to compute shortest paths, the switches periodically exchange control messages called bridge protocol data units (BPDUs). These messages carry at a minimum the Ethernet address that the transmitting switch believes to be the root, and the weight of the shortest path to the root from the transmitting switch.

There have been a number of improvements to STP, so that the original STP is now obsolete. One of the improvements, rapid spanning tree protocol (RSTP) (see Section 9.3.2) reduces the convergence time to compute a spanning tree when there is a topological change. RSTP precomputes backup paths to the root, so that they can be switched to when necessary. An extension of RSTP to VLANs is the multiple spanning tree protocol (MSTP). Each VLAN has its own spanning tree and blocks the links for its VLAN group. Links that are blocked by some VLANs may be part of the spanning trees of other VLANs. Unlike STP, all links can be utilized as long as each link is covered by some VLAN.

This chapter discusses the role of hypervisors and virtual Ethernet switches in modern data center networks. Topics include Type 1 and Type 2 hypervisors, and discussion of the major hypervisors in use today (including PowerVM, KVM, VMware, Xen, and zVM). This chapter also discusses virtual local area networks and other types of network virtualization and encapsulation; virtual Ethernet adapters (VNICs) including VLAG and IPv6 considerations; shared I/O adapters including SR-IOV and MR-IOV; ESX Server virtualization; sockets and VDE industry standards such as IEEE 802.1Q, EVB, VDP, VEB, and VEPA for virtual Ethernet switches; and examples such as the Open vSwitch, Cisco Nexus 5000V, and IBM 5000V virtual switches. Enterprise applications such as the Open System Adapter and HiperSockets are also discussed as they apply to mainframe logical partitions.

Introduction to industrial networking

In this book, an “industrial network” is any network that supports the interconnectivity of and communication between devices that make up or support an ICS. These types of ICS networks may be local-area switched networks as common with distributed control system (DCS) architectures, or wide-area routed networks more typical of supervisory control and data acquisition (SCADA) architectures. Everyone should be familiar with networking to some degree (if not, this book should probably not be read before reading several others on basic network technology and design). The vast majority of information on the subject is relevant to business networks—primarily Ethernet and IP-based networks using the TCP transport that are designed (with some departmental separation and access control) primarily around information sharing and collaborative workflow. The business network is highly interconnected, with ubiquitous wireless connectivity options, and are extremely dynamic in nature due to an abundance of host-, server-, and cloud-based applications and services, all of which are being used by a large number of staff, supporting a diversified number of business functions. There is typically a network interface in every cubicle (or access to a wireless infrastructure), and often high degrees of remote access via virtual private networks (VPN), collaboration with both internal and external parties, and Internet-facing web, e-mail, and business-to-business (B2B) services. Internet connectivity from a business network is a necessity, as is serving information from the business to the Internet. In terms of cyber security, the business network is concerned with protecting the confidentiality, integrity, and availability (in that order) of information as it is transmitted from source generation to central storage and back to destination usage.

An industrial network is not much different technologically—most are Ethernet and IP based, and consist of both wired and wireless connectivity (there are certainly still areas of legacy serial connectivity using RS-232/422/485 as well). The similarities end there. In an industrial network the availability of data is often prioritized over data integrity and confidentiality. As a result, there is a greater use of real-time protocols, UDP transport, and fault-tolerant networks interconnecting endpoints and servers. Bandwidth and latency in industrial networks are extremely important, because the applications and protocols in use support real-time operations that depend on deterministic communication often with precise timing requirements. Unfortunately, as more industrial systems migrate to Ethernet and IP, ubiquitous connectivity can become an unwanted side effect that introduces significant security risk unless proper design considerations are taken.

Table 5.1 addresses some of the many differences between typical business and industrial networks.

Table 5.1. Differences in Industrial Network Architectures by Function

Note that these differences dictate network design in many cases. The requirement for high reliability and resiliency dictates the use of ring or mesh network topologies, while the need for real-time operation and low latency requires a design that minimizes switching and routing hops or may dictate purpose-built network appliances. Both of these requirements may result in a vendor requiring the use of specific networking equipment to support the necessary configuration and customization necessary to accomplish the required functionality. The use of specific protocols also drives design, where systems dependent solely upon a given protocol must support that protocol (e.g. serial network buses).

The network shown in Figure 5.1 illustrates how the needs of a control system can influence design (redundancy will not be shown on most drawings for simplicity and clarity). While on the surface the connectivity seems straightforward (many devices connected to Layer 2 or Layer 3 Ethernet devices, in a star topology), when taking into account the five primary communication flows that are required, represented as TCP Session 1 through 5 in Figure 5.1, it becomes obvious how logical information flow maps to physical design. In Figure 5.2, we see how these five sessions require a total of 20 paths that must be traversed. It is therefore necessary to minimize latency wherever possible to maintain real-time and deterministic communication. This means that Ethernet “switching” should be used where possible, reserving Ethernet “routing” for instances where the communication must traverse a functional boundary. This concept, represented in Figure 5.1 and 5.2 as subnets, is important when thinking about network segmentation and the establishment of security zones (see Chapter 9, “Establishing Zones and Conduits). It becomes even more obvious that the selection of Ethernet “firewalls” deployed low in the architectural hierarchy must be designed for industrial networks in order to not impact network performance. One common method of accomplishing this is through the use of “transparent” or “bridged” mode configurations that do not require any IP routing to occur as the data traverses the firewall.

Figures 5.1 and 5.2 illustrate a common design utilizing Ethernet switches for low-latency connectivity of real-time systems, such as data concentrators and controllers, and a separate router (typically implemented as a Layer 3 switch) to provide connectivity between the multiple subnets. Note that in this design, the total end-to-end latency from the HMI client to the controller would be relatively high—consisting of 11 total switch hops and 3 router hops. An optimized design, represented in Figure 5.3, would replace the router with a Layer 3 switch (an Ethernet switch capable of performing routing functions2). Layer 3 switches provide significantly improved performance, and by replacing separate Layer 2 and Layer 3 devices with a single device, several hops are eliminated.

In Figure 5.4, a design typical of one vendor’s systems has been provided. Redundancy is provided here by connecting systems to two separate Ethernet connections. While Figure 5.4 shows a very simple redundant network, more sophisticated networks can be deployed in this manner as well. The use of spanning tree protocol will eliminate loops (in a switched environment) and dynamic routing protocols will enable multipath designs in a routed environment. In more sophisticated designs, redundant switching and routing protocols, such as VSRP and VRRP, enable the use of multiple switches in high-availability, redundant configurations.

As we get lower into the control environment, functionality becomes more specialized, utilizing a variety of open and/or proprietary protocols, in either their native form or adapted to operate over Ethernet. Figure 5.5 illustrates a common fieldbus network based on FOUNDATION Fieldbus using serial two-wire connectivity, and reliant upon taps (known as couplers) and bus terminations. Many fieldbus networks are similar, including PROFIBUS-PA, ControlNet, and DeviceNet.

It should be evident by now that specific areas of an industrial network have unique design requirements, and utilize specific topologies. It may be helpful at this point to fully understand some of the topologies that are used before looking at how this affects network segmentation.

3.2 Evolution of Ethernet

The 10Base-T standard in 1990 spurred the adoption of Ethernet in LANs because it allowed the use of widely available UTP wire pair with RJ45 connector using hubs, and since use of wire pair provided separate paths for transmitting and receiving signals, collision was not detected due to elevated voltage in the medium, but MAU had to indicate to MAC if it detected simultaneous activities on both Tx and Rx paths because back plane of the hub acted as shared medium, and therefore the MAC layer had to implement backoff algorithm in case of collision. The use of hubs allowed the use of star topology with distance of each LAN link of 100 m and a collision diameter of 2500 m. This standard was based on a physical layer implementation of 10-MHz clock speed and a data path of 1 bit in the PLS layer as shown in Fig. 3.3. It should be noted that the master clock speed for Manchester encoding, which was used for encoding at 10 Mbps, always matches the data speed, and this determines the carrier signal frequency, so for 10-Mbps Ethernet, the carrier frequency is 10 MHz.

With the rapid growth of Ethernet LANs and evolution of Ethernet switch, it became clear that higher bandwidth would be required to support aggregation of 10-Mbps links. The IEEE 802.3u task group was tasked with developing 100 Mbps Ethernet specification. After deliberations, they set the following goals that have served well in the Ethernet evolution:

ease of migration and seamless integration with installed base;

implementation over widely available UTP wire pair;

a 10-fold increase in performance should not increase price by more than a factor of 2;

leverage existing technology from (fiber distributed data interface) FDDI; and

analyze market research to ensure adoption by various industries and research organizations.

The goal of 100 Mbps bandwidth could be achieved by increasing the clock speed from 10 to 100 MHz but that would reduce the slot time to 5.12 μs from 51.2 μs, and as a consequence, the collision diameter would be reduced from 2500 to 250 m. Both were not acceptable. Therefore, the 100 Mbps standard changed the clock speed to 25 MHz and increased the data path to 4 bits. Increasing clock speed from 10 to 25 MHz was an increase by 2.5 times, and combining this with 4-bit–wide data path gave a 2.5 × 4 = 10-fold increase in the bandwidth without reducing the collision diameter substantially. In addition to this, the encoding was changed from Differential Manchester to first non-return-to-zero (NRZ) and later to non-return-to-zero, inverted (NRZI) to meet the needs of increased bandwidth. This encoding/decoding function was moved from PLS layer to a physical coding sublayer (PCS). Also, the MAC layer was decoupled from the physical layer (PHY) by replacing PLS layer with a Medium-Independent Interface (MII) layer which is equivalent to AUI layer as far as functionality is concerned except that MII functions at higher clock speed of 25 MHz and has a 4-bit data path. This MII layer allowed to keep MAC and LLC layer unchanged from 10 Mbps to 100 Mbps evolution. Instead of MII, one could use Reduced MII (RMII) also which uses 50-MHz clock instead of 25-MHz clock and 2-bit data path instead of 4-bit data path. RMII allows the use of eight-pin interface instead of 16-pin interface needed for MII. For discussion purposes, however, we will refer to MII only. To allow for ease of migration, the IEEE 802.3u standard made MII layer support both 10 and 100 Mbps by autonegotiation (AN) sublayer to detect attached end station or device to determine the speed that the device supported and also to detect if device supported duplex or HDX capability. To keep MAC layer unchanged by the introduction of 4-bit data path, the standard introduced a reconciliation sublayer to account for this 4-bit data path. Although with the use of FDX wire pair and switch, CSMA/CD for collision handling became redundant, there still was a need for flow control which thus far was handled indirectly by CSMA/CD function. For this flow control, the standard provided a MAC control sublayer in the MAC layer. This sublayer detected and processed a special MAC control frame or pause frame to affect flow control. Functionally, MAU, which is the combination of PMA and MDI sublayers, remained same and provided an MDI. In addition a physical medium–dependent (PMD) sublayer was added below PMA sublayer so that PMA can get unserialized data from PCS during Tx path and pass serialized data to PMD and receive serialized data on the receive path from PMD and pass unserialized data to PCS. Fig. 3.9 shows the comparison of 10 and 100 Mbps data link and physical layers and also maps to OSI seven-layer model. This figure also shows other standards including 1, 10, 40, and 100 Gbps.

By 1995, 100 Mbps also known as Fast Ethernet was widely adopted and implemented. By that time, the emergence of high-bandwidth applications, which were time critical, required further increase in bandwidth. This time, the IEEE 802.3z task group was tasked to develop the 1000 Mbps or 1 Gbps Ethernet standard. They kept similar goals in mind as the earlier task group had kept while developing 100-Mbps standard. The proposal became standard in 1998 to meet these demands. With the support for VLAN adopted in 1998, the 1-Gbps standard also included support for VLAN and was based on using 8-bit data path and a clock speed of 125 MHz. These two combinations gave a 1000-Mbps bandwidth. However, increasing the clock speed to 125 MHz reduced the slot time to 0.512 μs and collision diameter to 25 m. Both of these were not acceptable. So this time, the standard increased the Tx time to 4096-bit times from 512-bit times by adding a carrier extension while keeping the frame size unchanged. This effectively increased the slot time by a factor of 8 and collision diameter to 200 m. It should be noted that the slot size issue and the frame extension is really only an issue for HDX transmission which is exceedingly rare at 1000 Mbps and somewhat rare today even at 100 Mbps.

The MII sublayer was modified to gigabit MII (GMII) sublayer to handle higher clock speed and 8-bit data path. This standard also introduced frame bursting to improve performance. Here, if an end station had multiple Ethernet frames waiting for transmission, then the end station would keep the channel occupied by transmitting a control character rather than initiate IFG. This control character was followed by an IFG of 0.096 μs which is the IFG at 1000 Mbps and then transmits the next frame. This control character prevented another end station from capturing the channel. This process is followed to a maximum of 8000 bytes of data before relinquishing the channel. The encoding in PCS sublayer was also changed from NRZI to 8B/10B, and the AN functionality was combined within PCS layer. One Gbps standard also allowed for asymmetric flow control in the sense that switch controlled the flow from an attached device, but the device was not allowed to control traffic from switch. This mechanism allowed buffering on the device in case of flow control and switch did not have to add buffers for flow control. The physical layer used fiber channel technology for encoding in PCS layer and connectors in the MDI layer. The PMD layer defined the standard for converting electrical signal to optical signal for transmitting over fiber optic cable including multimode fiber (MMF) or single-mode fiber (SMF). The distance that the signal could travel depended on the wavelength of the light source, fiber diameter, and type of the fiber. One-Gbps Ethernet used Gigabit Ethernet Interface Converter (GBIC) type of transceivers which supported short- and long-wavelength lasers using MMF (62.5- and 50-μm diameter) and SMF (10-μm diameter) optical fibers as well as short copper wire–pair physical interfaces. This also was based on fiber channel technology.

With the success of 1 Gbps standard and increasing need for higher bandwidths to aggregate 1-Gbps links, there was a need to further increase bandwidth. IEEE 802.3ae task group started work, and in 2002 the standard for 10 Gbps was published. Ten Gbps offered considerable saving over Ethernet over synchronous optical network (SONET) deployments. Ten Gbps was based on 64B/66B encoding in PCS sublayer, and in 10-gigabit MII sublayer, clock speed was increased to 156.25 MHz, and the data path was increased to 64-bit path with 32 bit for Tx data path and 32 bit for receive data path. Increase of clock speed from 125 to 156.25 MHz was a 1.25 times increase this coupled with eight times increase in data path increased the bandwidth by 1.25 × 8 = 10 times from 1 to 10 Gbps. This standard operated only on fiber optics medium in FDX mode with 1310- and 1550-nm lasers giving a range of 40 km which extended the reach of Ethernet from LAN to MAN. For larger distances covered by WAN, this standard also provided for a WAN Interface Sublayer (WIS) that formatted the frames to be SONET compatible, so that SONET infrastructure could be used for transport over WAN. This standard also needed a new PMD layer because fiber channel technology–based PMD sublayer used in 1-Gbps standard could not be used. Later, this 10-Gbps standard was modified to support copper wire pair for up to 100 m for data center applications and also to support transport over DWDM with Reconfigurable Optical Add–Drop Multiplexer (ROADM) so that frames could be sent over RAN and WAN without using SONET technology and that provided considerable cost savings. The GBIC transceivers and (10 gigabit small form factor pluggable (X is Roman character for 10)) XFP transceivers were replaced with small-form factor pluggable (SFP) and SFP + type of transceivers.

With the spread of 10-Gbps Ethernet, stage was set for another 10-fold increase in bandwidth to aggregate 10-Gbps links and also to provide support to increasing data center applications due to the onset of cloud-based applications. The IEEE 802.3ba task group started work on this 40 and 100 Gbps standard in 2008 and published the standard in 2010. The deployment of these 40 and 100 Gbps switches has commenced just now in 2015. The layers for these are also shown in Fig. 3.9. The increase in bandwidth was achieved by retaining 64B/66B encoding, and 156.25-MHz clock speeds in 40- and 100-gigabit MII sublayers from 10 Gbps standard but by increasing the number of lanes or wavelengths in PMA layer to 4- for 40-Gbps and 10- for 100-Gbps bandwidths. This standard also provides for the use of SFP, SFP+, and QSFP (Quad SFP, i.e., 4 SFP transceivers in one) type of transceivers. This standard removed the WIS layer (the layer that provided SONET support) so that Ethernet could use DWDM for WAN applications.

By tweaking clock speeds, data bit path, encoding/decoding methods, slot time, number of lanes or wavelengths for serialization/deserialization, and by leveraging improved ASIC chip based hardware, and enhancements to software, the IEEE 802.3 task groups were able to increase bandwidth from 10 Mbps to 100 Gbps while retaining the original Ethernet frame format shown in Fig. 3.7. Table 3.3 below gives a list25 of Ethernet evolution, and it is very interesting to track all the important developments related to Ethernet that is so profoundly changing the data networks. It also shows that the work has started on 400 Gbps. Table 3.3 also shows that the standard using multiple 25/50Gbps lanes is expected by 2017. There are already talks on starting the work on developing 1-Tbps (Terabit per second) Ethernet standard. So far, the basic Ethernet frame format shown in Fig. 3.7 has remained unchanged, and it will be interesting to see if it will remain so in the case of 1 Tbps as well.

Table 3.3. Evolution of Ethernet

EPON, Ethernet passive optical network; IoT, internet of things; LLDP, link layer discovery protocol; MIB, management information base; RFC, request for comments; TBD, to be determined.

Now that we have explored the definition of the Ethernet and its evolution, it will be helpful to briefly examine the components related to hardware of a typical Ethernet switch because it is the hardware that forms the data network and it is the network that moves bits from source to destination and on this movement of bits, ride the services needed by customers.

7.5 Ethernet Fabrics in the Data Center

Traffic engineering in the data center is challenging using traditional Ethernet switches arranged in a typical hierarchy of increasingly powerful switches as one moves up the hierarchy closer to the core. Although the interconnecting links increase in bandwidth as we move closer to the core, these links are normally heavily oversubscribed, and blocking can easily occur. By oversubscription we mean that the aggregate potential bandwidth entering one tier of the hierarchy is greater than the aggregate bandwidth going to the next tier.

An alternative to a network of these traditional switches is the Ethernet fabric [7]. Ethernet fabrics are based on a nonblocking architecture whereby every tier is connected to the next tier with equal or higher aggregate bandwidth. This is facilitated by a topology referred to as fat-tree topology. We depict such a topology in Figure 7.8. In a fat-tree topology, the number of links entering one switch in a given tier of the topology is equal to the number of links leaving that switch toward the next tier. This implies that as we approach the core, the number of links entering the switches is much greater than those toward the leaves of the tree. The root of the tree will have more links than any switch lower than it. This topology is also often referred to as a Clos architecture. Clos switching architecture dates from the early days of crossbar telephony and is an effective nonblocking switching architecture. Note that the Clos architecture is not only used in constructing the network topology but may also be used in the internal architecture of the Ethernet switches themselves.

For Ethernet fabrics, the key characteristic is that the aggregate bandwidth not decrease from one tier to the next as we approach the core. Whether this is achieved by a larger number of low-bandwidth links or a lower number of high-bandwidth links does not fundamentally alter the fat-tree premise.

As we approach the core of the data center network, these interconnecting links are increasingly built from 10 Gbps links, and it is costly to simply try to overprovision the network with extra links. Thus, part of the Ethernet fabric solution is to combine it with ECMP technology so that all of the potential bandwidth connecting one tier to another is available to the fabric.