FlexE Rollout to Begin in 2019 across Cloud and Telecom Service Providers Networks

The data rates that have been standardized by the IEEE do not always align with the needs of cloud and telecom service provider networks, and creating new rates can take four or more years to complete. Flexible Ethernet, or FlexE, provides a layer of flexibility to close the gap between what networks require and the Ethernet rates that are standardized, while also enabling lower cost solutions. FlexE has the ability to support large packet flows across multiple ports very efficiently, which enables it to support rates beyond the highest standardized Ethernet rate, and to do that at the lowest cost per bit by leveraging low-cost, high-volume optics. With the availability of flexible rate coherent optical solutions used to interconnect data centers and central offices, FlexE enables the fiber links to be configured to the optimal rate, even if that rate is not a standardized Ethernet rate. Use of FlexE greatly improves network capacity while reducing fiber plant capital expenditure.

For these reasons and additional benefits, the ecosystem is coming together to enable FlexE roll out in networks in 2019, with the portions of the network using FlexE shown in Figure 1. FlexE will be used between data centers (East-West traffic), from data centers to points of presence (North-South traffic), and within telecom service provider networks.

Figure 1. FlexE usage.

FlexEthernet Ecosystem and Deployment

The initial FlexE standard, which is based on n x 100G connectivity, was finalized in the Optical Interconnect Forum (OIF) in 2016. In the time since, the industry has been building the necessary technology to enable deployments of FlexE, with the key components as follows and shown in Figure 2.

1. Packet processor capable of supporting a 400G or larger packet flow
2. FlexE shim
3. OTN processor supporting a FlexE shim
4. Coherent DSP that is FlexE aware
5. Router equipment integrating these components
6. Optical transport equipment integrating these components.

Figure 2: FlexE components.

To enable FlexE capable optical transport equipment, a FlexE aware coherent DSP and an optical transport network (OTN) processor with FlexE support (such as this one) are required. Although many of the benefits of FlexE are tied to the transport network, FlexE needs to also originate on the router or switch, where a FlexE shim is required. The complete set of components and systems to enable end-to-end FlexE are on track to be available in 2019, allowing service providers to start full FlexE trials.

FlexE specification work launched because of requirements from hyperscale cloud service providers and was supported by companies such as Google and Microsoft in the standardization process. The hyperscale cloud service providers are also expected to be among the first to deploy the technology, with a focus on FlexE bonding and subrating. As detailed below, FlexE subrating enables transmission rates to be configured with a low granularity (at 25 Gbps and 50 Gbps increments), optimizing the fiber resource and providing capital expenditure savings. As cloud service providers are extremely focused on cost, FlexE bonding will enable them to support high rate logical ports from the router without the need for new transport equipment, as described by authors from Google in [1]. FlexE bonding can also be used as a more efficient alternative to Link Aggregation Grouping (LAG).

Telecom service providers have also started initial FlexE demonstrations, such as the one announced by Vodafone in June 2018. Full FlexE trials are anticipated for 2019, driven by benefits similar to those the cloud service providers expect, along with the potential to use FlexE channelization along with the OTN switched networks that have already been deployed around the world. Also, for service providers using MPLS, FlexE bonding enables a larger label switch path, simplifying network management.

FlexE Modes and Benefits

There are three modes of FlexE, adding flexibility to solve different types of rate mismatches (Figure 3):

1. Bonding: When the desired rate is higher than the port rate
2. Subrating: When the desired DWDM rate does not match a standardized Ethernet rate
3. Channelization: When the desired client rate does not match the port rate.

Figure 3. The three modes of FlexE.

FlexE Bonding. FlexE bonding can be used to support very large packet flows over multiple lower-speed ports, as shown in Figure 3A. There are many benefits to this, enabling both capital expenditure and operational efficiencies. In the example shown, a 600G packet flow is being bonded across six 100G ports. This enables the support of greater than 100G rates while leveraging high-volume, low-cost 100 Gigabit Ethernet (GbE) optics (e.g., QSFP28).

To support greater than 100G rates, 400G client optics are also now starting to become available. However, 400G QSFP-DD and OSFP optical modules are bringing new challenges to the industry to support PAM4 and an eight-lane electrical interface within a small form factor. It is expected that optical module vendors will need some time to ramp to the high-volume production needed to achieve lower price points. An alternative to QSFP-DD or OSFP optics is to use FlexE to bond 400G across 4 x 100GbE with QSFP28 transceivers, which enables the use of modules that are shipping in the millions of units today and have achieved a lower price point in terms of cost/bit.

FlexE bonding also provides service providers the ability to upgrade their routers and switches to support 400G or higher rates without needing to upgrade their deployed transport equipment supporting 100G clients. With 400G bonded across 4 x 100GbE, the transport equipment can be “FlexE Unaware,” meaning that it does not need to know that FlexE is being used in the system; it will treat the clients as though they are just 100GbE. The router or switch on the other end of the link then uses the deskew capability built into FlexE shims to recombine the multiple 100Gs back into the larger packet flow.

There are alternatives that exist today to bond ports together, such as LAG. However, FlexE is a more efficient approach. LAG uses a hashing algorithm to assign traffic to each link in a way that can be 20% to 30% inefficient. FlexE uses a calendar-based system that enables full bandwidth usage, reduces the number of ports required, and therefore reduces capital expenditures. LAG also has a limitation that limits the largest flow to the rate of a single link, whereas FlexE can have the flow be the same as the combination of all the links.

FlexE also enables operational benefits. Cloud service providers need to support many terabits of traffic, so a scalable approach is paramount. Instead of the network management system having to manage many individual 100G links, FlexE can be used to bond links together, significantly reducing the number of links that need to be managed. This reduces complexity, saves cost, and enables the level of scalability that is required.

FlexE Subrating. A second mode of FlexE, FlexE subrating, can be used to take advantage of advances in coherent DSP technology that support flexible modulation schemes. The coherent DSPs shipping in volume in 2018 generally support 100G per wavelength using DP-QPSK or 200G per wavelength using DP-16QAM. Coherent DSPs that will begin to ship in 2019 will both increase the baud rate to achieve higher data rates and add the ability to select a modulation scheme to configure the data rate across each wavelength at a 25- or 50-Gbps granularity. The low granularity enables more optimal bandwidth efficiency across the fiber and reduces the number of router and transport ports required, reducing capital expenditures.

An example is shown in Figure 3B. In this case, the long-haul optical link can support a 250G wavelength, but there is no Ethernet rate that corresponds to this. FlexE can enable support of the 250G rate. It uses the FlexE calendar system and bonding to have the 250G go between the router and transport equipment appearing as 3 x 100GbE, so that off-the-shelf 100GbE optics can still be used. On the transport platform, the FlexE aware DSP can use a standard 3 x 100GbE interface to the client optics, then “crunch” to the desired 250G rate to send across the fiber. Without FlexE, the network would have been required to use a 200G wavelength that aligns with an IEEE standard rate; allowing the use of the 250G wavelength provides 25% more capacity.

Doing an analysis to quantify the benefits that technology like FlexE subrating can enable on their long-haul data center network, Microsoft found that using 50-Gbps granularity provided a capacity gain of up to 70%, and a 25-Gbps granularity could provide a capacity gain of 86%[2].

It is expected that 400ZR optics will become available in 2019, supporting 400G with coherent optics in a small, pluggable form factor. Following this will be 400ZR+, adding support for 100G, 200G, and 300G rates over longer reaches. As 300G is not an IEEE standard rate, the packet-processor-to-DSP interface needs a way to support it. FlexE can use bonding to create a 3 x 100GbE interface to the DSP, without any bandwidth inefficiency.

FlexE Channelization. A third mode of FlexE is FlexE channelization. This mode enables client rates that do not match the port rate to be supported, as shown in Figure 3C. In this figure, the router on the left side is using FlexE to multiplex together different client rates and then bond these across multiple 100G ports. This ability enables the router to support a high number of low-rate clients without being limited by the optics that can fit on the faceplate. For example, ten 10G clients can go over a single QSFP28 port, which enables a higher faceplate density than using QSFP+ in a breakout mode or using SFP+. This is similar to one of the advantages enabled by using multi-link gearboxing (MLG). The advantage of FlexE is that it provides additional flexibility in how different rates are supported and multiplexed together and could enable any combination of clients spread across n x 100G links.

FlexE channelization can also be supported within an OTN switched network. An OTN processor with FlexE capabilities can terminate the FlexE stream, then map the FlexE clients to flexible OTN containers to be switched in the OTN domain. Using FlexE on both router and transport enables optimized faceplate density and eliminates the need for transport equipment to use expensive and high-power packet processors on Ethernet virtual private line (EVPL) cards.

FlexE channelization also enables service providers to reconfigure their client rates without a truck roll. The combination of 10G, 25G, and 50G clients shown in the example in Figure 3 can be reconfigured through software. The FlexE layer can continue to mux the different clients onto the same 100G ports already connected to the OTN switched transport network, reducing operational expenses and the time needed to bring up new services.

FlexE Future

Beyond the initial rollouts in 2019, FlexE was designed to be inherently scalable. It is already defined in a way that supports next-generation bonding and subrating across 400GbE optics and can leverage these once they are available in large quantity and low prices.

While FlexE was conceived to solve cloud service provider network challenges, the FlexE concept is being leveraged in applications beyond the cloud. These include FlexE channelization in next-generation mobile backhaul networks as a way to isolate certain traffic classes to meet requirements for latency and isolation. These additional use cases will drive further expansion of components and systems supporting FlexE and further drive a larger FlexE ecosystem.

References

[1] T. Hofmeister, V. Vusirikala, and B. Koley, "How can Flexibility on the Line Side Best be Exploited on the Client Side?" in Optical Fiber Communication Conference, OSA Technical Digest (online) (Optical Society of America, 2016), paper W4G.4.

[2] Filer, et al., “Elastic Optical Networking in the Microsoft Cloud,” Journal of Optical Communications and Networking, July 2016, Vol 8: pp. A45-A54.

Stephen Docking is manager, product marketing, Communications Business Unit at Microchip Technology.