The advantages and disadvantages of serial links in communication system design
Serial links, based on flexible PHYs, are frequently more efficient than parallel alternatives.
Michael T. Moore
Cypress Semiconductor
Today's communications systems are pushing the speed limit well beyond the traditional 1-Gbit/sec barrier. Traditionally, buses are limited in terms of scalability, bandwidth, reliability, and distance. Even PCI-X, the latest flavor of PCI, reaches a limit at just over 1 Gbyte/sec.
The primary limiting factor for shared-bus topologies is that of high I/O and limited distance. Reliability is a problem with shared buses, as a failure in any single pin or connection will cause the whole interface to fail. Shared buses lack the expansion and I/O capabilities to enable individual servers and storage racks to grow into multiple servers separated by distance.
A new paradigm is required to enable both chip-to-chip and inter-board communication at high speed, with lower I/O count and greater reliability. Serial links can replace both the traditional intra-box (e.g., PCI) and the inter-box (e.g., Ethernet or FibreChannel) standards with one solution operating over printed circuit board (PCB) traces, copper cable, or optical media. Newer interfaces such as Infiniband or Gigabit Ethernet may be used for high-speed applications.
Flexibility of serial links in communication system design
Serial links offer significant flexibility over parallel buses in terms of the media used. Both Fibre Channel and Gigabit Ethernet standards specify interfaces to fiber-optic and copper interfaces.
Module-based serial connectors can be used, enabling the user to switch between copper and a variety of fiber types simply by switching the module type on a port. For example, the Gigabit Interface Converter (GBIC) module is a removable, hot-swappable unit that allows easy conversion between media simply by changing the module. This module also allows blind-mating with the receptacle, enabling the user to easily replace (or test) defective modules on a board. The user can also switch between copper or fiber after the system has been designed and installed in a system. This is impossible with current parallel backplane technology.
Standards used in communication system design
Some of the more popular communication technologies currently in use and in development are described next along with their advantages and disadvantages.
Infiniband: Infiniband is a recent development in high-speed interconnect for servers and peripherals. It is focused on distributed I/O, dealing with the connections from server to server and to peripherals such as storage, switches, etc. Ethernet and Fibre Channel currently dominate this application space. Infiniband is also defined as an in-chassis connection protocol, but the complex software overhead associated with this protocol may initially ensure that Infiniband is limited to inter-box rather than intra-box connectivity.
Infiniband is a switch fabric I/O, where sources are dynamically connected to destinations through networks of crossbar switches and point-to-point links, thereby offering excellent electrical characteristics and frequency of operation. Because of the dynamic topology of switch fabrics, data can be routed from any node to any node over several possible paths, allowing the system to "route around" failed nodes.
The minimum implementation of Infiniband provides a full-duplex 250 Mbytes/sec bandwidth over four wires (two for transmit, two for receive) clocked at 2.5 GHz. The Infiniband spec also provides for x4 and x12 links, using 4 or 12 parallel implementations of the basic link. Infiniband uses the IBM 8B/10B encoding scheme to deliver data, control information, and the clock on a single differential pair.
Fibre Channel: Fibre Channel is an established distributed I/O interface, used in server to disk storage interfaces. It has low latency and high bandwidth efficiency, due to its low software overhead. Fibre Channel resembles Ethernet, but lacks the IP stack. Fibre Channel delivers a bandwidth of 100-200 Mbytes/sec today, using a 1- or 2-Gbit/sec physical layer
Gigabit Ethernet: Gigabit Ethernet (802.3z) is an extension of the popular Ethernet (802.3 10Base) and Fast Ethernet (802.3u 100Base) standards. Gigabit Ethernet extends the data rate and signaling conventions defined for the earlier standards to the 1-Gbit/sec rate, and is backward-compatible with the earlier flavors of Ethernet. The Gigabit Media Independent Interface (GMII) is defined for a variety of fiber-optic media types as well as for copper cable.
Both Gigabit Ethernet over fiber and Fibre Channel use 8B/10B encoding to provide an equal transition density of 1's and 0's in the data stream. This is to ensure that the clock can be recovered from the data.
Other Standards: SMPTE is a serial standard for video communication. SMPTE-292 operates at a rate of 1.482 Gbits/sec, and one application is to carry uncompressed HDTV information.
The Table lists today's bandwidth requirements for these protocols.
Both Gigabit Ethernet and Fibre Channel have defined roadmaps to the next generation of technology. Fibre Channel is expected reach 1.25 GBytes/sec in 2002 with a physical layer operating at 10 Gbits/sec. A new Fibre Channel spec (2X Fibre Channel) operating at twice the current speed is also awaiting adoption.
The IEEE recently ratified the 10-Gigabit Ethernet (10GE) standard (802.3ae), which continues the evolution of Ethernet in terms of distance and speed. It uses the 802.3 Ethernet MAC protocol and the same frame formats and minimum and maximum frame sizes.
The 10GE specification defines two types of PHYs: the LAN PHY and the WAN PHY. The terms "LAN" and "WAN" are not related to distance here. The WAN PHY specification is an extension of the LAN and is distinguished by the Physical Coding Sublayer (PCS).
The WAN PHY enables connection of 10GE to existing and future SONET/SDH equipment, based upon the OC-192 rate. The 10GE WAN PHY includes a simplified SONET/SDH framer in the WAN Interface Sublayer (WIS). This will enable Ethernet to grow from the LAN to the MAN and WAN in its native format. The LAN PHY has a payload rate of 10 Gbits/sec, whereas the WAN PHY has a payload of 9.29 Gbits/sec.
Serial solutions versus parallel buses
Some of the advantages of serial solutions are:
1) Module-based serial connectors enable the user to switch between copper and optical fiber, enabling maximum flexibility in terms of distance, noise immunity, and performance.
2) 8B/10B data encoding is used for DC balance (clock recovery) and error detection.
3) Error detection/correction coupled with improvements in PHY technology enable higher data rates over existing installed fiber infrastructure.
4) Correctly terminated differential serial signals can carry data at a rate of 3 Gbits/sec on regular PCB trace.
5) Differential serial links can drive longer distance than parallel buses and are less susceptible to noise.
6) Circuit board routing is simpler, as only two serial lines need to be routed, replacing a wide bus and the associated clock. Integrated clock data recovery (CDR) is particularly beneficial here.
7) Fewer connector pins are used, improving board area utilization and reliability.
8) Fiber-optic media offers improved electrical safety in harsh environments.
9) Point-to-point links have greater efficiency than shared buses like PCI.
Disadvantages of serial solutions in comparison to parallel buses include:
1) Parallel buses, in particular PCI, are very widely deployed and abundant cheap silicon is available.
2) Many engineers are experienced in designing PCI parallel buses.
3) There is a widely installed base of PCI-compatible equipment and software.
4) For lower data rates, PCI performs adequately.
5) Hot-swappable PCI solutions are available, allowing users to add or remove PCI adapter cards without having to shut down the system.
SERDES requirements
8B/10B encoding decoding: One of the requirements of many serial standards (Infiniband, Fibre Channel, Gigabit Ethernet, ATM, ESCON, and DVB-ASI) is that 8B/10B encoding and decoding are used on the original data stream. There are several reasons to use 8B/10B encoding/decoding on the data stream:
- Transition Density - The purpose of this is to guarantee enough signal transitions in the transmitted data stream to ensure correct phase lock loop (PLL) operation. Without this 8B/10B encoding scheme, a string of 1's or 0's in the data could cause the receive clock to drift or lose synchronization, causing loss of data.
- DC Balance - 8B/10B also ensures that the signal is DC balanced, which means that no DC-offset develops in the link over time. The 8B/10B code maps bytes (8-bit chunks) of the user data to multiple 10-bit output values that are chosen to be DC balanced.
- Error Correction - 8B/10B encoded characters follow specific rules that allow many signaling errors to be immediately detected.
- Special Characters - The 10-bit characters used by 8B/10B encoding offer 4x the characters of an 8-bit character space. This enables special characters to be encoded into the data stream, which are not decoded as data characters. These special characters can be used as delimiters or other commands.
CDR can be used to encode the clock signal with the data to eliminate clock skew problems. Traditionally, separate CDR chips have been used to encode and recover the clock from the data to be transmitted. However, advanced devices -- the "high speed" and the "frequency agile" programmable PHY devices described later in this article -- include integrated CDR on every channel. This is a significant benefit to the designer in terms of saving board space, routing, and power, and reducing design complexity.
Tight jitter performance, demonstrated by an eye diagram: Jitter is a high-frequency semi-random displacement of a signal from its ideal location. These displacements can occur in amplitude, phase, and pulse width and are generally categorized as either deterministic or random. For data communications parts, measurement and specification of jitter is normally restricted to timing displacements.
The jitter performance of a serial communications transceiver is largely attributable to the CDR's ability to recover the clock from the data, despite limited edge rates and the presence of noise. Good CDR circuit performance can mean the difference between a clean, open eye and a poor-quality closed eye diagram. It is important for communications system designers to choose a transceiver with good jitter performance, as this will enable greater reliability of communication in noisy, high-speed environments.
Flexible interface operation: The frequency-agile programmable PHY is well suited to backplane applications supporting standards such as Gigabit Ethernet, FibreChannel, SMPTE-292, HDTV, and others. A smooth integration of SERDES, CDR, and programmable logic enables a user to quickly create custom designs and concentrate on the parallel side of their interface.
To implement these interfaces with the serial transceiver, programmable logic offers the best choice of flexibility, performance, and time to market. Designers require the flexibility of using multiple data channels independently or taking advantage of channel bonding to form a wider data path. The programmable PHY interfaces to standard fiber-optic modules and also directly drives copper cables and circuit board traces for maximum flexibility. The frequency-agile programmable PHY provides four to eight serial channels, operating from 0.2 to 1.5 Gbits/sec, with an integrated receive framer for single or multi-byte alignment.
The high-speed programmable PHY can support 2.5-Gbit/sec operation for standards such as Infiniband or SONET. This gives significant flexibility to the user in terms of standards supported.
Combined programmable logic and SERDES: There are many advantages to a combined programmable logic and SERDES device. The programmable logic architecture is required to enable the user's custom functionality in the programmable PHY. This programmable architecture offers rich logic and memory resources, a flexible timing model with abundant routing resources, in-system re-programmability, and other features. This combination enables fast time to market, low costs for prototyping, and the ability to add to designs or change pinouts at any stage in the design cycle. This device integrates the high-performance PHY, programmable logic, and configuration FLASH memory in one package, minimizing board space requirements and design complexity.
Integrated memory and control logic with SERDES: The integrated programmable PHY requires abundant high-speed memory, including integrated control logic. This control logic consists of dual port arbitration and FIFO flag logic, integrated directly on silicon, ensuring that the memory operates at Fmax. This also saves the user from having to implementing the control logic in the programmable section, thereby saving gates.
Ideally, the memory intellectual property (IP) should be from an established supplier of high-performance memory for communications applications. This is because the memory performance is fundamental to the overall performance of the device.
Software support for Programmable PHY: The most effective use of the programmable PHY requires software that enables a seamless programming interface to allow design engineers to easily integrate custom IP with the SERDES via HDL blocks, HDL text, or graphical state machines. The software offers design entry, synthesis, and verification of both programmable and SERDES functionality in an intuitive integrated design flow.
Integrated configuration FLASH memory: Having an integrated configuration flash memory enables a one-chip solution integrating the PHY, programmable logic, and configuration block in single package. This reduces the board space required for the components and reduces the complexity of board design and layout.
General backplane applications
Backplanes are used in virtually all communication and storage systems to link line cards to a switch fabric. This architecture enables the switching of data from one a port of a line card to a distant port on another card.
A typical line card architecture is shown in Figure 1. The section highlighted shows where high-performance SERDES and programmable logic and bandwidth-optimized memory are used to control the interface between the "network processing engine" on the line card and the backplane.
Serial communication with fiber optics has many advantages over electrical or "wire-based" interfaces. Some of the advantages of fiber over copper backplanes are:
- Complete electrical isolation
- Small media size
- Light weight
- No radiated emissions from optical media
- Immunity from external electrical influences
- High data-carrying capacity.
The physical attributes of fiber-optic cable make is desirable in applications where safety may be a concern. When cut, optical fibers cannot cause fire or electric shock hazards and cannot cause a short circuit. When used with LEDs or low-power lasers, there is no optical eye hazard. Fiber-optic cable is also much lighter and physically smaller than copper cable for a given level of bandwidth. Optical fiber is a safe communications medium in physically hostile environments such as underwater, in harsh weather, or in the presence of explosive gases.
As highlighted in the Table, the frequency-agile programmable PHY described earlier provides high speed and flexibility for such applications as backplanes for storage area network (SAN), wide area network (WAN), wireless infrastructure (WIN), and local area network (LAN) switches. This highly integrated device combines a SERDES, programmable logic with abundant memory, and boot FLASH into a space- and power-saving, single-chip solution that eases board design. Designers have the flexibility of using its four data channels independently or taking advantage of channel bonding to form a wider data path.
As described below, the high-speed programmable PHY is a single or dual channel PHY and can implement an Infiniband link. This combination provides a point-to-point communications building block for high speed, fabric-based switching networks.
Infiniband applications
The high-speed programmable PHY with the InfiniPHY Core provides complete parallel-to-serial and serial-to-parallel conversion in a single chip. It integrates a 2.5-Gbit/sec transmit clock generation PLL, a clock and data recovery PLL, a SERDES, and an 8B/10B encoder/decoder. The device also provides an HSTL I/O parallel bus interface and superior jitter performance to ensure robust system operation. The available 50k gates of programmable logic with 144 kbits of available memory allow the user to further integrate custom functionality into the device.
The transceiver supports InfiniBand-compliant byte alignment and link training sequence, ensuring interoperability. This highly integrated solution results in significant savings in board real estate and power consumption.
Conclusion
This ideal serial solution offers a powerful integration of programmable logic and this high-performance serial interface. This flexible solution offers a variable data rate, and compliance with a broad range of standards. This is the ultimate data communications device, enabling a seamless integration of user's logic with high-speed flexible serial interfaces.
References
1) 10GEA technology overview white paper.
2) "The once and future Ethernet," Communications System Design, May 2001.
3) "Building terabit systems with multiple-gigabit multi-channel transceivers," EDN, June 21, 2001.
