NYSE Euronext and 100G: The drive to zero latency

By Dimple I. Amin and Andrew F. Bach

Overview

With a decrease of 1 ms in latency equating to up to $100 million for its trading partners, NYSE Euronext decided to launch the first commercial application of serial 100-Gbps technology.

According to a TABB Group report, it is estimated that a 1-ms advantage can equate to more than $100 million in financial transaction revenue per year. Perhaps that is why high-performance networks and data centers are considered the new trading floors of the 21st century.

A company that knows this all too well is NYSE Euronext. The world’s most diverse exchange group, NYSE Euronext offers a broad and growing array of financial products and services in Europe, North America, and Asia that includes cash equities, futures, options and exchange-traded products, bonds, market data, and commercial technology offerings. When looking to upgrade its networking infrastructure, it concluded that offering the highest data rate possible would help lower latency and provide its customers with a competitive edge. That’s why it decided to be the first enterprise to deploy 100-Gbps technology.

Low latency, high volume

With more than 7,800 listed issues globally, NYSE Euronext’s equities markets—the New York Stock Exchange, Euronext, NYSE Arca, and NYSE Amex—represent approximately 40% of the world’s cash equities trading volume, the most liquidity of any global exchange group.

A key focus for the exchange is an ongoing pursuit to build the most global, diverse, and technologically advanced marketplace to deliver significant operating leverage for its customers. NYSE Euronext’s business unit in charge of these electronic trading efforts, NYSE Technology, offers comprehensive global commercial technology, connectivity, and market data products and services to support sell- and buy-side trading customers.

The increasingly higher traffic volumes algorithmic trading has generated illustrate the networking challenges NYSE Technology faces. Such trading requires high-frequency systems to execute trades and market orders automatically. Financial services firms that rely on this trading for a large portion of their revenues require communications infrastructures that can help them aggregate hundreds of different market data points and then execute quickly on the resulting trade decisions.

Even when described in basic terms, the complexity of algorithmic trading support is apparent. Data is sent to the trading desks and aggregated into an automated trading system. The hedge fund’s or trader’s computer algorithms then dissect the data and compare it to predetermined trade triggers. If the data matches the trading triggers, the automated systems immediately execute the buy or sell command. Hedge funds and brokerage firms can buy and sell the same stock automatically and several times in just a few hours—moving millions of shares for a gain or loss of a fraction of a penny—using these algorithmic trading platforms.

In such a fast-paced, data-driven, and interconnected trading environment, lowering network latency can result in a significant competitive advantage if a financial firm is able to trigger its algorithmic trading and deliver the trade to the market faster than the competition. With latency on WAN connectivity often in the tens or hundreds of milliseconds, many market makers and major financial trading firms have increasingly begun to collocate their trading servers as close as possible to an exchange’s servers to further lower latency—often in a carrier hotel or other collocation site as well as an exchange’s own access centers.

NYSE Technology had already recognized this trend in its own business. Many of its trading customers were already consuming one or more Gigabit Ethernet (GbE) connections to its access centers; its own network was going to eclipse 10-Gbps WAN speeds in early 2009 while potentially pushing 1 terabit of data to the market later that year. So the exchange looked to build a new infrastructure to dramatically expand its bandwidth capacity and deliver ultralow latency trading to its customers.

The path to 100 Gbps

In early 2008, NYSE Technology created a cross-functional team to define, create, and deploy a new infrastructure—including data centers, metro and wide area networks, and trading systems—that would leverage the exchange’s existing Secure Financial Transaction Infrastructure (SFTI). The goal was to place NYSE Euronext at the forefront of ultralow-latency trade executions and market data distributions, with the ability to handle what the company estimates will be several billion daily transactions comprising petabytes of information by 2010.

SFTI (pronounced “safety”), managed by NYSE Technology, is NYSE Euronext’s ultralow-latency, highly available network for connecting members of the financial community with the infrastructure of the world’s leading markets for trading systems, clearing and settlement systems, market data distribution, and other core financial industry utilities over a single IP-driven platform. NYSE Euronext created SFTI to simplify customer connections by replacing multiple point-to-point circuits with access to a highly redundant, geographically diverse network offering a number of distributed access points that are interconnected to each other and the NYSE Euronext data centers, as well as the world’s leading market centers (see figure).

Market makers and other major financial trading firms can connect their equipment to SFTI directly in the NYSE Euronext access centers via Ethernet connections to the SFTI edge routers. Featuring independent, self-healing optical rings, SFTI is engineered for high performance, security, and resiliency with all connections, equipment, power supplies, and network links being redundant.

In metro New York City, NYSE Technology’s goal was to consolidate customer traffic from the SFTI access centers and backhaul it to its data centers. Currently NYSE Euronext operates three legacy data centers, but they are being consolidated into a new facility in New Jersey.

In the past, efforts to reduce latency had been implemented with superfast computers, faster software, and faster storage connections. These accomplishments have resulted in the communications network representing a far greater component of the overall trade/market data path than in the past. Today, saving fractions of milliseconds by reducing data transport latency in network systems and fiber-based transport networks is receiving more attention. In addition, since low-latency market data is uncompressed and trading volumes are increasing so rapidly, much more bandwidth capacity is required.

With customers using multiple GbE connections to SFTI and starting to move toward 10GbE connections, NYSE Technologies knew it needed an order-of-magnitude increase in capacity to ensure sufficient bandwidth and low latency. It was clear that moving to 40 Gbps would be a very short-term solution given the traffic growth and customer demand.

NYSE Euronext therefore decided to deploy the first true commercial 100-Gbps data stream over a single wavelength using the Ciena CN 4200 RS FlexSelect Advanced Services Platform. NYSE Euronext announced its plans to deploy the CN 4200 RS 100-Gbps capabilities in May 2009 as a core infrastructure component to SFTI to support the anticipated bandwidth demands of the exchange’s new state-of-the-art data centers in New Jersey and the greater London metro area.

The 100-Gbps link uses a full C-band tunable “muxceiver” in Ciena’s CN 4200 RS with dual-polarization return-to-zero differential quadrature phase-shift keying (DQPSK) modulation, allowing for deployment alongside 10- and 40-Gbps channels and enabling scalable capacity up to 8 Tbps. In addition, the muxceiver’s hardware technology employs a flexible architecture that can adapt to any standard 10-Gbps client rate using standard XFP optical modules, while the firmware allows for rapid adaptation to emerging OTN standards.

Soon after announcing the 100-Gbps network plans, NYSE Euronext decided to accelerate its schedule and deploy the new capabilities sooner than originally planned. In July 2009, the exchange deployed Ciena’s CN 4200 RS with 100-Gbps muxceiver cards in two SFTI access centers in metro New York City. These links began carrying live customer traffic—from market data feeds to trade executions—over a single 100-Gbps wavelength back to the existing data centers.

In addition to the 100-Gbps capacity, NYSE Euronext configured its CN 4200 RS chassis as a multidegree reconfigurable optical add/drop multiplexer (ROADM) to enable on-demand routing of any service anywhere on the network using a dynamic wavelength routing module based on wavelength-selective switch (WSS) technology. This functionality gives the exchange the flexibility to manage traffic across the SFTI infrastructure between access centers and data centers.

Future expansion

As of this writing, the initial 100-Gbps links have been operational for more than five months and the high utilization rates on the single 100-Gbps wavelengths validate NYSE Euronext’s decision to jump to that data rate. In addition, early testing of the 100-Gbps technology saw latency improvements of 60 to 70 µs, which have also been validated in the production environment.

By early 2010, the New Jersey data center will be operational with multiple CN 4200s and 100-Gbps muxceivers initially providing connections to the legacy data centers for the transfer of NYSE Euronext’s existing applications, traffic, and content. In the second quarter of 2010, the facility will officially open to customer traffic and will feature 100-Gbps links from the data center to SFTI access centers in and around New York City.

The next phase of the deployment will bring NYSE Euronext’s customers directly into the facility to further reduce latency. A significant portion of the data center’s overall footprint can be dedicated to collocation services that financial trading customers can lease to bring their algorithmic trading servers and applications within the shortest distance possible of the exchange’s equipment. Demand for collocation space exceeded initial estimates and is another part of the network where the 100-Gbps capability will provide the capacity and low latency for NYSE Euronext to offer its customers a high-performance trading environment.

In supporting the financial services industry, latency equates to a large extent to market share—whoever connects the order faster gets the order. Deploying 100 Gbps is a good example of the commitment NYSE Euronext is making to provide its customers with the lowest latency possible.

Dimple I Amin is vice president, products and technology, at Ciena Corp.

Andrew F. Bach is senior vice president, network services, at NYSE Euronext. Bach led the cross-functional team that devised the company's 100-Gbps strategy.