Cluster Opinion: The Ethernet Opportunity for Linux Clusters

Bernard Daines
Linux Networx

Over the last few years, cluster systems have become a proven alternative to traditional supercomputers in the high performance computing market. According to Top500.org (see Resources), cluster systems currently make up 149 of the 500 fastest computer systems in the world. Presently, the third most powerful supercomputer in the world is a Linux cluster. In most computing architectures, Linux clusters have proven their ability to perform as well as, or better than, supercomputers, at a significantly lower cost to the purchaser.

Just because Linux clusters are gaining in popularity, traditional supercomputers will not disappear. Supercomputers are good at solving highly specialized, complex problems involving computations in which interactions among many variables are analyzed simultaneously. Because their specialized processors and other components are designed to directly and rapidly access a large shared memory resource, supercomputers speed up these type of complex calculations, but at the cost of high price and reduced flexibility.

Linux clusters, with each node of a cluster having its own operating system and often its own private memory, excel with computationally intensive problems that can be broken into many smaller, independent parts. As the popularity of Linux clusters increases, so does the number of applications that work on Linux clusters. Currently, such clusters are helping their users design better rockets, improve engine block molds, and create safer cars. They also predict the watering needs of farmers and ranchers, and assist with national security needs. Just a few years ago, all of these tasks were performed by expensive, proprietary supercomputers.

Oracle is aggressively adopting Linux both internally and for its products. Recently, the company's chief executive Larry Ellison pledged to run the company's entire business on the Linux operating system. Oracle's current cluster solution is its 9i Real Application Cluster, which it promotes over conventional "heavy iron" database implementations for jobs such as online transaction processing (OLTP) and enterprise decision support.

More than a computing resource

Linux clusters are more than just teraflops supercomputers. Smaller Linux clusters are very beneficial to organizations with moderate computing needs. For example, Audi is using a 17-node, dual processor Linux cluster to simulate components for future car models. As Audi's future computing demands increase, expanding their computing capacity is as simple as adding additional nodes to their cluster system (see Resources).

Unlike traditional supercomputers, clusters also have the advantage of being able to run several applications simultaneously. A 250-node cluster, for example, can be broken into five subclusters of 50 nodes each, or one subcluster of 150 and another of 100 nodes. Each subcluster has the ability to work on different calculations and run different applications.

With these benefits notwithstanding, Linux clusters have yet to make a significant dent in the enterprise and high availability markets. Yet, those environments are ideal for Linux cluster systems. More accurately simulating the probable outcomes of complex real-world activities is critical to the survival and success of many organizations. For example, many financial organizations need to model the effects of long-term market volatility on financial planning scenarios. As well, the reliability and flexibility of Linux clusters also make them ideal for data centers which are increasingly populated with database, Web, and file servers.

Wanted: An open interconnect

Costing less while being more flexible, reliable, and powerful are not enough for Linux clusters to succeed in the enterprise and high availability markets. An open-standards based interconnect with low latency and high bandwidth would make clusters even more attractive in those environments. Currently, low latency, high bandwidth cluster interconnects are also proprietary and expensive. An open-standards based interconnect would allow more cluster features and functions to become available and easier to integrate in a data center environment.

From the enterprise and high availability perspective, an open-standards based interconnect requires seamless connectivity, high scalability, simple management, the ability to play well with others, standard components, and a lower cost. While current interconnects offer bits and pieces of the puzzle, none currently contain all the necessary requirements.

Several high-performance interconnects are available for clusters today. While they deliver on performance, they make it difficult for cluster nodes to communicate with other machines in the data center. Less expensive interconnect options may communicate better with other machines, but suffer from higher latency, lower bandwidth, and are often too cumbersome when compared with the proprietary interconnects.

InfiniBand was to be the next standards interconnect, but has yet to gain industry-wide support (see Resources). InfiniBand documentation is open to the industry, but currently too few organizations are investing in it to make InfiniBand a strong market force.

Ethernet: An Open-Standards Solution for Linux clusters?

While proprietary interconnect and InfiniBand vendors may be working to address these issues, there is another option with a broad market potential, and technical and economic feasibility: Ethernet.

Today, off-the-shelf Ethernet lacks the technical capabilities required for Linux clusters in the enterprise and high availability environment. For example, while 10 Gigabit Ethernet has sufficient bandwidth to handle many complex problems, its latency is too high for many critical cluster applications.

Various organizations are attempting to reduce Ethernet latency by adding off-load engines for higher layer protocols (e.g., TCP/IP) (see Resources). While the latency of Ethernet with off-load will be better than without off-load, that does not fix many inherent latency issues common in Ethernet switching today. While off-load engines offer a small class of enterprise solutions with limited process-to-process communication ability, the general interconnect problem remains unresolved.

Delays in the uptake of 10 Gigabit Ethernet, due primarily to market conditions, are unfortunate. Extensive development in this space is being starved. Even so, while this technology will certainly be delayed, there is little reason to doubt that it will ultimately thrive as all lower speed Ethernet technologies have before.

According to some estimates, more than 85 percent of all network connections (95 percent of end connections) are Ethernet (see Resources). An improved Ethernet interconnect will allow clusters to make incredible inroads into the enterprise and high availability markets. Ethernet has the broadest skill set in the industry, and is also the most trusted network technology. It contains the lowest-risk roadmap to the future, and we can anticipate the best cost/performance ratio from it.

The New Ethernet

To be successful in the enterprise and high availability markets, the new Ethernet will need:

• Universal Ethernet frames
• Common components
• Common management
• High bandwidth/low latency
• Lower cost
• Datacenter operational focus
• Ability to communicate with other machines in the enterprise arena

Success will go to the organization(s) that integrate these capabilities in harmony. A new, powerful data center Ethernet has the potential to fulfill this vision. As long as the new solution is based on Ethernet, it will ensure broad appeal and industry-wide support, and will gain a large piece of the enterprise and high availability interconnect market.

Resource:

Top 500

The InfiniBand Trade Association

About Ethernet, CISCO Systems, Inc.

Gigabit Ethernet Over Copper Deployment Guide, Intel Corporation.

Audi Deploys Linux Networx Cluster System for Speedy Automotive Design Simulations

iWarp interface spec could bust IP bottlenecks, EETimes.com, 2001.