Case Study
How the Max Planck Institute for Gravitational Physics Built the World's Fastest Ethernet Compute Cluster in Its Quest to Detect Gravitational Waves
The Quest to Detect Gravitational Waves
With an international race to detect gravitational waves, Professor Bruce
Allen would leave nothing to chance. Professor Allen is the director of
the Max Planck Institute for Gravitational Physics in Hannover Germany,
and he is on a mission to observe the last unverified prediction of Einstein's
famous general theory of relativity.
Gravitational waves were first predicted by Albert Einstein in 1917 as part of
his General Theory of Relativity. A gravitational wave is a fluctuation in the
curvature of space-time that propagates through the fabric of space-time at the
speed of light. Such waves are produced by the rapid acceleration of very compact
and massive astrophysical objects, such as orbiting neutron stars or black holes.
They can also result from exotic processes in the early universe.
Special detectors are now being deployed to find evidence of these waves. These detectors utilize very precise laser interferometers at right angles to measure perturbations at long distances of up to 4 kilometers. The challenge to detecting gravitational waves requires correlating measurements from multiple detectors around the globe and searching for very weak signals in the data stream. And that requires evaluating very large numbers of theoretical models, each of which involves extensive number-crunching on a massive amount of data.
Gravitational waves were first predicted by Albert Einstein in 1917 in his revolutionary General Theory of Relativity. The ATLAS Compute Cluster built by the Max Planck Institute for Gravitational Physics is now being used to search for direct evidence of these waves.
Building a Compute Cluster
Professor Allen knows that the scientific challenge is enormous. According to Allen, “Gravitational wave research is one of the most exciting fields of science. It will open a complete new window to the universe and requires very large-scale and sophisticated computing technologies. But it’s hard to beat Mother Nature. We are trying to dig very weak signals out of detector noise. In many cases, the sensitivity of our search is limited by the available computer power. In some cases, even using every computer on planet Earth would not be enough to do the most sensitive possible search.”
With sufficient funding to do the job, Allen began by constructing a suitable data center. The facility would need to house thousands of servers and over a petabyte (or a million gigabytes) of storage. Allen and his team specified and oversaw the design of the power and cooling systems and then had the building retrofitted to accommodate the pipes and conduits required to house the high-performance computing (HPC) cluster.
The networking requirements for the HPC cluster were quite demanding: a highly scalable network with non-blocking throughput and low latency. The network would need to support both loosely-coupled inter-processor communications and the Message Passing Interface (MPI) for parallel processing, as well as provide fast direct access to over one petabyte of storage. And Allen wanted to keep the networking costs below 20% of the total cluster cost, which meant using a standard commodity solution like Ethernet and avoiding specialty interconnects.
Max Planck Institute Chooses Woven’s Ethernet Fabric for Its ATLAS Compute Cluster
Max Planck chose the Woven Systems Ethernet Fabric Switch to meet its stringent data center requirements. “Our research is pushing the state-of-the-art in computational analysis, and Woven’s innovative technology gives us a higher-performing and more flexible 10 Gigabit Ethernet (GE) network than traditional HPC switch suppliers,” Allen claims. “The price/performance and flexibility of the Woven 10 GE Fabric is unmatched by any other networking solution we could find.”
The new ATLAS Compute Cluster is located at the Institute’s data center in Hannover, Germany. The division of Observational Relativity and Cosmology, which oversees the project, is part of a collaborative effort employing the latest generation of sensitive detectors in the United States (LIGO), Italy (VIRGO), and Germany (GEO600). These detectors generate over a terabyte of data daily, which is available to all participating members for analysis.
The cluster must process a multitude of data analysis algorithms that scrutinize past and present measurements for patterns expected to result from passing waves. Each algorithm must process thousands of measurements and variables, and make numerous extrapolations. And new algorithms are constantly being added, including various combinations and permutations of existing ones.
The ATLAS Compute Cluster, which is currently configured with 1342 compute nodes, is housed in the Institute’s data center in Hannover, Germany.
The cluster itself consists of 1342 compute nodes occupying 32 full racks (42 nodes each). The use of Intel® Quad-core processors gives the cluster over 5,000 CPU cores. Each compute node has its own dedicated 1 GE connection to a Woven TRX 100 Ethernet switch, which has four separate 10 GE uplinks to the EFX 1000 Ethernet Fabric Switch at the core of the Ethernet fabric. Because this configuration is not over-subscribed, it assures non-blocking throughput performance for all the compute nodes.
Storage consists of both direct-attached disk drives and 42 separate file servers. The storage array has twelve 10 GE connections directly to the core EFX 1000 switch, and 30 GE connections to the Ethernet fabric via a TRX 100 switch. Once again, a non-blocking configuration is employed to ensure peak performance. Daily data from the four wave detectors is uploaded to the file server array, where it can be accessed by all of the compute nodes as needed. In total the system has over 1300 terabytes (or 1.3 petabytes) of data storage capacity available.
The Woven Ethernet Fabric, which can be deployed on a single or multiple EFX 1000 switches, utilizes multiple active paths to create a resilient fabric topology. Woven’s switch architecture incorporates patented vSCALE™ packet processing technology with Dynamic Congestion Avoidance to automatically redirect traffic to available, uncongested paths to deliver peak performance. Utilizing cut-through switching, the EFX 1000 achieves port-to-port latency of just 1.6 µs through a single switch and 4.6 µs across a multi-switch fabric. In the Atlas cluster, the latency from compute node to compute node (including hops across two edge switches) is about 10 µs from Gb/s port to Gb/s port.
The World’s Fastest Ethernet Compute Cluster
During the Institute’s extensive acceptance testing, the Woven Ethernet Fabric achieved a performance rating of 32.8 Teraflops using the HPC Linpack benchmark, which placed the cluster in the #34 position on www.Top500.org list from November of 2007 (In the following Top-500 list, released in June 2008, Atlas is in the 58th position world-wide). It is significant to note that not a single one of the higher-ranking clusters utilizes Ethernet, making ATLAS the world’s fastest Ethernet compute cluster.
Based on Woven’s vSCALE Dynamic Congestion Avoidance capability in a non-blocking 10 GE fabric, the cluster was able to reach 64% of the theoretical peak efficiency. This is also a significant achievement, according to Professor Allen: “The HPC Linpack experts we consulted tell us that they have never seen such high Gigabit Ethernet efficiencies on such a large cluster.”