NCSA NEWS

How do you set the usage record for a single month on a supercomputer? Start with a really big scientific problem, one that requires massive computing time to solve.

Astrophysicists Wai-Mo Suen of Washington University in St. Louis and Ed Seidel of the Albert Einstein Institute in Potsdam, Germany, and the University of Illinois at Urbana-Champaign, study black holes and neutron stars -- work that uses big computing time to examine big scientific problems.

For years, Suen, Seidel and their international research team have been racking up the hours on NCSA's SGI Origin2000 supercomputing system.

In June, the team virtually owned NCSA's 256-processor Origin2000 for a capability computing run of more than two weeks. Capability computing refers to superjobs that require dedicated use of a high-performance computing system for days or weeks. By the time Suen and Seidel had finished their simulations, they had output nearly a terabyte of data and logged an astonishing 140,000 CPU-hours on the Origin2000.

That's equivalent to more than 16 years on a single-processor machine. Never before has any research team exceeded 100,000 CPU-hours in one month on an NCSA Origin system.

Although Suen and Seidel are the lead researchers on the project, Suen credited the success of the June runs to "more than a dozen of us on both sides of the Atlantic cutting our daily allowance of sleep to less than five hours for two weeks." The runs were monitored nonstop in both Germany and the U.S. Whoever first noticed a problem took the initiative to kill, correct, and restart the job.

"The tight collaboration and the close monitoring have been crucial for successfully carrying out these unprecedented large-scale simulations of the unexplored territories of black holes and neutron stars," added Suen.

June's huge effort was the latest step in years of ongoing research. Suen and Seidel study gravitational waves, black holes, and neutron stars in an effort to understand the basic properties of these phenomena and to shed light on fundamental questions that range from the nature of gravitational waves emitted by violent events in the universe to the nature of space and time.

Over the last several years the research team simulated gravitational phenomena that occur when black holes or neutron stars experience head-on or grazing collisions. Their simulations use Einstein's general relativity theory, which consists of extremely complex sets of coupled, nonlinear, partial differential equations. The research group developed a large-scale, collaborative simulation code called Cactus to perform these simulations.

Past cosmic collision computations could be run, with few exceptions, only by simulating collisions in relatively simple configurations of black holes and neutron stars. The complexity of simulations was limited by available computing power. But during June's capability computing runs, the research team could simulate more details in gravitational phenomena than ever before. "All these phenomena can be studied in detail for the first time ever [this year]," said Seidel. "The large-scale capability computing provided by NCSA's high-end systems has been a major catalyst for this progress."

The research team could finally perform detailed calculations of two colliding black holes that were completely nonsymmetrical and therefore much more complicated. They were also able to add more physics to the simulations, including spin on each black hole and orbital angular momentum. June's colliding black hole simulations enabled the researchers to compute details of the formation of the black hole that is created when two spinning black holes of unequal mass graze one another.

They could also compute the gravitational radiation emitted in this collision process, a timely area of research because gravitational waves from violent events involving black holes and neutron stars are expected to be detected within the next 10 years by the gravitational wave observatories now being built in the U.S. by the LIGO project and in Europe by the VIRGO and GEO projects. According to Suen, the simulations marked the first time the research team was able to extract detailed data from this type of simulated black hole collision. When the simulations are further extended, Suen added, they will provide valuable information about gravity waves to the researchers who will be using the new gravitational wave observatories.

Dedicated time on the 256-processor Origin2000 array also allowed Suen and Seidel to begin numerical simulations of the coalescence of binary neutron stars based on Einstein's full theory of gravity. Scientists theorize that the merging of these binary neutron stars is related to the phenomena of gamma ray bursts -- one of the biggest mysteries of modern astronomy. Studying such events is the final goal of a NASA Grand Challenge project in which this research team is participating.

The neutron star simulations enabled the study of bodies under the influence of strong gravity. The simulations followed the colliding neutron stars to the point of gravitational collapse, where the neutron stars may form a black hole with an "event horizon" past which nothing -- not light or matter -- can escape. The team also conducted numerical studies of gravitational waves that evolve to form black holes under their own gravity. The terabytes of data generated by these simulations will be the focus of intense study for the next few months.

"These simulations could simply not be done without sustained access to a machine with the memory and horsepower of NCSA's 256-processor Origin," said Seidel. "It is really exciting. We are now at a stage where the algorithms, codes, and collaborations have matured enough that we can do new physics, but we are limited by how much time we can get on such large machines. I hope very much that NCSA will be able to continue to make such extraordinary high-end resources available in order to make this kind of science more routine."