The NCSA Astrophysics Group's area of study is the entire universe--ranging in scale from its history and formation to the properties of individual structures within it. With Princeton University's Jeremiah Ostriker and other collaborators from across the country (see names below), they are tackling problems on a wide range of length-scales, including the structure of supersonic jets and cosmology.
The multiscale nature of these problems, as well as their nonlinear character, classify them as Grand Challenges. Group Leader Mike Norman states the group's goal as "setting the benchmark of excellence in these disciplines."
To study these phenomena, Norman; Greg Bryan, graduate student in astronomy; and Jim Stone, former postdoctoral research associate, have employed the CMHOG (Connection Machine Higher-Order Godunov) code on NCSA's 512-node CM-5. CMHOG implements the PPM (piecewise parabolic method) algorithm, developed by Paul Woodward at the Army High Performance Computing Research Center.
"CMHOG is the fastest implementation of the best algorithm on one of the largest parallel supercomputers in the world," Norman says. "It is like a big hammer. What do you do with a hammer? You go around squashing things. We are going to beat on a lot of classical problems."
The NCSA team's most recent jet calculation contained 32.5 million zones and ran at 12.5 Gflops on the CM-5. A striking visualization of this jet appears much like fire.
Immediate plans are to run a 100 million-zone calculation. Bryan says this large calculation will allow the group to probe why a jet seems stable for great distances and then suddenly stops.
"This phenomenon has a lot to do with turbulence," Bryan says. "The key thing is that it works on many scales. We want to resolve as many of those as possible."
A visualization of this simulation (above) shows the large- scale structure as intersecting filaments. These filaments represent sheets of galaxies that are typically 100 million light-years from end to end. The intersections are clusters of galaxies, ranging in length from 1 to 10 million light-years. Galaxies are typically less than 100,000 light-years in diameter.
NSF is funding further work in this area through a five-year HPCC Program Grand Challenge grant of $3.2 million; NCSA will receive approximately $650,000 of the grant.
The principal investigator is Princeton University's Jeremiah Ostriker, with whom Norman has collaborated for several years. The other researchers in the Grand Challenge Cosmology Consortium (GC3) are David Spergel (Princeton University), Ralph Roskies (Pittsburgh Supercomputing Center), Dennis Gannon (Indiana University), Daniel Reed (UIUC), Edmund Bertschinger (MIT), and Lars Hernquist (University of California, Santa Cruz). The consortium is going to use NCSA's CM-5 to simulate the formation of galaxy clusters and compare them with real clusters in the universe. These clusters are represented by the red regions in the filament visualization shown below.
"One can do that if what is going on outside has very little influence. Unfortunately, that is not the case if gravity is the prime influence, which it is in this field," he says. "These structures are entirely the result of gravity operating on initially small density fluctuations and [then] amplifying them."
From the galaxy sheets to a single galaxy is a factor of 1,000 in length-scale, or 3 orders of magnitude. "Our goal is to resolve the process of galaxy formation. You need at least a grid fine enough to put 10 zones across a galaxy--a grid resolution of 10,000 light-years," Norman says.
There are two possible approaches to simulating this multiscale problem at the needed resolution. One is what Norman calls the "brute force" method, running a simulation with 10,000 zones on a side. "This is feasible in 2D using current computers, but out of the question in 3D."
He imagines doing an 10,0002 simulation and getting resolution of 100 million zones but says it is not now possible to consider all the scales at 1012, which is (104)3, or the factor difference between the large-scale universe and single galaxy grids cubed.
A much more intelligent approach, says Norman, is refining locally using a multiscale algorithm. An inter-disciplinary project called HAMR (Hierarchical Adaptive Mesh Refinement) is the key here.
"Using HAMR, we will be able to achieve effective resolution of 10,0003, which is what we need to do the galaxy formation problem," Norman says. "We hadn't had a big enough computer to solve a multiscale algorithm. HAMR [on the CM-5] will crack this open."
Norman says HAMR is "like a Russian doll--a doll within a doll within a doll." However, "if you had HAMR, you would still have quite a bit of development remaining to have a cosmology or combustion code," he adds. "Our plan is to take HAMR and build a cosmology code--NC Hammer." NC stands for Numerical Cosmology.
The basic idea of HAMR is to use coarse zones where possible, only refining grid resolution where needed. Norman says one starts a simulation on a coarse grid. "As structure develops, the system has built into it various thresholds . . . that refine the grid locally. The key word is locally. We are trying to get away from refining the whole grid to the desired resolution."
HAMR will introduce finer subgrids in necessary places. "There is no restriction on the number of subgrids or levels of refinement," Norman adds.
Norman compares the grid hierarchy to UNIX directories. The large scale is the root directory, or base grid. A simulation is begun on this base grid at a low resolution, perhaps 643 or 1283. "Then the simulation proceeds, and structures develop. Level 1 subgrids are introduced, centered around regions where structure has developed.
"If those structures are adequately resolved by the subgrids, stop," he says. "But usually a cascade of structures form within the subgrid, and they need further refinement--level 2 subgrids."
At the higher level subgrids, Norman explains that they can use the same number of zones but get a higher resolution because of the smaller size of the "box" they are considering.
In one possible example, Norman imagines starting with a 643 calculation, which would give birth to a bunch of 643 calculations. "At the end, you may have over 100 grids at various levels of the hierarchy. What has happened is that this system has spawned all of these calculations, but these calculations are not independent. The boxes need to know what is going on in their neighbors' [boxes].
"The grids adapt hierarchically to resolve new structures, but they also adapt in the sense of tracking features," Norman says. "For instance, if a galaxy were moving, a subgrid would move with it."
HAMR thus must manage the grid hierarchy, but "What equations are actually being solved on these grids?" Norman asks. "So far, they are just the lattices. To make HAMR application- specific, it is necessary to tell it what equations to solve on each grid. Furthermore, you have to tell it the boundary conditions of each grid.
"To construct a hierarchical adaptive mesh application, you need two neatly divided pieces: an adaptive mesh refinement system that is application-independent and physics solvers," Norman states. Neeman is developing the first, HAMR. Bryan is developing physics solvers that will constitute NC Hammer.
Using HAMR's approach results in huge memory savings, Norman says. "It takes the completely impossible and makes it possible. The price to pay is that hierarchical adaptive mesh refinement is complicated.
"The computer science challenge along with the [science] Grand Challenge is how to implement HAMR on the CM-5," Norman says, stressing that researchers need a completely new programming language. As of early 1994, Norman says his group has the physics solvers (CMHOG, KRONOS) running on the CM-5 and that HAMR is almost running on conventional supercomputers.
"The Grand Challenge of cosmology is one of the thrusts that marry the two," Norman says.
