High-Performance Computing Research Center
After several years of planning and more than a year of construction, Princeton University's High-Performance Computing Research Center opened its doors the week of Nov. 28, 2011. The facility gives researchers on campus new capacity to tackle some of the world's most complex scientific challenges.
Situated on Princeton’s Forrestal campus, the 47,000-square-foot building is the new home to powerful research computers that are capable of generating models of galaxy formation, tracking the motion of a single molecule and simulating the seismic forces of an earthquake, among other highly technical tasks.
The facility, which will come fully online when all systems are operational in January, is the centerpiece of Princeton's innovative plan to provide robust computing resources to all faculty members and researchers. Since Jan. 1 alone, the University's existing high-performance computers located at sites across campus — and which now will be united at the new center — have provided resources for 214 unique researchers from 57 faculty research groups and two undergraduate classes spread among 15 academic departments.
"Princeton's approach is really unique in that it is making these powerful computing resources available to all researchers," said Jeroen Tromp, the Blair Professor of Geology and the director of the Princeton Institute for Computational Science and Engineering (PICSciE), which will oversee the new facility in conjunction with the Office of Information Technology (OIT). "At most universities, researchers work department-by-department or individually to get the computing resources they need."
Today's scientific challenges involve tough problems and data amounts so large, they're measured in petabytes, or quadrillions of bytes. High-performance computing involves the use of supercomputers and computer clusters that can tackle difficult calculations and these large data sets. For example, the computers are enabling scientists at Princeton to model shock waves caused by supernova explosions, explore the feasibility of carbon dioxide storage, and design a cheaper and more efficient fuel cell. Such computers also help researchers understand the complexity of the human brain, how schools of fish decide which way to swim and how flocks decide which way to fly. (Read more about the research projects that will be supported by the center.)
About 70 percent of the computing power at the new center is to be dedicated to high-performance research computing, while the other 30 percent runs the email, databases and other computing services needed to support campus. The computers were moved from computing facilities at 87 Prospect, the New South Building, and Lewis Library on the University's main campus.
The high-performance research computers are part of Princeton's existing TIGRESS High-Performance Computing Center and Visualization Laboratory in the Lewis Library. TIGRESS is short for Terascale Infrastructure for Groundbreaking Research in Science and Engineering. While the five computers that make up TIGRESS have been moved to the new building, the TIGRESS staff, educational resources, and the visualization laboratory will remain in the Lewis Library on the main campus.
Keeping cool with an eye toward sustainability
The new facility at Forrestal was specially designed to provide power to the computers while using as little energy as possible. The entire ground floor of the new building is dedicated to maintaining the computers. One part of the floor is dedicated to electrical switching, another to back-up power supplies in the form of large batteries, and another to creating the cool air that will continuously bathe the machines.
Of all the features in the new Forrestal computing research center, air conditioning is among the most important, Hillegas said. "Anyone who has used a laptop knows that computers generate heat. These high-performance computers generate massive amounts of heat." Three massive water chillers, each the size of an ice-cream truck, cool thousands of gallons of water per day to a temperature of about 45 degrees Fahrenheit. The chilled water in turn cools air that is propelled by fans up and into the second-floor computer room to keep the mighty machines comfortably at room temperature.
In case of a power failure, a 100,000 gallon tank of pre-cooled water stands next to the building, ready for use until a back-up generator rumbles to life. The water is cool enough to be used without additional cooling, said Edward T. Borer Jr., manager of Princeton University’s energy plant. "We have a tank full of cold water that we can immediately circulate into the data center to start removing heat," he said.
On the second floor stand rows of cabinets that house the computer servers. Each cabinet generates as much heat as 200 continuously burning light bulbs packed into a space the size of a kitchen refrigerator.
To offset these cooling and power needs, the building has many energy-saving features. During winter, the air conditioning system can be switched off, and giant louvers on the south facing wall can be opened to let in cold outside air. This feature takes advantage of modern technical standards that allow computers to be exposed to a greater range of temperatures and humidity levels, said Borer.
Other sustainability measures include cooling towers that enable the chillers to be turned off when the outside temperature is near freezing. A second backup generator, this one gas-powered, has a co-generation feature that harnesses waste heat as energy to chill the water. It runs on natural gas and has a lower carbon footprint than the electricity provided by the power utility. The natural gas generator will be switched on when electricity prices are high, thus saving money and lowering Princeton’s carbon footprint.
One of the challenges of planning this building was to "right-size" it for current and future research computing needs, the facility's planners said.
"Research computing is not predictable because the technology changes fairly rapidly," said John Ziegler, director of real estate development at Princeton. The solution was to plan the existing building so that someday it can be doubled in size by constructing a mirror image of the existing facility, with the loading dock and freight elevator at the center. "We developed the site with an eye to the future and the expansion capabilities," Ziegler said.
Offices across campus continue to invest in the future of research computing at Princeton. Support for the new computers comes from a variety of sources, including PICSciE, OIT, the engineering school, the Lewis-Sigler Institute for Integrative Genomics, the Princeton Institute for the Science and Technology of Materials and PPPL, as well as a number of academic departments and faculty members. The acquisition of the new computers was made possible through grants provided by the National Science Foundation, the U.S. Department of Energy, the Air Force Office of Scientific Research, and the David and Lucile Packard Foundation.