The photo shows our new, water cooled compute cluster which is now used to produce a large fraction of our results. It consists of a Sun Blade Constellation system with a total of 384 2.9 GHz cores and 2.3 TB memory, wired over a 72 port 40 Gb/s Infiniband-Switch and 48 1Gb/s Ethernet ports, connected via Ethernet and Infiniband to a set of three Sun fileservers with a total of 120 TB filespace. Two additional Sun compute nodes with a total of 48 2.6 GHz cores and 320 GB memory complete the system. Hidden behind the left rack there's also a second Sun blade system consisting of 10 blades with a total of 80 2.8 GHz cores and 640 GB memory. All nodes and servers are using Opensolaris with the new filesystem ZFS as operating system. The whole cluster of 512 cores currently delivers about 5 TFlops computing power.

The blade system is cooled with a Sun Cooling Door for 35 kW max lost heat, both racks have water cooled back doors for 8 kW max lost heat. The heart of our water chiller is shown in the picture below, it can cope with a total of 40 kW lost heat.

We are especially thankful to Circular Informationssysteme GmbH and Sun Microsystems for their patient help configuring this highly experimental system, Kortüm und Zoldan GmbH as well as GETA GmbH for the custom built water cooler system, and, last but not least, the department TID of  KIT for the fast help during the realization of this complex project.

All pictures ©Markus Breig, KIT.

right rack:
Sun Blade 6048 Constellation with Sun Cooling door 5200, 24 Sun X6275 blades with 2 nodes each, each node has two Intel Xeon 5270 2.9 GHz "Nehalem" quad-core processors with 48 GB memory/node, 384 cores with 2.3 TB memory in total; all nodes are connected with a 40 Gb/s QDR Infiniband network over a Sun Datacenter 72 port IB switch and have a 1 Gb/s Ethernet connection. The (half-filled) system is rated at about 4.5 TFlops.

central rack:
Three Sun X4540 Fileservers, two with 48 TB and one with 24 TB, one Sun X4600 with 8 AMD 885 2.6 GHz dual-core processors/64 GB and one Sun X4600 M2 with 8 AMD 8360SE 2.5 GHz quad-core processors/256 GB, connected over 40Gb/s Infiniband to the blades.

left rack:
GLORIA and Balloon data processor (IMK-FFB) in its infant state, consisting of two Sun X4200 servers with 4 AMD 2216 2.4 GHz cores/16 GB and 32 GB memory each, a Sun X4275 Server with 16 cores, 36 GB memory, and 15 TB disk space, and a Sun X4540 fileserver with 48 TB disk space, 40Gb/s Infiniband and 10 Gb/s Ethernet interconnection.

behind left rack:
A Sun Blade 8000P with 5 Sun Blade X8420 and 5 Blade X8440, each equipped with 4 AMD 8220/8222 2.8 GHz/3.0 GHz dual-core processors and 64 GB memory, for a total of 80 cores and 640 GB memory.

Above the racks you can see the pipes which supply the cool water to the cooling doors.

The main pumps are configured to work redundantly. Directly behind the pumps the chiller exchanges the lost heat to the main 120 kW  cooling system. The red pot in front of the pumps is an equalizing tank.

 Inside the Constellation System:  The cooling door on the right directly cools the outward airflow of the numerous fans.

Impression of network cable chaos in the central rack: Three CISCO Gigabit Switches interconnected as a stack.

The backyard of the central rack: Three filservers Sun X4540 and two compute nodes Sun X4600. The thick black cables are the Infiniband connections. The thin red threads are fiber optic cables.

Same as above. At the right you see the Knürr cooling door, at the top there are thick gray cables, which lead to the KVM Switch below the monitor.