DoC Private Cloud Commodity Hardware Investigations

CSG are investigating the hardware that could be used to realise the DoC cloud. We have identified two basic infrastructure and server requirements; we provide details and cost estimates on this page. The basic idea is that we buy two of everything and then have one notional rack in the Huxley 221 Data Centre containing half the equipment and another notional rack in the ICT Mech Eng Data Centre containing the remaining half. All prices quoted below exclude VAT.

Infrastructure Requirements

£31,400 total for these infrastructure requirements. Note that the specifications and number of UPS and power distribution units have been selected from the outset to cater for a full rack. This takes into account cloud expansion.

Server requirements

We identify two types of server for the DoC private cloud: a compute node and a storage node. A compute node contains a large number of CPUs/cores. Its primary role in the cloud is one of computation (virtual machine hosting, distributed computing and the like). A storage node contains a large local (or directly attached) storage capacity. Its primary role in the cloud is to provide storage (for compute nodes and to house distributed file-systems).

Compute Nodes

We have considered a number of manufacturers for compute nodes including IBM, Dell and HP. We want dense compute nodes (a large number of CPUS/cores per cubic volume) so as to make best use of rack-space - power and cooling requirements notwithstanding. We have found that Dell offer the densest server configurations in the form of the PowerEdge C6000 series. We identify two servers in particular: the AMD-based Dell PowerEdge C6145 and the Intel-based Dell PowerEdge C6220

PowerEdge C6220

This is a 4-node, 2U server, each node comprising a self-contained two-socket system with its own memory, VGA, 10GbE NIC, USB and LOM port. We have a quote from Dell of £19917 for the following configuration:

• 2 x Intel Xeon E5-2670 2.60GHz 8-core/16-thread CPUs per node.
• 128GB RAM per node (512GB RAM total in the 2U server)
• 2 x 1TB SATA 7.2k 2.5in hard-drives per node (eight in total in the 2U server)

The nodes are separate systems housed in the same 2U server chassis. They are not otherwise interconnected.

One of these servers has sixty-four physical CPU cores in total - 128 via hyper-threading. Given 512GB total RAM, this means 8GB of RAM per physical core or 4GB of RAM per 'thread'.

Note that this server contains Intel Sandy Bridge Xeon processors. These support (amongst other things) on-board CPU energy usage monitoring.

PowerEdge C6145

This is a 2-node, 2U server, each node comprising a self-contained four-socket system with its own memory, VGA, 1GbE NIC, USB and LOM port. We have an old quote from Dell of £13,700.20 for the following configuration:

• 4 x AMD Opteron 6282SE 2.6GHz, 16-core CPUs per node.
• 256GB RAM per node (512GB total in the 2U server)
• 1 x 1TB SATA 7.2k 2.5in hard-drive per node (two in total in the 2U server).

Note: 10GbE NIC *not* included in the above quote: a two-node server would require two such cards and transceivers, priced at £616.95 + VAT each. Equipping the C6145 accordingly would bring the total to £14,933.90.

The nodes are separate systems housed in the same 2U server chassis. They are not otherwise interconnected.

One of these servers has 128 CPU cores in total. Given 512GB total RAM, this means 4GB of RAM per physical core.

IBM System x3690 X5

We have a quote on a 4U, 1TB RAM, 40-core (80 threads) server from IBM (courtesy of AMS): £36,909.25.

• 4 x ten-core Intel Xeon E7-8870 2.4Ghz CPUs (eighty threads with SMT)
• 1TB DDR3 @ 1066MHz ECC RAM.
• 2 x 146GB 15000RPM SAS hard-disks.

Note that this server is based on the Westmere-EX architecture: there are no four-way Sandy Bridge CPUs available at the moment - only two-way.

Storage Nodes

Under this category, we require servers with many disk-slots. We chose a 2.5in disk form-factor over a 3.5in one since we prefer more spindles rather than raw capacity. In addition, RAID re-build speeds will be faster with smaller-capacity 2.5in disks. The current maximum capacity of Enterprise 2.5in drives, however, is 1TB.

We considered Dell, Sun/Oracle, HP and IBM.

HP

HP ProLiant DL380p Gen 8 Server Quote from ProLinx (ICT's server supplier): £10,208.00.

• 2 x Intel Xeon E5-2650 2.00GHz, eight-core CPUs.
• Sixteen 2.5in drive-bay slots (populated with 1TB SAS 7200RPM hard-disks).
• 64GB DDR3 @ 1600MHz ECC RAM.

Dell

Dell PowerEdge R720xd Quote from Dell: £7,173.99.

• 2 x Intel Xeon E5-2640 six-core 2.5Ghz CPUs
• Twenty-four 2.5in drive-bay slots (populated with 1TB SAS 7200RPM hard-disks).
• Two rear 2.5in drive-bay slots (populated with 146GB SAS 15000RPM hard-disks).
• 64GB DDR3 @ 1600MHz ECC RAM.
• 2 x 10GbE NICs.

Alternatively, it costs £5,965.00 for the same configuration with twelve 3.5in drive-bay slots populated with 2TB SAS 7200RPM hard-disks.

IBM

IBM System x3650 M4 Quote from QAssociates: £10,188.70.

• 2 x Intel Xeon E5-2640 six-core 2.5Ghz CPUs.
• fourteen 2.5in 1TB SAS 7200RPM hard-disks.
• two 146GB SAS 15000RPM hard-disks.
• 64GB DDR3 @ 1600MHz ECC RAM.
• 2 x 10GbE NICs.

SUN/ORACLE

Sun Fire X4270 M3 Server No quote; known to be quite expensive. Can be configured like the Dell PowerEdge R720xd.

Summary

• At the moment, Dell seems to offer the best prices for a given specification.
• There are other costs involved, such as:
  • ICT networking.
  • Power feed organisation and cooling.
  • Software licenses (if needed).

• it is possible to get different compute & storage node configurations for a given price-target. Consider the following targets (inclusive of VAT and the infrastructure cost):

  • £100K: two C6220s and two R720xds.
  • £150K (1): four C6220s and two R720xds.
  • £150K (2): two C6220s and eight R720xds.
  • £200K: four C6220s and eight R720xds.