Summer 2012: DoC Private Cloud Hardware Investigations

During the summer of 2012, CSG investigated the hardware that could be used to realise the DoC cloud in a scalable and expandable way, before purchasing agreed cloud hardware in August 2012. These notes are largely historic, and describe our summer 2012 investigations.

Susan expressed a preference for open source software running on commodity hardware. A strong advantage of this approach is that commodity hardware could be repurposed, even if the cloud project failed completely, or got scaled down to use a tiny proportion of the hardware.

However, Peter Pietzuch recommended that we also investigate commercial scalable storage systems - specifically NetApp "scalable NAS" units - which his colleagues at the University of Cambridge Computing Laboratory have used for years. CSG have discussed NetApps with colleagues in ICT and Martyn Johnson at the Cambridge Computing Lab, and have now got a realistic price for a NetApp model that we imagine is suitable (see below).

Looking at commodity hardware first, we identify two types of server for the DoC private cloud: a compute node and a storage node:

We envisage that multiple compute nodes and multiple storage nodes would be needed. The question of what software to run to "glue together" several storage nodes into some form of distributed multi-replica object store (aka Amazon S3) or distributed POSIX filesystem remains unclear.

Alternatively, if we go for a NetApp storage system, we would still need the distinct compute nodes, but a single NetApp system (or ideally two, one per machine room) would (apparently) provide all the storage, performance and redundancy.

In addition, one should not overlook infrastructure (eg. racks to put the servers in, Uninterruptible Power Supplies to power the servers etc!):

Infrastructure Requirements

The basic idea is that we buy two of everything and then have one rack in the Huxley 221 Data Centre containing half the equipment and the other rack in the ICT Mech Eng Data Centre containing the remaining half. All prices quoted below exclude VAT.





APC Smart-UPS RT 6000VA RM 230V



4200 Watts / 6000 VA UPS

ICT LOM switch



(Est.) Remote-management

ICT 10GbE switch



(Est.) Fast intra-cloud comms

1U Rack PDU



(Est.) Rack power & monitoring

Rittal 46U rack



(Est.) Houses servers & UPSes

£31,400 total for these infrastructure requirements. Note that the specifications and number of UPS and power distribution units have been selected from the outset to cater for a full rack. This takes into account cloud expansion.

Compute Nodes

Commodity Intel architecture rack mount servers with the highest possible core density (and lowest price per core). These will host the VMs and provide the raw computing capability. Lloyd has done a lot of work getting quotes for commodity compute nodes, shown below:

Commodity Compute Node Details

Storage Nodes: Commodity Intel Architecture

Commodity Intel architecture rack mount servers with the highest possible storage density, ideally with the maximum number of disk spindles possible for performance. These will host the VM images, any associated data, possibly knitted together using some distributed multi-replica filesystem for performance and fault tolerance.

Lloyd has done a lot of work getting quotes for commodity compute nodes, shown below:

Commodity Storage Node Details

Storage: Commercial NetApp Network Attached Storage

Giuseppe and Duncan have done a lot of work investigating NetApp storage systems and here are the details:

NetApp Scalable NAS Details


project/privatecloud/hardware (last edited 2013-10-22 12:02:22 by dcw)