|
Size: 7195
Comment:
|
← Revision 51 as of 2012-05-17 12:46:47 ⇥
Size: 4495
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = DoC Private Cloud Commodity Hardware Investigations = | = DoC Private Cloud: Compute Servers = |
| Line 3: | Line 3: |
| CSG are investigating the hardware that could be used to realise the DoC cloud. We have identified two basic infrastructure and server requirements; we provide details and cost estimates on this page. The basic idea is that we buy two of everything and then have one notional rack in the Huxley 221 Data Centre containing half the equipment and another notional rack in the ICT Mech Eng Data Centre containing the remaining half. All prices quoted below exclude VAT. |
CSG are investigating the hardware that could be used to realise the DoC cloud. This page describes various commodity compute node options. |
| Line 7: | Line 6: |
| == Infrastructure Requirements == | We want dense compute nodes (a large number of CPUS/cores per cubic volume) so as to make best use of rack-space - power and cooling requirements notwithstanding. We have found that Dell offer the densest server configurations in the form of the PowerEdge C6000 series. We identify two servers in particular: the AMD-based [[http://www.dell.com/uk/enterprise/p/poweredge-c6145/pd|Dell PowerEdge C6145]] and the Intel-based [[http://www.dell.com/uk/business/p/poweredge-c6220/pd|Dell PowerEdge C6220]] |
| Line 9: | Line 8: |
| ||'''Item'''||'''Amount'''||'''Price'''||'''Comments'''|| ||[[http://www.businessdirect.bt.com/products/apc-smart-ups-rt--6000va-4200w-4V4C.html|APC Smart-UPS RT 6000VA RM 230V]]||6||£2200||4200 Watts / 6000 VA UPS|| ||<rowstyle="background-color: #D3D3D3;">ICT LOM switch||2||£1000||(Est.) Remote-management|| ||ICT 10GbE switch||2||£5000||(Est.) Fast intra-cloud comms|| ||<rowstyle="background-color: #D3D3D3;">1U Rack PDU||12||£350||(Est.) Rack power & monitoring|| ||Rittal 46U rack||2||£1000||(Est.) Houses servers & UPSes|| |
We have considered a number of manufacturers for compute nodes including Dell, IBM and HP. HP have no comparable core-dense servers (the best we found had half the cores, half the memory and yet was more expensive than the Dell options below) so have been eliminated. |
| Line 16: | Line 11: |
| '''£31,400 total''' for these infrastructure requirements. Note that the specifications and number of UPS and power distribution units have been specified from the outset to cater for a full rack. This takes into account cloud expansion. | == Dell PowerEdge C6220 == |
| Line 18: | Line 13: |
| == Server requirements == We identify two types of server for the DoC private cloud: a ''compute node'' and a ''storage node''. A compute node contains a large number of CPUs/cores. Its primary role in the cloud is one of computation (virtual machine hosting, distributed computing and the like). A storage node contains a large local (or directly attached) storage capacity. Its primary role in the cloud is to provide storage (for compute nodes and to house distributed file-systems). === Compute Nodes === We have considered a number of manufacturers for compute nodes including IBM, Dell and HP. We want dense compute nodes (a large number of CPUS/cores per cubic volume) so as to make best use of rack-space - power and cooling requirements notwithstanding. We have found that Dell offer the densest server configurations in the form of the PowerEdge C6000 series. We identify two servers in particular: the AMD-based [[http://www.dell.com/uk/enterprise/p/poweredge-c6145/pd|Dell PowerEdge C6145]] and the Intel-based [[http://www.dell.com/uk/business/p/poweredge-c6220/pd|Dell PowerEdge C6220]] ==== PowerEdge C6220 ==== |
|
| Line 43: | Line 30: |
| ==== PowerEdge C6145 ==== | == Dell PowerEdge C6145 == |
| Line 45: | Line 33: |
| four-socket system with its own memory, VGA, 1GbE NIC, USB and LOM port. We have an old quote from Dell of '''£13,700.20''' for the following configuration: | four-socket system with its own memory, VGA, 10GbE NIC, USB and LOM port. We have an old quote from Dell of '''£14,933.90''' for the following configuration: |
| Line 49: | Line 37: |
| . 1 x 1TB SATA 7.2k 2.5in hard-drive per node (two in total in the 2U server). Note: 10GbE NIC *not* included in the above quote: a two-node server would require two such [[http://www.dell.com/us/enterprise/p/broadcom-netxtremeii-57711/pd|cards]] and transceivers, priced at £616.95 + VAT each. Equipping the C6145 accordingly would bring the total to '''£14,933.90'''. |
. 1 x 1TB SATA 7.2k 3.5in hard-drive per node (two in total in the 2U server). |
| Line 58: | Line 44: |
| ==== IBM System x3690 X5 ==== We have a quote on a 4U, 1TB RAM, 40-core (80 threads) server from IBM (courtesy of AMS): '''£36,909.25'''. |
== Dell PowerEdge R820 == We have a more-recent quote on a [[http://www.dell.com/uk/business/p/poweredge-r820/pd?~ck=anav|Dell PowerEdge R820]]. This is a 2U, 128GB RAM, 2U 32-core (64 threads) server costing '''£16,735.29'''. . 4 x eight-core Intel Xeon E5-4650L 2.60GHz CPUs (sixty-four threads with SMT) . 128GB DDR3 @ 1600MHz ECC RAM. . 2 x 300GB 2.5in 10000RPM SAS hard-disks. . 14 x 1TB 2.5in 7200RPM SAS hard-disks. Note that this is a four-way server with CPUs based on the Sandy Bridge architecture. == IBM System x3690 X5 == We also have a quote on an [[http://www-03.ibm.com/systems/uk/x/hardware/enterprise/x3690x5/index.html|IBM System x3690 X5]]. The configuration - courtesy of AMS - is a 4U, 1TB RAM, 40-core (80 threads) server costing '''£36,909.25'''. The department had a loan server of this configuration - again, courtesy of AMS - during February - March 2012. |
| Line 63: | Line 59: |
| . 2 x 146GB 15000RPM SAS hard-disks. | . 2 x 146GB 2.5in 15000RPM SAS hard-disks. |
| Line 65: | Line 61: |
| Note that this server is based on the Westmere-EX architecture: there are no four-way Sandy Bridge CPUs available at the moment - only two-way. | Note that this is a four-way server with CPUs based on the Westmere-EX architecture. |
| Line 67: | Line 63: |
| === Storage Nodes === Under this category, we require servers with many disk-slots. We chose a 2.5in disk form-factor over a 3.5in one since we prefer more spindles rather than raw capacity. In addition, RAID re-build speeds will be faster with smaller-capacity 2.5in disks. The current maximum capacity of Enterprise 2.5in drives, however, is 1TB. |
== General Notes == As explained above, we do not include quotes for HP computer servers because we could not find any models that matched the general requirements (CPU core density, modernity, performance, rack-size and price). Nevertheless, we highlight the following general HP models: |
| Line 70: | Line 66: |
| We considered Dell, Sun/Oracle, HP and IBM. ==== HP ==== [[http://h10010.www1.hp.com/wwpc/uk/en/sm/WF05a/15351-15351-3328412-241475-241475-5177957.html?dnr=1|HP ProLiant DL380p Gen 8 Server]] Quote from ProLinx (ICT's server supplier): '''£10,208.00'''. . 2 x Intel Xeon E5-2650 2.00GHz, eight-core CPUs. . Sixteen 2.5in drive-bay slots (populated with 1TB SAS 7200RPM hard-disks). . 64GB DDR3 @ 1600MHz ECC RAM. ==== Dell ==== [[http://www.dell.com/uk/enterprise/p/poweredge-r720xd/pd|Dell PowerEdge R720xd]] Quote from Dell: '''£7,173.99'''. . 2 x Intel Xeon E5-2640 six-core 2.5Ghz CPUs . Twenty-four 2.5in drive-bay slots (populated with 1TB SAS 7200RPM hard-disks). . Two rear 2.5in drive-bay slots (populated with 146GB SAS 15000RPM hard-disks). . 64GB DDR3 @ 1600MHz ECC RAM. . 2 x 10GbE NICs. ==== IBM ==== [[http://www-03.ibm.com/systems/uk/x/hardware/rack/x3650m4/index.html|IBM System x3650 M4]] Quote from QAssociates: £10,188.70. . 2 x Intel Xeon E5-2640 six-core 2.5Ghz CPUs. . fourteen 2.5in 1TB SAS 7200RPM hard-disks. . two 146GB SAS 15000RPM hard-disks. . 64GB DDR3 @ 1600MHz ECC RAM. . 2 x 10GbE NICs. ==== SUN/ORACLE ==== [[https://shop.oracle.com/pls/ostore/f?p=dstore:product:3661837664325387::NO:RP,6:P6_LPI,P6_PPI:114383329111691951956617|Sun Fire X4270 M3 Server]] No quote; known to be quite expensive. Can be configured like the Dell PowerEdge R720xd. == Summary == . At the moment, Dell seems to offer the best prices for a given specification. . There are other costs involved, such as: . ICT networking. . Power feed organisation and cooling. . Software licenses (if needed). . it is possible to get different compute & storage node configurations for a given price-target. Consider the following targets (inclusive of VAT and the infrastructure cost): . £100K: two C6220s and two R720xds. . £150K (1): four C6220s and two R720xds. . £150K (2): two C6220s and eight R720xds. . £200K: four C6220s and eight R720xds. |
. [[http://h10010.www1.hp.com/wwpc/uk/en/sm/WF25a/15351-15351-3328412-3328422-3328422-4142916.html?dnr=1|HP ProLiant DL580 G7]]: 4U; four Intel Westmere Xeons. . [[http://h10010.www1.hp.com/wwpc/uk/en/sm/WF25a/15351-15351-3328412-3328422-3328422-4194641.html?dnr=1|HP ProLiant DL585 G7]]: 4U; four AMD Interlagos Opteron CPUs. . [[http://h10010.www1.hp.com/wwpc/uk/en/sm/WF25a/15351-15351-3328412-4222584-4222584-4231377.html|HP ProLiant DL980 G7]]: 8U; 10 Intel Westmere Xeons. |
DoC Private Cloud: Compute Servers
CSG are investigating the hardware that could be used to realise the DoC cloud. This page describes various commodity compute node options.
We want dense compute nodes (a large number of CPUS/cores per cubic volume) so as to make best use of rack-space - power and cooling requirements notwithstanding. We have found that Dell offer the densest server configurations in the form of the PowerEdge C6000 series. We identify two servers in particular: the AMD-based Dell PowerEdge C6145 and the Intel-based Dell PowerEdge C6220
We have considered a number of manufacturers for compute nodes including Dell, IBM and HP. HP have no comparable core-dense servers (the best we found had half the cores, half the memory and yet was more expensive than the Dell options below) so have been eliminated.
Dell PowerEdge C6220
This is a 4-node, 2U server, each node comprising a self-contained two-socket system with its own memory, VGA, 10GbE NIC, USB and LOM port. We have a quote from Dell of £19917 for the following configuration:
- 2 x Intel Xeon E5-2670 2.60GHz 8-core/16-thread CPUs per node.
- 128GB RAM per node (512GB RAM total in the 2U server)
- 2 x 1TB SATA 7.2k 2.5in hard-drives per node (eight in total in the 2U server)
The nodes are separate systems housed in the same 2U server chassis. They are not otherwise interconnected.
One of these servers has sixty-four physical CPU cores in total - 128 via hyper-threading. Given 512GB total RAM, this means 8GB of RAM per physical core or 4GB of RAM per 'thread'.
Note that this server contains Intel Sandy Bridge Xeon processors. These support (amongst other things) on-board CPU energy usage monitoring.
Dell PowerEdge C6145
This is a 2-node, 2U server, each node comprising a self-contained four-socket system with its own memory, VGA, 10GbE NIC, USB and LOM port. We have an old quote from Dell of £14,933.90 for the following configuration:
- 4 x AMD Opteron 6282SE 2.6GHz, 16-core CPUs per node.
- 256GB RAM per node (512GB total in the 2U server)
- 1 x 1TB SATA 7.2k 3.5in hard-drive per node (two in total in the 2U server).
The nodes are separate systems housed in the same 2U server chassis. They are not otherwise interconnected.
One of these servers has 128 CPU cores in total. Given 512GB total RAM, this means 4GB of RAM per physical core.
Dell PowerEdge R820
We have a more-recent quote on a Dell PowerEdge R820. This is a 2U, 128GB RAM, 2U 32-core (64 threads) server costing £16,735.29.
- 4 x eight-core Intel Xeon E5-4650L 2.60GHz CPUs (sixty-four threads with SMT)
- 128GB DDR3 @ 1600MHz ECC RAM.
- 2 x 300GB 2.5in 10000RPM SAS hard-disks.
- 14 x 1TB 2.5in 7200RPM SAS hard-disks.
Note that this is a four-way server with CPUs based on the Sandy Bridge architecture.
IBM System x3690 X5
We also have a quote on an IBM System x3690 X5. The configuration - courtesy of AMS - is a 4U, 1TB RAM, 40-core (80 threads) server costing £36,909.25. The department had a loan server of this configuration - again, courtesy of AMS - during February - March 2012.
- 4 x ten-core Intel Xeon E7-8870 2.4Ghz CPUs (eighty threads with SMT)
- 1TB DDR3 @ 1066MHz ECC RAM.
- 2 x 146GB 2.5in 15000RPM SAS hard-disks.
Note that this is a four-way server with CPUs based on the Westmere-EX architecture.
General Notes
As explained above, we do not include quotes for HP computer servers because we could not find any models that matched the general requirements (CPU core density, modernity, performance, rack-size and price). Nevertheless, we highlight the following general HP models:
HP ProLiant DL580 G7: 4U; four Intel Westmere Xeons.
HP ProLiant DL585 G7: 4U; four AMD Interlagos Opteron CPUs.
HP ProLiant DL980 G7: 8U; 10 Intel Westmere Xeons.