|
Size: 10340
Comment:
|
Size: 7062
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Wiki page for notes on Jan-April 2012 DoC private cloud discussions = | = DoC Private Cloud = |
| Line 3: | Line 3: |
| == Intro == | == Project Goal == |
| Line 5: | Line 5: |
| Sometime in early 2012, Susan told DCW that DoC were thinking of hiring someone (Jeremy Cohen) for 6 months into CSG, specifically tasked with building a DoC private cloud [definition unclear]. Essentially, Exec Committee has found some money and needs to spend it quick! |
Build a DoC '''Infrastructure-as-a-service private cloud, very like Amazon EC2''' ("Elastic Compute Service") which presents a ''secure and convenient web interface'' which enables users of DoC to ''specify and create VMs and associated storage, automatically install OSes on them and deploy them''. |
| Line 10: | Line 8: |
| She explained the core idea was "virtualisation even for research clusters": at present, research groups buy clusters when they have money, CSG set them up, install "linux du jour" on them, configure fileservers (if part of cluster), tape backups (if part), processing node special software etc. |
The main goal is to virtualize most research servers, decoupling the OS image from the hardware for greater flexibility. Sharing (amortizing) the costs of each machine. One driver of this is EPSRC deciding to only provide 50% of any hardware bid over £10K in future, with the Dept expected to pay the remaining 50%. |
| Line 15: | Line 12: |
| Then the servers age, the OS is essentially frozen (it's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes). They become "fragile". Sometimes it's hard to even retire them on schedule (4/5/6 years or whatever). Also these clusters are often only accessible by members of that research group so the resource may not be fully utilised. |
This project will definitely proceed, having been approved by Executive Committee and by two open meetings of Academic staff. |
| Line 22: | Line 14: |
| Susan's vision: setup a private cloud, researchers add hardware to that | Peter McBrien (PJM) is leading the project, and has laid out two stages: 1. a 6 month phase in which CSG (advised by an academic working group) will design and build a prototype cloud, recruiting a "Cloud Manager" person to join CSG, possibly for 6 months in the first instance. The Department will spend some significant amount of money to build the prototype cloud, perhaps in the £100-200K range. 2. assuming the prototype cloud is successful, it will move into production and the "Cloud Manager" become permanent. Researchers would then be encouraged to add research-funded hardware to the cloud and given some form of preferential treatment on "their hardware". All members of CSG are enthusiastic to gain cloud-related skills from the "Cloud Manager". (By the way, the Cloud Manager will do non-cloud-related systems administration too). Most crucially: (despite not knowing the exact set of services to provide, let alone how to implement them, or having yet appointed a Cloud Manager person) the Department's substantial initial investment '''must be fully spent before the end of July'''. This means that all kit must be ordered, delivered and paid for before the 31st of July. With the Olympics making deliveries difficult, this means that everything must have been ordered by 1st July. Re: amount to spend, Anne O'Neill (AON) and Peter McBrien (PJM) suggested CSG prepare possible collections of commodity hardware, on the assumptions that either £100K, £150K and £200K (inc vat) would be spent. == The Problem We're Trying to Solve == At present, research groups buy clusters when they have money, CSG set them up, install the current supported Linux or Windows release on them (the CSG supported Linux release currently changes each year), optionally configuring storage and fileserver nodes, arranging tape backups of important data, adding special software etc. Important note: CSG do '''not currently backup physical server disk images''', instead we attempt to determine from the owners what data is IMPORTANT and back that up instead. Then the servers age, after the first year the OS becomes essentially frozen apart from minor security updates. It's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes. They become "fragile", and eventually a security risk. Sometimes it's hard to retire them when the hardware becomes more than 4-5 years old, because of the "fragile" software setup on them. A second problem is that these clusters are often only accessible by members of the specific research group that bought them, so the resource may not be fully utilised. Instead, the idea is to setup a private cloud, researchers add hardware to that |
| Line 24: | Line 52: |
| tied (1-1 at first) to their own hardware, CSG install that virtual cluster node's OS, researchers work as before - but each node is encapsulated inside a VM. Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased. |
tied (1-1 at first) to their own hardware, the creation process should automatically install a CSG-supported operating system (historically supported Linuxes and Windows versions) on the new VM. Researchers work as before on each VM - but each node is encapsulated inside a VM. |
| Line 29: | Line 57: |
| Suppose, for instance, the group needed N nodes x 100% of underlying VM host x M months [and then less thereafter]. |
Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased and the VM migrated to it. |
| Line 32: | Line 60: |
| Susan also added "and it should just scale without limits, manage itself magically", which is less realistic:-) Saving RAs significant informal sysadmin time is a goal. |
We would also gain to flexibility to create short-term VMs for specific "run this software on 16 nodes" experiments. Such short-term VMs would then be automatically destroyed. |
| Line 36: | Line 63: |
| Various discussions with PJM and AON followed, Jeremy decided not to accept the job, DoC still wants to hire a "Cloud Manager" as part of CSG. Most crucially: the Dept decided it has money now, not next year, and that (despite not knowing the exact spec, services to provide, let alone how to implement them) we therefore needed to purchase all the kit having it delivered in July 2012, before the Olympics. PJM added "build a private cloud like Amazon EC2 does", AON suggested a budget of £100K, £150K or even £200K - we are tasked with providing possible plans for these price levels. |
We could even give every DoC user (students and staff!) their very own VM when they join, with full root/admin access - or at least the ability to create one when they first need it (lazy evaluation:-)). |
| Line 46: | Line 67: |
| DWM has spent a lot of time evaluating Ceph as a possible S3/Elastic Block Store like storage system for supporting VM storage and possibly very high speed filesystems eg. staging areas for VM data (scaleout NAS with replication). So far: it's not there yet. Alternatives need to be looked at as well.. |
== Open Staff cloud meetings == |
| Line 52: | Line 69: |
| == Working Group: 3rd April 2012 meeting == | In April 2012, the discussion was opened out to all interested staff, and (so far) two open staff cloud meetings have been held. Here are some notes taken by DCW and LDK of the discussions at both meetings. |
| Line 54: | Line 72: |
| A working group of academics has been set up, this met on 3rd April 2012 for the first time. Things discussed: |
[[internal/project/privatecloud/meeting-2012-04-03|Open Staff Meeting 1 - April 3rd 2012]] |
| Line 57: | Line 74: |
| - PJM/Susan: background (spend money now, define services later), acknowledged unusual approach.. added (PJM) idea that a group can have a VM per project per year if they need, so they build new apps on the latest supported OS, while maintaining the ability to run their old versions on the older OS, allows people to try old code on new OS releases without "big bang" server upgrade problems. old VMs can eventually wither away.. |
[[internal/project/privatecloud/meeting-2012-04-25|Open Staff Meeting 2 - April 25th 2012]] |
| Line 64: | Line 76: |
| - PJM: start with concept of: every student gets a VM as they walk in through the door, keep while at College, have root access [need to fix/avoid NFS problem]. users should have the ability to create more VMs programmatically, both short term and long term ones. |
== Cloud Services and Software Investigations == |
| Line 69: | Line 78: |
| - PJM: also, are we all agreed: it's got to be a realiable production system. | CSG have been performing initial investigations of hardware to buy, and possible software that might help with the project. |
| Line 71: | Line 81: |
| - JAMM: use cases - projects into cloud technologies, pervasive computing exercises could be made more flexible [not sure how], some of her research involves streaming data from sensors, need high-capacity filestores. |
== Cloud Hardware == |
| Line 75: | Line 83: |
| - PRP: EPSRC call "every research grant puts in for a small cluster" by the name "vanity clusters". EPSRC favouring shared resources (Dept, College, federated) - will allocate at most first £10K of equipment, then excess must have matching funds from Dept! favours (for example) shared services, grids, clouds and HPC. |
Susan expressed a serious preference for open source software running on commodity hardware. A strong advantage of this approach is that commodity hardware could be repurposed, even if the cloud project failed completely, or got scaled down to use a tiny proportion of the hardware. |
| Line 81: | Line 87: |
| - PRP added: VMs can really speed up provisioning of research project kit, instead of purchasing kit, waiting for it to arrive, installing and configuring it, continuing to maintain it, then (after project) decide what to do with it, can create 16 short term VMs bound to suitable hardware very quickly, do quick experiments and release the VMs resources. If spare hardware capacity is in hand, of course! Like Julie, Peter added that research into cloud and distributed systems performance could be improved if we had a cloud which we could monitor and tweak. |
Peter Pietzuch strongly recommended that we also investigate commercial scalable storage systems - specifically NetApp "scalable NAS" units - which his colleagues at the University of Cambridge Computing Laboratory have used for years. CSG have discussed NetApps with colleagues in ICT and Martyn Johnson at the Cambridge Computing Lab, are investigating these and attempting to get possible prices for (perhaps) suitable models. We don't yet have any documentation written to cover NetApp investigations because we haven't got a single quote or ballpark estimate of the price yet! |
| Line 90: | Line 95: |
| - JD: 2 important aspects of cloud here: 1. easily provisioned VMs; 2. amortization of all resources over multiple projects. The latter requires that researchers don't require all of their "own" resources "all" of the time - otherwise none spare! |
Lloyd Kamara (LDK) has done a lot of work on getting quotes for possible Compute and Storage nodes (conventional Dell/HP/IBM rack mount servers with, respectively, as many cores and as many disks as possible per unit cost. His results are here: |
| Line 95: | Line 99: |
| - PJM/Susan: the matching funds model allows Dept to demand up to 50% of these shared resources [on average over time, perhaps front-loaded so "owners" get the majority of time up front, release nearly all resources later for general use]. |
[[internal/project/privatecloud/hardware|Commodity Hardware Options]] |
| Line 100: | Line 101: |
| - CCADAR: will sometimes need exclusive access to all "your" cluster VMs on all your hardware for experiments - repeatability is especially important. => need ability to pin VMs onto particular classes of node. |
== Software Investigations == |
| Line 104: | Line 103: |
| - PRP: Yes, and sometime experiments need to happen directly on the bare metal. but only a small minority! |
CSG have been familiarising themselves with various possible open source cloud or storage software systems that might be able to implement some/all of the required iaas cloud services, and performing some initial investigations of a few of them. While the Cloud Manager will of course be responsible for designing and building the cloud, existing members of CSG are concerned to '''map the terrain''' to find out where the dragons are lurking and to provide an '''existence proof''' to reduce the risk that after buying the hardware, no software can be added to build the desired cloud. |
| Line 107: | Line 110: |
| - JAMM: performance monitoring very important. | Here are our notes: |
| Line 109: | Line 112: |
| - WJK: yes, including power monitoring of the physical VM hosts, a la picards. very useful. - GCASALE: agreed, subtle point about frequency of monitoring being very different between cheap power mon and expensive power mon.. LDK discussing with him. - SUSAN: Maja had mentioned that she makes a very large amount of use of Matlab, on Windows clusters, buying extra parallel licenses etc. PJM: why not use College standard license? DCW: believe extra modules and parallel licenses not included in College Matlab license. DCW added: Note that ICT HPC kit doesn't support Matlab for same reason! - TORA: Lab are very interested in more continuous autotesting, need a better sandbox: like a short term VM to run student code in! Also very interested in scalable storage (didn't say why?) - JD/SUSAN discussed: where are other Computing Depts with clouds? at any level (Dept, College, federated?) - answer seems to be: none known in production. - DWM added that LESC had done lots of "cloud v1" - grid - related work, and mentioned the similarities between grids, private clouds and HPC. - PRP said that we should make more use of ICT's HPC, as we're paying for it. Susan said: some use (PHJK, Kanwal), have found HPC team not very welcoming to DoC, sniffy about Java code. DCW said: yes, real programmers in HPC:-) DCW added: lots of money still going in though - let's use it. ICT also upgrading to VMware ESX 5, which "supports cloud" (buzzword alert). DCW added: HPC doesn't even let you access College home dirs cos they're "not fast enough". - PJM asked re: this - does everyone want DoC home dirs and research volumes accessible from VMs? everyone agreed, and several people pointed out that existing fileservers can be saturated by Condor so need to scale more. => cloud storage needs to hold VM images and (some) scalable filesystem data too. not clear how much. - DCW asked: what about Amazon S3 - simple distributed (key,value) storage system - do we need that? some people said "might be useful" but noone had a solid use case. - WJK added that he'd love to do experiments using different speed storage eg. flash and raid levels. - TORA added that a large scalable block storage system would be very useful, but neglected to say why. - DWM said there seems to be a need for scalable storage at some level as part of the cloud, there are a variety of technologies - open source and commercial - to look at. - PJM channeled PRP in saying that "commercial filers" should be looked into, think he meants NetApp/EMC stuff. Susan said DoC prefers open source if possible, PRP added that cloud storage is NetApp's bread and butter and their support and scalability was really good. DCW: look at. - SUSAN reported that DR had initially said - CSG do everything his group needs, why need a cloud. However, when she asked him - want more scalable storage, his eyes lit up! - DWM: so we conclude that scalable storage is very important? - GCASALE asked: what type of cloud? private? DCW/PJM: yes. what about cloudbursting, Giuliano asked ? [what's that we said] - upload VMs to Amazon after development (or when need short term resources). PJM: useful if possible. - "tall chap in green shirt": what about network bandwidth? 10Gb links? may also need bandwidth reservation in switch fabric. DWM: talking with ICT networking about 10Gb. - "natasha's phd student in her place": their group are very interested in virtualizing algorithms and still using FGPAs and GPUs, and again more scalable storage is needed here. - WL agrees, saying some VM hosts definitely need to have GPUs and FPGAs (he can provide details and costs). He added that he'd be very interested in "getting under the hood" and tweaking and monitoring how various aspects of the cloud operate. PJM said: may be contrary to production cloud - but perhaps a "sandpit cloud" could fork off the main cloud on occasion, grab some hardware etc. WJK agreed. DCW added that Amazon EC2 had VMs with access to GPUs and FPGAs etc in their pricing model. - PJM talked about a cost accounting model, enforcing 50% maximum usage, WJK wondered whether anything that heavy was needed. (god knows how that's even implemented! perhaps logging use for post-analysis). - JD asked: would we give access to people outside of DoC? DCW: no. PJM: might be open to sharing with ICT. JD: power of clouds - federating. |
[[internal/project/privatecloud/investigations|Software Investigations]] |
DoC Private Cloud
Project Goal
Build a DoC Infrastructure-as-a-service private cloud, very like Amazon EC2 ("Elastic Compute Service") which presents a secure and convenient web interface which enables users of DoC to specify and create VMs and associated storage, automatically install OSes on them and deploy them.
The main goal is to virtualize most research servers, decoupling the OS image from the hardware for greater flexibility. Sharing (amortizing) the costs of each machine. One driver of this is EPSRC deciding to only provide 50% of any hardware bid over £10K in future, with the Dept expected to pay the remaining 50%.
This project will definitely proceed, having been approved by Executive Committee and by two open meetings of Academic staff.
Peter McBrien (PJM) is leading the project, and has laid out two stages:
- a 6 month phase in which CSG (advised by an academic working group) will design and build a prototype cloud, recruiting a "Cloud Manager" person to join CSG, possibly for 6 months in the first instance. The Department will spend some significant amount of money to build the prototype cloud, perhaps in the £100-200K range.
- assuming the prototype cloud is successful, it will move into production and the "Cloud Manager" become permanent. Researchers would then be encouraged to add research-funded hardware to the cloud and given some form of preferential treatment on "their hardware". All members of CSG are enthusiastic to gain cloud-related skills from the "Cloud Manager". (By the way, the Cloud Manager will do non-cloud-related systems administration too).
Most crucially: (despite not knowing the exact set of services to provide, let alone how to implement them, or having yet appointed a Cloud Manager person) the Department's substantial initial investment must be fully spent before the end of July. This means that all kit must be ordered, delivered and paid for before the 31st of July. With the Olympics making deliveries difficult, this means that everything must have been ordered by 1st July.
Re: amount to spend, Anne O'Neill (AON) and Peter McBrien (PJM) suggested CSG prepare possible collections of commodity hardware, on the assumptions that either £100K, £150K and £200K (inc vat) would be spent.
The Problem We're Trying to Solve
At present, research groups buy clusters when they have money, CSG set them up, install the current supported Linux or Windows release on them (the CSG supported Linux release currently changes each year), optionally configuring storage and fileserver nodes, arranging tape backups of important data, adding special software etc.
Important note: CSG do not currently backup physical server disk images, instead we attempt to determine from the owners what data is IMPORTANT and back that up instead.
Then the servers age, after the first year the OS becomes essentially frozen apart from minor security updates. It's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes. They become "fragile", and eventually a security risk.
Sometimes it's hard to retire them when the hardware becomes more than 4-5 years old, because of the "fragile" software setup on them.
A second problem is that these clusters are often only accessible by members of the specific research group that bought them, so the resource may not be fully utilised.
Instead, the idea is to setup a private cloud, researchers add hardware to that cloud's core resources, then create a VM for each cluster node, perhaps tied (1-1 at first) to their own hardware, the creation process should automatically install a CSG-supported operating system (historically supported Linuxes and Windows versions) on the new VM. Researchers work as before on each VM - but each node is encapsulated inside a VM.
Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased and the VM migrated to it.
We would also gain to flexibility to create short-term VMs for specific "run this software on 16 nodes" experiments. Such short-term VMs would then be automatically destroyed.
We could even give every DoC user (students and staff!) their very own VM when they join, with full root/admin access - or at least the ability to create one when they first need it (lazy evaluation:-)).
Open Staff cloud meetings
In April 2012, the discussion was opened out to all interested staff, and (so far) two open staff cloud meetings have been held. Here are some notes taken by DCW and LDK of the discussions at both meetings.
Open Staff Meeting 1 - April 3rd 2012
Open Staff Meeting 2 - April 25th 2012
Cloud Services and Software Investigations
CSG have been performing initial investigations of hardware to buy, and possible software that might help with the project.
Cloud Hardware
Susan expressed a serious preference for open source software running on commodity hardware. A strong advantage of this approach is that commodity hardware could be repurposed, even if the cloud project failed completely, or got scaled down to use a tiny proportion of the hardware.
Peter Pietzuch strongly recommended that we also investigate commercial scalable storage systems - specifically NetApp "scalable NAS" units - which his colleagues at the University of Cambridge Computing Laboratory have used for years. CSG have discussed NetApps with colleagues in ICT and Martyn Johnson at the Cambridge Computing Lab, are investigating these and attempting to get possible prices for (perhaps) suitable models. We don't yet have any documentation written to cover NetApp investigations because we haven't got a single quote or ballpark estimate of the price yet!
Lloyd Kamara (LDK) has done a lot of work on getting quotes for possible Compute and Storage nodes (conventional Dell/HP/IBM rack mount servers with, respectively, as many cores and as many disks as possible per unit cost. His results are here:
Software Investigations
CSG have been familiarising themselves with various possible open source cloud or storage software systems that might be able to implement some/all of the required iaas cloud services, and performing some initial investigations of a few of them. While the Cloud Manager will of course be responsible for designing and building the cloud, existing members of CSG are concerned to map the terrain to find out where the dragons are lurking and to provide an existence proof to reduce the risk that after buying the hardware, no software can be added to build the desired cloud.
Here are our notes: