DoC Private Cloud

Services

Initially, something like the following services will be needed:

• Virtual-machine hosting / automated provisioning facility (Amazon EC20-a-like).
• Persistent backing-store for VM images (perhaps Amazon S3-style, perhaps Amazon Elastic Block Service-style, perhaps distributed filesystem).
• High-performance POSIX file-store access / scratch areas (perhaps in-cloud or perhaps separate from the cloud).

In relation to scalable storage options, we've already been thinking about these for a while; see also: Storage-NG.

Candidate software to investigate, solving all or part of the private iaas cloud problem, includes:

• OpenStack: claims to do most of the job - VM provisioning, interface with KVM, Xen etc (subproject "Nova") and S3-alike storage system (project "Swift"). Also can integrate to (eg. Ceph, GlusterFS) for filesystem-based distributed storage. Also, Nova has an iSCSI client storage plugin too.

• CloudStack: Citrix open source project, donated to Apache Foundation, rather similar to the above.

• VM seed/runtime image storage:

  • OpenStack Swift distributed object store. (Implements Amazon S3 only)

  • Sheepdog distributed image storage.

• High-performance distributed POSIX file-stores:

  • Ceph distributed object store / block-device / filesystem. DWM evaluated: Conclusion - not ready yet.

  • Gluster distributed filesystem. DWM evaluating. Looking rather promising. No "storage head node", i.e. bottleneck!

  • MooseFS distributed filesystem.

  • Fraunhofer Parallel File System distributed filesystem, no storage head node, BUT NO REPLICATION, pure striping. Discovered by SM. Conclusion: not suitable.

At present, the options for scalable high-performance general purpose NFS filers suitable for storing home dirs, research volumes etc are either expensive or not yet mature. The above POSIX filestores may well be suitable for the "inner problem" of VM storage.

Upgrading existing NFS fileservers to 10Gb is definitely also worth trying (first).

• Virtualization software:

  • Xen paravirtualization tools.

  • KVM (para)-virtualization tools.

  • VMware ESX virtualization software - commercial.

  • libvirt VM abstraction and management layer.

  • Ganeti VM management system.

  • OpenStack Nova VM management system [uses Xen, KVM or VMware under the hood].

The virtual-machine management layer will need to support accounting for resource utilization by the VMs spawned for a given user or group, live migration of VMs from one host to another, and will likely need to support automated backups / snapshots of at least SOME historical virtual-machine disk state. (Note that this differs from existing doctrine, which specifies that the machine-local OS data is expendable, and can be regenerated.)

The use of seed images, data de-duplication, and/or copy-on-write would also be valuable for minimising storage requirements.

Background

Sometime in early 2012, Susan told DCW that DoC were thinking of hiring someone for 6 months into CSG, specifically tasked with building a DoC private cloud. Essentially she said that Exec Committee has found some significant pot of money which needs to be spent this financial year.

She explained the core idea was "virtualisation even for research clusters": at present, research groups buy clusters when they have money, CSG set them up, install "linux du jour" on them, configure fileservers (if part of cluster), tape backups (if part), processing node special software etc.

Then the servers age, the OS is essentially frozen (it's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes). They become "fragile". Sometimes it's hard to even retire them on schedule (4/5/6 years or whatever). Also these clusters are often only accessible by members of that research group so the resource may not be fully utilised.

Susan's vision: setup a private cloud, researchers add hardware to that cloud's core resources, then create a VM for each cluster node, perhaps tied (1-1 at first) to their own hardware, CSG install that virtual cluster node's OS, researchers work as before - but each node is encapsulated inside a VM. Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased.

Various discussions with PJM and AON followed. Project will definitely proceed. In two stages:

1. a 6 month phase to build a prototype cloud, recruiting a "Cloud Manager" person to join CSG, either temporary or permanent. The Department will spend some significant amount of money, perhaps in the £100-200K range.
2. assuming the prototype cloud is successful, it will move into production and the "Cloud Manager" become permanent. Researchers then have the option of adding hardware to the cloud. All members of CSG will become skilled in cloud-related topics, and the "Cloud Manager" will do non-cloud-related problem solving too.

Most crucially: (despite not knowing the exact spec, services to provide, let alone how to implement them) we therefore need to purchase all the kit having it delivered in July 2012, before the Olympics. PJM added "build a private cloud like Amazon EC2 does", AON suggested a budget of £100K, £150K or even £200K - we will provide possible plans for these price levels.

DWM has spent a lot of time evaluating Ceph as a possible S3/Elastic Block Store like storage system for supporting VM storage and possibly very high speed filesystems eg. staging areas for VM data (scaleout NAS with replication). So far: it's not there yet, at least as a fast POSIX filesystem. Alternatives need to be looked at as well..

Staff Working Group meetings

Staff Working Group Meeting 1 - April 3rd 2012

Staff Working Group/Open Meeting 2 - April 25th 2012