DoC Private Cloud: Investigations into possible services and possible software

Services

We believe that something like the following services will be needed to provide the DoC iaas private cloud:

• Basic Virtualization:- conventional VM system hosting VMs on one or more compute servers.

• Automated VM provisioning facility:- like Amazon EC2, a web interface to allow DoC users to specify and create VMs, install OSes on them and deploy them.

• VM Image Storage:- Persistent backing-store for VM images (perhaps Amazon S3-style, perhaps Amazon Elastic Block Service-style, perhaps distributed filesystem).

• Scalable NFS/CIFS fileservers:- Existing fileservers can already be maxed out by Condor jobs on 200 Lab PCs all hitting /vol/vipdata at Gig speed. Much faster, high-performance, ideally much more scalable conventional POSIX fileservers (or collections of fileservers running some type of distributed filesystems) are clearly needed even for Condor, let alone for hundreds of cloud VMs too.

Note that the scalable NFS/CIFS fileservers could be considered part of the cloud storage system - if for example a single storage system provided (say) distributed or high-performance POSIX fileshares AND VM image storage - or could be considered entirely separate from the cloud storage - which then only needs to provide VM image storage.

In relation to scalable storage options, we've already been thinking about these for a while; see also: Storage-NG.

Candidate software to investigate

Possible open source software which may solve all or part of the private iaas cloud problem includes:

Whole System

• OpenStack: claims to do most of the job - VM provisioning, interface with KVM, Xen, VMware etc (subproject "Nova") and S3-alike storage system (project "Swift"). Also can integrate with distirbuted file storage systems (eg. Ceph, GlusterFS) for filesystem-based distributed storage. Also, Nova has an iSCSI client storage plugin too ("nova-volume").

• CloudStack: Citrix open source project, donated to Apache Foundation, rather similar to the above.

Scalable Storage

• VM seed/runtime image storage:

  • OpenStack Swift distributed object store. (Implements Amazon S3 like service, distributed object (key,value) store)

  • Sheepdog distributed image storage.

• High-performance distributed POSIX file-stores:

  • Ceph distributed object store / block-device / filesystem. DWM evaluated: Conclusion - not ready yet.

  • Gluster distributed filesystem. DWM evaluating. Looking rather promising. No "storage head node", i.e. bottleneck!

  • MooseFS distributed filesystem.

  • Fraunhofer Parallel File System distributed filesystem, no storage head node, BUT NO REPLICATION, pure striping. Discovered by SM. Conclusion: not suitable.

At present, the options for scalable high-performance general purpose NFS filers suitable for storing home dirs, research volumes etc are either expensive or not yet mature. The above POSIX filestores may well be suitable for the "inner problem" of VM storage.

Upgrading existing NFS fileservers to 10Gb is definitely also worth trying (first).

Virtualization software

• Xen paravirtualization tools.

• KVM (para)-virtualization tools.

• VMware ESX virtualization software - commercial.

• libvirt VM abstraction and management layer.

• Ganeti VM management system.

• OpenStack Nova VM management system [uses Xen, KVM or VMware under the hood].

The virtual-machine management layer will need to support accounting for resource utilization by the VMs spawned for a given user or group, live migration of VMs from one host to another, and will likely need to support automated backups / snapshots of at least SOME historical virtual-machine disk state. (Note that this differs from existing doctrine, which specifies that the machine-local OS data is expendable, and can be regenerated.)

The use of seed images, data de-duplication, and/or copy-on-write would also be valuable for minimising storage requirements.