|
Size: 13907
Comment:
|
Size: 7812
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = DoC Private Cloud = | ## page was renamed from internal/project/privatecloud = DoC Private Cloud: 2012 - 2013 = |
| Line 3: | Line 4: |
| == Services == | == Project Goal == |
| Line 5: | Line 6: |
| Initially, the following services will be needed: | In early summer 2012, CSG were tasked with building a DoC '''Infrastructure-as-a-service private cloud, very like Amazon EC2''' ("Elastic Compute Service") which presents a ''secure and convenient web interface'' which enables users of DoC to ''specify and create VMs and associated storage, automatically install OSes on them and deploy them''. |
| Line 7: | Line 9: |
| * Virtual-machine hosting / automated provisioning facility. * Persistent backing-store for VM images. * High-performance POSIX file-store access / scratch areas. |
The main goal is to virtualize most research servers, decoupling the OS image from the hardware for greater flexibility. Sharing (amortizing) the costs of each machine. One driver of this is EPSRC deciding to only provide 50% of any hardware bid over £10K in future, with the Dept expected to pay the remaining 50%. |
| Line 11: | Line 13: |
| Candidate software tooling includes: | This project was approved by Executive Committee and by two open meetings of Academic staff. Peter McBrien (PJM) led the project, and laid out two stages: |
| Line 13: | Line 15: |
| * Storage: * [[http://ceph.newdream.net/|Ceph]] distributed object store / block-device / filesystem. * [[http://openstack.org/projects/storage/|OpenStack Swift]] distributed object store. * [[http://www.osrg.net/sheepdog/|Sheepdog]] distributed image storage. * [[http://www.gluster.org/|Gluster]] distributed filesystem. |
1. a 6 month phase in which CSG (advised by an academic working group) will design and build a prototype cloud, recruiting a "Cloud Manager" person to join CSG, possibly for 6 months in the first instance. The Department will spend some significant amount of money to build the prototype cloud, perhaps in the £100-200K range. |
| Line 19: | Line 17: |
| We've been thinking along these lines for a while; see also: [[internal/project/Storage-NG|Storage-NG]]. | 2. assuming the prototype cloud is successful, it will move into production and the "Cloud Manager" become permanent. Researchers would then be encouraged to add research-funded hardware to the cloud and given some form of preferential treatment on "their hardware". All members of CSG are enthusiastic to gain cloud-related skills from the "Cloud Manager". |
| Line 21: | Line 19: |
| Upgrades to the network connections for existing NFS filers may also be warranted. | Most crucially: the Department's substantial initial investment '''had to be fully spent before the end of July 2012'''. This means that all kit was ordered, delivered and paid for before the 31st July 2012. Nearly £300K inc vat was spent on the project. The Cloud Manager, Thomas Joseph, was appointed about a year later - in July 2013. |
| Line 23: | Line 21: |
| * Virtual-machines: * [[http://www.xen.org/|Xen]] paravirtualization tools. * [[http://www.linux-kvm.org/|KVM]] (para)-virtualization tools. * [[http://libvirt.org/|libvirt]] VM abstraction and management layer. * [[http://code.google.com/p/ganeti/|Ganeti]] VM management system. |
== The Problem We're Trying to Solve == |
| Line 29: | Line 23: |
| The virtual-machine management layer will need to support accounting for resource utilization by the VMs spawned for a given user or group, live migration of VMs from one host to another, and will likely need to support automated backups / snapshots of historical virtual-machine disk state. (Note that this differs from existing doctrine, which specifies that the machine-local OS data is expendable, and can be regenerated.) | At present, research groups buy clusters when they have money, CSG set them up, install the current supported Linux or Windows release on them (the CSG supported Linux release currently changes each year), optionally configuring storage and fileserver nodes, arranging tape backups of important data, adding special software etc. |
| Line 31: | Line 29: |
| The use of seed images, data de-duplication, and/or copy-on-write would also be valuable for minimising storage requirements. | Then the servers age, after the first year the OS becomes essentially frozen apart from minor security updates. It's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes. They become "fragile", and eventually a security risk. |
| Line 33: | Line 34: |
| == Background == | Sometimes it's hard to retire them when the hardware becomes more than 4-5 years old, because of the "fragile" software setup on them. |
| Line 35: | Line 37: |
| Sometime in early 2012, Susan told DCW that DoC were thinking of hiring someone for 6 months into CSG, specifically tasked with building a DoC private cloud. Essentially she said that Exec Committee has found some significant pot of money which needs to be spent this financial year. |
A second problem is that these clusters are often only accessible by members of the specific research group that bought them, so the resource may not be fully utilised. |
| Line 40: | Line 40: |
| She explained the core idea was "virtualisation even for research clusters": at present, research groups buy clusters when they have money, CSG set them up, install "linux du jour" on them, configure fileservers (if part of cluster), tape backups (if part), processing node special software etc. |
Instead, the idea is to setup a private cloud, researchers add hardware to that cloud's core resources, then create a VM for each cluster node, perhaps tied (1-1 at first) to their own hardware, the creation process should automatically install a CSG-supported operating system (historically supported Linuxes and Windows versions) or a non-CSG supported "standalone" operating system on the new VM. Researchers work as before on each VM - but each node is encapsulated inside a VM. |
| Line 45: | Line 48: |
| Then the servers age, the OS is essentially frozen (it's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes). They become "fragile". Sometimes it's hard to even retire them on schedule (4/5/6 years or whatever). Also these clusters are often only accessible by members of that research group so the resource may not be fully utilised. |
Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased and the VM migrated to it. |
| Line 52: | Line 51: |
| Susan's vision: setup a private cloud, researchers add hardware to that cloud's core resources, then create a VM for each cluster node, perhaps tied (1-1 at first) to their own hardware, CSG install that virtual cluster node's OS, researchers work as before - but each node is encapsulated inside a VM. Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased. |
We would also gain to flexibility to create short-term VMs for specific "run this software on 16 nodes" experiments. A fleet of such short-term VMs might be created today, run for a couple of days, and then be destroyed at the end of the experiment. |
| Line 59: | Line 55: |
| Various discussions with PJM and AON followed, post will be called "Cloud Manager" and be part of CSG, and do non-cloud things too. Could be permanent, could be 6 months in the first instance. |
We could even give every DoC user (students and staff!) their very own VM when they join, with full root/admin access - or at least the ability to create one when they first need it (lazy evaluation:-)). |
| Line 63: | Line 59: |
| Most crucially: (despite not knowing the exact spec, services to provide, let alone how to implement them) we therefore need to purchase all the kit having it delivered in July 2012, before the Olympics. PJM added "build a private cloud like Amazon EC2 does", AON suggested a budget of £100K, £150K or even £200K - we will provide possible plans for these price levels. |
== Open Staff cloud meetings == |
| Line 70: | Line 61: |
| DWM has spent a lot of time evaluating Ceph as a possible S3/Elastic Block Store like storage system for supporting VM storage and possibly very high speed filesystems eg. staging areas for VM data (scaleout NAS with replication). So far: it's not there yet. Alternatives need to be looked at as well.. |
In April 2012, the discussion was opened out to all interested staff, and (so far) two open staff cloud meetings have been held. Here are some notes taken by DCW and LDK of the discussions at both meetings. |
| Line 76: | Line 64: |
| == Working Group: 3rd April 2012 meeting == | [[project/privatecloud/meeting-2012-04-03|Open Staff Meeting 1 - April 3rd 2012]] |
| Line 78: | Line 66: |
| A working group of academics has been set up, this met on 3rd April 2012 for the first time. Things discussed: |
[[project/privatecloud/meeting-2012-04-25|Open Staff Meeting 2 - April 25th 2012]] |
| Line 81: | Line 68: |
| * PJM/Susan: background (spend money now, define services later), acknowledged unusual approach.. added (PJM) idea that a group can have a VM per project per year if they need, so they build new apps on the latest supported OS, while maintaining the ability to run their old versions on the older OS, allows people to try old code on new OS releases without "big bang" server upgrade problems. old VMs can eventually wither away.. want to save RAs (and CSG?) sysadmin time. | == Summer 2013: Cloud Access URL == |
| Line 83: | Line 70: |
| * PJM: start with concept of: every student gets a VM as they walk in through the door, keep while at College, have root access [need to fix/avoid NFS problem]. users should have the ability to create more VMs programmatically, both short term and long term ones. | The end-user interface for the DoC Private Cloud is now available for departmental users via [[http://cloudstack.doc.ic.ac.uk/client|cloudstack.doc.ic.ac.uk/client]]. Please use your normal college user-name and password for authentication; the domain should be ''imperial''. |
| Line 85: | Line 73: |
| * PJM: also, are we all agreed: it's got to be a realiable production system. Noone disagreed (but see later discussions). | == Summer 2012: Cloud Hardware we bought == |
| Line 87: | Line 75: |
| * JAMM: use cases of interest to her - projects into cloud technologies, pervasive computing exercises could be made more flexible [not sure how], some of her research involves streaming data from sensors, need high-capacity filestores. | Here is the hardware we have bought for the cloud. More can be added later (eg. by research groups opting in): |
| Line 89: | Line 77: |
| * PRP: EPSRC call "every research grant puts in for a small cluster" by the name "vanity clusters". EPSRC favouring shared resources (Dept, College, federated) - will allocate at most first £10K of equipment, then excess must have matching funds from Dept! favours (for example) shared services, grids, clouds and HPC. | . 4 x Dell [[www.dell.com/uk/business/p/poweredge-c6220/pd|PowerEdge C6220]] compute servers. This is a very dense compute server, with four independent nodes in a two unit chassis. Each node contains two Intel Xeon E5-2690 8-core 2.9GHz processors (32 threads with hyper-threading), 128GB of RAM and two 1TB hard drives. . 2 x IBM [[http://www-03.ibm.com/systems/uk/x/hardware/rack/x3750m4/|System x3750 M4]]. Each server has four Intel Xeon E5-4650 8-core 2.7GHz processors (64 threads with hyper-threading), 512GB of RAM, two 300GB hard drives and twelve 1TB hard-drives. . 4 x Dell [[www.dell.com/uk/enterprise/p/poweredge-r720/pd|PowerEdge R720]]. Each server has two Intel Xeon E5-2640 2.50GHz six-core 2.5Ghz processors (24 threads with hyper-threading), 64GB of RAM, two 300GB hard drives and 24 1TB hard-drives. . 1 x NetApp [[http://www.netapp.com/uk/products/storage-systems/fas2200/fas2200-product-comparison.aspx|NetApp F2240A-2]] dual-controller Filer and disk-shelf; raw storage capacity 60TB. . 4 x Extreme [[http://www.extremenetworks.com/products/summit-x670.aspx|Summit X670]] 10GbE switches; these form 2 pairs of switches, one pair in the DoC machine room (Huxley) and the other pair to be installed in the ICT machine room (MechEng). |
| Line 91: | Line 83: |
| * PRP added: VMs can really speed up provisioning of research project kit, instead of purchasing kit, waiting for it to arrive, installing and configuring it, use and maintain it, then (after project) decide what to do with it, can create 16 short term VMs bound to suitable hardware very quickly, do quick experiments and release the VMs resources. If spare hardware capacity is in hand, of course! | We identified two types of server for the DoC private cloud: a ''compute node'' and a ''storage node'': |
| Line 93: | Line 85: |
| * PRP agreed with Julie that research into cloud and distributed systems performance could be improved if we had a cloud which we could monitor and tweak. | * A '''compute node''' contains a large number of CPUs/cores. Its primary role in the cloud is one of computation (virtual machine hosting, distributed computing and the like). The Dell C6220 and IBM 3750s mentioned above are variant types of compute nodes. * A '''storage node''' contains a large number of locally attached disks providing a chunk of fault tolerant storage. Its primary role in the cloud is to provide storage (for VM images and associated research filesystems). The Dell R720s and the NetApp are both storage heavy nodes. |
| Line 95: | Line 88: |
| * JD: 2 important aspects of cloud here: 1. easily provisioned VMs; 2. amortization of all resources over multiple projects. The latter requires that researchers don't require all of their "own" resources "all" of the time - otherwise none spare! | We envisage that multiple compute nodes and multiple storage nodes would be needed. Here are our old notes: |
| Line 97: | Line 91: |
| * PJM/Susan: the matching funds model allows Dept to demand up to 50% of these shared resources [on average over time, perhaps front-loaded so "owners" get the majority of time up front, release nearly all resources later for general use]. | [[project/privatecloud/hardware|Hardware Investigations]] |
| Line 99: | Line 93: |
| * CCADAR: will sometimes need exclusive access to all "your" cluster VMs on all your hardware for experiments - repeatability is especially important. => need ability to pin VMs onto particular classes of node. | == Software Investigations == |
| Line 101: | Line 95: |
| * PRP: Yes, and sometime experiments need to happen directly on the bare metal. but only a small minority! | CSG have been familiarising themselves with various possible open source cloud or storage software systems that might be able to implement some/all of the required IaaS cloud services, and performing some initial investigations of a few of them. While the Cloud Manager will of course be responsible for designing and building the cloud, existing members of CSG are concerned to '''map the terrain''' to find out where the dragons are lurking and to provide an '''existence proof''' to reduce the risk that after buying the hardware, no software can be added to build the desired cloud. |
| Line 103: | Line 102: |
| * JAMM: performance monitoring very important. | Here are our notes: |
| Line 105: | Line 104: |
| * WJK: yes, including power monitoring of the physical VM hosts, a la picards. very useful. * GCASALE: agreed, added a subtle point about frequency of monitoring being very different between "cheap" power mon and "expensive" power mon.. LDK discussing with him. * SUSAN: Maja had mentioned that she makes a very large amount of use of Matlab, on Windows clusters, buying extra parallel licenses etc. PJM: why not use College standard license? DCW: believe extra modules and parallel licenses not included in College Matlab license, which is why ICT HPC kit doesn't support Matlab either! * TORA: Lab are very interested in more continuous autotesting, need a better sandbox: like a short term VM to run student code in! Also very interested in scalable storage (didn't say why?) * JD/SUSAN discussed: where are other Computing Depts with clouds? at any level (Dept, College, federated?) - answer seems to be: none known in production. * DWM added that LESC had done lots of "cloud v1" - grid - related work, and mentioned the similarities between grids, private clouds, batch processing and HPC. * PRP said that we should make more use of ICT's HPC, big resource. Susan said: some use ICT extensively (eg PHJK). PJM added that PHJK has found ICT HPC support very helpful, has invested money in more HPC kit, and believes we should make more use of college HPC. SUSAN added that she/Khanwal have found HPC team a bit sniffy about running Java code on HPC kit. * DCW said: yes, real programmers in HPC:-), and added that lots of money still going in though - let's use it. DCW added: HPC doesn't even let you access College home dirs cos they're "not fast enough" (source: Simon Burbidge, ICT), and mentioned that ICT also upgrading to VMware ESX 5, which "supports cloud" (but DCW doesn't know what that means). * PJM asked re: this - does everyone want DoC home dirs and research volumes accessible from VMs? everyone agreed, but several people pointed out that existing fileservers can be saturated by Condor so fileservers will need to scale more to cope. * DCW asked: what about Amazon S3 - simple distributed (key,value) storage system - important to DoC? some people said "might be useful" but noone had a solid use case. * WJK added that he'd love to do storage speed experiments using different speed storage eg. flash and raid levels. * TORA added that a large scalable block storage system would be very useful, but neglected to say why. * DWM said there seems to be a need for scalable storage at some level as part of the cloud, there are a variety of technologies - open source and commercial - to look at. Amazingly, he didn't even say "Ceph":-) * PJM said that commercial filers should be looked into, such as NetApp/EMC. PRP added that cloud storage is NetApp's bread and butter and their support and scalability was really good. Susan said DoC should consider these, but has a preference for open source if possible, DCW: CSG need to investigate NetApp with PRP/Cambridge/ICT help. * SUSAN reported that DR had initially said - CSG do everything his group needs, why need a cloud. However, when she asked him - could your group use more scalable storage, his eyes lit up! * DWM: so we conclude that scalable storage is very important? general vague agreement. * DCW summary: so cloud storage needs to hold VM images, it's not clear whether the same cloud storage subsystem should also support scalable filesystems, or whether fileservers are separate (but need to scale more). No estimate of size! S3 probably not important (optional extra). * GCASALE asked: what type of cloud? private? DCW/PJM: yes. what about cloudbursting, he asked ? DWM: what's that? GCASALE: ability to upload VMs to Amazon after development (or when need short term extra resources, maybe downloading VMs from Amazon too, general inter-Amazon operability). PJM: useful if possible. * "tall chap in green shirt": what about network bandwidth? 10Gb links? may also need bandwidth reservation in switch fabric. DWM: talking with ICT networking about 10Gb, they can also discuss bandwidth reservation. * "natasha's phd student in her place": their group are very interested in virtualizing algorithms but still using FGPAs and GPUs, and again more scalable storage is needed here. * WL agreed, saying some VM hosts definitely need to have GPUs and FPGAs (he can provide details and costs). DCW added that Amazon EC2 had VMs with access to GPUs and FPGAs etc in their pricing model. * WL added that he'd be very interested in "getting under the hood" and tweaking and monitoring how various aspects of the cloud operate. PJM said: may be contrary to production cloud - but perhaps a "sandpit cloud" could fork off the main cloud on occasion, grab some hardware etc. WJK agreed. * PJM talked about a cost accounting model, enforcing 50% maximum usage, sounded very complicated (DCW: god knows how that's even implemented! perhaps logging use for post-analysis). WJK wondered whether anything that heavy was needed. * JD asked: would we give access to people outside of DoC? DCW: no, our resources, our users. JD: power of clouds (and interesting research topics) is when you get to federating. * PJM: might be open to sharing with ICT, maybe specific research projects later? Quick round up of other comments at end, useful services to check? * CacheDB useful (JD) * OpenNebula (GCASALE) - DCW: looks very interesting, open source data centre virtualization, very cute, supposed to be able to "integrate your existing storage, hosts etc". * MooseFS (DCW added) - DCW: cluster file system that openNebula can use for shared storage. LDK: uses FUSE layer. * Eucalyptus (JD). * OpenStack (PJM) * Hadoop/Mapreduce (green shirt bloke) * DCW asked about size of storage: helpful answer was "TBs to PBs". === PJM's summary of meeting === I think that three basic conclusions should be drawn from yesterday's discussion regarding the specification of hardware: 1. Our concept of buying compute nodes with large numbers of cores and large memory, with disc storage for virtual machines images, and with 10G network will support the main objective of providing a DoC Cloud that provides virtual computers to the DoC students and staff for teaching and research purposes. We need to refine exactly which machines and configuration will be purchased. 2. We should look at input from research groups as to what is the cost of GPU, FGPA, and hardware monitoring, and see if this can be incorporated at this stage. 3. We do need to look at fast network storage options. === Next meeting === Next Working Group meeting: 25th April 1pm, level 4 common room |
[[project/privatecloud/investigations|Software Investigations]] |
DoC Private Cloud: 2012 - 2013
Project Goal
In early summer 2012, CSG were tasked with building a DoC Infrastructure-as-a-service private cloud, very like Amazon EC2 ("Elastic Compute Service") which presents a secure and convenient web interface which enables users of DoC to specify and create VMs and associated storage, automatically install OSes on them and deploy them.
The main goal is to virtualize most research servers, decoupling the OS image from the hardware for greater flexibility. Sharing (amortizing) the costs of each machine. One driver of this is EPSRC deciding to only provide 50% of any hardware bid over £10K in future, with the Dept expected to pay the remaining 50%.
This project was approved by Executive Committee and by two open meetings of Academic staff. Peter McBrien (PJM) led the project, and laid out two stages:
- a 6 month phase in which CSG (advised by an academic working group) will design and build a prototype cloud, recruiting a "Cloud Manager" person to join CSG, possibly for 6 months in the first instance. The Department will spend some significant amount of money to build the prototype cloud, perhaps in the £100-200K range.
- assuming the prototype cloud is successful, it will move into production and the "Cloud Manager" become permanent. Researchers would then be encouraged to add research-funded hardware to the cloud and given some form of preferential treatment on "their hardware". All members of CSG are enthusiastic to gain cloud-related skills from the "Cloud Manager".
Most crucially: the Department's substantial initial investment had to be fully spent before the end of July 2012. This means that all kit was ordered, delivered and paid for before the 31st July 2012. Nearly £300K inc vat was spent on the project. The Cloud Manager, Thomas Joseph, was appointed about a year later - in July 2013.
The Problem We're Trying to Solve
At present, research groups buy clusters when they have money, CSG set them up, install the current supported Linux or Windows release on them (the CSG supported Linux release currently changes each year), optionally configuring storage and fileserver nodes, arranging tape backups of important data, adding special software etc.
Then the servers age, after the first year the OS becomes essentially frozen apart from minor security updates. It's often difficult to persuade researchers that we should reinstall their fileservers, webservers and compute nodes. They become "fragile", and eventually a security risk.
Sometimes it's hard to retire them when the hardware becomes more than 4-5 years old, because of the "fragile" software setup on them.
A second problem is that these clusters are often only accessible by members of the specific research group that bought them, so the resource may not be fully utilised.
Instead, the idea is to setup a private cloud, researchers add hardware to that cloud's core resources, then create a VM for each cluster node, perhaps tied (1-1 at first) to their own hardware, the creation process should automatically install a CSG-supported operating system (historically supported Linuxes and Windows versions) or a non-CSG supported "standalone" operating system on the new VM. Researchers work as before on each VM - but each node is encapsulated inside a VM.
Later, these VMs could share resources - when the group don't need 100% resources, or new more powerful hardware is purchased and the VM migrated to it.
We would also gain to flexibility to create short-term VMs for specific "run this software on 16 nodes" experiments. A fleet of such short-term VMs might be created today, run for a couple of days, and then be destroyed at the end of the experiment.
We could even give every DoC user (students and staff!) their very own VM when they join, with full root/admin access - or at least the ability to create one when they first need it (lazy evaluation:-)).
Open Staff cloud meetings
In April 2012, the discussion was opened out to all interested staff, and (so far) two open staff cloud meetings have been held. Here are some notes taken by DCW and LDK of the discussions at both meetings.
Open Staff Meeting 1 - April 3rd 2012
Open Staff Meeting 2 - April 25th 2012
Summer 2013: Cloud Access URL
The end-user interface for the DoC Private Cloud is now available for departmental users via cloudstack.doc.ic.ac.uk/client. Please use your normal college user-name and password for authentication; the domain should be imperial.
Summer 2012: Cloud Hardware we bought
Here is the hardware we have bought for the cloud. More can be added later (eg. by research groups opting in):
4 x Dell PowerEdge C6220 compute servers. This is a very dense compute server, with four independent nodes in a two unit chassis. Each node contains two Intel Xeon E5-2690 8-core 2.9GHz processors (32 threads with hyper-threading), 128GB of RAM and two 1TB hard drives.
2 x IBM System x3750 M4. Each server has four Intel Xeon E5-4650 8-core 2.7GHz processors (64 threads with hyper-threading), 512GB of RAM, two 300GB hard drives and twelve 1TB hard-drives.
4 x Dell PowerEdge R720. Each server has two Intel Xeon E5-2640 2.50GHz six-core 2.5Ghz processors (24 threads with hyper-threading), 64GB of RAM, two 300GB hard drives and 24 1TB hard-drives.
1 x NetApp NetApp F2240A-2 dual-controller Filer and disk-shelf; raw storage capacity 60TB.
4 x Extreme Summit X670 10GbE switches; these form 2 pairs of switches, one pair in the DoC machine room (Huxley) and the other pair to be installed in the ICT machine room (MechEng).
We identified two types of server for the DoC private cloud: a compute node and a storage node:
A compute node contains a large number of CPUs/cores. Its primary role in the cloud is one of computation (virtual machine hosting, distributed computing and the like). The Dell C6220 and IBM 3750s mentioned above are variant types of compute nodes.
A storage node contains a large number of locally attached disks providing a chunk of fault tolerant storage. Its primary role in the cloud is to provide storage (for VM images and associated research filesystems). The Dell R720s and the NetApp are both storage heavy nodes.
We envisage that multiple compute nodes and multiple storage nodes would be needed. Here are our old notes:
Software Investigations
CSG have been familiarising themselves with various possible open source cloud or storage software systems that might be able to implement some/all of the required IaaS cloud services, and performing some initial investigations of a few of them. While the Cloud Manager will of course be responsible for designing and building the cloud, existing members of CSG are concerned to map the terrain to find out where the dragons are lurking and to provide an existence proof to reduce the risk that after buying the hardware, no software can be added to build the desired cloud.
Here are our notes: