Cloud Computing Principles and Paradigms: 10 comet cloud-an autonomic cloud engine

Editor's Notes

• #4: CometCloud is an autonomic computing engine (framework) for cloud and grid environments to realize a virtual computational cloud with resizable computing capability, which integrates local computational environments and public cloud services on-demand.Specifically, CometCloud enables policy-based autonomic cloudbridging and cloudbursting. Autonomic cloudbridging enables on-the-fly integration of local computational environments (data centers, grids) and public cloud services (such as Amazon EC2 [10] and Eucalyptus [20])autonomic cloudbursting enables dynamic application scale-out to address dynamic workloads, spikes in demands, and other extreme requirements
• #5: CometCloud is based on a peer-to-peer substrate that can span enterprise data centers, grids, and clouds. Resources can be assimilated(جذب کردن، تلفیق کردن) on-demand and on-the-fly into its peer-to-peer overlay to provide services to applications. CometCloud is composed of a programming layer, a service layer, and an infrastructure layerInfrastecture Layer:The infrastructure layer uses the Chord self-organizing overlay, and the Squid information discovery and content-based routing substrate built on top of Chord.The routing engine supports flexible content-based routing and complex querying using partial keywords, wildcards, or ranges. It guarantees that all peer nodes with data elements that match a query/message will be located. Nodes have different roles and, accordingly, different access privileges based on their credentials and capabilities.This layer also provides replication and load balancing services, and it handles dynamic joins and leaves of nodes as well as node failures.Every node keeps the replica of its successor node’s state, and it reflects changes to that as well as notify predecessor.Service Layer:The service layer provides a range of services to supports autonomics at the programming and application level. This layer supports the Linda-like tuple space coordination model.Programming Layer:The programming layer provides the basic framework for application development and management. It supports a range of paradigms including the master/worker/BOT. Masters generate tasks and workers consume them. Scheduling and monitoring of tasks are supported by the application framework.The task consistency service handles lost tasks. Even though replication is provided by the infrastructure layer, a task may be lost due to network congestion. In this case, since there is no failure, infrastructure level replication may not be able to handle it. This can be handled by the master, for example, by waiting for the result of each task for a predefined time interval and, if it does not receive the result back, regenerating the lost task.
• #6: The goal of autonomic cloudbursts is to seamlessly and securely integrate private enterprise clouds and data centers with public utility clouds on-demand,to provide the abstraction of resizable computing capacity.CometCloud considers three types of clouds based on perceived security/trust and assigns capabilities accordingly. The first is a highly trusted, robust, and secure cloud, usually composed of trusted/secure nodes within an enterprise, which is typically used to host masters and other key (management, scheduling, monitoring) roles.The second type of cloud is one composed of nodes with such credentials—that is, the cloud of secure workers.The final type of cloud consists of casual workers. These workers are not part of the space but can access the space through the proxy and a request handler to obtain (possibly encrypted) work units.If the space needs to be scale-up to store dynamically growing workload as well as requires more computing capability, then autonomic cloudbursts target secure worker to scale up. But only if more computing capability is required, then unsecured workers are added.
• #7: Load Dynamics. Application workloads can vary significantly. This includes the number of application tasks as well the computational requirements of a task. The computational environment must dynamically grow (or shrink) in response to these dynamics while still maintaining strict deadlines.Accuracy of the Analytics. The required accuracy of risk analytics depends on a number of highly dynamic market parameters and has a direct impact on the computational demand—for example the number of scenarios in the Monte Carlo VaR formulation. The computational environment must be able to dynamically adapt to satisfy the accuracy requirements while still maintaining strict deadlines.Collaboration of Different Groups. Different groups can run the same application with different dataset policies . Here, policy means user’s SLA bounded by their condition such as time frame, budgets, and economic models. As collaboration groups join or leave the work, the computational environment must grow or shrink to satisfy their SLA.Economics. Application tasks can have very heterogeneous and dynamic priorities and must be assigned resources and scheduled accordingly. Budgets and economic models can be used to dynamically provision computational resources based on the priority and criticality of the application task. For example, application tasks can be assigned budgets and can be assigned resources based on this budget. The computational environment must be able to handle heterogeneous and dynamic provisioning and scheduling requirements.Failures. Due to the strict deadlines involved, failures can be disastrous. The computation must be able to manage failures without impacting application quality of service, including deadlines and accuracies.
• #8: Autonomic cloudbridging is meant to connect CometCloud and a virtual cloud which consists of public cloud, data center, and grid by the dynamic needs ofthe application.Hence, types of used clouds, the number of nodes in each cloud, and resource types of nodes should be decided according to the changing environment of the clouds and application’s resource requirements.The scheduling agent manages autonomic cloudbridging and guarantees QoS within user policies. Autonomic cloudburst is represented by changing resource provisioning not to violate defined policy. We define three types of policies:Deadline-Based. When an application needs to be completed as soon as possible, assuming an adequate budget, the maximum required workers are allocated for the job.Budget-Based. When a budget is enforced on the application, the number of workers allocated must ensure that the budget is not violated.Workload-Based. When the application workload changes, the number of workers explicitly defined by the application is allocated or released.
• #9: it support fault-tolerance in two ways which are in the infrastructure layer and in the programming layer. The replication substrate in the infrastructure layer provides a mechanism to keep the same state as that of its successor’s state, specifically coordination space and overlay information. Every node has a local space in the service layer and a replica space in the infrastructure layer. When a tuple is inserted or extracted from the local space, the node notifies this update to its predecessor and the predecessor updates the replica space. Hence every node keeps the same replica of its successor’s local space. When a node fails, another node in the overlay detects the failure and notifies it to the predecessor of the failed node. Then the predecessor of the failed node merges the replica space into the local space, and this makes all the tuples from the failed node recovered. Also the predecessor node makes a new replica for the local space of its new successor.To address packet loss, in programming layer the master checks the space periodically and regenerates lost tasks.
• #10: VaR for measuring the risk level of a firm’s holdings and image registration for medical informatics. VaR calculation should be completed within the limited time, and the computational requirements for the calculation can change significantly.VaR we will focus on how autonomic cloudbursts work for dynamically changing workloads. Image registration is the process to determine the linear/nonlinear mapping T between two images of the same object or similar objects that are acquired at different time.that data size of image registration is much larger than that of VaR. For image registration, because it usually needs to be completed as soon as possible within budget limit, we will focus on how CometCloud works using budget-based policy.
• #11: Total application runtime of CometCloud-based (a) VaR and (b) image registration on Amazon EC2.If the computed data size is large and it needs more time to be completed, then workers will have less access the proxy and the communication overhead of the proxy will decrease.
• #12: When the application workload increases (or decreases), a predefined number of workers are added (or released), based on the application workload. we defined workload-specific and workload-bounded policies. In workload-specific, a user can specify the workload that nodes are allocated or released. In workload-bounded, whenever the workload increases by more than a specified threshold, a predefined number of workers is added. Similarly, if the workload decreases by more than the specified threshold, the predefined number of workers is released.
• #13: we set the maximum number of available nodes to 25 for TW(private datacenter) and 100 for EC2.Costs for the TW data center included hardware investment, software, electricity, and so on.initially allocated 10 nodes each from TW and EC2.Total 500 tasks.