CS 6343: CLOUD COMPUTING Term Project

Transcription

1 CS 6343: CLOUD COMPUTING Term Project For all projects Each group will be assigned a cluster of machines Each group should install VMM on each platform and use the VMs to simulate more machines See VMM installation document You are always welcome to use open source solutions for any required functionality you need to develop But you should not downgrade the requirement such that you can use an open source solution Group A TA: Yongtao Huang Project A1: Cloud file systems Explore the features of various cloud file systems Consider different file systems, such as HDFS Swift: Ceph: Project A2: NoSQL DB Explore the features of various nosql databases Consider different nosql DB, such as HBase: Cassandra Redis: Project steps Install the file/db systems on a cluster Install each component of each file/db system on a VM and create the corresponding VM image Clone the VM image at any platform to support easy experimentation Our goal is to use all the machines for A group to conduct final experimental studies Make sure to use qemu format for the VM virtual disk so that you do not need to have an overblown image size Identify the features to be compared Lookup time (lookup a file or a key) Normal situation During file transfers due to load balancing or node join/leave Table update time due to load balancing or node join/leave Access latency, access throughput, directory service latency, etc. Load balancing features and performance Consistency model and solutions, availability solutions, etc. Performance benchmarking E.g., MapReduce programs, TCP-W, etc. Other special features that are unique to a certain file system Evaluate the file systems through request generator Use File Generator to generate the file system

2 Make sure that your file system is external to the VMs so that your VMs can have a reasonable size and can be cloned You can create the file systems at the KVM host, and let VM mount to it. Create one file system for each VM (this is just to simulate the file system of each VM, avoiding keeping the files in the VM). Generate small file systems in the testing stage and extensive ones at the evaluation stage Modify Request Generator to send requests to each of the file systems If cannot finish all 3, at least finish 2 Run the request generator with different workloads Record the measurements for the identified metrics Generate direct requests to explore basic features of the file/db systems Consider creating load unbalanced scenarios in the request generator Evaluate the DB systems through YCSB Papers Benchmarking Cloud Serving Systems with YCSB Benchmarking Top NoSQL Databases A comparison between several NoSQL databases with comments and notes Use YCSB with different settings Modify YCSB to create unbalanced workload (target workload on a few nodes) Design application programs to evaluate the file systems during computation Set up MapReduce on top of the file systems Build VM images after MapReduce installation Port several MapReduce sample programs to run on top of the system and evaluate the performance of the file system For fair and thorough evaluation, you may need to Modify the source code to explore some features E.g., capture the lookup time E.g., capture the transfer time after adding new nodes or after load balancing (need to also generate requests to access files/keys that have been transferred and measure their access latencies) Write code to have a fair comparison of different designs Focus on the comparison of the naming service (sharding) solutions Implement different DHT solutions of different file/db systems and compare their performance Consider Ceph, Swift, Cassandra, Redis. Compare the lookup time. Compare the lookup table update time Simple implementation of the DHT solutions for Swift and Ceph DHT table design and DHT computation Hash function should be in a separate class, replaceable Access requests Table initialization Physical node insertion and removal Virtual node insertion and removal Lookup (file-name, blockid) returns node Ids Should at least have the initialization and lookup capabilities Should have a system configuration file to specify Number of physical nodes and weight of each physical node

3 Virtual node unit weight Initial Table (OSD map, Ring) Client caching for lookup requests (allow on/off switch) Can use caching packages such as Java s EhCache, Guava Caching Utilities, cache2k Or simply use a HashMap to implement your own cache Cached object version check and invalidation DHT table update request handling according to the protocols of various systems Can assume that the request is issued by a control client to simulate table changes after load balancing and after node addition/removal For load balancing: load-balance (heavy-pid, light-pid); Node addition and failure: pid = add-physical-node (); remove-physical-node (pid); Swift and Ceph table update protocol Centralized, the Proxy in Swift and the Monitor in Ceph Cassandra and Redis table update protocol Directly use the scuttlebutt protocol Complete implementation of the DHT solutions Client side needs to call file system read/write to perform and actual file accesses Collect load information and make load balancing decisions, then perform actual file transfers based on load balancing decisions. Load balancing decision in Cassandra: only between neighboring nodes. Load balancing decision in Redis: by user commands, addslot adds a hash node to a data server; deleteslot removes a hash node from a data server Support node insertion and removal Actually change table according to load balancing and node addition/removal Project B TA: Nidhi.Solanki Project: SaaS exploration Develop a SaaS application on GAE and Azure Project Steps Study the PaaS platforms and summarize their APIs GAE Overview: Azure Design an application Convert the standalone RoboCode to a web based application Or take the converted code and improve it Or consider a different application Develop the SaaS application on GAE and/or Azure Develop the web application (no PaaS element needed in this case) Develop a router to route user requests to the web application hosts Should balance the load on the hosts Use PaaS access control support to implement login and access control Use PaaS data store to implement tenant data management Develop a client program to issue http requests to the SaaS application Deploy the SaaS application in a secure manner Consider a hybrid cloud

4 Install VMM on hosts in the lab to build the private cloud Public cloud could be Azure or GAE. GAE is a simulator and may have to be hosted locally. Azure should be hosted on Microsoft site Web application is deployed as a VM image Create VMs using VM images of your OS choices Deploy your web application on one VM and save a new VM image with your app (let s refer to this VM as appvm). Note: Use the qumu solution with dynamic virtual disk size to ensure that you will not have a VM with a huge image size Clone several copies of the appvm on multiple machines. Need to change some configurations manually to support communication Deploy the router program on one VM, also save it as a VM image (let s refer to it as routervm) Access control and data management are deployed on the PaaS platform Communication Client issues requests to your SaaS through the public cloud URL to perform access control After access control the request is redirected to the routervm, the routervm decides which appvm should pick up the request and further redirect the request to the selected appvm From this point on, appvm interact with the client to process the request (RoboCode edit/play functions) During execution of the appvm, if it is necessary to access the database, then perform a remote access to the cloud data store Security After user login, a secure session should be maintained, i.e., there is no need for the user to login again, but no one else should be able to pretend to be the user to issue the app requests. Use session mechanism on the web application side hosted locally. From public cloud access control to web application hosts, a secure protocol should be implemented to ensure secure communication. The protocol should sustain replay attacks The communication mechanism should be implemented as a framework so that other SaaS applications can use the framework as well SaaS RoboCode functionalities Support RoboCode editing, compilation, and playing Support scoring and rewards for individual users Use cloud data store to store data related to individual users, including their RoboCodes, scores and rewards they obtained, their specific configurations in the SaaS, etc. Enforce access control Consider system level data Consider user specific data Consider user groups, support group based sharing (can be considered as tenants) Study the performance of the application system on different PaaS Document the steps and issues for converting standalone software to SaaS Document the performance study results Project submissions Regular meetings

5 There will be weekly meetings with the TA and bi-weekly meetings with the Professor Schedule and milestones will be defined during the meetings Mid-term submissions Your working code Though your code is not for the entire project, it should implement some functions required in the project and should be fully working Midterm project report (contents is the same as final report) Final submissions Your working code Final project report Final report will be progressively revised from midterm report (revised report should be submitted till it is satisfactory) Final demo will be schedule during midterm demo Submission guideline Each team only needs one submission, team leader should make the submission through e- learning During submission, attach the doc file and the file name has to be <team-label>.doc or <teamlabel>.docx Do not submit pdf Code submission Zip or tar your code, file name has to be <team-label>.zip Attach the zip file during e-learning submission VM submission Zip or tar your code, file name has to be <team-label>.zip Prepare all the VMs that are needed to test your system (should minimize your VM size) Upload the zip file and the VMs to MS OneDrive, provide the url in your report Required information in the report (can include more than listed) In the cover page, provide team-label, title, and team members Introduction Goal of the project What you would offer in your system and why what you offered is important Study of related work Summary of related works (in paper or similar products available) How your project is different from or is similar to some existing works Approach System architecture Activity diagram (workflow) Architecture diagrams (from high level to low level decomposition of the system) Description of the nodes in the architecture Detailed design Including the APIs of each important component, the algorithms used in some components, etc. Implementation details List the code files and VMs that are in your system Provide a structure of the code files and describe each code file and each VM in the list. The relations of the code files and what they are doing should have been in the detailed design section, you can relate the code files to those in the design section Problems encountered and how they are resolved This is important, it is the basis for us to decide the efforts you have actually put in the project

6 ... Experimental results Clearly state the control parameters and the metrics for measurement Experimentation needs to be thorough and results need to be easy to read (e.g., use graphs and table) Some teams need to coordinate with each other in experimentation A1 and A2 D and E and TA Installation manual, including existing open source used (no need to give installation guide for open sources, just their urls) and how to setup your programs User manual, discussing how to use your system Workload distribution Include a high level summary and a detailed list The detailed list include tasks done by each member in the team week by week (just attach the log you prepared every week)