Chat Channels Via Kafka (with Grading Notes)

Transcription

1 SE424: Distributed Systems Assignment 2 Semester Due: 10 Jan 2017 Chat Channels Via Kafka (with Grading Notes) In this assignment, we ll take the work from the previous assignment on channelized chat and have it use a distributed system as a backend instead of a custom server. The distributed system that we ll use is the one that we saw in class recently: Kafka. Documentation Site You ll find a lot of help in using Kafka and its Java binding at the following links: (General Kafka documentation) (Kafka Java classes Javadocs) 1 The Communication Protocol The clients will speak a protocol built on the following message type: sendername hour:minute:second - sentence The line begins with the name of the sender followed by one space (spaces aren t allowed in the name). It s followed by a time stamp in the format hour:minute:second (ex. 12:05:07) followed by a single space, a dash (-), and then single space. After the second space comes a sentence which may contain spaces. The message ends with a new line character. The sendername is the sender name as chosen by the sender. It is not managed by Kafka and does not need to be globally unique. The timestamp is the local time at the sender s computer. Messages of the above type are prepared and sent by Kafka Producer objects for distribution to clients who have Consumer objects for the message s topic. Note that the message format is different from the previous assignment. Since Kafka will forward messages based on topics and handle all topic registration and leaving, there is no need for other message types. Depending on how you build your clients, you may not need to parse the above messages at all since Kafka will distribute the messages between the Consumer objects automatically. 2 Kafka and ZooKeeper Setup While there are classes which you can use to programmatically set up ZooKeeper servers and Kafka broker instances, we will assume that they are setup and configured using an external set of tools. We ll set up the following Kafka and ZooKeeper topology: 1

2 You ll need to create four (4) configuration scripts for the setup: 1. A Kafka Broker 0 script (KB0) ( ) 2. A Kafka Broker 1 Script (KB1) ( ) 3. A Kafka Broker 2 Script (KB2) ( ) 4. A ZooKeeper Script (ZS) ( ) You may use different IP addresses during development and testing, but for uniformity, the scripts must be submitted with the configuration parameters in the figure above 3 Working with Producers and Consumers You will take the client program that you built for the previous assignment and change it to use Kafka instead of the single server. The client s interface should remain similar, but its internals will be changed to as follows: Register When a user requests to register for a topic, the client must start a Kafka Consumer object which connects to the configured broker and starts to receive messages from the broker. Leave When a user requests to unregister for a topic, the client must stop any active Kafka Consumer objects for that topic. Send When a user requests to send a message with a given topic, the client must use a Kafka Producer object to send the message using the configured broker. 4 The Client Program The client program will remain more or less the same as from the previous assignment with some changes as a noted below. When the client program starts, it first reads from a configuration file which contains the following information: (1) The Kafka broker IP (2) The Kafka broker port It stores the Kafka broker IP and server port for later, when the user requests to connect. This will let the user point the client program to a different Kafka broker without needing to recompile the code and without the user needing to enter the Kafka broker IP and port manually. The configuration file will be read when the client program starts up and doesn t need to be read again during execution. Note: You choose how to implement the configuration file, but you must document the configuration file and how it works in the README file for the assignment. The client program lets the user perform the following operations: Login The user enters in a user name (ex. Alice) which will be used for all subsequent messages to the Kafka broker. The user name is not managed by Kafka and does not need to be checked as globally unique (though you may optionally enforce global uniqueness). Quit Close the client program. If the client currently has any open Kafka Consumer objects, consumer threads, or producer threads, they must be closed. The client then closes. 2

3 Register for a topic The user enters a topic to register for. If the user is already registered for the topic (i.e. a Kafka Consumer object for the topic already exists), the client program shows an error message. Otherwise, the client program then performs the following steps: If the topic already does not already exist in the cluster, the client creates the topic with a replication factor of 3 and one (1) partition (see the classes AdminClient, KafkaAdminClient, NewTopic for how to do this). The client starts a Kafka Consumer object for the topic from the Kafka broker given in the configuration file. It configures the Consumer to receive all subsequent messages (e.g. it will not sign up to receive messages sent in the past). The client listens on the Kafka Consumer object in a separate listening thread. Once registered for a topic, the client will begin listening for messages from the Kafka broker on a Kafka Consumer object held on a dedicated listening thread. It will show messages that it receives on the screen in the following format: ( topic ) sendername hour:minutes:seconds - sentence The receiver must add the topic name and parenthesis before the received message (which has the rest of the fields in the line). For instance, the following are some messages that might be shown: ( horses ) Alice 12:10:45 - I like horses! ( horses ) Bob 12:10:50 - So do I! ( cats ) Alice 12:11:15 - Meow ( cats ) Alice 12:11:20 - Feed me! Output Alternative As an alternative to showing the messages on the console, the tool may place all messages received from the topic in a text file in the execution directory of the tool. The text file must be called (topicname).txt (ex. horses.txt for the topic horses and cats.txt for the topic cats ). For example, in the above case, the tool would use two files with the content showed below: horses.txt cats.txt ( horses ) Alice 12:10:45 - I like horses! ( horses ) Bob 12:10:50 - So do I! ( cats ) Alice 12:11:15 - Meow ( cats ) Alice 12:11:20 - Feed me! The tool must enter each message into the text file on a separate line. If a file with the topic s name already exists in the program s execution directory when the user registers for the topic, all new messages must be appended to the existing content. Leave a topic The user enters a topic to leave. If the user isn t registered for the topic, an error message is shown to the user. Otherwise, the client program closes any open Kafka Consumer object or listening threads associated with the topic. Send a message The user selects a topic and enters a sentence to send to the configured Kafka Broker. The user does not need to be registered to the topic to send a message under it. The client converts the sentence to a message to send to the configured Kafka Broker using a Kafka Producer object. The message sent to the Kafka Broker must have the following format: 3

4 sendername hour:minutes:seconds - sentence The sendername is the name that the user provided during the login step. The time is the current hour, minute, and second at the sender s computer. The Kafka broker will be in charge of forwarding the message to all clients who are registered on the topic. Additional Client Requirements 1. The client program must offer either a GUI (Java Swing or JavaFX) or text based (command line) interface with the above operations available. 2. The user must not be required to enter raw protocol messages; the client must build the messages for the user. 3. Protocol messages must be sent by the client program in response to user requested operations using the GUI or text based interface. 4. The choice of the interface and its design is up to you. 5. You must document the user interface and how it works in the README file for the assignment. 5 Additional User Interface Requirements The client program must support the following requirements: 1. Language: The client program must be written in Java. 2. Robustness: The client program must handle all errors, not crashing due to unhandled exceptions. 3. Responsiveness: The client program must enable the user to interact with the user interface when listening or processing. This implies that network communications must not be done on the same thread as the user interface. 4

5 6 What to turn in by 10 Jan 2017 You may work in groups of up to two (2) students. There are two methods you may use for submitting your work: Moodle Github The following rules apply for each of the methods: 6.1 Moodle Submission Each member of the group must turn in a single ZIP file via Moodle which contains the following items: The configuration scripts KB0, KB1, KB2, ZS in clearly named and properly formatted configuration files. The full source code for all parts of the system. A compiled JAR of the client program runnable on any Windows 7 or 10 computer, along with any required configuration files. A file called README.pdf which has the following information: A list of the students in the group, including names and ID numbers A detailed list of which student worked on what and for approximately how many hours Instructions for using the scripts KB0, KB1, KB2, and ZS to setup the ZooKeeper and Kafka cluster. Instructions for compiling the source code for the client program Instructions for running the client program Documentation of the client configuration file 6.2 GitHub Submission The group must use the private repository provided for the assignment on GitHub. The repository must have the following properties and contents: The configuration scripts KB0, KB1, KB2, ZS in clearly named and properly formatted configuration files. The full source code for all parts of the system. A ZIP file containing the following files: A compiled JAR of the client program runnable on any Windows 7 computer, along with any required configuration files. A file called Readme.md which has the following information: A list of the students in the group, including names, GitHub user names, and ID numbers A detailed list of which student worked on what and for approximately how many hours Instructions for using the scripts KB0, KB1, KB2, and ZS to setup the ZooKeeper and Kafka cluster. 5

6 Instructions for compiling the source code for the client program Instructions for running the client program Documentation of the client configuration file A commit to the master branch with the text Final commit for submission 7 Grading The weights for the grading will be assigned as follows: Topic creation, registration, leaving: 30% Message forwarding and display: 40% Setup scripts: 15% Robustness: 15% 7.1 Penalties and Errors The following penalties will be assessed for the errors or omissions below: Grading Notes (General): Hard coding file names, IP addresses, or port numbers into the code. (-15) points Not handling quit correctly. (-15) points Missing item from required list of readme file elements. (-4) points (maximum 20 points off) Tool works only if the client and cluster servers (Kafka, ZooKeeper) are on the same computer. (-15) points Grading Notes (Client): Client doesn t allow multiple instances to be opened. (-5) points Client requires the user to enter the complete message format as well as the sentence. (-10) points Can t rejoin a topic after leaving it. (-5) points Doesn t show all received messages from the cluster for a particular topic. (-10) points When registering for a topic, doesn t create it with the correct replication parameters. (-5) points When registering for a topic, asks for all messages in the topic s history. (-5) points Grading Notes (Robustness): Unhandled exception. (-3) points each 6

7 7.2 Testing Regimen The following is the testing regimen I will use for the grading. It s worthwhile to try out the testing regimen yourself before submission. As a note, the three clients below will be run on three different computers in Rooms 815, 817, or 819 in Bet Achi. 1. Change Kafka and ZooKeeper IP addresses to something other than what s in the configuration file (will use 4 different computers). 2. Turn on the ZooKeeper server. 3. Turn on the Kafka Brokers KB0, KB1, KB2. 4. Configure client1 to work with KB0. 5. Turn on client1. 6. Send a message with topic1 7. Register for topic1 8. Send a message with topic1 9. Send a message with topic2 10. Register for topic2 11. Leave topic1 12. Send a message with topic1 13. Send a message with topic2 14. Close client1 15. Reopen client1 16. Send a message with topic1 17. Register for topic1 18. Send a message with topic1 19. Configure client2 to work with KB1 20. Turn on client2 21. Client1: Send a message with topic1 22. Client2: Send a message with topic1 23. Client2: Register for topic1 24. Client1: Send a message with topic1 25. Client1: Send a message with topic2 26. Client1: Register for topic2 27. Configure client3 to work with KB2. 7

8 28. Turn on client3 29. Client1: Send a message with topic1 30. Client2: Send a message with topic1 31. Client3: Register for topic2 32. Client1: Send a message with topic1 33. Client1: Send a message with topic2 34. Client1: Register for topic2 35. Client2: Send a message with topic2 36. Client3: Send a message with topic3 37. Client1: Leave topic2 38. Client1: Send a message with topic2 39. Close client3 40. Configure client3 to work with KB1 41. Turn on client3 42. Client3: Send a message with topic2 43. Client1: Send an illegal message (no topic, command not found) 44. Find which broker is managing topic1 and close it 45. Close any client connected to the closed broker, reconfigure the clients to use another broker, and restart. 46. Client1: Send a message with topic1 47. Client2: Send a message with topic1 48. Client3: Send a message with topic1 49. Close the servers 8