CS11 C++ DGC. Spring Lecture 6

Transcription

1 CS11 C++ DGC Spring Lecture 6

2 The Spread Toolkit A high performance, open source messaging service Provides message-based communication Point-to-point messaging Group communication (aka broadcast or multicast communication) Heavy focus on safety and reliability Source, binaries, and docs available at

3 Spread vs. TCP, UDP, etc. Many networking applications use TCP/IP Transmission Control Protocol A reliable, stream-based protocol Spread is message-based; less suitable for streaming apps Some applications use UDP User Datagram Protocol Can implement broadcast (to subnet) or multicast with UDP UDP is unreliable and unordered! Spread can provide either unreliable or reliable broadcast and multicast As usual, pick the right tool for the job.

4 How Spread Works Spread consists of two parts: A library of functions that your program uses SP_connect, SP_multicast, SP_receive, etc. A set of daemons that forms a Spread network Configuration of network driven by a spread.conf file Daemons must be running for Spread programs to communicate! Spread can leverage multiple underlying communication layers TCP, UDP, multicast, etc.

5 Spread Groups Spread is based on notion of groups Clients join one or more groups Messages are sent to an entire group Each client also has its own private group No one else can join that group Can send messages to a particular client s private group, for point-to-point communication Spread provides group membership change notifications If a client joins, leaves, or is disconnected from a particular group, membership messages are sent to the whole group

6 Ordering and Reliability For group messaging, two major concerns: Do messages actually show up at all? What order do messages show up at a receiver? What order with respect to a particular sender? What global order with respect to all senders? Messaging libraries provide reliability guarantees and ordering guarantees Spread provides six different levels of service Different combinations of reliability and ordering guarantee Can specify level of service when each message is sent

7 Spread Guarantees UNRELIABLE_MESS No reliability guarantee an unreliable message may be dropped! No ordering guarantee any unreliable message can arrive before or after any other unreliable message RELIABLE_MESS Spread will make a best effort to deliver the message to all group members No ordering guarantee just like UNRELIABLE_MESS

8 Spread Guarantees (2) FIFO_MESS Messages are delivered reliably (best effort) Messages from a particular source are received in the same order that the source sent them Clients may still get different total message order Sender 1 Sender 2 A 2 B 2 A 1 B 1 Spread Network B 2 A 2 B 1 A 2 B 2 B 1 A 1 A 1 Client 1 Client 2

9 Spread Guarantees (3) CAUSAL_MESS Messages are delivered reliably Messages arrive in a causal ordering General idea: If the receiving of one message could have caused the sending of another message, then they are received in that order by all clients. Example: Message M 1 is received by a client, and then the client sends message M 2. Other clients that receive both M 1 and M 2 receive M 1 first. Only specifies a partial ordering, not a total order over all messages

10 Spread Guarantees (4) AGREED_MESS Specifies a total order of all messages, over all senders All receivers get messages in exact same order SAFE_MESS Same ordering guarantee as AGREED_MESS Spread doesn t deliver message to recipients until every single daemon is ready to deliver message Idea: Make sure every member gets this message! If membership changes occur, Spread provides appropriate synchronization messages

11 Spread API Spread API is written for C programming Can use from C++, but no exceptions, classes, etc. Errors indicated by an int return code Value less than 0 always indicates an error A value of 0 sometimes means success Some functions return number of bytes sent on success (normally > 0) For error situations: void SP_error(int error) Prints error description to stdout (probably using a printf family function) Can t really incorporate into C++ stream IO and cout/cerr

12 Spread Types Group names specified as char* Maximum length: MAX_GROUP_NAME Can use string literals to specify group names e.g. "chat-group" Or, use std::string, and c_str() function All communication operations require a mailbox Provides communication context for Spread API Spread provides a mailbox type (in Spread 3, it s an int)

13 Connecting Use SP_connect to connect to a Spread daemon int SP_connect(const char *spread_name, const char *private_name, int priority, int group_membership, mailbox *mbox, char *private_group) spread_name specifies connection details for a Spread daemon Valid forms: 4803 Port on local machine 4803@localhost Same (Windows friendly) 4803@host.domain.tld Port on a remote machine 4803@ Same

14 Connecting (2) int SP_connect(const char *spread_name, const char *private_name, int priority, int group_membership, mailbox *mbox, char *private_group) private_name specifies name of local connection Must be unique on the machine! Can pass in NULL (or 0) Spread daemon will assign a random, unique name to connection priority specifies low (0) or high (1) priority Currently ignored; pass in 0. group_membership indicates whether client wants to receive membership-change messages 0 = no, 1 = yes

15 Connecting (3) int SP_connect(const char *spread_name, const char *private_name, int priority, int group_membership, mailbox *mbox, char *private_group) mbox and private_group are out-parameters Pass in pointers to variables to receive the values On successful connect, Spread sets these to valid values mbox is the mailbox to use for Spread operations private_group is the name of the connection s private group, for use with point-to-point communication

16 Example Spread Connect Code // Hard-coded connection info (not the best) const char * SpreadName = "4803@localhost"; // Variables to receive output from Spread mailbox mbox; char priv_group_name[max_group_name]; // No private name; priority = 0; recv group msgs = 1 rc = SP_connect(SpreadName, 0, 0, 1, &mbox, priv_group_name); if (rc!= ACCEPT_SESSION) { // Might be better to throw some kind of exception SP_error(rc); return -1; } cout << "Connected. Private group name = \"" << priv_group_name << "\"." << endl;

17 Spread Connection Details Each connection will have a mailbox and a private group name Make a struct/class to manage this info Can easily pass Spread connection info to various functions Don t enable membership messages unless you really want/need them If you don t need them, you will have to receive (and ignore) them

18 Disconnection Disconnection is much simpler: int SP_disconnect(mailbox mbox) Pass in mailbox, and that s that. Check return code for errors, of course. Only real error in this situation is invalid mailbox Check anyway make debugging easier down the road!

19 Joining and Leaving Groups To join a group: int SP_join(mailbox mbox, const char *group) Group s name is a string Length shorter than MAX_GROUP_NAME (32 chars, but don t bank on that ) Terminated by a 0, like all good C strings Can include letters, numbers, and punctuation from the set -/_.\ (specifically not #!) If group doesn t already exist, it is created To leave a group: int SP_leave(mailbox mbox, const char *group) If group doesn t exist, this is a no-op.

20 Sending Messages Can send messages to one group, or to several groups SP_multicast SP_multigroup_multicast In memory, message can be contiguous or fragmented SP_scat_multicast SP_multigroup_scat_multicast Functions are all similar in capability

21 Sending Messages (2) Normal send message function: int SP_multicast(mailbox mbox, service service_type, const char *group, int16 mess_type, int mess_len, const char *mess) service_type specifies delivery guarantees Can be set to: UNRELIABLE_MESS RELIABLE_MESS FIFO_MESS CAUSAL_MESS AGREED_MESS SAFE_MESS Can also bitwise-or in SELF_DISCARD, if desired: RELIABLE_MESS SELF_DISCARD Otherwise, a client also receives its own messages

22 Sending Messages (3) int SP_multicast(mailbox mbox, service service_type, const char *group, int16 mess_type, int mess_len, const char *mess) group specifies group to send the message to Must be less than MAX_GROUP_NAME bytes in length If sender is in group, they receive the message too, unless SELF_DISCARD was specified mess_type is an application-defined message type value 16-bit value: to Keeps applications from having to look into message body to determine message s type very useful!

23 Sending Messages (4) int SP_multicast(mailbox mbox, service service_type, const char *group, int16 mess_type, int mess_len, const char *mess) mess_len specifies message s length Usually, messages must be less than a few hundred KB Probably want to stay under a few dozen KB max! mess is the actual message data! Whatever you decide to send

24 Receiving Messages Normal receive message function: int SP_receive(mailbox mbox, service *service_type, char sender[max_group_name], int max_groups, int *num_groups, char groups[][max_group_name], int16 *mess_type, int *endian_mismatch, int max_mess_len, char *mess) Receives both application messages and membershipchange messages unless client specifies no membership messages to SP_connect, of course Most parameters are out-parameters! SP_receive is a blocking call won t return until a message was received, or an error occurs Also a scatter-receive version: SP_scat_receive

25 Receiving Messages (2) int SP_receive(mailbox mbox, service *service_type, char sender[max_group_name], int max_groups, int *num_groups, char groups[][max_group_name], int16 *mess_type, int *endian_mismatch, int max_mess_len, char *mess) Return value: Value < 0 indicates an error On success, result is number of bytes received Meaning of arguments depends on whether message is from application, or a membershipchange message See API documentation for all the details

26 Receiving Messages (3) int SP_receive(mailbox mbox, service *service_type, char sender[max_group_name], int max_groups, int *num_groups, char groups[][max_group_name], int16 *mess_type, int *endian_mismatch, int max_mess_len, char *mess) service_type recieves the kind of message REGULAR_MESS and MEMBERSHIP_MESS constants are used to tell what kind of message These are bit-fields; need bitwise logic operations to check For application messages, sender receives private group name of sender Meaning is different for membership messages

27 Service-Type Value Different values are represented with different bit patterns: UNRELIABLE_MESS 0x RELIABLE_MESS 0x FIFO_MESS 0x CAUSAL_MESS 0x AGREED_MESS 0x SAFE_MESS 0x REGULAR_MESS 0x f REG_MEMB_MESS 0x TRANSITION_MESS 0x CAUSED_BY_JOIN 0x CAUSED_BY_LEAVE 0x CAUSED_BY_DISCONNECT 0x CAUSED_BY_NETWORK 0x MEMBERSHIP_MESS 0x00003f00

28 Service-Type Value (2) Use bitwise Boolean operators to check service_type value (service_type & REGULAR_MESS) is nonzero when the message is an application message (service_type & MEMBERSHIP_MESS) is nonzero when the message is a membership message Or, use macros provided by Spread API Is_regular_mess(service_type) Is_membership_mess(service_type) Is_unreliable_mess(service_type) Is_reliable_mess(service_type) etc.

29 Receiving Messages (4) For application messages: int SP_receive(mailbox mbox, service *service_type, char sender[max_group_name], int max_groups, int *num_groups, char groups[][max_group_name], int16 *mess_type, int *endian_mismatch, int max_mess_len, char *mess) max_groups, num_groups, and groups array report what groups message was sent to Meaning changes for membership messages! mess_type receives app-specific message type mess points to buffer that receives message data Specify max_mess_len correctly to avoid buffer overflows!

30 Insufficient Buffer Space If message is larger than provided buffer, Spread won t deliver the message Some APIs simply truncate the message down to the buffer size, so this is nicer Also, if groups array doesn t have enough entries for all groups, Spread won t deliver the message In these cases, Spread will report how much space is actually needed See API documentation for details Can allocate more space, then retry the receive operation To force Spread to truncate message/group data Set service_type = DROP_RECV before calling SP_receive function (usually not recommended)

31 Endian Issues SP_receive also provides endian_mismatch property Spread sets this value based on sender and receiver endian-ness 1 = sender stores words in a different byte-order than receiver 0 = sender stores words in the same byte-order as receiver Spread ensures that mess_type is converted properly Message contents may not be converted properly! Spread doesn t know what your message format is Have to do this yourself See htons, htonl, ntohs, ntohl functions, for example

32 Example Spread Receive Code service svc_type; char sender[max_group_name]; int max_groups = 16; // Max groups we have space for char groups[16][max_group_name]; int num_groups; // Number of actual groups int16 msg_type; int endian_mismatch; int max_msg_length = 1024; char message[1024]; svc_type = 0; // or DROP_RECV to truncate rc = SP_receive(mbox, &svc_type, sender, max_groups, &num_groups, groups, &msg_type, &endian_mismatch, max_msg_length, message);

33 Making Spread C++ Friendly Some suggestions for making Spread easier to work with in C++ Make a spread_error exception type that derives from runtime_error Just like POSIX threading exception type Include return-code value for exceptions Make a struct to hold Spread connection info Mailbox and private group name Could also turn this into a full-fledged class Constructor connects, destructor disconnects, etc.

34 Making Spread C++ Friendly (2) Create a class to wrap Spread messages Include both message data, and list of groups Provide functions to read and write primitives perhaps in network byte-order to avoid endian issues Dynamically grow the buffer as needed Write wrappers for both SP_multicast and SP_receive that use the message class Avoid all the crazy in/out parameters Can even handle insufficient space issues by growing the necessary buffer, then retrying

35 This Week s Lab Create a simple echo-server using Spread Your program connects to a Spread daemon and joins a well known (i.e. hard-coded) group Program can be started in one of two modes: In send mode, your program requests user input, then sends typed messages to Spread group In receive mode, your program receives the Spread messages and prints them out When user types quit the whole thing stops Spread Toolkit is installed on the CS cluster Lab write-up tells how to build your program to use Spread, how to start Spread running, etc.