Coupling Page 1 Coupling Thursday, October 21, 2004 3:23 PM Two kinds of multiple-processor systems Tightly-coupled Can share efficient semaphores. Usually involve some form of shared memory. Loosely-coupled Processes use processors with independent memory. Semaphores are slow.
Message-passing One process "sends" a message to another. Two primitives "send" and "recv". Send: send a message to a process or group of processes. Recv: receive a message from a process. "mailbox": buffer that can be sent to or received from. Primitives Create mailbox Delete mailbox Send to mailbox Receive from mailbox Coupling Page 2
Coupling Page 3
Coupling Page 4
Coupling Page 5
Coupling Page 6
Coupling Page 7
Coupling Page 8 Time has no meaning in a distributed system: clock drift means that no clocks will agree. No sense of time, only precedence.
Coupling Page 9 Multiple sends Mailboxes must be queues, or sends will block.
Coupling Page 10 Synchronization: "Brain-dead sync": semaphore is reactive process Notes: Semaphore is single-threaded daemon Queues requests; acts in order of receipt Requires queue for strong behavior
Coupling Page 11 Aside: we are not far from creating a database server here. basic thing that a database server provides: atomicity over distributed operations. serialization: queues into some (random) order, plays back requests in serial stream. only difference between lock server and db server is complexity of queries.
Coupling Page 12 Network Semaphore Properties Queues (serialized FIFO) Slow (waits for ACK) Lots of communications.
Coupling Page 13 Good and bad news Good news: can construct semaphores from message-passing. Bad news: it's reallllly slow.
Coupling Page 14 Kinds of message-passing Blocking send and receive Sender doesn't block, receiver blocks. (normal) (annoying: sends are asynchronous, process won't die until it's done with them) Neither blocks
Coupling Page 15 Synchronization Thursday, October 21, 2004 3:26 PM Generally, no form of "tight" synchronization for processes in a loosly coupled environment. Clocks don't agree. Messages take unknown time to travel. New "loose" form of synchronization: "barriers". A group of processes waits till all are in desirable states. Implementable via message-passing.
Key to synchronization: A message receipt is a synchronization event. It says that the process that sent the message is now in a specific state. Coupling Page 16
Example of barrier synchronization Coupling Page 17
Optimal barrier sync (lowest network overhead without broadcast) Arrange nodes into a tree "collect" responses from children One node (root) gets all responses Root forwards responses to all descendants Coupling Page 18
Coupling Page 19 Network resource sharing Thursday, October 21, 2004 3:29 PM Network resource sharing Locking daemons Distributed voting
Coupling Page 20 Locking daemons Thursday, October 21, 2004 3:30 PM Locking daemons Create an artificial domain of atomicity by creating a resource for which one must wait. Implicitly serialize resource requests. If applications do not proceed until they are granted a lock, then their operations are atomic (within the set of cooperating processes sharing a lockable resource). Caveat: participation is voluntary.
Coupling Page 21 Common daemons with atomicity properties Database servers. Object servers ("web services") File servers (e.g., NFS lockd) Common daemons without atomicity properties Web servers Session servers (e.g., sshd)
Coupling Page 22
Coupling Page 23 File sharing Thursday, October 21, 2004 3:43 PM Problem of file sharing: what is exclusive access? int fd = open("/g/111/class/a2/t2.o", O_EXCL O_CREAT O_WRONLY); // exclusive write O_WRONLY: open for writing O_CREAT: create if not present O_EXCL: fail if other processes have it open. Same problems all over again: Readers/writers races. Deadlock waiting for resources to be allocated. Catch: this file cannot be local. It's on a file server (netapp01) how to make this "predictable". what's reasonable?
Coupling Page 24 Local domain: global knowledge of file descriptors Can serialize writes in the kernel. Network file system: No global knowledge of file descriptors. No good knowledge of race competitors. asynchronous No way for kernel to serialize creat is not atomic without another mechanism to assure it.
Coupling Page 25
Coupling Page 26 Summary Monday, October 24, 2005 4:36 PM Kernel serializes local requests On network, no concept of serial kernel => no intrinsic ability to serialize. Relying upon locking daemons (essentially glorified semaphores in network domain) provides serialization that kernel provided before.