Pipes, connections, channels and multiplexors

Transcription

1 Pipes, connections, channels and multiplexos Fancisco J. Ballesteos ABSTRACT Channels in the style of CSP ae a poeful abstaction. The ae close to pipes and connections used to inteconnect system and netok components to build distibuted systems, but they ae not quite the same thing. this pape e descibe changes made to Go channels and tools built upon those to povide system and netok-ide sevices fo a ne OS unde constuction Channels and pipes A channel is an atifact that can be used to send (typed) data though it. The Go language opeations on channels include sending, eceiving, and selection among a set of send and eceive opeations. Go pemits also to close a channel afte the last item has been sent. On most systems, pocesses and applications talk though pipes, netok connections, FIFOS, and elated atifacts. shot, they ae just file desciptos once open, pemit the application to ite data (fo sending) and/o ead data (fo eceiving). Some of these ae duplex, but they can be consideed to be a pai of devices (one fo each diection). hat follos e ill efe to all these atifacts as pipes (e.g., a netok connection may be consideed as a pai of simplex pipes). Thee is a mismatch beteen channels and pipes and this pape shos hat e did to ty to bidge the gap beteen both abstactions fo a ne system. The aim is to let applications leveage the CSP style of pogamming hile, at the same time, let them ok acoss the netok. We assume that the eade is familia ith channels in the Go language, and descibes only ou modifications and additions. Close and eos When using pipes, each end of the pipe may close and the pipe implementation takes cae of popagating the eo to the othe end. That is not the case ith standad Go channels. Futhemoe, upon eos, it is desiable fo one end of the pipe to lean about the eo that did happen at the othe end. We have modified the standad Go implementation to: 1 Accept an optional eo agument to close. 2 Make the send opeation etun false hen used on a closed channel (instead of panicing; the eceive opeation aleady behaves nicely the case of a closed channel). 3 Povide a pimitive, ceo, that etuns the eo given hen the channel as closed. 4 Make a close of an aleady closed channel a no-opeation (instead of a panic). With this modified tool in hand, it is feasible to ite the folloing code:

2 - 2 - va inc, outc chan[]byte... fo data := ange inc { ndata := modify(data) if ok := outc <-ndata;!ok { close(inc, ceo(outc)) beak close(outc, ceo(inc)) Hee, a pocess consumes data fom inc and poduces ne data though outc fo anothe one. The image to have in mind is poc1 inc poc2 outc poc3 hee the code shon coesponds to the middle pocess. Pehaps the fist pocess teminates nomally (o abnomally), in hich case it ould close inc. this case, ou code closes outc as expected. But this time, the eo given by the fist pocess is knon to the second pocess, and it can even foad such eo to the thid one. A moe inteesting case is hen the thid pocess decides to cease consuming data fom outc and calls close. No, ou middle pocess ill notice that ok becomes false hen it ties to send moe data, and can beak its loop cleanly, closing also the input channel to singal to the fist pocess that thee is no point in poducing futhe data. this second example, the last call to close is a no-opeation because the output channel as aleady closed, and e don t need to add unnecessay code to pevent the call. The impotant point is that temination of the data steam is easy to handle fo the pogam ithout esoting to exceptions (o panics), and e kno hich one is the eo, so e can take hateve measues ae convenient in that case. Channels and pipes Thee ae thee big diffeences beteen channels and pipes (e ae using pipe to efe to any file descipto used to convey data, as stated befoe). One is that pipes may have eos hen sending o eceiving, but channels do not. Anothe one is that pipes cay ony steams of bytes and not sepaate messages. Yet anothe is that channels convey a data type but pipes convey just bytes. The fist diffeence is mostly dealt ith the changes made to channels as descibed in the pevious section. That is, channels may have eos hile sending and o eceiving, consideed the changes made. Theefoe, the code using a channel must conside eos in vey much the same ay it ould do if using a pipe. To addess the thid diffeence e ae going to conside channels of byte aays by no. The second diffeence can be dealt ith by ensuing that applications using channels to speak though a pipe peseve message boundaies ithin the pipe. With this in mind, a ne nchan package povides ne channel tools to bidge the gap beteen the channel and the pipe domains. The folloing function ites each message eceived fom c into as it aives. If peseves message boundaies, that is enough. The second function is its contepat. func WiteBytesTo( io.wite, c <-chan []byte) (int64, eo) func ReadBytesFom( io.reade, c chan<- []byte) (int64, eo) Hoeve, is most cases, the tanspot does not peseve message boundaies. Thus, the next function ites all messages eceived fom c into, but pecedes each such ite ith a heade indicating the message length. The second function can ely on this to ead one message at a time and foad it to the given channel.

3 - 3 - func WiteMsgsTo( io.wite, c <-chan []byte) (int64, eo) func ReadMsgsFom( io.reade, c chan<- []byte) (int64, eo) One inteesting featue of WiteMsgsTo and ReadMsgsFom is that hen the channel is closed, its eo status is checked out and foaded though the pipe. The othe end notices that the message is an eo indication and closes the channel ith said eo. Thus, code like the except shon fo ou middle pocess in the steam of the pocesses ould ok coectly even if the input channel comes fom a pipe and not fom a anothe pocess ithin the same pogam. ections The nchan package defines a connection as type stuct { Tag sting // debug <-chan []byte chan<- []byte This joins to channels to make a full-duplex connection. A pocess talking to an extenal entity elies on this stuctue to bidge the system pipe used to a pai of channels. Thee ae utilies that leveage the functions descibed in the pevious section and build a channel inteface to extenal pipes, fo example: func Ne( io.readwiteclose, nbuf int, in, out chan bool) The function ceates pocesses to feed and dain the connection channels fom and to the extenal pipe. Futhemoe, if suppots closing only fo eading o iting, a close on the input o output channels ould close the espective halves of the pipe. Because of the message potocol explained in the pevious section, eos ae also popagated acoss the extenal pipe and the pocess using the connection can vey much ignoe that the souce/sink of data is extenal. It is easy to build pipes hee the channel sends elements though the channel: func NePipe(nbuf int) And, using this, e can ceate in-memoy connections that do not leave the pocess space: func NePipe(nbuf int) (, ) This has been vey useful duing testing, because this connection can be ceated ith no buffeing and it is easie to spot dead-locks that involve both ends of the connection. Once the pogam is eady, e can eplace the connection based pipe ith an actual system povided pipe. Multiplexos Upon the channel based connections shon in the pevious sections, the nchan package povides multiplexos. type Mux stuct { chan... func NeMux(c, iscalle bool) *Mux func (m *Mux) Close(e eo) func (m *Mux) () chan<- []byte func (m *Mux) Rpc() (outc chan<- []byte, epc <-chan []byte) A pogam speaking a potocol usually ceates a ne connection by dialing o accepting connections and then ceates a Mux by calling NeMux to multiplex the connection among multiple equests.

4 - 4 - Mux pipe The nice thing of the multiplexed connection is that equests may cay a seies of messages (and not just one message pe equest) and may o not have eplies. Replies may also be a full seies of messages. Both ends of a multiplexed connetion (the pocess using the mux and its pee at the othe end of the pipe) may issue equests. Thus, this is not a client-seve inteaction model, although it may be used as such. To issue ne outgoing equests though the multiplexo, the pocess calls (to issue equests ith no expected eply): oc := mux.() oc <- []byte("no eply") oc <- []byte("expected") close(oc) O the pocess may call Rpc (to issue equests ith an expected eply). c, := mux.rpc() c <- []byte("anothe") c <- []byte("equest") close(c) fo m := ange { Pintf("got %v as pat of the eply\n", m) Pintf("and the final eo status is %v\n", ceo()) the fist case, the multiplexo etuns a to the calle ith just the channel. Of couse, this can be done multiple times to issue seveal concuent outgoing equests: Mux pipe the figue, the to connections of the left ee built by to calls to mux.(), hich etuns a ith an chan to issue equests. The pocess using the channel may issue as many messages as desied and then close the channel. If the equest depicted belo equies a eply, mux.rpc() is be called finstead of mux.() and the esulting pictue is as shon. Mux pipe The impotant pat is that messages (and eplies) sent as pat of a equest (o eply) may be steamed ithout affecting othe equests and eplies, othe than by the usage of the undelying connection. That is, an

5 - 5 - idle steam does not block othe steams. The inteface fo the eceiving pat of the multiplexo is a single channel that conveys one pe incoming equest. The equest has only the channel if no eply is expected, and has both the and channels set if a eply is expected. Mux pipe To eceive equests fom the othe end of the pipe, the code might look like this: fo call := ange mux. { // call is a fo m := ange call. { Pintf("got %v as pat of the equest\n", m) if call.!= nil { call. <- []byte("a eply") call. <- []byte("as expected, but...") close(call., "Oops!, failed") Fo example, if a pocess eceived to equests, one ith no eply expected and anothe ith a eply expected, the pictue ould be: Mux pipe Hee, the to connections on the left epesent equests that ee eceived though the channel depicted on top of the multiplexo. The impotant thing to note is that pocesses may no issue steams of equests, o eplies, though channels and they ae fed to extenal pipes (o fom them) as equied. The intefaces shon have geatly simplified pogamming fo (netoked) system seviced being itten fo the ne system. Acknoledgements We ae vey gateful to Roge Peppe and Chales Fosyth fo thei insights and help.