Twitter Adaptation Layer Submitted for Drexel University s CS544

Transcription

1 Twitter Adaptation Layer Submitted for Drexel University s CS544 Josh Datko 9 June Description of Service The Twitter Adaptation Layer (TWAL) provides connected, best-effort-end-to-end delivery of tweets with limited flow and error control. It provides logical sockets over a broadcast medium (Twitter) to simulate unicast communication. While theoretically flexible to allow various higher protocols, the TWAL is customized for use for the Hangman protocol. Typically, Twitter is a broadcast medium. Tweets are posted to a users timeline via a RESTful HTTP POST and once accepted by the Twitter server, it is added to the user s timeline. Tweets at this point are not directed at anybody in particular, but rather they broadcast to the Twitterverse. However, by using mentions, that tweet is meant for the mentioned user, even though it is still broadcast. By adding mentions to tweet, destination addressing is achieved. Source addressing (who sent the tweet) is embedded in the metadata of the tweet itself. The extra credit is described as the discovery protocol in Section which uses the message format in Section Quality of Service (QoS) QoS: Connected As entities progress through the DFA they will exchange messages to each other to establish the OPEN state. In this state, there is assurance that the messages are being sent to the receiving party since each party has participated in a connection handshake QoS: Best-effort-end-to-end-delivery TWAL is simliar to SMTP in this regard; it can guarantee delivery to the mail exchanger, but it can t guarantee receipt by the end user. Tweets are posted to Twitter through a REST interface that acknowledges receipt of the posted tweet to the user s timeline. Therefore, delivery to twitter is guaranteed. Also, if the tweet contains a mention during the ESTABLISHED or OPEN state, it will show up in the other users timeline, but there is no guarantee that the remote end will consume the tweet. Therefore, it is best-effort. Just like SMTP can t guarantee the user read his . If a socket in TCP network programming is defined by {srcport,srcaddr,dstport,dstaddr}, then the TWAL socket is defined as {srcusername,dstusername}. Delivery of data is provided by Twitter and to continue with the SMTP example, a user s timeline is like their mailbox. When in the OPEN state, tweets with the mention will show in the destination timeline and if they contain the tag, see Section 2.5, the tweet will have unique addressing to differentiate it from another conversation to support concurrent conversations. 1

2 1.1.3 QoS: Flow Control and Ordering Twitter enforces rate limiting based on its API calls. Posting status updates is limited to 1,000 tweets per day [1], which is broken down into hourly intervals. Therefore the TWAL is meant to service relatively low, short-lived conversations. For the Hangman Protocol, the server can t maintain continuos conversations for very long with numerous clients, since the Hangman protocol is a proof-of-concept, we considered this limitation acceptable. Any GET method using the REST API, when authenticated with OAuth, is limited to 350 requests per hour [3]. Through a combination of the three different twitter APIs (REST, Streaming and Search), this limit can be worked-around. If only using the REST API, an entity can t received faster than once every 12 seconds (about) for a steady basis, unless using the streaming API, where it is unlimited. The bottom line is that Twitter itself enforces the flow control and provides HTTP error codes to inform the account when its near / over the rate limits. Also, tweets are given 64 bit unsigned integer identifiers that are no-longer sequential, but are based on time [6]. Therefore, tweets relatively close to each other are sequential, but there is no guarantee that all tweets will maintain this straight-sequential ordering. Combined with the timestamp however, order can be determined. Twitter stores tweets in a user s timeline in like a list where newest tweets are inserted into the head of the list. Therefore, if the sender is tweeting more than the consumer is consuming, tweets are stored by twitter and can be retrieved. Tweets are never dropped, unless, of course Twitter is down and the fail whale is flying. In this case, senders and receiving will be aware of the outage on the next send / receive, when they try to reach twitter. Simliar to pulling the plug on ethernet when using TCP with the exception then even the receiving entity will know immediately and won t have to wait for a timeout QoS: Preventing Duplicate Messages Twitter prevents duplicate tweets. Unfortunately, what it considers duplicate is a bit vague. Definitely, two exacts tweets in are row are considered duplicate and the second produces an error. However, it may consider duplicate a tweet in the last x seconds, where x doesn t appear to be clearly defined. Twitter seems to have purposely obfuscated these details because they are used for spam detection and providing such heuristics would allow for bypassing spam detection. The TWAL does not sending duplicate messages because it is not allowed by Twitter and each message is acknowledged when posted to Twitter QoS: Error Control Error control is provided between the sending account and Twitter since Twitter will respond with the sender s message. At this point, the tweet has been posted, but it provides a final level check among the other protocols that twitter uses (HTTP and REST). There is no end-to-end error control. 1.2 TWAL Stack and Sublayering The TWAL is further internally layered. Besides the main TWAL DFA, it contains a PUSH, PULL and DISCOVERY protocol. Figure 1 shows the internal layers as well as relations to the Hangman Protocol and to Twitter PUSH Protocol The PUSH protocol provides the message sending service. Outgoing messages and handled to the PUSH protocol, which makes the appropriate Twitter POST message to the Twitter service. Twitter provides an acknowledgment, although this is not shown in the diagram. The PUSH protocol does not provide extra rate limiting, but relies on error messages from Twitter to inform the TWAL DFA when it has exceeded a rate limit. This could be improved to monitoring rate limiting prior to a Twitter warning, but was beyond the scope of this project. Lastly, the PUSH protocol uses the Twitter REST API [4] which performs a HTTP POST to the Twitter service, and receives a response on the POST. 2

3 Figure 1: TWAL Stack PULL Protocol The TWAL DFA polls the PULL protocol to receive message and provides the receive service to the TWAL DFA. However, the PULL protocol is actually consuming tweets with two separate Twitter APIs. First, it polls the Twitter service with the REST API. It makes a HTTP GET request every 11.5 seconds to see if there are new status updates for this account. The time is to ensure that Twitter rate limits are not violated (350 queries per hour)[3]. It also uses the Twitter Streaming API [5] to receive tweets. The Streaming API is a different technique to receive tweets by making a persistent HTTP connection and being pushed messages from the Twitter server. This results in near instantaneous tweet retrieval and does not count against the rate limits. However, Twitter describes this service as unreliable, specifically duplicate tweets can be received among other things. By consuming both APIs, near instantaneous receives can occur and reliable messaging can be reconciled every 11.5 seconds Discovery Protocol The Discovery protocol provides a server discovery service to the Hangman protocol. It uses the Twitter Search API [7] to perform real-time searches. The Search API is rate-limited to 150 requests per hour, therefore the discovery protocol checks every 30 seconds. This protocol searches the global hashtag #Hangman. It then examines retrieved tweets and attempts to match a valid announce message to detect a Hangman server. The message is further discussed in Section 2.6. Once detected, it presents the server account name to the Hangman Protocol when prompted. It is a passive discovery service, it does not attempt to contact the server during the discovery phase. Since discovery does not guarantee connection, the discovery services provides the name of a potential server and relies on the TWAL DFA to connect to the server and see if it is online. The Discovery Protocol grabs the last 15 tweets with the given hashtag. Depending on how #Hangman is trending, this could be tweets that are days old. Anecdotal observations showed that 15 tweets with 3

4 this hashtag retrieved tweets 6 hours old. Servers should announce their presence using the discovery protocol using the Message in Section 2.6 on startup. A server may choose not to announce itself on startup, if for example, it has recently aready announced itself. If the server will run continuously, it should announce it s presence every hour to facilitate discovery. Clients using the discovery protocol must search for the announce message described in Section 2.6. Clients must only contact the server during the TWAL DFA states and not during discovery. Servers should ignore messages in reply to the announce message. 2 Messages The following messages are defined in TWAL: 2.1 Following The following message is defined by the Twitter service and is beyond the scope of this specification. The TWAL sends the follow message through a twitter API to another account. By following a user, the followed user s status updates appear in the following user s timeline. 2.2 Hello The hello tweet contains the string: ^hello^ and must contain a TWAL tag described in Section 2.5, as Figure 2 does. The basis of this is to allow an upper layer protocol, which may be a human typing on a twitter client, a straightforward hello message. This may cause confusion if that exact string is used in the payload of the higher protocol, in that case, that upper layer protocol must handle that situation. The hello message should contain other data to make the tweet unique to prevent it from being blocked as a duplicate, see Section However, considering that a human could use the TWAL, there are no restrictions. The TWAL for the Hangman protocol uses a date string with a tag described in Section 2.5. ^hello^ Sun Jun 03 21:28:04 EDT 2470/t> Figure 2: Example Hello message 2.3 Bye Simliar to the Hello message in Section 2.2, the bye message is contains the string ^bye^ and must contain a TWAL tag, as Figure 3 shows. ^bye^ Sun Jun 03 21:31:51 EDT 1048/t> Figure 3: Example Bye message 2.4 Data Transfer The data transfer message is not a hello, bye, follow or announce message and must contain a tag. It is a carrier for a higher layer PDU. It will only be recognized in the OPEN state. 2.5 Tag All TWAL messages contain a tag. The TWAL tag is defined by the regular expression: <t.*?/t>. The tag must mention the destination username, with the exception of the announcement message described in Section 2.6. Additionally, the tag should contain characters to ensure that the tweet does not get 4

5 flagged as a duplicate as Section indicates. For the Hangman protocol, it contains a one-up sequence number initialized at a random number between 0 and This number is not used as a traditional sequence number, since the Twitter service provides ordering as Section describes, but is useful in visual debugging to see the messages of the conversation. The tag has a maximum length of 40 characters, leaving 100 characters for the payload data. The maximum length of a Twitter username is 15 characters. The tag format, based on the above regular expression, has a length of 6 characters. The mention symbol consumes one character. The remaining 18 characters are for unique data strings to prevent tweets from being flagged as duplicates and future options. 4378/t> Figure 4: Example Message with Tag 2.6 Announce The announce message is used to broadcast the establishment of a Hangman server. This message must contain: The hashtag #hangman The keyword ^discover^ A TWAL tag defined in Section 2.5 Also, the announce message should contain random, unique characters to prevent Twitter from flagging the tweet as a duplicate. An example announce message is shown in Figure 5. #hangman^discover^ Sun Jun 03 22:34:47 EDT 2012<t ANNOUNCE/t> Figure 5: Example Announcement Message 3 Deterministic Finite Automata (DFA) The DFA for the TWAL is straightforward. Starting out in the DEAD state, the conversation moves to the ESTABLISHED state only when both parties follow each other. The client follows the server then the server follows the client. Mutual following is required to ensure that the other users tweets show up correctly in the timeline. It is not necessary to always unfollow and transition to the DEAD state, in fact, Twitter could black list an account for aggressive following: following and unfollowing the same user repeatedly[2]. So, clients should not unfollow at the end of each session. The client sends a HELLO message to the server and waits for a HELLO message back. Upon receipt, the client transitions to the OPEN state. The server transitions once it sends a HELLO message to the client. The conversations stays in the OPEN as long as it is sending and receiving data transfer messages, as defined Section 2.4. Either party sending the BYE message will move the state back to ESTABLISHED and unfollow will send it back to DEAD. 4 Extensibility The TWAL provides a flexible payload format and tag definition based on regular expressions, therefore additional options can be added later. Additionally, extra characters have been left in the tag for future expansion. The TWAL was designed to specifically support the Hangman Protocol, but could be further refined to be protocol agnostic with future work. 5

6 Figure 6: Twitter Adaptation Layer DFA 5 Security The TWAL provides limited security as outlined below and it assumes that Twitter is cooperative (nonmalicious) intermediately. The TWAL should not be used to transport sensitive data. As the following sections show, the TWAL is in general, not very secure. 5.1 Authentication End-to-end message authenticity is supported in TWAL. Since a Twitter account is required to POST messages, and all tweets are stamped with the sender account (authenticated with OAuth). Assuming that Twitter is not malicious and does not manipulate the tweet, the receiver has reasonable assurance that the sender is the real sending account. 5.2 Access Control There are no specific access controls enforced by the TWAL, however once in the OPEN state, higher layer protocols may provide additional policies. 5.3 Data Integrity The TWAL provides data integrity as far as POSTing to Twitter, since twitter responds with the posted tweet. Even this is a limited check on data integrity, since the tweet is already posted. There is no end-to-end data integrity. 5.4 Confidentiality TWAL currently provides zero confidentiality. Tweets are public and broadcast to the Twitterverse. 6

7 5.5 Availability Unfortunately, TWAL is susceptible to Denial of Service attacks. Due to Twitter rate limiting, numerous accounts could perform a distributed denial of service attack by connecting to one server, which would in turn tweet each one. The server would quickly exceed it s rate limits and would effectively be silenced. 5.6 Non-Repudiation As a result of Twitters no duplicate policies, a limited degree of non-repudiation is enforce. However, this non-repudiation is ephemeral, since after a short period time the tweet can be duplicated. Also, Twitter does not keep tweets forever (after a certain number). 5.7 Trust The foundation of trust in TWAL is trusting the Twitter service. Twitter is provides the limited security features described above and is trusted to actually transport the data. 6 Definitions Tweet: Also known as a status update. This is a string of at most 140 UTF8 characters. Mention: A tweet that contains a twitter username preceded by the References [1] Twitter. About twitter limits (update, api, dm, and following). articles/15364-about-twitter-limits-update-api-dm-and-following, June [2] Twitter. Following rules and best practices following-rules-and-best-practices, June [3] Twitter. Rate limiting. June [4] Twitter. Rest api resources. June [5] Twitter. The streaming apis. May [6] Twitter. Twitter ids, json and snowflake. June [7] Twitter. Using the twitter search api. June