Internet Connectivity with Introduction The purpose of this workshop is to help you g et acquainted with the basics of internet connectivity by leveraging ARM mbed tools. If you are not already familiar with these tools check out the Getting Started with mbed doc. In this workshop, we will approach internet connectivity from a product development perspective. To this end we will cover the existing standards, how TCP/UDP work, and how to make the most of an internet-enabled system. The prerequisites for this class are having an mbed enabled board with an internet capable interface (Ethernet, WiFi, etc) and an internet connection. The slides that accompany this document can be found here https://goo.gl/su5lvh. Here is a permalink to this document in case you are viewing it offline: https://goo.gl/waaw6u Table of Contents: Introduction Network Layers Berkley Sockets Types of Sockets UDP UDP Example Server Code: Client code: TCP TCP Example Server code: Client code: Protocols HTTP What is it? Benefits: Issues: Example: GET Request Request Response Websockets What is it? Benefits: Issues:
Network Layers The entirety of the internet is based on the OSI seven layer networking model. This model specifies how the internet works, from the highest application level (7) with a web application to the lowest level (1) where the bits are twiddles across at the physical level. In general the seven layers can be simplified as below. Layers 1 4 will get lumped together into the Data Transport Layer. This part of the model is comprised of drivers for the device and is either hard coded in hardware or provided by the vendor of the device. As a developer you should almost never have to worry about this space. If you do wind up in this area it will usually be to port a pre existing driver with minimal changes. Levels 5 7 are the Application Layers where web servers, web pages, applications and all the use cases for technology live.
Berkley Sockets The Berkeley Socket API is an abstraction layer that sits between layers four and five and acts as a standard interface between them. This allows for the application layer and the data transport layer to be modular, in turn allowing high level applications to be written without needing to know how to use Ethernet, cellular or WiFi; each side interfaces to the Berkeley Socket API and doesn't care about what is on the other side. It just works. The Berkeley Socket abstraction has a few well defined function calls. Some functions are only used for TCP while others are only used for UDP. Some are only used by servers, some are only used by clients. The list below is not a full list of the Berkeley Socket API. For the full list see here. Accept() creates a socket for a new connection. Bind() associates a socket with a local endpoint. Close() closes a socket connection and releases any associated resources. Connect() establishes a connection to a remote host. Listen() allows a socket to listen for incoming connections. Send_to() sends data to an unbound socket. Send_all() sends data to a bound socket. Receive() receives data from an unbound socket. Receive_all() receives data from a bound socket and pushes the data into a buffer array.
Types of Sockets The Berkeley Socket Interface is designed to work with either Unix sockets or TCP/UDP sockets. Unix sockets are only used inside a computer for sharing information between threads or programs, so for the purposes of this document we will only be discussing TCP/UDP sockets that allow you to talk to remote resources (ie the internet). Both TCP and UDP have their own advantages and disadvantages depending on the type of application. UDP UDP is a stateless protocol, which means it does not remember what its last action was. It sends the packet and forgets about it no checking the packets for delivery or the order in which they arrive. This means that the UDP header is simple and the transaction is faster, but has no guarantee of delivery or data integrity. UDP is good for high throughput transfers like video streaming because you don t care if every single packet makes it there, a certain amount of packet loss is expected and corrected for on the other end with algorithms. In UDP, the transactions are as follows: 1. Server side: create socket and bind to port. 2. Server listens to bound socket for connection. 3. Client sends data to remote port on server. 4. Server receives data. 5. Server sends data back to client (optional). 6. Client receives data from server (optional, would need to bind to a socket to receive). 7. Close the socket.
A standard UDP transaction can be seen below: It is worth noting that if you want to receive data from a socket on a given port you must bind the socket to that port. This is why the server side had a bind; it would be necessary on the client as well if it wished to receive data. UDP Example This example demonstrates a server client connection over UDP. Server Code: The source code for this example can be found here: https://developer.mbed.org/teams/mbed/code/udpechoserver/ Start off by including the necessary header files and defining your server port: #include "mbed.h" #include "EthernetInterface.h" #define ECHO_SERVER_PORT 7 In the main program, declare objects for your Ethernet interface, then initialize and connect: int main ( void ) { EthernetInterface eth ;
eth. init (); //UseDHCP eth. connect (); printf ( "\nserveripaddressis%s\n", eth. getipaddress ()); Declare the UDP socket object and bind it to the previously declared server port. Declare an endpoint client: UDPSocket server ; server. bind ( ECHO_SERVER_PORT ); Endpoint client; Create a 256 byte character buffer and store the packets coming from the client using receivefrom socket method: char buffer [ 256 ]; while ( true ) { printf ( "\nwaitingforudppacket...\n" ); int n = server. receivefrom ( client, buffer, sizeof ( buffer )); buffer [ n ] = '\0'; Print out the contents of the packets until the transfer is complete: } } printf ( "Receivedpacketfrom:%s\n", client. get_address ()); printf ( "Packetcontents:'%s'\n", buffer ); printf ( "SendingPacketbacktoClient\n" ); server. sendto ( client, buffer, n ); To test this with a partner, have them load the client code to test your UDP connection. Else, you can load the python script below onto your PC: import socket ECHO_SERVER_ADDRESS = '10.2.131.195' ECHO_PORT = 7 sock = socket. socket ( socket. AF_INET, socket. SOCK_DGRAM ) sock. sendto ( "HelloWorld\n", ( ECHO_SERVER_ADDRESS, ECHO_PORT ))
response = sock. recv ( 256 ) sock. close () print response Client code: This example is based on playing the client in the UDP socket relationship. The source code for this example can be found here: https://developer.mbed.org/teams/mbed/code/udpechoclient Start with the appropriate header files: #include "mbed.h" #include "EthernetInterface.h" Define the server address and the server port: const char * ECHO_SERVER_ADDRESS = "192.168.2.2" ; const int ECHO_SERVER_PORT = 7; Declare the Ethernet interface object, initialize it and start a connection: int main () { EthernetInterface eth ; eth. init (); //UseDHCP eth. connect (); printf ( "\nclientipaddressis%s\n", eth. getipaddress ()); Declare the object for the UDP socket and initialize it: UDPSocket sock ; sock. init (); Declare the endpoint for the server and use it to set the server address: Endpoint echo_server ; echo_server. set_address ( ECHO_SERVER_ADDRESS, ECHO_SERVER_PORT );
Create a character buffer with the message you would like to send to the server and send it to the previously set address: char out_buffer [] = "HelloWorld" ; printf ( "Sending message'%s' toserver(%s)\n", out_buffer, ECHO_SERVER_ADDRESS ); sock. sendto ( echo_server, out_buffer, sizeof ( out_buffer )); Receive the echoed message in an additional buffer and print it out. char in_buffer [ 256 ]; int n = sock. receivefrom ( echo_server, in_buffer, sizeof ( in_buffer )); in_buffer [ n ] = '\0' ; printf ( "Receivedmessagefromserver:'%s'\n", in_buffer ); } sock. close (); while ( 1 ) {} eth. disconnect (); To test this with a partner, have them load the server code to test your UDP connection. Else, you can load the python script below onto your PC: import socket ECHO_PORT = 7 print 'ServerRunningat', socket. gethostbyname ( socket. gethostname ()) sock = socket. socket ( socket. AF_INET, socket. SOCK_DGRAM ) sock. bind (( '', ECHO_PORT )) while True : print "waitingforudpdatapacket..." data, address = sock. recvfrom ( 256 ) print "Receivedpacketfrom", address, "withdata", data print "Sending packetbacktoclient" sock. sendto ( data, address) TCP Unlike UDP, TCP is a stateful protocol, which means that the protocol will error check for packets in the wrong order and any packets that didn t make it. This makes for a denser overhead and a longer time for transaction (because it has to resend every packet that
doesn t make it before it will allow the next one to proceed). This is best for applications where you need to guarantee the integrity of the data and the time is less critical. All HTTP transactions are based on TCP because it guarantees the web page is delivered to the user. The socket transactions are as follows: Server: 1. Create the TCP socket. 2. Bind the socket to a port number. 3. Wait for or/ accept a connection. 4. Wait for or send a message. 5. Continue until end of file notification. 6. Close the socket. Client: 1. Create a TCP socket. 2. Connect to server. 3. Wait for or send a message. 4. Close the connection.
Here is an example diagram of the TCP socket interaction: TCP Example This example demonstrates a server client connection over TCP. Server code: This example is based on playing the server in the TCP socket relationship. The source code for this example can be found here: https://developer.mbed.org/teams/mbed/code/tcpechoserver/docs/a5ead1402704/main_8cp p_source.html Include the relevant headers and define the server port that will be used: #include "mbed.h" #include "EthernetInterface.h"
#define ECHO_SERVER_PORT 7 In the main program, declare the Ethernet interface object, initialize it and connect: int main ( void ) { EthernetInterface eth ; eth. init (); //UseDHCP eth. connect (); printf ( "\nserveripaddressis%s\n", eth. getipaddress ()); Declare the socket server object, bind it to the previously set port, and start listening for connections: TCPSocketServer server ; server. bind ( ECHO_SERVER_PORT ); server. listen (); Wait for a new connection, declare the socket connection object and accept the connection: while ( true ) { printf ( "\nwaitfornewconnection...\n" ); TCPSocketConnection client ; server. accept ( client ); client. set_blocking ( false, 1500 ); // Timeoutafter(1.5)s Establish a character buffer to capture the incoming data: printf ( "Connectionfrom:%s\n", client. get_address ()); char buffer [ 256 ]; while ( true ) { int n = client. receive ( buffer, sizeof ( buffer )); if ( n <= 0 ) break ; // printreceivedmessagetoterminal buffer [ n ] = '\0' ; printf ( "ReceivedmessagefromClient:'%s'\n", buffer ); Reverse the message to be sent back to the client:
//reversethemessage char temp ; for ( int f = 0, l = n - 1 ; f < l ; f ++, l --){ temp = buffer [ f ]; buffer [ f ] = buffer [ l ]; buffer [ l ] = temp ; } Print out the new message and send it to the client: // printreversedmessagetoterminal printf ( "SendingmessagetoClient:'%s'\n", buffer ); } // Echoreceivedmessagebacktoclient client. send_all ( buffer, n ); if ( n <= 0 ) break ; Now that everything is complete for this transaction, close the connection: } } client. close (); If you have no one else for the client, run this python script on your PC to complete the test: import socket ECHO_SERVER_ADDRESS = "192.168.2.2" ECHO_PORT = 7 message = 'Hello, world' s = socket. socket ( socket. AF_INET, socket. SOCK_STREAM ) s. connect (( ECHO_SERVER_ADDRESS, ECHO_PORT )) print 'Sending', repr ( message ) s. sendall ( message ) data = s. recv ( 1024 ) s. close () print 'Received', repr ( data) Client code: As before, make sure to include the important headers:
#include "mbed.h" #include "EthernetInterface.h" Set the server address and the server port: const char * ECHO_SERVER_ADDRESS = "192.168.2.2" ; const int ECHO_SERVER_PORT = 7; In the main program, declare the Ethernet interface object, initialize it and connect: int main () { EthernetInterface eth ; eth. init (); //UseDHCP eth. connect (); printf ( "\nclientipaddressis%s\n", eth. getipaddress ()); Declare the object for the socket connection and set a loop to wait for a connection with the server: // ConnecttoServer TCPSocketConnection socket ; while ( socket. connect ( ECHO_SERVER_ADDRESS, ECHO_SERVER_PORT ) < 0 ) { printf ( "Unabletoconnectto(%s) onport(%d)\n", ECHO_SERVER_ADDRESS, ECHO_SERVER_PORT ); wait ( 1 ); } printf ( "ConnectedtoServerat%s\n", ECHO_SERVER_ADDRESS ); Now that the connection is secure, set a character string for a message of your choice and send it over the socket: // Sendmessagetoserver char hello [] = "HelloWorld" ; printf ( "Sending messagetoserver:'%s'\n", hello ); socket. send_all ( hello, sizeof ( hello ) - 1 ); Create a character buffer and use the receive socket method to collect the incoming bytes: // Receivemessagefromserver char buf [ 256 ];
int n = socket. receive ( buf, 256 ); buf [ n ] = '\0' ; printf ( "Receivedmessagefromserver:'%s'\n", buf ); Close the connection at this point: //Cleanup socket. close (); eth. disconnect (); } while ( true ) {} If you have no one else for the server, run this python script on your PC to complete the test: import socket print 'ServerRunningat', socket. gethostbyname ( socket. gethostname ()) s = socket. socket ( socket. AF_INET, socket. SOCK_STREAM ) s. bind (( '', 7 )) s. listen ( 1 ) while True : conn, addr = s. accept () print 'Connectedby', addr while True : data = conn. recv ( 1024 ) if not data : break conn. sendall ( data ) conn. close () Protocols A level up the abstraction ladder are the application layers where protocols live. Internet protocols build on the Berkeley Socket abstraction and use either TCP or UDP (in most cases). Protocols can be built independent of hardware considerations, and will then run on any hardware that pairs to the socket abstraction. HTTP What is it? HTTP, the Hyper Text Transfer Protocol, is a stateless method for transferring data across TCP. It works by having the client send a request header to the server, then the server
responds with the status, more headers, and the content payload. Then the connection is closed. This is done for every resource: open, send, close. So, for example, if you have a webpage with ten resources there will be ten different HTTP requests, each one opening its own socket. Benefits: TCP based. Widely used. Stateless (open, grab, close). Request and response. Issues: Bulky (1KB 1MB). Implementation varies. Every resource requires a new connection. Example: GET Request This is an example of a GET request to an HTTP server. It demonstrates the content being sent back and forth for a simple web page load. It is meant to give you an understanding of the complexity involved in even a simple HTTP exchange. Take note of how big some of the data fields can become, and think about how you would implement that on an embedded system. Request Here we will be sending data to the server to request something. In this case it is a GET request of the web page hosted at mbed.org.
Send request Type: GET using HTTP 1.1: GET / HTTP / 1.1 Request headers: Host : mbed. org User - Agent : Mozilla / 5.0 Gecko / 20100101 Firefox / 37.0 Accept : text / html, application / xhtml + xml, application / xml ; q = 0.9,* /*;q=0.8 Accept - Language : en - US, en ; q = 0.5 Accept - Encoding : gzip, deflate DNT : 1 Cookie : messages =...// removedduetosizecontraints Connection : keep - alive There is no payload being sent with this request. GET requests do not have payloads, but POST requests do. Response The response from the server is displayed below. The server responds with the HTTP version number and the status, which in this case is 200 OK meaning it is a valid request and here is the response : HTTP / 1.1 200 OK Other valid responses are 404 not found, 401 redirects, 500 server timeouts etc. Response headers are then sent. The response headers in this case list various things like cookie updates, content type and connection type:. Server : nginx / 1.7. 10 Date : Wed, 06 May 2015 18 : 23 : 03 GMT Content - Type : text / html ; charset = utf -8 Content - Length : 8429 Connection : keep - alive Vary : Cookie, Accept - Language, Accept - Encoding Content - Language : en Set - Cookie : csrftoken = 2r4hBeRBW5vEwMSZKduhPY6unHKs4NZN ; expires = Wed, 04 - May - 2016 18 : 23 : 03
GMT ; Max - Age = 31449600 ; Path =/ Content - Encoding : gzip X - Upstream - L3 : 172.17. 42.1 : 8090 X - Upstream - L2 : wcd - 20150506-1 - nginx Response payload is included at the bottom of the response. In the case of a webpage this is the HTML data that represents the webpage. In the case of APIs this would be a JSON formatted data section: <HTMLResponsePayload>... Websockets What is it? Websockets are a type of upgraded HTTP/TCP connection that essentially allows for a real time pipe of data to be created between the client and server. Websockets are useful for embedded systems because once established they allow you to send and receive data to the server as if it were another peripheral. For real time sensor data such as location it is imperative to have a low latency; websockets provides this along with guaranteed connections. A websocket will start life as an HTTP connection, then send an upgrade request that will switch the connection into a bi directional TCP pipe. A Websocket connection can be closed by either side.
Benefits: TCP based, port 80 meaning the protocol is secure. Live connection in other words, instead of having to open up a new connection for every pair, the same initial connection can be used for the duration of the conversation, saving overhead and time. Server client the client and the server communicate both ways in the communication. Native support (web browser), Real time. Issues: Inconsistent implementation between browsers and servers. Scalability. In general, websockets are a great solution for getting embedded data to a remote server in real time. They have a relatively low overhead and open up a bi directional pipe that is easy to use. The one thing to keep in mind is that each websocket you open is a new TCP connection, which means each device will require a persistent socket connection to the server to operate. This is not an issue when you have fewer than 10,000 devices, but at a certain point you are going to start overloading the server. This can be alleviated by using a distributed server model or by switching to a different protocol.