CISC 4700 L01 Network & Client-Server Programming Spring Harold, Chapter 7: URLConnections

Transcription

1 CISC 4700 L01 Network & Client-Server Programming Spring 2016 Harold, Chapter 7: URLConnections

2 URLConnection: abstract class representing active connection to resource specified by URL. Provides more control over server interaction than URL class. Can inspect header sent by a server, and set client's header fields. Can send data to web server with HTTP request methods (POST, PUT, etc.). URLConnection is part of Java's protocol handler mechanism. A protocol handler separates details of processing a protocol from the other things that a web client might do. If browser encounters unknown scheme, it can download protocol handler for same and proceed. Abstract URLConnection classes in java.net package. Concrete subclasses in the sun.net package hierarchy. Many methods (including ctor) and fields in URLConnection are protected. URLConnection objects are not usually directly instantiated. They are generally created by runtime environment, as needed, using the forname() and newinstance() methods of java.lang.class.

3 Opening URLConnections Sequence? 1. Construct URL object. 2. Invoke its openconnection() method to retrieve URLConnection object for same. Configure the URLConnection. Read header fields. 1. Get input stream, read data. 2. Get output stream, write data. 3. Close connection. If defaults are acceptable, can skip some steps. 3

4 Constructor for URLConnection class: protected URLConnection(URL url); Unless subclassing to handle new kind of URL, create indirectly: try { URL u = new URL(" URLConnection uc = u.openconnection(); // read from URL... } catch (MalformedURLException ex) { System.err.println(ex); } catch (IOException ex) { System.err.println(ex); } If subclassing URLConnection must implement public abstract void connect() throws IOException; When URLConnection first constructed, it is unconnected. Use connect() to establish connection (usually via TCP sockets). Some methods getinputstream(), getcontent(), getheaderfield(),... requiring connection will call connect() if connection isn't already open. 4

5 Reading Data from a Server 1. Construct a URL object. 2. Invoke its openconnection() to get a URLConnection. Invoke URLConnection's getinputstream(). 1. Read from the input stream. Example: Differences between URL and URLConnection? URLConnection provides access to HTTP header. 1. URLConnection can configure request parameters sent to server. 2. URLConnection can write to and read from server. 5

6 Reading the Header Header has a lot of useful information: content type length charset expiration date modification date Can get this info easily with Java methods. 6

7 Retrieving Specific Header Fields public String getcontenttype() returns MIME media type of response body. Returns null if content type is unavailable. Assumes that web server isn't lying. If content type is text/*, header may also contain charset info; return value of getcontenttype() will contain this as well. See rceviewer.java for program to download web page with correct charset. public int getcontentlength() tells how many bytes in content; if no Content-length header, it returns -1. If resource length exceeds , returns -1. In this case, use public int getcontentlengthlong()instead. Example: Download a binary file. Previously, used openstream() to download text file, reading until server closed connection. Unsafe for binary file, since server doesn't always close server exactly when data is finished. More reliable: Use getcontentlength() to find file length. See 7

8 public String getcontentencoding(): If content is unencoded, returns null. x-gzip: most commonly used encoding on the Web. public long getdate(): Returns long (number of milliseconds since Epoch), or 0 if no Date: field. public long getexpiration() public long getlastmodified() Example: Read URLs from command line, determine content type, length, encoding, and dates (modification, expiration, current). 8

9 Retrieving arbitrary header fields Since there's no limit to the variety of header fields... public String getheaderfield(string name) returns value of named header field (not case-sensitive, no closing colon). Returns null if no such field. If you want a numeric value, convert String to long or int. Example: Get value of the Content-Type field via String contenttype = uc.getheaderfield("content-type"); public String getheaderfieldkey(int n) returns key of nth header field. public String getheaderfield=(int n) returns value of nth header field. Example: Print out entire HTTP header. va 9

10 public long getheaderfielddate(string name, long default) retrieves header filed specified by name argument and tries to convert it to a long specifying number of milliseconds since Epoch. Internally, it uses parsedate() from java.util.date. If parsedate() doesn't understand the date or if getheaderfielddate() can't find requested header field, it returns the default argument. public int getheaderfieldint(string name, int default) retrieves value of header field specified by name argument and tries to convert it to an int. If it fails, returns default argument. Example: int contentlength = uc.getheaderfieldint("content-length", -1); 10

11 Caches Browsers cache web resources upon first encounter, reloading from cache when needed, unless cached version is stale. Pages accessed via GET over HTTP should be cached. Pages accessed via PUT should not be cached. This can be adjusted, depending on HTTP header contents: Expires gives expiration date Cache-control offers fine-grained control Last-modified ETag unique resource ID that changes when the resource does. Simple Java class for parsing and querying Cache-control headers If resource's representation is available in cache... if (expiry date hasn't arrived) use cached value else { check server with HEAD if (resource hasn't changed) use cached value else GET resource } 11

12 Web Cache for Java Default: Java caches nothing To install system-wide cache that URL class will use, you need concrete subclasses of RepsonseCache, CacheRequest, CacheResponse. Install your subclass of ResponseCache that works with your subclass of CacheResponse by passing it to RepsonseCache.setDefault(). JVM only supports a single shared cache. When system tries to load new URL, it first looks in cache. If present, URLConnection doesn't need to connect to remote server. Otherwise, protocol handler will download it, putting response into cache for next time. Storing/retrieving data from cache: public abstract CacheResponse get(url uri, String reuqestmethod, Map<String, List<String>> requestheaders) throws IOException; public abstract CacheRequest put(url uri, URLConnection connection)throws IOException; CacheRequest: abstract class with getbody() and abort() methods Example of concrete subclass of CacheRequest at t.java 12

13 get() method in ResponseCache retrieves data and headers from cache, returning them wrapped in a CacheResponse object. (If desired URI isn't in cache, returns null.) CacheResponse has two abstract methods: getheaders(), getbody(). Here we have a CacheResponse subclass tied to a SimpleCache and a CacheControl. Shared references pass data from request class to response class. onse.java Here we have a simple ResponseCache subclass that stores and retrieves cached values as requested. 13

14 Configuring the Connection URLConnection class has seven protected instance fields: url, doinput, dooutput, allowuserinteraction, usecaches, ifmodifiedsince, connected Accessed and modified via obvious getters and setters. Can only modify before URLConnection is connected. There are also getter/setter methods defining default behavior for all instances of URLConnection: getdefaultusecaches(), setdefaultusecaches(), getdefaultallowuserinteraction(), setdefaultallowuserinteraction(), getfilenamemap(), setfilenamemap() These can be invoked at any time. Moderately straightforward. Example: Program that prints default value of ifmodifiedsince, sets its value to 24 hours ago, prints new value. If doc has been modified in last 24 hours, downloads and displays. 14

15 Can disable the cache (forcing a reload), by uc.setusecaches(false); We can't access usecaches directly in client code; it's protected. Can query and modify connect and read timeout values, using the obvious getters and setters. 15

16 Configuring the Client Request HTTP Header HTTP client sends server a request line and a header. Header Chrome sends for is Request URL: Request Method:GET Status Code:304 Not Modified Accept:text/html,application/xhtml+xml,application/xml;q=0.9,im age/webp,*/*;q=0.8 Accept-Encoding:gzip,deflate,sdch Accept-Language:en-US,en;q=0.8,de;q=0.6 Cache-Control:max-age=0 Connection:keep-alive Cookie: gads=id=7e17eed5cbf9389f:t= :s=alni_mz1wqsuqk 1DlYzGXzWrZ0aC4KuBQg; wt3_sid=%3b ; et cetera Host: If-Modified-Since:Sun, 12 Jan :04:39 GMT If-None-Match:"db1-4efcaa1906ef2" User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/ (KHTML, like Gecko) Chrome/ Safari/

17 Web server can user this information to do stuff: alter info and layout to fit web client get/set cookies authenticate cookies... Can add more headers to HTTP header using public void setrequestproperty(string name, String value) before opening connection. A property may have multiple values. Example: A cookie is a collection of (name, value) pairs, e.g., Cookie: username=bovik; password=go+tartans;session=42 Adding this to a connection: uc.setrequestproperty("cookie", "username=bovik; password=go+tartans;session=42"); Adding a new property: public void addrequestproperty(string name, String value); 17

18 No fixed list of legal headers. Servers can ignore headers they don't recognize. HTTP mildly restricts header name/value content: For instance, no whitespace or line breaks. Java only enforces "no line breaks". Easy to create malformed headers in Java. How a server reacts to same? Unpredictable. To inspect headers in a URLConnection: public String getrequestproperty(string name); Can get all request properties for a connection: public Map<String,List<String>> getrequestproperties(); 18

19 Writing Data to a Server... submitting form info using POST, GET... uploading file using PUT To get an output stream, use public OutputStream getoutputstream(); But first, call setdooutput(true); For an http URL, this changes request method from GET to POST. Remember... GET: only for safe operations POST: for unsafe operations Once you have OutputStream, buffer it by chaining it to BufferredOutputStream, BufferredWriter, DataOutputStream, OutputStreamWriter, or whatever's convenient. Sending data with POST is similar to GET, except must setdooutput(true); use getoutputstream()to obtain stream to write query, rather than attaching it to URL. 19

20 Java buffers data written onto output stream until closed. This allows Java to determine content length. [Use telnet to demo a complete transaction.] Example: FormPoster uses URLConnection class and QueryString class (Chapter 5) to post form data: Constructor sets the URL. add() builds query string. post() actually sends data to server. See for details. 20

21 Security Considerations for URLConnections URLConnection objects: subject to usual security restrictions: making network connections reading or writing files etc. Ex: A URLConnection can be created by untrusted applet iff URLConnection points to host from which applet came. Details can be tricky. Ex: A jar URL pointing into applet's own jar file: okay. A jar URL pointing into local hard rive: not okay. Is it okay to connect to a URL? public Permission getpermission() throws IOException; Returns java.security.permission objet specifying permission needed to connect to the URL. Returns null if no permission is needed (e.g., no security manager in place). Subclasses of URLConnection return different subclasses of java.security.permission, such as java.net.socketpermission. 21

22 Guessing MIME Media Types Can't depend on protocols and servers using standard MIME types. URLConnection has two static methods to help figure out MIME type public static String guesscontenttypefromname(string name); Uses content-types.properties (usually in jre/lib) /etc/mailcap (on Unix) public static string guesscontenttypefromstream(inputstream in); Looks in first few (at most 16) bytes of data in the stream (akin to Unix file). Input stream must support marking. Less reliable than guesscontenttypefromname(), e.g., XML doc beginning with comment is mislabeled as HTML. 22

23 HttpURLConnection Class Abstract subclass of URLConnection. Provides additional methods when working with http URLs: get/set request method decide whether to follow redirects get response code, msg determine whether proxy server is being used overrides getpermission() Can't directly create instances of same. Do something akin to URL u = new URL(" URLConnection uc = u.openconnection(); HttpURLConnection http = (HttpURLConnection) uc; 23

24 Request method Most common request method is GET. Other request methods: POST, HEAD, PUT, DELETE, OPTIONS, TRACE By default HttpURLConnection uses GET. Change via public void setrequestmethod(string method) throws ProtocolException where method can be any of GET, POST, HEAD, PUT, DELETE, OPTIONS, TRACE May need to perform other actions, as well. Ex: POST needs a Content-length header. Some servers (e.g., WebDAV) support additional non-standard request methods, not supported by Java. GET: Previously discussed POST: Previously discussed 24

25 HEAD Behaves much like GET. Only returns HTTP header, not the file. Ex: Get the time a URL was last changed DELETE Not supported by all servers. Response is implementation dependent (actual removal vs. move to trash). PUT Store file on web server; often used by web-aware text editors. User must know actual directory structure. Authentication needed. OPTIONS What options are supported for a URL? TRACE Echoes HTTP header server received from client. Useful to see whether a proxy server is intercepting. 25

26 Disconnecting from the Server HTTP 1.1 supports persistent connections over a fixed TCP socket. If Keep-Alive is used, server won't immediately close connection upon sending last byte of data to client. Server will time out after (say) 5 seconds of inactivity. Preferable to explicitly close connection when done. HttpURLConnection transparently supports Keep-Alive unless explicitly disabled. When done talking to host public abstract void disconnect() closes any open streams. NB: Closing all open streams does not close socket and disconnect. 26

27 Handling Server Responses First line of server response: numeric code and explanatory msg. Obtain via public int getresponsecode() throws IOException; and public String getresponsemessage() throws IOException; HttpURLConnection contains 36 named constants, e.g., HttpURLConnection.OK HttpURLConnection.NOT_FOUND Ex: SourceViewer program including the response code and msg, see Note that there's no method to specifically return HTTP version, so we fake it (as HTTP/1.x). However, uc.getheaderfield(0) returns entire first HTTP request line: HTTP/ OK 27

28 Error Conditions The server can be configured to sendx alternative error pages,giving more info than 404 Not found, see for an example. Can use public InputStream geterrorstream() to return InputStream containing such a page (or null, if not found). Generally inside catch block after getinputstream() fails. Ex: Yet another SourceViewer, reading from error stream if input stream fails. Redirects By default, HttpURLConnection follows redirects. "Globally" configurable via static methods public static boolean getfollowredirects(); public static void setfollowredirects(boolean follow); and locally (instance-by-instance) via methods public boolean getinstancefollowredirects(); public static void setinstancefollowredirects(boolean follow); Proxies Is this HttpURLConnection going through a proxy server? public abstract boolean usingproxy(); 28

29 Streaming Mode HTTP Catch-22: Need to write Content-length header before the body Only know the length after we count number of bytes in body Java caches everything you write until OutputStream is closed. Good for small responses. Bad for large responses. If you know the size of the response, can do long length =...; response.setheader("content-length", String.valueOf(length)); If not, use chunked transfer encoding, in which request body is sent in multiple pieces, each with its own content length. Enable same via public void setchunkedstramingmode(int chunklength); HTTP sent out by Java will be somewhat different, but this is transparent to the Java programmer if using URLConnection class (rather than raw sockets). It does get in the way of authentication and redirection. If you know size of request data in advance, can optimize, via public void setfixedlengthstreamingmode(int contentlength); public void setfixedlengthstreamingmode(long contentlength); See text for some caveats. 29