Programming for Chemical and Life Science Informatics

Transcription

1 Programming for Chemical and Life Science Informatics I573 - Week 10 (Web Applications) Rajarshi Guha 31 st March & 2 rd April 2008

2 Outline Web services Overview Software tools Examples REST services

3 Part I Web Services

4 What is a Web Service? Code running on an arbitrary machine that can be accessed in a platform- and language-independent fashion More generally, a WS framework is an XML- based distributed services system Suitable for machine-to-machine interaction Platform and language agnostic Follows standards

5 Basic Architectures: CGI vs WS

6 WS versus CGI When is WS called for? What to make algorithms available avoid downloads avoid support requests if the algorithm takes milliseconds (or longer) it s feasible to make it into a web service Avoid direct database access Why not CGI? Returns HTML (traditionally) Can be a security issues It s possible to use CGI & WS s together

7 What Can WS s Do For Us? Generally we have to get software Might involve compiling, debugging Usually involves administration A web service is basically a set of function calls that a remote user can send output to and get a return value - the RPC view

8 Famous Web Services Google Amazon Access the search engine Get historical data Use their database functionality EBI (UK), KEGG (Japan), IU (USA), NCBI (USA)

9 Web Service Caveats A number of security issues can arise when using web services What s happening to your data after it s sent to the service? Same problem applies to ordinary CGI services Provence - what is the servcie really doing? Requires a network connection Is the service reliable?

10 Web Service Components Server side Web server Application container Service code Client side Some libraries The client code Web servers and application containers are optional Helps to administer a setup (redirection, load balancing etc) As we ll see clients can range from web pages to custom command line code You can work with C, C++, Java, Python, Ruby, Perl, R,.Net, PHP

11 Server Setup Unless we re working a very large WS setup, we can skip the web server setup Since we ll primary focus on Java services, we use an applicatin container - Tomcat A short tutorial on setting up Tomcat is here One of the key components that you need to add to a Tomcat installation is the AXIS libraries These libraries allow us to communicate with the service using the SOAP protocol

12 Writing a Web Service The nice thing about writing a Java web service is that it s no different from writing any other type of Java code No need to bother about SOAP or Tomcat It can get complex, especially when you start using custom classes as input arguments or return values As long a you use Java primitives, there is no extra work So what makes a Java class a web service? It s where it s located that makes it into a web service

13 Writing a Fingerprint Service The CDK provides a fingerprint function Given an IAtomContainer object, returns a BitSet object A BitSet is not a primitive, so we design the service such that it takes a String (SMILES) and returns a String (comma separated values of the bit positions that are set to 1) This approach ensures that any language will be able to communicate with our web service It is possible to return the BitSet directly - and it ll work if your the SOAP library for your language understands what a BitSet is and how to represent it in your language

14 Writing a Fingerprint Service public class CDKws { public String getfingerprintstring(string s, int length, int depth) throws CDKException { String print = ""; IMolecule mol = null; SmilesParser sp = new SmilesParser(); mol = sp.parsesmiles(s); if (mol == null) { throw new CDKException("SmilesParser exception"); } Fingerprinter fprinter = new Fingerprinter(length, depth); try { print = fprinter.getfingerprint(mol).tostring(); } catch (Exception e) { throw new CDKException("Fingerprinter error"); } return(print); } }

15 Writing a Fingerprint Service The code is exactly the same as if it were meant to be a command line program Makes local testing easy Returning a String is easy, but places the burden of parsing on the users side SOAP natively supports arrays. Since arrays are available in all common languages, we could use that to make things more efficient

16 Deploying the Web Service After compiling and testing the class, you will copy the class file to the appropriate Tomcat directory Update a web service deployment descriptor (WSDD) Lets Tomcat know what classes and methods can be accessed as WS s <deployment xmlns=" xmlns:java=" <service name="cdkws" provider="java:rpc"> <parameter name="classname" value="cdkws"/> <parameter name="allowedmethods" value="getfingerprintstring getfingerp </service> <service name="cdkdesc" provider="java:rpc"> <parameter name="classname" value="cdkdesc"/> <parameter name="allowedmethods" value="getdescriptors"/> </service> </deployment>

17 Deploying the Web Service

18 Writing a Web Service Client We can write the client in a variety of languages We ll focus on Python - much less boiler plate code to write If you use Java, you ll need the AXIS libraries in your CLASSPATH For Python you need to install the SOAPpy module or the ZSI module SOAPpy is no longer maintained, but is much easier to use than ZSI

19 Web Service Description Language WSDL is an XML description of a web service Suitable for a parsing (pretty verbose for humans) Given a WSDL document, you can autogenerate code to access the service This essentially lets you look at the service as if it were a local function A WSDL document describes What type of objects the service accepts What type of objecs the service returns What exceptions might occur

20 Web Service Description Language <?xml version="1.0" encoding="utf-8"?> <wsdl:definitions targetnamespace=" xmlns:apachesoap=" xmlns:impl=" xmlns:intf=" xmlns:soapenc=" xmlns:tns1=" xmlns:wsdl=" xmlns:wsdlsoap=" xmlns:xsd=" <!--WSDL created by Apache Axis version: 1.2RC2 Built on Nov 16, 2004 (12:19:44 EST)--> <wsdl:types> <schema targetnamespace=" xmlns=" <import namespace=" <complextype name="cdkexception"> <sequence/> </complextype> </schema> </wsdl:types> <wsdl:message name="cdkexception"> <wsdl:part name="fault" type="tns1:cdkexception"/> </wsdl:message> <wsdl:message name="get3dcoordinatesresponse"> <wsdl:part name="get3dcoordinatesreturn" type="soapenc:string"/> </wsdl:message> <wsdl:message name="get3dcoordinatesrequest"> <wsdl:part name="in0" type="soapenc:string"/> <wsdl:part name="in1" type="soapenc:string"/> </wsdl:message> <wsdl:porttype name="cdkstruct3d"> <wsdl:operation name="get3dcoordinates" parameterorder="in0 in1">

21 Web Service Description Language We can write WSDL documents by hand This is the preferred way of designing web services It s possible to get Tomcat + AXIS to autogenerate WSDL documents for deployed web services This makes life much easier for the most part Certain clients cannot handle the tye of WSDL document that AXIS generates One of the key questions is: where do I find the WSDL document in the first place?

22 Web Service Discovery Web services can be provided by anybody, anywhere How do we know what is available? Universal Description Discovery and Integration XML registry for web services Supports annotation, so you can search for web services Someone must maintain a UDDI registry

23 Web Service Discovery Communication between the client and the UDDI registry is described in the specification Requires library support Java, Python, Perl, Ruby are supported Example UDDI or UDDI-like registries NASA BioMoby

24 Web Service Discovery Once you have the WSDL, you can directly communicate with the WS A conceptually equivalent way, but much more lightweight is to use RSS feeds to list available web services Allows you to keep track of new web services, or changes in WSDL s very easily Someone must collect the WSDL s and there s no standard way to annotate them An example is here

25 WSDL Components Types - used to define custom message types Messages - abstraction of request and response that a client and service need to communicate PortTypes - contains a set of operations, which organize WSDL messages Operation method name PortType Java interface Bindings - binds the PortType to a specific protocol Services - gives you one or more URL s for the service Lets us know where to go to execute CDKstruct3D

26 WSDL Examing the WSDL for the CDKstruct3D services indicates Location of the service It takes 2 inputs, both SOAP encoded String s It returns a SOAP encoded String It may throw an exception called CDKException But the WSDL does not say anything about what the arguments mean! Read the documentation if available Ask the author

27 Simple Object Access Protocol SOAP is an XML message format (encoding) Describes how an object is represented when sent to or received from a service SOAP mesages are sent to and received from a service SOAP is simply a standard way of sending different types of objects to and from web services As with most XML, it s very verbose for humans

28 A SOAP Message 0 or 1 header elements 1 body element Wrapped in an envelope The actual data being transfered is contained within the body Headers can contain Meta-data Security information QoS

29 A SOAP Message <?xml version="1.0" encoding="utf-8"?> <SOAP-ENV:Envelope SOAP- ENV:encodingStyle=" xmlns:soap-enc=" xmlns:xsi=" xmlns:soap-env=" xmlns:xsd=" > <SOAP-ENV:Body> <ns1:getfingerprintstring xmlns:ns1=" SOAP-ENC:root="1"> <v1 xsi:type="xsd:string">cc=coc</v1> <v2 xsi:type="xsd:int">1024</v2> <v3 xsi:type="xsd:int">6</v3> </ns1:getfingerprintstring> </SOAP-ENV:Body> </SOAP-ENV:Envelope>

30 What Types Does SOAP Support? Specifies a number of built in types int, double, float, boolean, String, Date, arrays Some compound types also allowed Arrays, hashes When making a SOAP request, we need to convert objects to the appropriate SOAP type Specified in the WSDL We use a SOAP toolkit to handle the conversions

31 A Java Client Initialization public CDKwsClient(String host) { String endpoint = " try { Service service = new Service(); call = (Call) service.createcall(); call.settargetendpointaddress( new java.net.url(endpoint)); } catch (Throwable t) { // handle the exception } }

32 A Java Client Calling the Service public String getfpstring(string[] args) { try { call.removeallparameters(); call.setoperationname( "getfingerprintstring" ); call.addparameter( "s", XMLType.XSD_STRING, ParameterMode.IN); call.setreturntype( XMLType.XSD_STRING ); String ret = (String)call.invoke( new Object[] { args[0] }); System.out.println(ret); return ret; } catch (Throwable t) { // handle the error } }

33 Being Lazy This is a lot of tedious code We can use AXIS to generate Java classes from the WSDL java -cp $CLASSPATH org.apache.axis.wsdl.wsdl2java \ -p javaworld.axis.proxy \ This will create a set of classes that correspond to the WSDL components So now the client code simplifies to...

34 Proxy Classes public Vector getfpvector(string[] args) throws Exception { CDKwsServiceLocator service = new CDKwsServiceLocator(); CDKws port = service.getport(); Vector ret = port.getfingerprintvector(args[0]); return ret } The AXIS generated code creates a proxy Rather than dealing with SOAP we now deal with the WS in terms of WSDL Analogous to HTTP - you don t write the raw HTTP request to view Essentially we treat the service as it were a local object

35 Python Web Services Python has a number of ways to deal with WS s SOAPpy - simple to use, no further development ZSI - more comprehensive, steep learning curve Python handling of of SOAP is not as well as developed as for Java But we gin the advanatges of Python

36 A Python Client Client code to use the fingerprint service import SOAPpy if name == main : service = SOAPpy.WSDL.Proxy( \ ) ret = service.getfingerprintstring( CC=COC ) print ret One would expect that calling the Vector version of the web service would return a list object SOAPpy will throw an exception Restricts the use of complex web services

37 More Python Client Fun Very easy to get available methods, parameter names etc >>> from SOAPpy import WSDL >>> service = WSDL.Proxy( >>> service.methods.keys() [u getmolecularweight, u getmolecularformula,... u gethtmlmolecularformula, u gettpsa ] >>> fpmethod = service.methods[ getfingerprintvector ] >>> fpmethod.inparams[0].name u in0 >>> fpmethod.inparams[0].type (u u string ) >>> >>> fpmethod.outparams[0].name u getfingerprintvectorreturn >>> fpmethod.outparams[0].type (u u Vector )

38 Cheminformatics Web Services A number of cheminformatics web services are hosted at IU Cheminformatics web services core.html Database web services db.html Statistical web services statistics.html Documentation and WSDL are available, so you can play around with them

39 Part II Representational State Transfer

40 Actions and Resources The traditional way the WWW was considered was an interconnected set of resources which were viewed A URL represented the location of a thing, or some feature of a thing is the location of a picture of me But we can also do things on the web and so a URL can also represent an action will get a picture of me When we start considering actions on the WWW, we are faced with many approaches Web services are one way

41 Representation State Transfer REST is an alternative network architecture that considers everything to be a uniquely identifiable resources First described by Roy Fielding in his Ph.D. thesis In this context, resources are abstract concepts. In practice what we deal with are representations of resources My picture could be JPEG image Plain text (Base64 encoded) A URL pointing to some actual representation

42 How Do We Do Things? If resources correspond to things, what are the actions? REST largely uses HTTP as a way to specify actions GET retrieve a resource from its URI POST add something to a resource PUT update a resource DELETE delete a resource All actions may not be applicable to all types of resources By have just these 4 operations, we are able to interact with all types of resources Note - how resources are made available is not the concern of REST. A resource could be made available as a CGI, WS, plain file links REST style architecture just ensures that it has a unique address

43 How Do We Do Things? If resources correspond to things, what are the actions? REST largely uses HTTP as a way to specify actions GET retrieve a resource from its URI POST add something to a resource PUT update a resource DELETE delete a resource All actions may not be applicable to all types of resources By have just these 4 operations, we are able to interact with all types of resources Note - how resources are made available is not the concern of REST. A resource could be made available as a CGI, WS, plain file links REST style architecture just ensures that it has a unique address

44 REST Services - 3D Structure Database Database access is a natural REST application A resource exists in a database, and we can create a unique URL for it An HTTP GET on such a URL retrieves the data from the database An address for the 3D structure of PubChem CID Client simply performs a GET on the URL Internally, we query the database and return the structure in SD format Some resource physically exists and we simply provide a way to address it

45 REST Services - 3D Structure Database Database access is a natural REST application A resource exists in a database, and we can create a unique URL for it An HTTP GET on such a URL retrieves the data from the database An address for the 3D structure of PubChem CID Client simply performs a GET on the URL Internally, we query the database and return the structure in SD format Some resource physically exists and we simply provide a way to address it

46 REST Services - Descriptors Given a molecule we would like to evaluate descriptors What are the resources we would like to make available? 1 Specific descriptors for the molecule 2 Values of specific descriptors for the molecule Note that we consider two possible resources, rather than just talk about values By allowing specific descriptors to be resources, we allow more flexibilty

47 REST Services - Descriptors

48 REST Services - Descriptors The resource identified by the previous link is an XML document, that references other resources The current resource is a list of all the descriptors for the specified molecule But we d like more detailed information - what is the value of the descriptor X for the specified molecule? Simply visit the link (say ALogP) and we get another resource (XML document) <DescriptorList> <Descriptor parent="alogpdescriptor" name="alogp" value=" <Descriptor parent="alogpdescriptor" name="alogp2" value=" <Descriptor parent="alogpdescriptor" name="amr" value="6.4383"/> </DescriptorList>

49 What s Happening? So we re able to GET resources Descriptors Descriptor values We can also book mark the results We are not specifying an action to calculate descriptors - rather these things appear to us as if they exist on some part of Internet But we could calculate descriptors for an infinite set of molecules All of them can t be precalculated or stored somewhere! But where does the value come from?

50 What s Happenning? In this case we have a traditional SOAP WS Added a Python CGI frontend The Python CGI decides what resource should be returned by examining the URL Generates the appropriate resource via the WS and returns it You could avoid the Python frontend and just have the web service check the URL and return the appropriate resource

51 Physical versus Logical URL s For the REST DB service, the resource was physically located in a database For the descriptor service, there is no actual descriptor value until we calculate it In both cases, the URL defining the resource is logical If we had to have 17M HTML pages, one for each entry in the DB, it would not be good design We cannot even do that for the descriptor service In both cases, the URL does not reveal implementation details One is using direct SQL queries The other is using a SOAP web service

52 Should Everything be REST? Sometimes it can be tricky to design an application without using actions For certain types of services, how do we send large, structured data? What about services that maintain state? X & data to a OLS regression service? What if we don t need the model just now? Everything need not be REST - it s just one way to design an application But makes life easier for the client No special libraries required Usually resources are in simple, well-known formats Aggregating and manipulation is now a lot easier

53 REST Principles Create a URL to each resource. The resources should be nouns, not verbs Categorize your resources according to whether clients can just receive a representation of the resource, or whether clients can modify (add to) the resource The resource should just return a representation of the resource and invoking the resource should not result in modifying the resource. Design to reveal data gradually. Don t reveal everything in a single response document.

54 REST Principles Create a URL to each resource. The resources should be nouns, not verbs Categorize your resources according to whether clients can just receive a representation of the resource, or whether clients can modify (add to) the resource The resource should just return a representation of the resource and invoking the resource should not result in modifying the resource. Design to reveal data gradually. Don t reveal everything in a single response document.

55 REST Principles Create a URL to each resource. The resources should be nouns, not verbs Categorize your resources according to whether clients can just receive a representation of the resource, or whether clients can modify (add to) the resource The resource should just return a representation of the resource and invoking the resource should not result in modifying the resource. Design to reveal data gradually. Don t reveal everything in a single response document.

56 REST Principles Create a URL to each resource. The resources should be nouns, not verbs Categorize your resources according to whether clients can just receive a representation of the resource, or whether clients can modify (add to) the resource The resource should just return a representation of the resource and invoking the resource should not result in modifying the resource. Design to reveal data gradually. Don t reveal everything in a single response document.

57 Practical REST One of the nicest aspects of REST is that your application can decide what to do by investigating the URL - might mean get the DB record for might mean return a PNG image of How do we handle errors? Use HTTP error codes If the requested resource is unavailable return a 404 (Not Found) If you only support GET, then return a 405 (Method Not Allowed) code for any other method If the URL seems invalid, return 400 (Bad Request)

58 POST versus GET GET interfaces are easy to use - just create a URL with everything in it But most browsers have a limit on the length of a GET request What if you want to send a SD file? What if you want to send CC#CCC? Either limit yourself to a fixed length input Encode URL (but still does not solve URL length limits) Most general approach is to use POST

59 POST versus GET Using POST in place of GET goes against core REST principles But doesn t matter to much, if your service is not databased backed By using POST requests you can send anything of any size to the server Any decent server-side program, should not be bothered if it s a GET or POST request (besides from being able to identify what type of request is made) Downside is that you cannot make a quick test by constructing a URL

60 Less Formal than SOAP (?) SOAP is more formalized since it defines schemata for inputs and outputs of a service (WSDL) Client libraries can automatically generate proxy objects by analyzing WSDL REST interfaces can return anything - have to read the documentation This can be problematic if there is no documentation But at least you can make a request and easily inspect the return value One possibility is WADL - but not a general solution

61 Practical REST - PHP <?php $url = $_SERVER[ PHP_SELF ]; $comps = split("/", $url); $ncomp = count($comps); $id = $comps[ $ncomp-1 ]; $resource = $comps[ $ncomp-2 ]; if ($resource == "db") { // call the DB service } else if ($resource == depict") { // call the depiction service } // return appropriate content-type // and data?>

62 Python Web Applications Web applications can vary in complexity A simple service, with no user front end (i.e. web page) Beautiful web pages with complex database backend Lots of tools to help write these things Generally fall in the realm of web frameworks Django Turbogear We will consider a lower level approach, directly using mod python mod python runs Python code within the Apache server Generally faster than writing Python CGI s - each time such a CGI is called Apache starts a new Python process The nice thing is that beyond some boilerplate, mod python code just reuses your pre-exisiting code

63 Basic mod python Installed on cheminfo. See tutorial Need to update permissions to allow mod python programs to run Edit a file called.htaccess in public html (make it if it s not there) SetHandler mod_python PythonHandler depict PythonDebug On <Files ~ "\.(gif html jpg png css)$"> SetHandler default-handler </Files> depict is the name of the Python program that will be called when a request is made for this directory By default, a method called handler will be called within this program

64 Handlers The underlying philosophy is that Apache will call a specific Python program when it sees a matching URL The program can then analyze the full URL and do what ever it wants In the preceding cases, lets say we put our program under public html/rest/depict When a browser requests rguha/rest/depict, it knows that it should call depict.py (and specifically the default handler) If the request is rguha/rest/depict/foo then the method foo in depict.py is called

65 The Default Handler The default handler is called handler A minimal form of the handler is from mod_python import apache def handler(req): ## do some work ret = do_something() ## set content type req.content_type = text/plain ## send the return value back to the browser req.write(ret) ## signal everything is OK return apache.ok

66 The Environment of a Handler Since the handler is called from Apache, you have access to the Apache environment. Things of interest The URL that was used by the client - allows you to decide what specific request was made and handle it appropriately The requested content types as well as setting the return content type - lets you indicate to the user the specific type of data that is coming back from the result The type of method - GET or POST All of this is provided by an mp request object - i.e., the req argument to a handler

67 Request Objects from mod_python import apache def handler(req): req.content_type = text/plain uri = req.uri if req.method == GET : ret = uri + Got with GET else: ret = uri + Got with POST req.write(ret) return apache.ok In general, there s very little boiler plate code

68 Getting Form Data Form data is easily accessible mod python hides details of GET / POST from you from mod_python import apache def handler(req): ## get first form element called mytextfield formelem = req.form.getfirst( mytextfield, None) ## get all form elements named colorbuttons otherelems = req.form.getlist( colorbuttons ) Must escape and validate data received from forms.

69 Sounds too simple... Handling requests is just the first step For REST services, the return type is usually a simple HTML or XML document or even a bunch of plain text lines Most applications will have some sort of web page frontend Writing dynamic HTML pages can be painful Good idea to use template systems Cheetah Mako Django Templates In general, if it s a large project better to work with a well designed framework that does a lot of the grunt work for you See here for a collection of mod python based REST services