Lesson 6: Portlet for job submission

Lesson 6: Portlet for job submission Mario Torrisi University of Catania - Italy (mario.torrisi@ct.infn.it) Sci-GaIA Winter School This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement n 654237

Outline MyJob portlet o G&C Engine monitoring module o UsersTracking database Portlet Template o Get portlet code o Build and deploy o Job submission and log inspection Portlet code o Code structure o Java Classes o processinputfile() method o submitjob() method o Customization script 2

MyJobs Portlet

MyJobs portlet Is a core component of the Catania Science Gateway Framework Allows users to manage their jobs on DCIs o Check jobs status o Retrieve output when jobs complete their execution Interacts with the Grid & Cloud Engine Monitoring module You can simply download and deploy MyJobs portlet into your Science Gateway MyJobs.war 4

MyJobs Portlet Download o http://grid.ct.infn.it/csgf/binaries/myjobs.war Deploy o cp /path/to/myjobs.war /path/to/plugin-sdk/dist o Check server.log file, you should see MyJobs was successfully deployed Add MyJobs to page 5

MyJobs Portlet Active Job List Done Job List Job Label Job status 6

User Tracking Database Each user action which involves the distributed infrastructure will be tracked by the UsersTracking Database Download sql script from here to create the Users Tracking Database schema The G&C Engine uses the GridOperations table to register applications and services accessing the distributed infrastructure mysql> describe GridOperations; +-------------+--------------+------+-----+---------+----------------+ Field Type Null Key Default Extra +-------------+--------------+------+-----+---------+----------------+ id int(11) NO PRI NULL auto_increment portal varchar(120) NO NULL description varchar(200) NO NULL +-------------+--------------+------+-----+---------+----------------+ 3 rows in set (0.00 sec) 7

GridOperations table How to get GridOperations values o id application identifier o portal portal name, see picture below to get the right value o description human readable application label +-------------+ Field +-------------+ id portal description +-------------+ Application registration in the GridOperations table is mandatory for the MyJobs portlet 8

MyJobs Portlet In order to MyJobs Portlet (and all the other applications) works properly you need to create a JDBC Connection Pool and JDBC Resources in the application server, used by the G&C Engine to communicate with the Users Tracking Database. o o JDBC Connection Pool is a group of reusable connections for a particular database 1 JDBC resource (data source) provides applications with a means of connecting to a database.[ ] A JDBC resource is identified by its Java Naming and Directory Interface (JNDI) name 1 1 https://docs.oracle.com/cd/e26576_01/doc.312/e24928/jdbc.htm#gsadg00576 9

JDBC Connection Pool/Resource Application server JNDI Resource Connection Pool G&C Engine lookup SQL connection connection connection connection connection Users Tracking Database JDBC Driver 10

JDBC Connection Pool/Resource To create JDBC Connection Pool and JDBC Resource you can 1. Edit the domain.xml file /path/to/liferay-bundle/glassfish-3.1.2/ /config/domain.xml Look for <resources> tag Add an new connection pool using the properties highlighted in next slide, pay attention to set the right address of DB container Tip check the docker inspect command Add two jdbc resources using the properties highlighted in next slide Look for <servers> tag Add two jndi resources as shown in the next slide to allow lookup Save and close file Restart domain 2. Configure them through the Glassfish Web Admin interface 11

JDBC Connection Pool Here what you should have in your domain.xml file <resources>... <jdbc-connection-pool driver-classname="" datasourceclassname="com.mysql.jdbc.jdbc2.optional.mysqlconnectionpooldatasource" restype="javax.sql.connectionpooldatasource" description="" name="usertrackingpool"> <property name="user" value="tracking_user"></property> <property name="url" value="jdbc:mysql:// :3306/userstracking"></property> <property name="password" value="usertracking"></property> </jdbc-connection-pool> <jdbc-resource pool-name="usertrackingpool" description="" jndiname="jdbc/usertrackingpool"></jdbc-resource> <jdbc-resource pool-name="usertrackingpool" description="" jndiname="jdbc/gehibernatepool"></jdbc-resource>... <resources/> <servers> <server name= server config-ref="server-config"> <resource-ref ref="jdbc/usertrackingpool"></resource-ref> <resource-ref ref="jdbc/gehibernatepool"></resource-ref>... <servers/> 12

JDBC Connection Pool 2 1 3 13

JDBC Resource 2 1 3 14

Template Portlet

Template Portlet The Template Portlet is a complete example of portlet able to submit a sequential Job into a distributed environment. Its Java code extends the Liferay MVCPortlet class and uses JSP pages to generate the input GUI 16

Template Portlet (Features) It provides a full example of: o Defining distributed infrastructure settings where the application will run o Managing input elements from web forms as application input o Managing upload file requests o Managing portlet configuration to handle distributed infrastructure settings o Viewing/Managing the pilot_script that contains the batch instructions that will be executed on the remote. It is a multi-infrastructure multi-middleware portlet to execute the same application on several infrastructures and middleware (currently supports: glite-based Grids, HPC Clusters, Cloud resources) It can be used as starting point to develop your own applications o A customization script is provided in order make its reuse easy 17

Template Portlet (Build & Deploy) Clone the source repository o cd /path/to/plugin-sdk/portlet o git clone https://github.com/sci-gaia/templateportlet.git o cd template-portlet o ant deploy Add to page 18

Template Portlet (Usage) template-portlet o Performs the hostname command on DCIs You can consider it like HelloWorld while approaching new programming languages o It accepts: An input file in order to show how to manage file as input of a real application A job label string in order to organize users interactions in the MyJobs portlet (Optional) o It has two buttons: Submit: to submit job Cancel: to reset form fields 19

Template Portlet (Job submission) Now you are almost ready to submit jobs A few mandatory steps are missing 20

Template Portlet (Configuration) View / Edit pilot_script Manage infrastructure Add infrastructure 21 Tip: insert the right values in GridOperations table

Template Portlet (Configuration Infrastructure Management) 22

Before testing the job execution Install Grid CA certificates, follow instruction for your system from here Download the tar at this link 2 and extract it in the /etc/grid-security/ folder of your liferay container. It contains a directory for each trusted VO. Inside each VO directory two types of files can be found: o o An LSC file contains a description of the certificate chain of the certificate used by a VOMS server to sign VOMS attributes. An X509 certificates used by a VOMS server to sign attributes Open the VPN or be sure the etokenserver allows incoming connections on port 8082 form your portal IP address Check the etokenserver service is reachable o o o Connect to http://etokenserver2.ct.infn.it:8082/etokenserver/ From the interface generate the robot proxy request Execute curl or wget on the generated request 2 http://italiangrid.github.io/voms/documentation/voms-clients-guide/3.0.3/ 23

Template Portlet (Job submission) Select a file (optional) Specify a label (optional) 24

Job Execution (Log inspection) Liferay server.log file reports: o o A full dump of the portlet preference values [mi_hostname_portlet:108] dump: Infrastructure #1 enableinfrastructure : 'yes' nameinfrastructure : 'EUMEDGRID-Support infrastructure' acronyminfrastructure: 'EUMEDGRID' The GridEngine Initialization INFO JSagaJobSubmission - Getting adaptor name... JSagaJobSubmission - Using adaptor: wms o The Robot proxy retrieval INFO RobotProxy - proxypath=/tmp/7f7e1e98-0fd1-4ebb-a1ae-0627efddf600 INFO RobotProxy - get proxy: http://etokenserver.ct.infn.it:8082/etokenserver/etoken/332576f78a4fe70a52048 043e90cd11f?voms=gridit:gridit&proxy-renewal=true 25

Job Execution (Log inspection) The JSAGA job submission string INFO JSagaJobSubmission - jobsandbox:/opt/liferay-portal-6.1.1-ce- ga2/glassfish-3.1.2/domains/domain1/autodeploy/mi-hostname-portlet/web- INF/job/pilot_script.sh>pilot_script.sh,/tmp/20130717133558_test_input_file. txt>20130717133558_test_input_file.txt,/tmp//joboutput/multiinfrastructurede mojobdescription1_1/<hostname- Output.txt,/tmp//jobOutput/multiinfrastructuredemojobdescription1_1/<hostnam e-error.txt The input sandbox file transfers Connecting to Gsiftp service at: wms014.cnaf.infn.it:2811... The job id and the job status thread execution 26 JSagaJobSubmission - Job Submitted: [wms://wms014.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms014.cnaf.infn.it:9000/kznsb62lcqcw0fxtitfr7q] UsersTrackingDBInterface - UpdateJobsStatusAsync running in Thread : Thread[pool-103-thread-1,5,grizzly-kernel]

Portlet Code

Portlet code (Code structure) 28

Portlet code (Java Classes) AppInfrastructureInfo.java o This class extends InfratructureInfo G&C Engine class to provide some other useful methods AppInput.java o This class is meant to collect all application GUI inputs and must be dynamically instantiated inside the action method as soon as the user press the Submit button AppPreferences.java o Class that contains all portlet s preferences values ConfigurationActionImpl.java o o This class implements ConfigurationAction Liferay interface Manage the code for portlet configuration TemplatePortlet.java o We will see this class more in depth later 29

Portlet code (Java Classes) Constats.java o This class is meant to collect application constants Utils.java o This class is used to manage useful methods that interact with: Application preferences file2string(), string2file() methods G&C Engine infrastructure parameters conversion 30

Portlet code (TemplatePortlet.java) 31

Action method Gets AppPreferences and AppInfrastructureInfo from portlet preferences @ProcessAction(name = "submit") public void submit(actionrequest actionrequest, ActionResponse actionresponse) throws IOException, PortletException { AppInput appinput = new AppInput(); PortletPreferences preferences = actionrequest.getpreferences(); String JSONAppPrefs = GetterUtil.getString(preferences.getValue( Constants.APP_PREFERENCES, null)); AppPreferences appprefs = Utils.getAppPreferences(JSONAppPrefs); String JSONAppInfras = GetterUtil.getString(preferences.getValue( Constants.APP_INFRASTRUCTURE_INFO_PREFERENCES, null)); Gets username using Liferay API ThemeDisplay themedisplay = (ThemeDisplay) actionrequest.getattribute(webkeys.theme_display); User user = themedisplay.getuser(); String username = user.getscreenname(); appinput.setusername(username); 32

Action method Extracts UploadRequest from ActionRequest, calls processinputfile() method and then submitjob() UploadPortletRequest uploadrequest = PortalUtil.getUploadPortletRequest(actionRequest); File uploadedfile = processinputfile(uploadrequest, username, timestamp, appinput); List<AppInfrastructureInfo> enabledinfras = Utils.getEnabledInfrastructureInfo(JSONAppInfras); if (enabledinfras.size() > 0) { InfrastructureInfo infrastructureinfo[] = Utils.convertAppInfrastructureInfo(enabledInfras); submitjob(appprefs, appinput, infrastructureinfo); 33

processinputfile() private File processinputfile(uploadportletrequest uploadrequest, String username, String timestamp, AppInput appinput) throws IOException { File file = null; String fileinputname = "fileupload"; String sourcefilename = uploadrequest.getfilename(fileinputname); if (Validator.isNotNull(sourceFileName)) { _log.debug("uploading file: " + sourcefilename + "..."); String filename = FileUtil.stripExtension(sourceFileName); _log.debug(filename); appinput.setinputfilename(filename); String extension = FileUtil.getExtension(sourceFileName); _log.debug(extension); // Get the uploaded file as a file. File uploadedfile = uploadrequest.getfile(fileinputname, true); File folder = new File(Constants.ROOT_FOLDER_NAME); // This is our final file path. file = new File(folder.getAbsolutePath() + Constants.FILE_SEPARATOR + username + "_" + timestamp + "_" + filename + ((!extension.isempty())? "." + extension : "")); FileUtil.move(uploadedFile, file); 34 } } return file;

submitjob() Initializes variable for JobSubmssion private void submitjob(apppreferences preferences, AppInput appinput, InfrastructureInfo[] enabledinfrastructures) { // Job details String executable = "/bin/sh"; String arguments = FileUtil.getShortFileName(pilotScript) + " " + appinput.getinputfilename(); String outputpath = "/tmp/"; String outputfile = "hostname-output.txt"; String errorfile = "hostname-error.txt"; String appfile = "hostname-files.tar.gz"; // InputSandbox (string with comma separated list of file names) String inputsandbox = pilotscript + "," + appinput.getinputsandbox(); // OutputSandbox (string with comma separated list of file names) String outputsandbox = appfile;... 35

submitjob() Creates GEJobDescription object // Prepare the GridEngine job description GEJobDescription jobdesc = new GEJobDescription(); jobdesc.setexecutable(executable); jobdesc.setarguments(arguments); jobdesc.setoutputpath(outputpath); jobdesc.setoutput(outputfile); jobdesc.seterror(errorfile); jobdesc.setoutputfiles(outputsandbox); jobdesc.setinputfiles(inputsandbox); // GridEngine' MultiInfrastructure job submission object MultiInfrastructureJobSubmission mijobsubmission = null Actual job submission... // Ready now to submit the Job mijobsubmission.submitjobasync(appinput.getusername(), portalipaddress, applicationid, appinput.getjoblabel()); 36

Customization script customize.sh 37

References Template portlet source code repository o https://github.com/sci-gaia/template-portlet Discussion Forum o http://discourse.sci-gaia.eu/ EGI IGTF Release o https://wiki.egi.eu/wiki/egi_igtf_release VOMS Clients guide o http://italiangrid.github.io/voms/documentation/voms-clientsguide/3.0.3/ 38

Thank you! mario.torrisi@ct.infn.it sci-gaia.eu