Co-ReSyF Hands-on sessions

Transcription

1 This project has received funding from the European Union s Horizon 2020 Research and Innovation Programme under grant agreement no Co-ReSyF Hands-on sessions Coastal Waters Research Synergy Framework Hervé Caumont (Terradue) Summer School Meeting, Thursday 13th July 2017 LNEC, Lisbon, Portugal

2 Coastal Waters Research Synergy Framework 2

3 Aiming at Platform services 3 Research Apps User profiles Novice: - EO data access helpers Processing as-a-service Cloud services Intermediate: - Data Processing Algorithm integration Collaborative research Workflow composer service Reuse of existing modules Expert: - Cloud Sandbox service Software integration

4 Platform Services v1.0 Front-end 4 Back-end Data Search Processor as a Service OGC WPS Hosted processor OGC WPS, Hadoop Workflow Composer Data Agency OGC OSxGT Cloud Controller User Support Workflow Engine Workflow Manager Third-party tools repos Visualisation Co-ReSyF toolkit repository & Analysis AAA yum, conda

5 Innovation & automation Benefits for Platform users Hands-on focus Setup Workflow Develop Jobs Design Service Sandbox Environment Validate App Deploy Service Monitor Service Operational Cloud Environment 5

6 6 An application lifecycle

7 Design & Libraries Declaration7 (1/4) Processing libs developed in any programming language supported by CentOS 6.5 C/C++, Java, Python (most common) Matlab and IDL Other languages shall be previously analyzed Then synchronized on the Platform via software packages management tools (RPM repos, GitHub, )

8 Service integration (2/4) 8 Python, CDAT, R, BEAM, BEAT Service developer uses the sandbox tools to implement, test and validate a scalable workflow Sandbox includes tools to proof test data discovery, parallel processing and access to results

9 Cloud Compute deployment (3/4) 9 After validation, the developer publishes the Service for integration on a selected Cloud Infrastructure A Processor as a Service accesses complete pools of datasets, allows application intensive computing, and stores processor outputs, e.g. back to the lab

10 Service Operations (4/4) Processing Services are operated on a selected Cloud provider Operators manage the service via their user account on Terradue Cloud Platform Resources can scale up/down The development environment remains active for maintenance activities on the Service Service End Users can access the service s products from ad-hoc community portals Includes support for standard protocols like OGC Web Services 10

11 11 Hands-on sessions

12 Hands-on sessions overview 12 Session 1 - Data access and adding your own data to the platform Cloud storage Session 2 - Develop your own processing chain, using existing modules What are the Platform APIs, along with integrated tools for data staging within Cloud Apps, that deliver custom data management services to registered developer users on the Platform? How registered users are able to create a work environment with state of the art tools like Git and GitHub, Maven, Oozie workflows, and work collaboratively for the development of powerful Apps?

13 Session 1 Data access and adding your own data to the platform Cloud storage Get the instructions Visit Select simple-da-training-part-1 13

14 Session 2 Integrate and test your own processing chain, using existing modules Get the instructions Visit Select simple-dcs-training-part-1 14

15 Session 3 15 Resources at your disposal Co-ReSyF Project EC funding Terradue Cloud Platform, sign-up Co-ReSyF Website Private Cloud VPN setup Co-ReSyF GeoPortal Cloud Dashboard Co-ReSyF Twitter handle Dev Cloud Sandbox (DCS) documentation Co-ReSyF repositories on GitHub DCS Community shared applications T2 API documentation User forum Operations Support

16 16 Hervé Caumont Cloud Platform Operations, Terradue Looking forward hearing from you! CO-RESYF SUMMER SCHOOL 2017

17 17 Back-up slides (from RPM-1)

18 Choose a language 18

19 The sen2cor processing tool Sen2Cor is a processor for Sentinel-2 Level 2A product generation and formatting; it performs the atmospheric, terrain and cirrus correction of Top-Of- Atmosphere Level 1C input data. Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain and cirrus corrected reflectance images; additional, Aerosol Optical Thickness, Water Vapor, Scene Classification Maps and Quality Indicators for cloud and snow probabilities

20 Understand the application structure function main() { local reference=${1} ciop-log "INFO" "**** Sentinel-2 Atmospheric Correction ****" ciop-log "INFO" " " ciop-log "INFO" "Input S-2 L1C product reference: ${reference}" ciop-log "INFO" " " ciop-log "INFO" "STEP 1: Getting input product" local_product=$( get_data "${reference}" "${TMPDIR}" ) return ${ERR_GET_DATA} ciop-log "INFO" " " ciop-log "INFO" "STEP 2: SEN2COR tool" output=$( sen2cor "${reference}" "${local_product}" ) return ${ERR_GET_DATA} ciop-log "INFO" " " ciop-log "INFO" "STEP 3: Publishing results" publish_data "${output}" return ${ERR_PUBLISH} ciop-log "INFO" " " } 20

21 Understand the application structure for reference in sys.stdin: reference = reference.strip('\n') ciop.log('info', '**** Sentinel-2 Atmospheric Correction ****') ciop.log('info', ' ') ciop.log('info', 'Input S-2 L1C product reference: ' + reference) ciop.log('info', ' ') ciop.log('info', 'STEP 1: Getting input product') local_product = get_data(reference, os.environ['tmpdir']) ciop.log('info', ' ') ciop.log('info', 'STEP 2: SEN2COR tool') output = sen2cor(reference, local_product) ciop.log('info', ' ') ciop.log('info', 'STEP 3: Publishing results') publish_data(output) ciop.log('info', ' ') 21

22 Inspect the logs - Job 22

23 Inspect the logs - Task 23

24 Inspect the logs - Attempt 24