2016 Giovanni Di Matteo (giovanni.di-matteo@ext.jrc.ec.europa.eu), Csaba Wirnhardt (csaba.wirnhardt@ext.jrc.ec.europa.eu) Approved by Pär Johan Åstrand (par-johan.astrand@jrc.ec.europa.eu) V 1.2 (17/06/2016) IES/H04/2016/21795
TABLE OF CONTENTS Document versions... 4 Abbreviations and Acronyms... 4 1. Sentinel-2 Requests management... 5 1.1 Creation of a Sentinel-2 Request... 5 1.1.1 Acquisition Window-related Sentinel-2 Requests... 6 1.1.2 Floating Sentinel-2 Requests... 7 1.2 Sentinel-2 Requests results... 8 1.2.1 Sentinel-2 alerting notification... 8 1.2.2 Acquisition Window-related SRs results... 9 1.2.3 Floating SRs results... 10 1.2.4 Downloading the Sentinel-2 data... 11 1.3 ESA accounts remarks... 17 IES IES (Institute for Environmental Sustainability) Page 2 of 17
TABLE OF TABLES Table 1 - Abbreviations and Acronyms... 4 Table 2 - Aria2 parameters for Metalink URLS download... 12 TABLE OF FIGURES Figure 1 - Sentinel-2 Requests Visual navigation icon... 5 Figure 2 - AW-related Sentinel-2 Requests definition... 6 Figure 3 - AW-related SR, AW information on mouse-over... 6 Figure 4 - Floating Sentinel-2 Requests definition... 7 Figure 5 - The number of already defined floating SRs for a Zone is displayed in the last column... 7 Figure 6 - Sentinel-2 alert e-mail sample... 8 Figure 7 - Acquisition Request containers for CwRS Acquisitions and Sentinel-2 imagery... 9 Figure 8 - Floating Acquisitions action button... 10 Figure 9 - Floating SR Acquisitions module... 10 Figure 10 - Download Product icon... 11 Figure 11 - Mask to fill in ESA credentials... 11 Figure 12 - Metalink file download icon... 11 Figure 13 - Metalink download input form... 12 Figure 14 - Metalink download pop-up and zip file... 13 Figure 15 - Loading a metalink file in JDownloader... 14 Figure 16 - Select the metalink file in JDownloader... 14 Figure 17 - To provide ESA credentials to gether files information... 15 Figure 18 - Files contained in the metalink file are listed in the JDownloader GUI... 15 Figure 19 - Start the download for a single file or for all of them... 16 Figure 20 - Statistics about the downloads... 16 Figure 21 - JDownloader settings... 17 Page 3 of 17
Document versions Version Date Edited by Edits description 1.0 08/06/2016 JRC (Giovanni Di Matteo) First version of the document 1.1 15/06/2016 JRC (Giovanni Di Matteo) First public version 1.2 16/06/2016 JRC (Pär Johan Åstrand) Final Abbreviations and Acronyms Acronym AR AW CAP CwRS GUI HR IDQA IP IR JRC MS QL QLB S2 VHR Description Acquisition Request Acquisition Window Common Agricultural Policy of the EU Control with Remote Sensing (Campaign) Graphical User Interface High Resolution imagery Input Data Quality Assessment Image Provider Image Request Joint Research Centre of the European Commission Member State Quick Look Quick Look Browser Sentinel-2 Very High Resolution imagery Table 1 - Abbreviations and Acronyms Page 4 of 17
1. Sentinel-2 Requests management Starting from G 4 CAP 1.5, the JRC is experimenting the possibility to exploit Sentinel-2 imagery in the context of CAP CwRS. The aim is to prove a seamless integration of the ESA APIs into the normal G 4 CAP system used for the management of the Campaigns, in order to allow Member State Administrations to request, check and download useful Sentinel-2 images over the CwRS control Zones. New Sentinel-2 Requests over an already defined control Zone can be defined by selecting the Sentinel-2 Request menu item available via CwRS Sentinel-2 Requests or clicking on the Sentinel-2 Requests icon in the visual navigation bar. Figure 1 - Sentinel-2 Requests Visual navigation icon 1.1 Creation of a Sentinel-2 Request MS Administrations have the possibility to create two different kind of Sentinel-2 Requests (SR): Acquisition Window (AW)-related Sentinel-2 Requests, i.e. to request Sentinel-2 imagery over a CwRS Zone in an already defined CwRS AW; Floating Sentinel-2 Requests, i.e. to request Sentinel-2 imagery over a CwRS Zone in a brand new time window. Sentinel-2 Acquisitions for the former SRs will be listed in the AR for the related AW and their quicklooks (QLs) will be displayed in the same window of the G 4 QLB. Sentinel-2 Acquisitions for the latter SRs will be listed in a dedicated module, as there is no correlation with other AWs and ARs. Page 5 of 17
1.1.1 Acquisition Window-related Sentinel-2 Requests To define this kind of SR, the user just has to select the AWs of interest. To allow a simple selection of all the AWs defined for a specific Country in a specific Campaign year, G 4 CAP provides a table view split by CwRS Zone and Period, like in the following Figure. Figure 2 - AW-related Sentinel-2 Requests definition Every row in the table is dedicated to a single Zone within the selected Country in the selected year. Every row contains all the AWs already defined, split by Period; every column represents a Period and contains all the AWs already defined for that period. Every AW is represented with its ID and a checkbox to select whether the user wishes a SR for that AW. Moving the mouse over the AW element, a pop-up message will provide the user with more details, like start and end dates. Figure 3 - AW-related SR, AW information on mouse-over After having selected all desired AWs, the user has to define the cloud cover (CC) threshold he wants to set for all these requests. The only Sentinel-2 imagery that will be proposed will be the ones having a CC percentage of less than or equal to the specified one over the CwRS Zone. The CC is calculated by intersecting the CwRS Zone AOI with the opaque cloud mask in the detector footprint (DETFO) given in the S2 metadata. This single value can be set in the ad hoc compulsory field CC threshold for Sentinel Requests (%). Every granted user will be allowed to select or deselect AWs at any moment. Page 6 of 17
The GUI allows the user also to select and deselect all the AWs checkboxes in one click, in order to speedup the procedure. After having selected the AWs and having set the CC threshold, please remember to click on the SAVE button, else all the changes will be lost. 1.1.2 Floating Sentinel-2 Requests The process to create a floating-sr is similar to the one described for AW-related SRs, but the user needs to fill in manually the time-window of interest. To proceed with the definition, the user has to click on the calendar icon in the last column of the table. In this way, a new pop-up window will open and the user will be able to choose the Zone for which to create the SR and all the already defined floating SRs for that Zones will be listed in a table, if any. Figure 4 - Floating Sentinel-2 Requests definition To create a new floating SR, the user will just have to fill in the desired CC threshold and the start and end dates of the SR window, then he needs to click on the Add link in the last column. Doing this, the SR will be saved and consequently displayed in the table. The format of the dates is dd/mm/yyyy, e.g. 05/07/2014. Selecting a different Zone from the dedicated Combo box, the user can define other floating SRs for other Zones of his Country. Closing the pop-up window, the user will be redirected to the main SR module, where the number of defined floating-srs for a specific Zone will be enumerated in the last column of each row. Figure 5 - The number of already defined floating SRs for a Zone is displayed in the last column Page 7 of 17
1.2 Sentinel-2 Requests results 1.2.1 Sentinel-2 alerting notification The Sentinel-2 daemon will run in background every night, looking for possible Sentinel-2 Acquisitions that fulfill the users requests. In case one or more matches are found, G 4 CAP will send an automatic e- mail to users having requested this kind of notifications (under Settings e-mails selection menu item), containing a link to the corresponding AR or floating SR, as in the following example. Figure 6 - Sentinel-2 alert e-mail sample Page 8 of 17
1.2.2 Acquisition Window-related SRs results All the Sentinel-2 Granules satisfying an AW-related SR will be displayed in the relative AR in a dedicated panel, like in the following figure. Figure 7 - Acquisition Request containers for CwRS Acquisitions and Sentinel-2 imagery CwRS Acquisitions and S2 imagery are kept separate. Moreover there is no need to perform any kind of workflow on the Sentinel-2 items: the only possible interaction with the user is to display their QLs on the G 4 QLB, or download their Product, or their Metalink. This is because Copernicus Sentinel-2 is free of charge and openly accessible 1 via the Copernicus Scientific Data Hub 2. Direct download for a Granule is still not available on the ESA side. 1 https://sentinel.esa.int/web/sentinel/sentinel-data-access-description 2 https://scihub.copernicus.eu/ Page 9 of 17
1.2.3 Floating SRs results Floating SRs do not have any correlated ARs, as they are defined on a brand new AW, so the Sentinel-2 imagery identified by G 4 CAP will be stored in an ad hoc module where all the floating SRs for a specific Zone will be grouped and displayed. This module is accessible from the SR module, clicking on the Floating S2-Acquisitions action button above the SR table. The button is highlighted in the next Figure. Figure 8 - Floating Acquisitions action button In this new module the user will be able to choose the Zone of his interest and see all the defined Floating SRs with their Sentinel-2 imagery satisfying the requirements. Figure 9 - Floating SR Acquisitions module Page 10 of 17
1.2.4 Downloading the Sentinel-2 data G 4 CAP tries to ease its users experience in getting access to Sentinel-2 imagery. Unluckily, at the moment, ESA APIs do not provide direct access to a single S2 Granule, so there is no one-click option for this goal. Anyway G 4 CAP users are provided with a direct link to download the whole S2 Product, even if just one portion of that image (Granule) will be of interest. To download the whole Product, just click on the dedicated icon and fill in your ESA username and password. The download of the compressed archive will begin automatically. Figure 10 - Download Product icon Figure 11 - Mask to fill in ESA credentials Another possibility provided by G 4 CAP is to download a metalink file, which is a standard (see https://en.wikipedia.org/wiki/metalink) allowing the parallel or sequential download of a set of Web resources. In this file generated by the system, the user will be able to find the URLs of all the files to be downloaded to have the complete Granule image. There are several free software able to read this format and download automatically all the URLs defined in it, a non-comprehensive list can be found here http://www.metalinker.org/implementation.html. Below we provide two different examples: aria and JDownloader. G 4 CAP allows the user to download the simple metalink file, or a zip archive containing the metalink file and a batch executable to automatically allow Aria to download all the URLs housed in that file. To choose one of those two options, the user has to click on the metalink file download icon. Figure 12 - Metalink file download icon Page 11 of 17
Once the user clicks on this icon, G 4 CAP displays a mask to provide information needed by Aria for the download procedures, please refer to the next section to gather more information about this. In case the user just wants to download the metalink file, he has to click on the Just Metalink button. Figure 13 - Metalink download input form Aria Aria (https://aria2.github.io/) is a command line tool able to read metalink files. One of the advantages in using Aria is that it does not need a proper installation, but the user just needs to download it on its system. Using the following command, it will download automatically all the files referenced in the metalink provided by G 4 CAP. aria2c.exe -M path\file.metalink -j 2 --http-user=username --httppasswd=password -d destinationfolder The specified options have the following meanings: Option Description -M It indicates you are passing a metalink file -j It indicates the maximum number of parallel download (remember ESA APIs allow a maximum of 2 parallel downloads per user) --http-user It indicates your ESA username --http-passwd It indicates your ESA password -d It indicates the folder where to store the downloaded Granules files Table 2 - Aria2 parameters for Metalink URLS download To ease stakeholders jobs, the JRC is proposing a dedicated solution based on this free software. First of all the binary is available in the Documentation section of G 4 CAP and can be downloaded by everybody and copied on the user s personal computers. Page 12 of 17
Moreover G 4 CAP is generating a batch file to avoid users to get lost with command line tools and their parameters. Obviously the user has to provide to G 4 CAP the information it is not able to collect or guess, like ESA system credentials and the physical path on the user s PC where to download the Sentinel-2 images. This batch file will contain the information provided by the user and will be stored in a zip archive together with the metalink file. The user has just to extract these two files in the aria2 folder and double-click on the batch executable and the multiple download will start automatically. Figure 14 - Metalink download pop-up and zip file Here after we summarise the process for the automatic downloads: 1. to click on the metalink download icon; 2. to fill in the requested information (ESA username, ESA password, physical folder where to download the S2 images); 3. to press the OK button; 4. to extract the zip archive to the aria2 folder; 5. to double-click on the batch file. Please note the JRC is not storing your ESA credentials. Page 13 of 17
JDownloader JDownloader is a download manager written in Java that can be downloaded for free at http://jdownloader.org/. This tool, differently from Aria, has a Graphical User Interface (GUI) and is more Windows-friendly. To load a metalink file, the user has to select the Load Linkcontainer item in the File menu. Figure 15 - Loading a metalink file in JDownloader Then just select the metalink file from the file chooser. Figure 16 - Select the metalink file in JDownloader To gather all the information on the files to be downloaded and to actually download them, JDownloader needs the ESA credentials to be used. Page 14 of 17
Figure 17 - To provide ESA credentials to gether files information Once filled in the credentials, JDownloader starts collecting information on all the files linked in the metalink file and displays them in a list in the GUI. Figure 18 - Files contained in the metalink file are listed in the JDownloader GUI Page 15 of 17
The user can choose to select a single file of the list and download it, or to download all of them in a batch, just right-clicking on any item of the list. Figure 19 - Start the download for a single file or for all of them The GUI shows the status of every download and the download speed. Figure 20 - Statistics about the downloads There are several options that can be set in the tool, the most important one, as in aria, is the maximum number of parallel download. Please remember that ESA allows a maximum of two simultaneous downloads per user. In case this constraint is violated, the user account will be disabled. Page 16 of 17
Figure 21 - JDownloader settings 1.3 ESA accounts remarks We want to stress again every user will need to request for an account at ESA in order to be able to download Sentinel data. Nowadays to create and account every user will need to fill in its personal information at https://scihub.copernicus.eu/dhus/#/self-registration. In case of issues, please check the ESA user guide available at https://scihub.copernicus.eu/userguide/1selfregistration. Our batch command uses 2 parallel threads to download S2 data, so every ESA user can launch just one of our batch command at a time, else its account will be de-activated. The same caution has to be paid if a user will download Products, or Granule files by itself. Always remember about the two parallel downloads limit. Page 17 of 17