ESA Unclassified - For Official Use Summary of 2013 user satisfaction survey
This document provides an overview of the results of the 2013 user satisfaction survey performed by the ESA Earth Observation User Services team during the ESA Living Planet Symposium in Edinburgh. The user satisfaction survey covers aspects essentially related to data management, including data access and data quality. The 2013 survey is particularly representative as it was elaborated from 600 returned questionnaires. Overall the results are largely positive but also indicate the areas where ESA shall further improve or modify its approach. This annex is extracted from the User Satisfaction Survey Report elaborated by the company tasked to perform the user survey: Advanced Studies and Research Center Ltd (ASRC), from Bucharest (Romania). The Full Report of the 2013 user satisfaction survey is available for download on Earth Online in the Document Library 1 INTRODUCTION An User Satisfaction Survey was conducted at the request of the European Space Agency in September 2013, at the occasion of the ESA Living Planet Symposium in Edinburgh (UK). This is the fourth survey performed to estimate the satisfaction of ESA s EO data users, following the ones performed in 2004 (during the Envisat Symposium in Salzburg, Austria), in 2007 (during the Envisat Symposium in Montreux, Switzerland) and in 2010 (during ESA Living Planet Symposium in Bergen, Norway). The scope of the survey was to analyse the users feedback regarding ESA EO services, with special emphasis on the new generation of EO satellites: Earth Explorer missions (CryoSat, GOCE and SMOS). The feedback was collected mainly during the ESA Living Planet Symposium in Edinburgh, UK (9-13 September 2013). The questionnaire remained available online for additional feedbacks until 30 September 2013. 621 questionnaires were returned, corresponding roughly to 1/3 rd of the symposium participants. This return is much higher than from the previous three surveys. The survey addressed the same topics than those addressed in the previous survey in order to have the possibility to make comparisons: online information provided, proposal submission and reporting, data access, catalogues available, tools and software, data ordering, help services provided and the Living Planet Symposium. In addition to the Bergen survey, the Edinburgh survey included new sections related to GOCE, SMOS and CryoSat.
Page 2 For each of the above topics, the participants were requested to rate three aspects: the overall satisfaction, the degree the respective item satisfies their needs and the importance for their activity of the item in question. The rates given as options for the response were from 1 (the lowest satisfaction / meeting needs / importance) to 4 (the highest). Statistics have been made on the response rates for each of the questions in the questionnaire used as the base for the survey, and on the ratings of the responses gathered. Customer Satisfaction Indexes (CSIs) have been calculated for each topic, on the same basis as the ones calculated in the Bergen survey, to evaluate the evolution of the users satisfaction. The calculated CSIs showed an increase of the users satisfaction compared to 3 years ago for all topics. The average ratings were high for all topics, as most of the respondents gave high marks on all the three aspects addressed. There was no area in the questionnaire where the respective majority rates were below 3. CSIs were calculated also for the parts of the survey without correspondence in the Bergen survey. The results obtained on these sections show as well a high degree of satisfaction, with CSIs between 77.7 and 86.6. Suggestions and comments were gathered from the users and were analysed within the survey. They represent the basis for the improvement recommendations that are formulated to ESA at the end of this document. 2 MAIN RESULTS 2.1 GENERAL SURVEY A total of 608 questionnaires were analysed (13 ESA respondents were removed from the survey analysis). Participants from 40 countries responded to the survey: Argentina, Australia, Austria, Belgium, Brazil, Bulgaria, Canada, China, Cyprus, Czech Republic, Denmark, Ecuador, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Israel, Italy, Ivory Coast, Latvia, Lithuania, Luxembourg, Netherlands, Norway, Poland, Portugal, Romania, Russia, Slovakia, South Africa, Spain, Sweden, Switzerland, Tanzania, Turkey, UK and USA. The distribution of the type of organization of the respondents is provided in the figure to the right. Regarding the most used EO missions data, most of the respondents are using data from the ERS/Envisat missions, followed by data from the Third Party Missions (TPMs). 97 respondents declared that in their activity they mostly use data from the ESA Earth Explorers (GOCE, SMOS or CryoSat) some of them in combination with ERS, Envisat or / and TPMs.
Figure 1: most used EO data by the survey respondents (multiple answers were possible) The questionnaire had specific sections dedicated to the Earth Explorers: 72 persons responded to the CryoSat questions 45 persons responded to the SMOS questions 49 persons responded to the GOCE questions The main survey result is that the majority of the participants rated very good (3) or excellent (4) the overall satisfaction on all topics: Information provided on the ESA Earth Observation dedicated web portal very good Information accessibility very good User registration process fast registration - excellent User registration process - full proposal registration very good Proposal submission and reporting very good Data ordering process very good Online data access very good Online data availability very good Data availability from archives very good Overall quality of ESA products - excellent Catalogues available - very good Tools and software provided - very good Documentation available - very good Technical support - excellent Quality of information and support provided by the EOHelp/EO-Support Team - excellent Edinburgh event - excellent Edinburgh event organization excellent Regarding the Earth Explorers missions, the following main results shall be noted. CryoSat: There is a need for the continuity of CryoSat data beyond CryoSat mission: 65 % ; The quality of L1b and L2 products is considered very good by the majority of the respondents; The access to CryoSat data is considered excellent by the majority; Information provided in the CryoSat Product Handbook and the support provided were mostly rated as excellent.
Page 4 SMOS: Most of the respondents considered the objectives of the SMOS mission related to their activity/needs achieved; ESA should provide a SMOS wiki, a user collaboration platform as available for CryoSat, where SMOS users can share information, results, resources and documents amongst them and with ESA; There is a need for the continuity of SMOS data beyond SMOS mission; The quality of SMOS products, the information and support provided were rated as very good by the majority of the respondents. GOCE: The majority of the respondents considered the objectives of the GOCE mission related to their activity/needs achieved; The majority of the respondents would like ESA to develop a GOCE wiki; The quality of GOCE products, the information and support provided are considered excellent by most of the respondents. 2.2 CUSTOMER SATISFACTION INDEX (CSI) The calculated CSIs are summarized in the following table. The trend of each CSI (increasing or decreasing) compared with the Bergen survey is shown in the table (between brackets: the difference between the two CSIs in 2013 and 2010). There is a general increasing trend for all CSIs. There are two exceptions concerning users coming from the educational environment, whose satisfaction slightly decreased regarding the tools/software provided by ESA and the data ordering process. Customer Satisfaction Index Category I: industry, government (31% of respondents) Category II: research (48% of respondents) Category III: education, other (21% of respondents) On-line Information 78.8 (8.4) 79.8 (3.8) 84.4 (9.7) Proposal submission and reporting 78.5 (13.8) 80.3 (5.3) 83.5 (3.5) Tools / Software 81.0 (9.3) 76.1 (2.8) 79.1 (-3.4) Catalogues 76.1 (8.8) 78.6 (7.6) 80.4 (7.6) On-line data access 75.8 (6.4) 76.9 (7.4) 79.7 (2.9) Data ordering 74.6 (1.9) 77.7 (2) 75.0 (-1.2) Help services 81.3 (5.9) 78.7 (3.3) 83.8 (6.9) Edinburgh Symposium 85.4 (7.5) 86.7 (3) 88.2 (15.8)
Customer Satisfaction Index Category I: industry, government (31% of respondents) Category II: research (48% of respondents) Category III: education, other (21% of respondents) General ESA products 1 77.7 78.3 82.2 CryoSat 1 77.9 80.9 86.3 SMOS 1 77.7 81.4 62.8 GOCE 1 83.4 86.6 87.7 Overall CSI 78.89 (6.92) 80.60 (5.17) 80.72 (5.69) The Full Survey report, available on Earth Online, provides all details on the survey results. 3 CONCLUSIONS The following conclusions can be derived as a result of the analysis of the data gathered within the survey: i. All the aspects investigated in the questionnaires are very important for the users. The exceptions are two items in the SMOS part of the survey: the quality of the SMOS products and the support provided. The majority of the questions were rated with 3 to 4 regarding both the overall satisfaction and the meeting needs aspects. No aspect was rated by the majority of the respondents with the rate 1 or 2. This leads to a first important conclusion that the degree of users satisfaction in general is high, with their needs basically covered by the products/services provided by ESA. ii. There are topics where the response rates and the percentages of the low value ratings on the importance of the item show that the respective topic is not of very big importance to the users, despite the fact that the majority of the respondents credit it as very important. This is the case of the new communication methods: Twitter, RSS, FB, Forums, Apps, Wiki, etc. For this topic, the response rate was very low, between 36 % and 44 % on the three aspects analysed (overall satisfaction, meeting needs and importance). The majority of the respondents to the survey choose not to respond to this question. The reason could be that the new communication methods are not sufficiently promoted; therefore the users are not aware of them. But at the same time, a very big percentage of the respondents to this question rated the importance of the item in question as not important. Therefore, the conclusion for this section is that the new communication methods are not well promoted and well known to the users, but at the same time, the users that are aware of these methods consider them as not important. The exception would be the dedicated wiki user collaboration platform, introduced for some missions, that could be considered important according to the responses gathered on this aspect within the CryoSat, SMOS and GOCE parts of the survey. 1 No comparison with 2010 survey, as no corresponding CSI was calculated.
Page 6 iii. 4 [ESA comments on ASRC conclusions: the initial analysis of the user replies indicates contradictory views from the respondents concerning new communication methods, though there is a clear support to the approach of collaboration platforms, in particular for the Earth Explorer missions. ESA will analyse in detail the replies to better understand user needs.] From the calculated CSIs and their evolution compared to the Bergen survey, two other main conclusions can be formulated: The satisfaction index is high/very high regarding the products and services provided by ESA; The trend of the satisfaction index is increasing over the last three years. There are nevertheless 2 aspects in the category of customers from educational environment where this index is lower than 3 years ago. These two topics are the tools/software provided by ESA and the data ordering process. On the data ordering, for the two other categories of clients, the satisfaction is higher than in 2010, but the increase is moderate to low: 1.9 for customers working in research area and 2.0 for the industrial/governmental customers. The situation is not the same for the tools and software provided by ESA, in which case the satisfaction index increase is 2.8 for the research related customers, and significant for the clients from industry/government: 9.3, despite the decrease of the satisfaction of the clients coming from education. Therefore the evolution in the last period analysed was in the good direction, but there are still aspects to be improved in order to better satisfy the needs of the clients from educational environment. Recommendations for improvement in this direction are formulated in the recommendations section. But, unfortunately, no relevant comments on these aspects have been formulated by the unsatisfied users working in education. Therefore, the recommendations are only formulated based on the input received from the clients in category I and II (industry/government and research). RECOMMENDATIONS [ESA comments on ASRC recommendations are indicated with ] 4.1 ONLINE INFORMATION The recommendation on the layout and information provided in the EO ESA web portal regards the way of finding the information needed. Most of the comments of the users, both on the layout and information provided and on the accessibility of the respective information, mentioned the difficulty in finding the information. Therefore a change in the layout/structure should be envisaged so that that is easier for the user to understand the logic behind and to find the relevant necessary information. As for the new communication methods, based on the conclusion in the previous section, the recommendation is to better promote the respective communication methods among the users and re-analyse the interest in the next survey, in order to decide on the continuation of using these new tools. Nevertheless, not much investment (time or costs) should be done until the confirmation, through a future survey, that these new communication methods are really useful and important. At the same time, there is the recommendation to continue with the implementation of Wiki collaboration platforms for different missions, as this appeared to be of interest for the users. Though, more promotional activities are needed in order to involve as many users as possible.
Regarding network performances of the website, the recommendation is that some improvements should be done to address the speed aspect, as the respondents frequent comment was that the website is often slow. The on-line documentation is currently revisited; this is a time-consuming effort as the information volume is huge. Particular attention is given to the information associated with the selection of new datasets during the user registration process. The concept of collaboration platforms are followed in the Thematic Exploitation Platforms approach and should also be gradually used for the Earth Explorers data user communities.] 4.2 PROPOSAL SUBMISSION AND REPORTING The recommendation formulated in this survey based on the feedback provided within the comments and suggestions from the users addresses two aspects to be envisaged by ESA: Simplify the registration/proposal submission process; Access data without the need to register/to submit a proposal. The self-registration process is gradually enforced; the Single-Sign-On approach will be further implemented allowing users to access more services/data with a single password. The self-registration process is the unique mean to gather statistical knowledge about the EO data users. The need to submit a proposal remains for the dataset requiring an on-request ordering (if a manual procedure is involved this is the case for SAR data) or in the case of large dataset requirements (like a full dataset or a very large portion on it), where a simple on-line download is inefficient and not feasible. The need to submit a proposal is substituted with the self-registration process as soon as a complete dataset is available on-line, e.g. with Envisat MERIS Full Resolution Level 1 dataset (140 TBytes) in November 2013. ] 4.3 TOOLS AND SOFTWARE PROVIDED The tools and software provided by ESA are of high interest for the users. Therefore, the recommendation is that ESA should continue providing them and should enlarge the offer as much as possible. New open source scientific toolboxes are in development to allow visualisation and processing of Sentinel data, in the context of EOEP-4 SEOM.] 4.4 CATALOGUES The satisfaction index is high regarding the catalogues available. The recommendations that can be formulated based on the comments of the users are: Improve the SAR data aspects; More attention should be paid to the elimination of the redundant information; Notify the users on the updates. The Next generation of User Services for Earth Observation (ngeo) will address many user recommendations related to catalogues and data ordering: 1- With ngeo, it will be possible to share as a simple URL:
Page 8 - a catalogue search request, - a shop cart (a list of products defined by the user), - a subscription to a catalogue search (an RSS feed to be notified of new catalogue entries matching use defined criteria), - an order for the systematic download of products (i.e. a standing order), - a particular product. 2- Shared URLs can be published as "links" in an email, on a web page, on a Wiki, on tweeter, Facebook, etc.. Clicking on such a link will send the user directly to the ngeo web client, in the right context, e.g.: - following a catalogue search URL, user ends up in the ngeo web client with the proper dataset selected and related search results visible on a map, - following a URL for a "standing order", user ends up on the ngeo web client with the order form ready to be submitted. 3- The ngeo web client is HTML based and will run on most recent web browsers of Windows, Mac, Linux. Access to the ngeo catalogue from a web site publishing EO related information will be immediate and will not require the installation of a stand-alone tool like EOLI-SA. 4- ngeo will let user subscribe to a catalogue search to receive notifications of new products via an RSS feed. 5- ngeo will support interferometric search (initially for Sentinel-1), search by polarisation, by sub-swath (the recommendation is too vague to give a detailed answer). 6- the elimination of the redundant information should greatly be improved as products are made available on-line on the Enhanced Online Archive EOA-2. Catalogues will be populated with metadata of existing products rather than with metadata of acquisitions made, the latter being a major cause of redundancies (overlapping acquisition stations).] 4.5 DATA ORDERING PROCESS The data ordering is an item of high interest and big importance for the users. Despite the fact that the process was rated by most of the respondents with high rate (3 to 4), there were, still, a lot of comments and suggestions. Having these in mind, the following recommendations can be formulated on this item: Order the data without the need of an additional software; Simplification of the process; Improvements of the existing tools (e.g. EOLISA): more friendly, simpler and existing bugs (if any, as the users mentioned some) corrected; Improvement of the quick views; Shortening of the process; Clearer data costs mention. Most of the replies/comments have been provided in sections 4.2 and 4.4. The following comments can nevertheless be added: Simplification of the process: there will be no more asynchronous ordering (as in EOLI) and no need for an order handling system as currently operated by the ESA order desk. With ngeo (and EOA-2), all products will be available for immediate download (or quasi immediate if on-the-fly processing is involved). Improvements of the existing tools:
- ngeo web client will feature nicer maps (OSM data), a time slider to easily navigate in the time dimension - resolution of bugs on the ngeo client will not require the deployment of a new software (like EOLI) to the user Improvement of the quick views: ngeo will provide a better view service where user can navigate (zoom/pan) in the product browse image, with the possibility to overlay map information (place names, roads, populated area,...). ] 4.6 DATA ACCESS The comments of the users have been analysed and the following recommendations were formulated: The possibility to access the data without the need of a specific tool (e.g. EOLISA); Provision of data samples/previews; Improvements could be considered regarding an easier mode for users to find the information/data; More data should be made available online; The possibility for the users to get access to data without the need to register. Most of the replies/comments have been provided in sections 4.2, 4.4 and 4.5. The following comments can nevertheless be added: Provision of data samples/previews: ngeo allows the grouping of products in any arbitrary ways into "logical datasets". This can be done to create datasets of sample products. Another possibility is to publish shop carts which would list sample products. The amount of data available on-line gradually increase (e.g. MERIS Full Resolution data during the last quarter). ] 4.7 TECHNICAL SUPPORT The technical support that the users received from ESA is one of the aspects best rated in the survey. Nevertheless, the recommendation is for a better promotion of the documentation available and a better visibility, as a lot of users declare that they have problems finding the needed documents. 4.8 GENERAL ESA PRODUCTS The following recommendations could be formulated on general ESA products: More data to be made available; A shorter time for data delivery should be considered; Better flexibility to the new users demands (this aspect has received a high percentage of dissatisfaction and not meeting needs: 7.4%, 5.8%). 4.9 EDINBURGH SYMPOSIUM In case that an ESA symposium similar to the Living Planet Symposium in Edinburgh will be organized again, the following recommendations are to be considered: A better evaluation of the number of participants at each session for better dimensioning the rooms; a very frequent comment addressed the fact that some of the sessions took place in rooms too small for the number of participants, while other sessions took place in rooms too big and empty, as not a lot of participants attended; A different way of organizing the parallel sessions should be envisaged, as lot of participants missed sessions that were relevant to their activity because of sessions overlapping;
Page 10 Introductory presentations to be foreseen for each section. Careful sizing of the communities vs available rooms is part of the standard organisation activities, however availability of rooms sized for 100, 300 or 600 places may result in an erroneous perception of the empty spaces from the participants. The programme tried to minimise conflicts between parallel sessions, however a few conflicts are unavoidable with 9 daily parallel sessions. Introductory presentations were implemented in the Earth Explorer sessions at the event and might be repeated in future events, when relevant. Note also: the Living Planet Symposium is conceived around data user presentations of own achievements, rather than on overview presentations.] 4.10 CRYOSAT The following recommendations can be formulated on CryoSat: Better promotion of the wiki collaboration platform; Continuation of CryoSat data beyond CryoSat; Improvements of the information provided in the Handbook, that is not always clear; Improvement of L2 data quality; A better consistency between different modes in GDR product and between L1b and L2 products. 4.11 SMOS For SMOS, based on the user suggestions, the recommendations are the following: Extending the coverage; Provision of a wiki collaboration platform to the users and the promotion of this tool once available; Continuity of similar data on long term; Improvement of the documentation provided. 4.12 GOCE The results of the study lead to the following recommendations on GOCE: Provision of a wiki collaboration platform to the users and the promotion of this tool once available; Continuation of providing Earth gravity measurements after GOCE, for applications such as the ones mentioned in the GOCE section of the Survey Final Report; An easier data access should be considered.