REVIEW OF THE RPS MAP OF EVIDENCE

Transcription

1 REVIEW OF THE RPS MAP OF EVIDENCE March 2016

2 1 Introduction The RPS Map of Evidence (MoE) is an online database which enables: The sharing and showcasing of good pharmacy practice and innovation; Increased knowledge exchange and learning in pharmacy and pharmaceutical sciences; and The collation of existing evidence and ongoing initiatives to inform policy and practice relating to pharmacy. The database was launched in 2011 with the intention of amplifying and mobilising pharmacy knowledge that is being created. Individuals are able to share evidence (for example, case studies, audit data [clinical and practice], service development, service evaluation, and research findings) by uploading the information directly onto the MoE. Evidence can also be retrieved by conducting keyword searches on the database. In the past few years, anecdotal evidence from members suggests that awareness and usage of the MoE is low and there are also issues with the functionality of the database. To appraise and improve the MoE, a review project was undertaken aiming to collect data relating to the usage, future directions and functionality of the database. 2 Aims of the Map of Evidence review The purpose of the RPS MoE review is to collect data relating to the usage and functionality of the database, with the purpose of: Ensuring the database is fit for purpose and is able to reach its full potential; Identifying amendments required to improve the usability/functionality of the system, including the interface and the development of guidance documents; Identifying the target (and potential) audience in order to inform promotion required to increase MoE usage, in terms of both uploads and searches. The purpose of this report is to summarise the key findings and recommendations from the MoE review in order to inform the re-engineering and relaunch of the database in

3 3 Methodology Sources of data used to inform the review. These are included in the table below: What/how Data provided Breakdown of current Analysis of existing data uploaded on the system. database portfolio Metrics in usage a) Usage in 2014 and 2015 b) Monthly monitoring of number of uploads and searches: Established using MoE web stats and analytics from 2014 and from 2015 Stakeholder views a) Users views and comments on their experience of the MoE and recommendations for improvement. b) Needs assessment of other potential user groups Established by user interview and discussion. A summary of recommendations can be found at the end of the report.

4 4 Findings 4.1 Breakdown of current portfolio All entries from the database were exported into Excel for analysis. A total of 355 entries have been inputted in the database since its inception in 2011, with 139 of these entered by RPS staff. Therefore, a total of 216 sources of information have been uploaded by external user since Figure 1 presents a breakdown of entries by category. Map of Evidence: breakdown of entries by category 7, 2% 14, 5% 15, 5% 29, 10% 103, 35% 127, 43% Research Audit Service Evaluation Service Development Case Studies Practice Examples Summary of Results The largest number of entries were recorded as research, with the second highest number of entries recorded as audit (over one-third). Analysis of the current portfolio highlighted issues around entries being incorrectly recorded (in the wrong category) by users, for example, audits being recorded as research. In addition, 27 entries were not recorded under any category. Subheadings were not included by users for the majority of entries. Out of the total number entries with subheadings (42), survey was the most frequently used subheading, followed by qualitative. There appeared to be some inconsistency in subheadings used with some users defining their research by approach, type of research, method and others by discipline, etc. A significant number of entries did not include any results or attachments (113 entries). 4

5 4.2 Analysis of current usage Table 1: The table below summaries the analytics and usage data collated Year Average monthly views New user led submissions The data presented above suggests that there has been little activity to increase the recorded evidence base over the past 2 years, however, the MoE is still attracting users seeking evidence. This information informs a number of direct recommendations around increasing usage of and engagement with MoE and provides a useful baseline for measuring and assessing future work in 2016 and beyond.

6 4.3 Stakeholder analysis Stakeholder views on aims of MoE Stakeholders were asked to provide information on the current aim, scope and functionality of MoE, how they would envisage using the MoE and feedback on what MoE outputs would be useful to them. Stakeholder Group Their interest or requirement from the MoE Perceived barriers/risks Early Career Researchers (including RPS members) Looking to share and showcase findings. Potential users of MoE, in terms of uploading information as well as searching for evidence to inform own projects (local / small-scale projects) and evaluate and develop own practice. Loss of interest Lack of time Quality of entry Local research leads Developing research culture within their setting - wish to encourage MoE usage as part of that. No information collected. Non-research users (policy stakeholders teams) A repository of high quality (peer reviewed) evidence to inform RPS policy and campaigning work. The MoE may not capture all relevant information on a topic. An easily accessible and simple to use database of policy relevant research would allow teams within the RPS to identify the evidence base and best practice examples for their work. Gaps in the evidence base will risk bias in the development of policy. Synopses aligned with RPS strategy and campaigning activity would give the policy teams an evidence base to work around and would help to identify key areas of research in need of further work. A good method of tracking member research activity. 6

7 The teams would be able to use the database more meaningfully if the quality of the information given for each project was improved. Academic Pharmacy/ Research Community Potential users and may also be searchers. Possible input re: scope and aim of MoE. Lack of interest and / or time Expert Advisors Committed to building the evidence base. The MoE to should cover all research undertaken and give clear information about the type and the value of evidence/results. Used as a resource for searching for research that is translational in nature or as a primer to develop research ideas from best practise. The map does not represent early stages of the research pathway. With limited evidence represented on the map it would be unfeasible to use the map for national level activity. The risk would be that, without capturing all relevant results in an area, the MoE display bias.

8 Stakeholder views outlining concerns and current issues with the MoE. Feedback on recommendations and suggested improvements to the MoE were categorised under two main areas; submissions and searches. Both the quality and standardisation or evidence were addressed. Functionality Findings from portfolio analysis and stakeholders Action Concerns over the quality of current MoE submissions. Existing data to be standardised and correctly categorised. Quality of submissions Submissions to the database lack rigour and do not have a review in place. Set up an authorisation and approval process to include notification to administrator and submitter on submission. Individual studies are inconsistently reported on the system. Data on who is using MoE to be tracked. No clear delineation between peer reviewed work versus non peer reviewed work. No clear delineation between research, service evaluation and audit. Low quality of submissions means the evidence cannot be used. Lack of delineation means it is more challenging for users to conduct meaningful searches. Poor quality submissions discourages others from uploading and sharing their research, or using it as a source of information. The quality of searches Some stakeholders found conducting a search on MOE relatively straightforward and that existing search terms and key words worked well. Others found the search function problematic and that key words they used did not always locate relevant documents. Believe all evidence captured should include information on the type and value of the evidence and results. High quality research to be easily identifiable from lower quality research. Overhaul of drop down menus: o Remove what is not required/useful e.g. LPF o Update/alter if needed e.g. county o Add additional categories e.g. research, service evaluation, peer reviewed, best practice, etc. Improve key word search function o Use information from standardising submissions (existing searchable database datasets (to define a list of key words and ensure new submissions use these terms. 8

9 o Re-categorise existing submissions with new key words and ensure new submissions use these terms.

10 5 Summary Feedback from stakeholders and web statistics suggests: o awareness and knowledge of the MoE is low; o limited understanding of the purpose of the database; and o some users have also experienced issues around the functionality of the system; o contrasting views from different stakeholders on the purpose, audience and scope of the database were noted. Low awareness of the database has resulted in: o limited number of users accessing the MoE to share their findings, particularly research conducted by early career researchers who often struggle to find relevant platforms to showcase their work; o low numbers of users accessing the MoE to using the database to search for relevant for evidence/findings. Issues around the functionality/usability of the site appear to have contributed to reduced usage of the database. 6 Recommendations Aims and objective of MoE 1. Embed the Map of Evidence in the wider RPS evidence strategy. Functionality 2. Improve functionality/usability of the site, in particular the search function (to allow detailed searching) and data capture form. 3. Develop guidance documents and help section for users with step-by-step instructions on how to upload and search information on the database. 4. Introduce an authorisation and approval process to monitor and ensure the quality of submissions. Data collection 5. More high quality evidence identified and uploaded onto the MoE. 6. Develop communication plan to promote the purpose and re-launch of a re-engineered MoE. To include targeted promotion of the benefits of the database to different potential user groups (users and submitters). Usage/ Improved Outputs 7. To measure and review usage in 2016 and compare with data collected in Create an operational plan for how collated evidence will be use and disseminated to stakeholders. 10

11 Copyright Royal Pharmaceutical 2016