EMQN UPDATE: GERMLINE SCHEMES 2017 USER FEEDBACK SURVEY

Transcription

1 UKAS accredited Proficiency Testing Provider EMQN Office Manchester Centre for Genomic Medicine, 6th Floor, St Mary s Hospital, Oxford Road, Manchester M13 9WL United Kingdom Tel: Fax: office@emqn.org EMQN UPDATE: GERMLINE SCHEMES 2017 USER FEEDBACK SURVEY TABLE OF CONTENTS TABLE OF CONTENTS... 1 SCOPE EMQN Services EMQN Educational Services Common feedback themes Next steps Summary Appendix... 3 SCOPE EMQN are committed to continually improving the services it provides and sent out a survey in Oct 2017 to request feedback on performance for the 2017 germline schemes. 126 germline scheme participants responded in total. There were approximately 575 laboratories registered in these schemes giving a response rate of 22% registered labs. This document provides a brief summary of the results of the survey, and the proposed actions in response to the feedback received. 1. EMQN Services The survey asked the respondents to rate their experience of participating in an EQA scheme with EMQN from 1-10 (where 10 is exceptional and 1 is very poor). The average score was 8.5, with 97% rating EMQN 7 or higher (Fig 1). Fig 1. Survey Rating of EMQN services On a scale of 1-10, where 10 is exceptional and 1 is very poor, please rate your overall experience of participating in an EQA scheme with EMQN Percentage 60% 40% 20% 0% Scale EMQN Educational Services The feedback from the survey was that the educational services EMQN provides is useful, with 88% finding the new example educational reports introduced in 2017 somewhat useful or very useful (Fig 2). Version 1 Page 1 of 5

2 98% of respondents rated the summary scheme reports and best practice documents provided by EMQN as very useful or somewhat useful. In response to comments, EMQN will review the best practice documents to determine if any need updating or if EMQN can provide new best practise guidance documents for areas not already covered. Fig 2. Response to question about EMQN Educational services How useful do you find our additional educational services? Very useful Somewhat useful Do not find useful Do not use/not applicable Summary scheme report 73.0% 25.4% 1.6% Educational reports (New for 2017 schemes) 64.3% 23.8% 0.8% 11.1% Best practise documents 79.4% 19.1% 1.6% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 3. Common feedback themes The EMQN team reviewed all the comments provided in the survey, and found that there were several common themes; - Positive feedback Most feedback was positive with users commenting that they were happy with the service they received which was great to hear. - Harmonisation of assessment Several people commented that they felt that assessment needed to be more standardised between schemes, and across different years. - Improvements to website There were some comments that the website could be more user-friendly. - Providing panel testing Many commented that they would like to see panel testing schemes introduce. EMQN have pilots for panel testing schemes in Cardiac Arrythmia and Cardiac Hypertrophic Cardiomyopathy in In the survey, users were asked if there were any new schemes they would like to see introduced. EMQN will review these and look into potential new schemes that could be offered in the future, including more panel testing. - Taking into account variation in national practices A number of respondents commented on the variation in national practices/regulations sometimes resulting in penalties for the labs. Participants can upload a document explaining this when they upload their reports to the website so that national variations can be taken into account, EMQN will make this guidance clearer in the 2018 scheme instructions. 4. Next steps Appendix 1 details the actions proposed in response to the feedback received in the survey. EMQN will take all of these suggestions to the quality team and will address them. 5. Summary Thank you very much to everyone who provided feedback, the results of the survey will be used to improve the quality of service EMQN provides in future schemes. Please contact EMQN if you would like to hear about the results in more detail or if you have any comments. Version 1 Page 2 of 5

3 6. Appendix Feedback Response Proposed Actions Samples: 4 participants commented that they would like to be provided with more DNA, and 2 commented that they often received less DNA than they were expecting according to their quantification methods. 2 participants commented on the quality of the DNA. The DNA samples are sent in small volumes at a high concentration (~200ng/ul), this is required for southern blots but the method is rarely used now. Any evaporation can mean loss of sample, and quantification methods can give very different concentrations on different platforms. There have been historical issues with DNA quality for MLPA using the chemagen for DNA extraction of cell lines, these have been addressed by changing the beads used by the chemagen platform but there may still be some quality issues. Send a sample survey in 2018 to get a better idea of how much DNA participants need for testing and at what concentration? Discuss whether there is any way to improve sample quality. Harmonisation: 7 people left comments regarding lack of harmonisation in scoring; this was between schemes, as well as between different years. It was felt that standardisation was needed. This is a recognised issue. EMQN are currently considering setting up scientific advisory groups (SAGs) for germline schemes which could help with harmonisation, for example the SAG could do a final review of the marking criteria etc at the end of a scheme year for implementation the following year Review the harmonised marking criteria and the HGVS criteria Consider SAGs for germline schemes Timelines: Feedback was conflicting with some requests to split schemes to multiple distributions, and some requests to send all schemes at the same time The feedback will be considered but in terms of cost for participants and harmonisation, it is more efficient to keep one round of germline distribution. None for now Website: 6 respondents commented that the web-site could be more user-friendly, particularly regarding documentation. Changes to the website are currently in progress Include some training for using the website eg a recorded webex showing where to find documentation etc Appeals: There were 4 ratings of poor for the appeals process. Across all the comments in the survey, 5 respondents made comments regarding disagreement with the outcome of the appeals. All appeals are fairly reviewed by a panel of experts in the field. In 2018 there were161 appeals across 32 schemes (2158 participations and 6474 reports assessed) 39% appeals were upheld/partially upheld Improve harmonisation (as above) 55% were rejected 6% were not appeals (comments/rcas) Variation in national practices: It is possible for the labs to submit a document explaining deviation from Make the wording clearer in the scheme instructions Version 1 Page 3 of 5

4 5 respondents commented on the variation in national practices/regulations sometimes resulting in penalties for the labs. reporting due to national regulations Scheme Reports: Some feedback commented on improving the content of the scheme reports The marking criteria gives participants some idea of the requirements by the assessors, and references are included for guidance on writing a clinical report Include more do s in the report, what the assessors want to see. A section is included to describe common findings each year. Best Practice: Three people surveyed commented that they would like more best practise guidelines available, for example for NGS, and one person commented that best practice guidelines should be updated. Reporting: Comments that the reporting process is onerous, should be more demanding, report format should be excluded from evaluation. EMQN are currently planning to put more resource into planning BP meetings Review which best practise documents need updating, and which disease areas have no best practise guidance (EMQN or external) Consider an optional report which is not marked ie. Won t count towards poor performance but for educational purposes Errors: Comments about errors such as corrections to scheme documentation Genotyping only schemes: 2 people requested options for genotyping and full schemes, and 1 request for a clinical interpretation scheme Purchasing: 3 comments about purchasing issues such as referencing the schemes purchased, incorrect addresses and timelines EMQN are continually auditing/improving procedures via quality management systems. Some errors are due to lack of resource which has been addressed by a recent recruitment EMQN are recognise the testing landscape is changing with some laboratories outsourcing/performing genotyping only. EMQN are currently considering the possibility of genotyping only/interpretation only modules to the EQA schemes The invoicing system has recently been updated which should resolve some issues Continual improvement of procedures in response to audits and errors captured as non-conformances TBC Review in 6 months Shipping: 3 comments about shipping due to incorrect shipping details, providing tracking information quickly, and issues shipping to certain regions. The Admin team now check the DHL contact details vs the EMQN contact details Review in 6 months Version 1 Page 4 of 5

5 Language: Request to use clear and unambiguous language This should be improved with better harmonisation (see above) Improve harmonisation (as above) Conflict of interest: 1 comment People who are presently working or were previously working in the lab which is evaluated should be excluded from the evalution from this particular laboratory. All assessors/organisers sign a CDA, EMQN ask all laboratories to anonymise their reports and ask if an assessor/organiser recognised a report that they refer it to someone else for assessment. There are always at least two assessors involved with marking each report, and a final moderation process to make which ensures assessment is fair and impartial. None 1 comment many of your mail are considered spam or junk mail by our hospital. Therefore, some mails might not be received, which could be a problem. EMQN are trying to increase the number of contact addresses so that alternative addresses are available if there are any issues contacting an individual or due to s being blocked. Unfortunately, it is not possible to avoid spam filters for all different IT systems. Add alternative contact details for participants Scheme instructions: 1 comment the "instructions" are too long HGVS: 1 comment EMQN could specify application of HGVS recommendations for the description of sequence variants, in the past the recommendations has changed frequently and these changes caused uncertainty about the correct or acceptable version EMQN take a pragmatic approach to HGVS and recommend use where clinically useful (eg not necessarily use for large deletions/mlpa). EMQN recognise that there is some confusion with the HGVS rules following the current release, and EMQN are currently having discussions about this. Review 2018 scheme instructions Follow up discussions and review HGVS criteria END Version 1 Page 5 of 5