1/26/16. Usability activities are NOT clinical Trials. Usability Testing Medical Devices. It s going to feel like a clinical trial

Transcription

1 Usability activities are NOT clinical Trials Usability Testing Medical Devices UPA 2012 Las Vegas, NV Generally the process will be less stringent: More loosely defined, faster Will likely involve a relatively small number of people per activity Initial studies stress simulation, with no medicinal agents or clinical impact. Dovetail with interaction design, visual design, and marketing activities Dan Berlin Experience Research dberlin@madpow.net Chris Hass SVP of Experience chass@madpow.net Usability Tes0ng Medical Devices 2 It s going to feel like a clinical trial Extremely detailed Usability Specification documents Usability Validation (Risk assessment and amelioration) Usability engineering file / Design Efficiency File Know Federal and International regulations: FDA Quality System Regulation HE: ISO/IEC 62366:2007 Knowledge of (and inspiration for) emerging standards Medical Device Manufacturers may be unfamiliar with your practices and lack of regulations Usability Tes0ng Medical Devices 3 Usability Tes0ng Medical Devices 4 1

2 Safeguarding participants may be more challenging Your safety protocols must be exacting, fully defined, piloted, and effective Even simulated product use may put participants at risk Target audiences may be physically or cognitively vulnerable Activity environments may be unusual Work with client and medical care specialists Reporting responsibilities, guidelines, formats, and requirements will be more complex Reporting guidelines, formats, and requirements will likely be extensive and complex Industry reporting standards (CIF) Specialized reporting formats Raw data may be reviewed again and again and again... Usability Tes0ng Medical Devices 5 Usability Tes0ng Medical Devices 6 Research locales may be more challenging Device use environments may be: Unfamiliar Sterile Stressful Germ-laden Emotionally distressing Gross Secure Rarely observable Complex or otherwise unusual to non-medical personnel (And fainting can be a factor!) Client Relationship A robust set of client team members is essential as is knowing their roles: Legal Medical advisor Safety advisor Compliance Adverse Event Specialist Physician is helpful, but not enough The usual suspects Clients may ask for proof of insurance But too much insurance could be a bad thing Source: Usability Tes0ng Medical Devices 7 Usability Tes0ng Medical Devices 8 2

3 FDA Relationship Working with an IRB If the study is for FDA approval learn about the client s relationship with the FDA: What are the relevant regulations? Who is the client s liaison to the FDA? What is the client s relationship with the FDA? Has the client met with the FDA? Have they received comments from the FDA? Does the client understand usability testing? Are we allowed to talk to the FDA? Source: IRB = Institutional Review Board Source: who_is_watching_the_irb_watchdogs_asks_nature For FDA approval, typically work with an independent IRB Their job: to ensure that research conducted on human subjects is carried out correctly and humanely Not as scary as you may think We are NOT evaluating medical efficacy or safety But we DO have to be conscientious Have your ducks in a row and it will be a smooth process Usability Tes0ng Medical Devices 9 Usability Tes0ng Medical Devices 10 Working with an IRB IRB Documents - Forms Adds about $2000-$5000 to the cost of the study Adds a day or two to the project for dealing with the IRB Be cognizant of any non-disclosure agreements Prior to the study, signature pages should be signed-off, circulated, and returned in a timely manner Everyone on the team will take this very seriously! Source: who_is_watching_the_irb_watchdogs_asks_nature Indemnification Agreement: So the IRB doesn t get sued Source: Financial Disclosure: The researchers don t have conflicts of interest Investigator Statement: The researcher code of conduct Satellite Site Application: May be needed if the study involves multiple locations Shipping & Invoice: Where documents are sent Study Application: Detailed information about the study Curriculum Vitae for all researchers Other forms that your IRB requires Usability Tes0ng Medical Devices 11 Usability Tes0ng Medical Devices 12 3

4 IRB Documents Study and Device Info IRB Documents Validation Plan Recruitment Screener: Be very precise Source: Recruitment Flyer: To hang in doctor s office to aid with recruiting Moderator s Guide: A second guide may be needed if there are child participants Consent and asset forms: For adults and children, respectively Safety Concerns: Outline potential risks Device Background: Detailed explanation of the device including indication and new features Device Instructions for Use: If they are also included in the study A detailed explanation of how the study will validate that the device is usable If you re used to winging it, you can t do that here Lay out every question and how it will be interpreted Lay out a rationale for every metric Break down the pieces of the tasks and what will be observed A detailed explanation of how the will be analyzed and presented Quantitative measures (asked and observed) Rating scales Yes/No observations Source: hep:// Usability Tes0ng Medical Devices 13 Usability Tes0ng Medical Devices 14 Quantitative Measures Rating Scales Asked Ease of use Instructions for Use clarity Design Ergonomics Observed Usability Help needed Number of errors Yes/No observations Did the participant do this, that, AND the other thing? Chart Color Ra0ng No Help Needed LiEle Difficulty Some Help Much Help Task Failure Explana0on Completed with no help 1-2 small errors 3-5 errors Moderator gets involved Failure Small errors: Doesn't press the green bueon Doesn't remove cap properly Etc. % of chart that is red % of chart that is black Recruiting Give your recruiter a heads up LONG before the study starts And start recruiting early Make a flyer to be hung in doctor s offices and distributed electronically which includes: The purpose of the study Why they would want to participate Compensation Recruiter contact information During the project, any screener changes need to be re-approved by the IRB Many institutions can t promote a study that is under their auspices Source: Usability Tes0ng Medical Devices 15 Usability Tes0ng Medical Devices 16 4

5 Sample Flyer Recruiting Don t be shy makes friends with local hospitals and clinics Nurses and admins are the gateway Have the sanitized IRB approval in-hand Find groups that can be a gateway to your desired audience Healthcare institutions Local and national support groups Federal agencies Community support groups Craigslist & Social Media Source: Plan for surprises like extensions to the timeline Usability Tes0ng Medical Devices 17 Usability Tes0ng Medical Devices 18 Working with Children Working with Children From a legal standpoint, a child is someone under 18 years old Children cannot consent to participate in the study, they can only assent An assent form explains the details of the study, in the first person You will treat a 17 year old different than an 8 year old Be cognizant of the protocol to ensure you are consistent in your data capture The parent conundrum: should they participate? At what point does it become co-discovery and is that okay? Source: Source: Put the child at ease: Explain the lab (to a point) Ask if s/he would like a break Give reminders that s/he is doing a great job in the study Structure the study so that it isn t drudgery Don t make them remember rating scales Not all children like to be spoken to like a child Be extra vigilant for signs that the participant is telling you want you want to hear As well as signs of fatigue and confusion You don t need to be their friend, but don t scare them away Source: Usability Tes0ng Medical Devices 19 Usability Tes0ng Medical Devices 20 5

6 Moderator s Guide Moderator s Guide Be precise with a tightly written protocol Must be well-defined in advance and piloted No changes to the protocol without revisiting the IRB, the client, the FDA approval minutes, etc. Clients may be used to clinical trials and view usability methods as thin We re not running a clinical trials, but that s no excuse for not adequately preparing Adhere to the protocol! Any changes require IRB approval Source: The moderator s guide should capture: The tasks Steps in the tasks What should be observed The questions to be asked of the participants Observer rating scales Basically, a shortened version of the Validation Plan Be precise in the metrics and methods They cannot vary Data reporting requirements will be significant Gaps in the reporting will be very visible Source: Usability Tes0ng Medical Devices 21 Usability Tes0ng Medical Devices 22 Notes Grid Notes Grid We are using micro-tasks and want to capture precise information The notes grid should facilitate the analysis yet be easy to use Every step and question on its own row Have clear study sections (introduction, etc.) Grey out areas that can be ignored Team note taking: moderator concentrates on certain items, note taker does the rest Source: Usability Tes0ng Medical Devices 23 Usability Tes0ng Medical Devices 24 6

7 Lab Preparation Participant Safety Capture the participants from two different angles A close-up view of the device interaction (over-the-shoulder) Straight-on to capture the entire field Have all device supplies at the ready As with any usability, participant comfort and safety is the top priority Must be mindful of participants with health conditions Will they need accessibility accommodations? Will they have an aide/caretaker? How will the condition affect their participation? Source: Alcohol wipes, extra needles, batteries, etc. Source: What you can do to ensure participant comfort and safety: Have a robust first aid kit (and possibly medical personnel) You ll use more supplies than you expect Provide water, juice, sugary, and non-sugary snacks Get a first aid kit and nitrile gloves Ensure they know that they will be working with a placebo Have a first aid plan and practice Accessible food Usability Tes0ng Medical Devices 25 Usability Tes0ng Medical Devices 26 Participant Safety Adverse Effects Warn participants of safety hazards while using the device, if it s part of the protocol Have a plan of action if someone is injured Establish a path of notification if a participant is injured Reporting may have many phases: reporting it internally, then to the client, to the FDA, and the IRB Educate yourself about the condition Source: An adverse effect (AE) event is when a participant tells you about a side effect or other experience while using a device or drug This is critical information for the client and must be reported, typically immediately Some clients may require you to go through AE training Source: Ensure the protocol does not exacerbate a disease or condition Be vigilant about fatigue, confusion, disorientation, dehydration, & stress Before the study begins, determine to whom AEs are reported AEs can be complex to identify, but are VERY important When in doubt, report! Usability Tes0ng Medical Devices 27 Usability Tes0ng Medical Devices 28 7

8 Adverse Effects Analysis In the course of a usability study for a medication website, a participant mentions that she stopped taking the drug. She explains that the blister packs and pills sometimes arrived crushed. While testing an infusion pump, the participant mentions that the tube always falls out of his previously used pump, which is the client s product. During a study, a participant mentions that her arms started to itch when she stopped taking the client s medication. A male participant mentions that his wife got pregnant two weeks after he started taking the client s medication, even though she used the birth control pill. The setup from the validation plan, moderator s guide, and notes grid should make analysis easy Dealing with gaps in the data can be tricky Did the question get asked? Was the behavior observed? Do we have to go back to the recordings? Make sure you HAVE the recordings! Rely on quantitative measures What percentage of participants passed the task? What ease of use ratings did participants give? How many participants committed the most common errors? Source: Usability Tes0ng Medical Devices 29 Usability Tes0ng Medical Devices 30 Analysis Thank You! Remember to think on a micro-task scale every little step of using the device should be observed It s difficult to watch and report on every step of a complex interaction and document it precisely, especially live Tallying needs to be specific, yet practical Be mindful of capturing and organizing the important qualitative data This can be especially tough when dealing with many participants Summarize qualitative data directly after the session don t wait Know what you re reporting there may be data to help marketing Sometimes the goal of these studies is to identify marketing claims, which can raise concerns Source: Conducting a usability study for a medical device is similar to a typical study, except with much stricter protocols Your safety and the safety & comfort of the participants are paramount Use quantitative measures and report percentages of participants that did or didn t perform an action It s all about proper planning! Questions/Comments? Dan Berlin Chris Hass Usability Tes0ng Medical Devices 31 Usability Tes0ng Medical Devices 32 8