A multi-perspective methodology for studying user interactions with a decision-support system

Transcription

1 A multi-perspective methodology for studying user interactions with a decision-support system Mor Peleg 1, Aviv Shachak 2, Dongwen Wang 3, Hana Sender 1, Shiri Shenhav 1 and Eddy Karnieli 2,4 1 Department of Management Information systems, University of Haifa, Israel; 2 Faculty of Medicine, Technion, Haifa, Israel 3 Biomedical Informatics Program, University of Rochester, NY, USA 4 Inst. of Endocrinology, Diabetes & Metabolism, Rambam Medical Center, Israel

2 Motivation for developing the decisionsupport system (DSS) for diabetic foot Diabetes foot problems may lead to amputation The American College of Foot and Ankle Surgeons developed a suitable guideline Implementing a guideline as a DSS may improve its effectiveness Goal: develop an effective implementation

3 Early steps in developing the DSS Focused on local adaptation, encoding, EMR integration lymphadenopathy =True or lymphangitis =True or absces = True. EMR Encoding in GLIF Execution with GLEE

4 Multiple methodologies for designing and evaluating a UI Collecting user needs Requirements analysis UI design Design evaluation

5 Collecting user needs Qualitative research methods: Field observation (M, W) Think aloud (M,W) Document analysis (M)» Instructional material» EMR Structured interviews (M, W, UI)

6 Structured interviews family practitioners, 1 diabetes expert, a vascular surgeon, and 2 nurses 7 questions related to clinical practice What kind of physical exams do you conduct? 8 questions related to the UI design

7 Questions regarding UI design User workflow Do you enter data into the EMR during the foot exam? User needs for the DSS Would you like to receive advice for referrals to a surgeon? User's preferences regarding system interaction Do you prefer an interactive flowchart, or a short textual summary at the end?

8 Requirements analysis via UML Sequence Diagrams Make inflamation decision<decision= no >, make severity classified decision<decision= yes >, Make referral decision <decision= st<2 > Make decision offloading done already <decision= no > Run offloading subguideline Show options for choosing offoading Set offloading selection Run Evaluation Frequency subguideline Make risk category decision<decision= 3> AddRecommedation ToBuffer ( Evalution frequency:1-3 month ) AddRecommedation ToBuffer ( Consider EMG )

9 Prototype/demo Interactive Powerpoint presentation Screenshots generated using Visual Basic We used two patient cases to draw user feedback - a simple case and a complex Printed educational material

10 get medical advice

11

12

13

14

15 Design Evaluation (Nielsen) Aesthetic and minimalist design Visibility of system status (was missing) Consistency and standards inconsistent positioning of screen elements Error prevention and error recovery (no undo) Match between system and the real world User control and freedom Recognition rather than recall Flexibility and efficiency of use

16 Think-aloud protocols Text size that was too small Lack of a Save button at the end of the system session If this were an active system, I'd use it tomorrow morning

17 Questionnaire based on the Technology Acceptance Model Usefulness (4,2,2) Using the system will make my work more effective To what extent do you agree with the decision support given by the system to patient #1? The explanations contributed to my understanding to a great extent Ease of use (6) I think that the system is easy to learn I think that the system is complex to use Intention to use (2) If I may have the option, I would prefer not to use the system

18 Questionnaire Results Mean rwg(j) Intention To use Perceived Ease of use Perceived usefulness Explanations usefulness Recommendation usefulness Physician # -likert scale

19 Summary We used multiple methodologies to develop DSS Acquiring and analyzing user needs Designing a suitable prototype Evaluating the prototype Proceeding to implementation The users valued the prototype s assistance They liked the minimal interaction design, patient education materials & explanations They thought that the UI was clear and easy They expressed high intention to use the DSS

20 Discussion Limitations: Heuristic evaluation done by one person Small num of physicians determined requirements and evaluated the prototype Currently implementing the system Our future plans involve suggesting this DSS to other healthcare organizations Evaluate the impact of the system on data collection, lab orders, and outcomes

21 Thanks!

22 How can the DSS assist users? Help in deciding How to treat infection When to refer to specialists Which tests to order How to stratify risk How often to follow up Provide patient education material Provide explanations