Statistical Methods in Trending. Ron Spivey RETIRED Associate Director Global Complaints Tending Alcon Laboratories

Transcription

1 Statistical Methods in Trending Ron Spivey RETIRED Associate Director Global Complaints Tending Alcon Laboratories

2 What s In It For You? Basic Statistics in Complaint Trending Basic Complaint Trending Concepts Basic Complaint Scorecard Concept

3 Key Points of Understanding This is not a statistics class There are different ways to analyze data There are different ways create scorecards

4 THIS IS ONE OF THE MOST IMPORTANT FACTORS IN TRENDING COMPLAINT DATA (In My Opinion)

5 Clarify Expectations What exactly are you counting? Complaints Customers Investigations Products As Reported As Found Complaint Descriptions Complaint Codes Occurrences How Many Times Did It Happen

6 Understand Your Trending Suppose Complaints Are Up This Month Compared to What? Previous Month Average of Previous Year Your Competitor If you normalized your data would the complaint count still be noticeable?

7 How Do Your Normalize Your Data? There Are Many Different Ways To Normalize Your Data Just be sure you know what is expected Product Sales Products Manufactured Procedures Whatever Method You Use Stick With It Be Consistent

8 Clearly Identify the Issue What Type of Complaint Is It? Safety/Adverse Event or Quality/Technical? How Do You Identify It? Safety Issue FDA Patient Codes MedDRA Medical Dictionary for Regulatory Activities Quality Issue Customized Internal Complaint Coding FDA Device Codes

9 Why Does It Matter? Your data may be based on internal coding Someone else may report on FDA coding The numbers may not match Who is Right Who is Wrong? Both are correct but. Senior Management needs to understand this Clearly Quickly

10 There Are Aspects of Trending That We All Do Every Day We Sort We Rank High to Low, Cheap to Expensive, Important to Unimportant We Arrange According To Some Pre-Determined Measurement

11 Think About: Grocery Shopping Sorting Vegetables By Condition Green Ripe Just Right Clothes Shopping Sorting By Price Cheap Affordable Too Expensive Washing Clothes Sorting By Stains Filthy Normal Slightly Dirty Wear Again (granted, this is a guy s perspective)

12 Washing Socks

13

14

15 Bell Curve

16 What Is A Bell Curve The term "bell curve" comes from the fact that the graph used to depict a normal distribution consists of a bellshaped line. The bell curve is also known as a normal distribution. The highest point in the curve, or the top of the bell, represents the most probable event. All possible occurrences are equally distributed around the most probable event, which creates a downward-sloping line on each side of the peak.

17 For Our Complaint Trending Purposes We Are More Concerned About the Plus Side Rather (the right side) Than the Minus Side (the left side) We Will Focus On What Is Greater Than The Average.

18 Each Vertical Line Represents A Deviation

19 Don t Be Intimidated By Statistical Terminology Just Focus On WORDS Commonly Used In Trending Average Mean Median Quartile Complaints Per Million* Standard Deviation Sigma (1, 2, 3) Outlier

20 Just So You Know Mean & Average Standard Deviation & Sigma Upper Control Limit & 3 Sigma (From a non-statistician point of view)

21 Standard Deviation A measured variation or "deviation A measured point away from the average (mean).

22 Separate The Words Standard an established requirement, an expected measurement. If you are a golfer, think of it as Par Deviate (noun) a person who strays from the norm. Deviate (verb) to depart or stay from the established course or standard.

23 Answer This Question If you have one month of complaint data that is above 3 standard deviations from your average monthly complaint data Wouldn t You Want To Know?

24 Why Is It Such A Big Deal If You Are Above 3 Standard Deviations?

25 What Are We Seeing? One standard deviation away from the mean in either direction on the horizontal axis (the yellow area on the graph) accounts for around 68 % of the data in this group. Two standard deviations away from the mean (the red and yellow areas) account for roughly 95% of the data. Three standard deviations (the red, yellow and blue areas) account for about 99 % of the data.

26 So Basically Less Than 1% of Your Data Should Ever Be Expected To Be Above 3 Standard Deviations From the Average. So I Repeat Wouldn t You Want To Know If That Ever Happens?

27 Calculating Standard Deviation , , , ,542 / Get Average of your data 2. Subtract each data point from the average 3. Square the value of each answer 4. Add together all squared values 5. Divide answer by your total number of data points minus one. (12 1 = 11) 6. Get square root of your answer The Square Root of 958 =

28 * * Calculating Sigma 1.Add 1 Std Dev to the Average (1 Sigma) 2.Add 2 Std Dev to the Average (2 Sigma) 3.Add 3 Std Dev to the Average (3 Sigma)

29 Let s Look At 12 Months of Complaints Before you start, clarify what the month represents. Date of 1 st Receipt-1 st Awareness Date Input In Your System Date Reported to a Regulatory Agency Date Investigation Opened

30 Mon/Yr Jan-10 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Complaint Count ,168 What Would This Look Like In a Control Chart

31 A Control Chart Is Divided Into Zones: + 3 Sigma Upper Control Limit (UCL) + 2 Sigma + 1 Sigma Mean (average) - 1 Sigma - 2 Sigma -3 Sigma Lower Control Limit (LCL) Also Referred to as Six Sigma

32 LCL (- 3 Sigma) Complaints Avg Complaints UCL (+ 3 Sigma) Jan-10 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

33 12 Months of Complaints How Do I Determine Complaints Per Million (CPM) Mon/Yr Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Sales 50,000 51,000 52,000 51,000 50,000 49,000 48,000 30,000 48,000 49,000 50,000 55, ,000 Complaint Count ,117 CPM 1,400 1,765 3,269 1,961 1,640 1,531 1,625 2,033 1,792 1,939 2,000 2, Complaints in January Divided By 50,000 Sales in January = * 1,000,000 = 1,400 Complaints Per Million

34 Setting Thresholds One Method Set Up Complaint Threshold On Previous Year of Data Think of a Threshold as a Limit

35 Setting Thresholds One Method The 3 Sigma Limit In 2013 Can Be Your Threshold For Use This Number as Comparative Analysis Or As A Limit To Compare To Each Month In 2014.

36 Other Comparative Suggestions You Shouldn t Just Compare Each New Month in 2014 to Your 2013 Threshold. You Should Shift Your Data Every Month (rolling 12 months) You Should Also Compare the Current Monthly Complaint Data to the Previous 11 Months of Complaint Data Example: Compare the Count This Month to the Average of the Previous 11 Months

37 Complaints Use Your Data To Make A Line Graph 400 4, , , UCL (3 Sigma) 2,500 2,000 1,500 Complaints Per Million 100 Avg Complaints 1, Jun-10 Jul Aug Sep Oct Nov Dec Jan-11 Feb Mar Apr May Complaints Avg Complaints Sales 50,000 51,000 52,000 51,000 50,000 49,000 48,000 30,000 48,000 49,000 50,000 58,000 CPM 1,500 1,529 1,173 1,686 2,000 2,571 2,708 2,167 1,292 1,531 1,740 2,241 UCL (3 Sigma) 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 0

38 Complaint Scorecard Separate From Your Complaint System Identify Key Issues or Trends Just the Highlights No Emotion, Focus on the Data Senior Management Review You Are Not The Owner of the Issue

39 Practical Functions of a Complaint Scorecard Potential Warning of Emerging Problems Serves as an early warning system for significant trends Senior Management Attention Endorses extended detailed investigations of unique issues. Ensures that investigations receive adequate support. Ensures that timely updates are provided until closure. Endorses defect thresholds for identified product issues. Complaint Topic Awareness Has the issue been seen before? How often are we seeing the issue? Was there a previous CAPA for this same issue / same product? Is there a similar CAPA relative to a different product?

40 Basic Rules for Scorecard Co-Originators (Gatekeepers) Be independent of the complaint process. Have consistent contact with key associates. Don t ask permission to add an issue. Analyze data before an issue is added: Product information Customer information Internal information Preliminary investigation results

41 Scorecard Data Year Opened Month Opened Date Opened Status (New,Carryover, Closed) The Issue (Give short description of this problem) Product (Provide most commonly known product identifier) Complaint Class (How the complaint is coded) Details (Give more details but still keep it short ) Alert Triggers (Above 3 Sigma; Above Established Threshold ) Total Sales for Month Current DPM Average DPM Original DPM Complaint Threshold (Based on Previous Year Data) Type (Quality, Safety) Division (Device, Pharma) QA Group Owner (QA Plant Director) Owner's Update Owner s Closure Summary Owner's Closure Approval CAPA? ( Yes or No) (If No, the owner needs to explain why not) CAPA Title / Ref # (Provide reference for quick identification) CAPA Open Date CAPA Due Date CAPA Close Date Effectiveness Check? Date of Effectiveness Check

42 Closing A Complaint Scorecard Topic The Owner Decides If It Is Ready to Close The Owner Must Provide Closure Summary All Information Must Be In the Complaint File Senior Management May Request More Data

43 THANK YOU Ron Spivey RETIRED