Statistical Methods in Trending Ron Spivey RETIRED Associate Director Global Complaints Tending Alcon Laboratories
What s In It For You? Basic Statistics in Complaint Trending Basic Complaint Trending Concepts Basic Complaint Scorecard Concept
Key Points of Understanding This is not a statistics class There are different ways to analyze data There are different ways create scorecards
THIS IS ONE OF THE MOST IMPORTANT FACTORS IN TRENDING COMPLAINT DATA (In My Opinion)
Clarify Expectations What exactly are you counting? Complaints Customers Investigations Products As Reported As Found Complaint Descriptions Complaint Codes Occurrences How Many Times Did It Happen
Understand Your Trending Suppose Complaints Are Up This Month Compared to What? Previous Month Average of Previous Year Your Competitor If you normalized your data would the complaint count still be noticeable?
How Do Your Normalize Your Data? There Are Many Different Ways To Normalize Your Data Just be sure you know what is expected Product Sales Products Manufactured Procedures Whatever Method You Use Stick With It Be Consistent
Clearly Identify the Issue What Type of Complaint Is It? Safety/Adverse Event or Quality/Technical? How Do You Identify It? Safety Issue FDA Patient Codes MedDRA Medical Dictionary for Regulatory Activities Quality Issue Customized Internal Complaint Coding FDA Device Codes
Why Does It Matter? Your data may be based on internal coding Someone else may report on FDA coding The numbers may not match Who is Right Who is Wrong? Both are correct but. Senior Management needs to understand this Clearly Quickly
There Are Aspects of Trending That We All Do Every Day We Sort We Rank High to Low, Cheap to Expensive, Important to Unimportant We Arrange According To Some Pre-Determined Measurement
Think About: Grocery Shopping Sorting Vegetables By Condition Green Ripe Just Right Clothes Shopping Sorting By Price Cheap Affordable Too Expensive Washing Clothes Sorting By Stains Filthy Normal Slightly Dirty Wear Again (granted, this is a guy s perspective)
Washing Socks
Bell Curve
What Is A Bell Curve The term "bell curve" comes from the fact that the graph used to depict a normal distribution consists of a bellshaped line. The bell curve is also known as a normal distribution. The highest point in the curve, or the top of the bell, represents the most probable event. All possible occurrences are equally distributed around the most probable event, which creates a downward-sloping line on each side of the peak.
For Our Complaint Trending Purposes We Are More Concerned About the Plus Side Rather (the right side) Than the Minus Side (the left side) We Will Focus On What Is Greater Than The Average.
Each Vertical Line Represents A Deviation
Don t Be Intimidated By Statistical Terminology Just Focus On WORDS Commonly Used In Trending Average Mean Median Quartile Complaints Per Million* Standard Deviation Sigma (1, 2, 3) Outlier
Just So You Know Mean & Average Standard Deviation & Sigma Upper Control Limit & 3 Sigma (From a non-statistician point of view)
Standard Deviation A measured variation or "deviation A measured point away from the average (mean).
Separate The Words Standard an established requirement, an expected measurement. If you are a golfer, think of it as Par Deviate (noun) a person who strays from the norm. Deviate (verb) to depart or stay from the established course or standard.
Answer This Question If you have one month of complaint data that is above 3 standard deviations from your average monthly complaint data Wouldn t You Want To Know?
Why Is It Such A Big Deal If You Are Above 3 Standard Deviations?
What Are We Seeing? One standard deviation away from the mean in either direction on the horizontal axis (the yellow area on the graph) accounts for around 68 % of the data in this group. Two standard deviations away from the mean (the red and yellow areas) account for roughly 95% of the data. Three standard deviations (the red, yellow and blue areas) account for about 99 % of the data.
So Basically Less Than 1% of Your Data Should Ever Be Expected To Be Above 3 Standard Deviations From the Average. So I Repeat Wouldn t You Want To Know If That Ever Happens?
Calculating Standard Deviation 61-36 1,296 70-27 729 75-22 484 78-19 361 82-15 225 86-11 121 90-7 49 100 3 9 100 3 9 126 29 841 130 33 1,089 170 73 5,329 97 10,542 / 11 1. Get Average of your data 2. Subtract each data point from the average 3. Square the value of each answer 4. Add together all squared values 5. Divide answer by your total number of data points minus one. (12 1 = 11) 6. Get square root of your answer The Square Root of 958 = 31 958
97 31 128 97 2*31 159 97 3*31 190 Calculating Sigma 1.Add 1 Std Dev to the Average (1 Sigma) 2.Add 2 Std Dev to the Average (2 Sigma) 3.Add 3 Std Dev to the Average (3 Sigma)
Let s Look At 12 Months of Complaints Before you start, clarify what the month represents. Date of 1 st Receipt-1 st Awareness Date Input In Your System Date Reported to a Regulatory Agency Date Investigation Opened
Mon/Yr Jan-10 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Complaint Count 70 90 170 100 82 75 78 61 86 100 126 130 1,168 What Would This Look Like In a Control Chart
A Control Chart Is Divided Into Zones: + 3 Sigma Upper Control Limit (UCL) + 2 Sigma + 1 Sigma Mean (average) - 1 Sigma - 2 Sigma -3 Sigma Lower Control Limit (LCL) Also Referred to as Six Sigma
200 180 160 170 140 120 126 130 LCL (- 3 Sigma) 100 80 60 70 90 100 82 75 78 61 86 100 Complaints Avg Complaints UCL (+ 3 Sigma) 40 20 0 Jan-10 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
12 Months of Complaints How Do I Determine Complaints Per Million (CPM) Mon/Yr Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Sales 50,000 51,000 52,000 51,000 50,000 49,000 48,000 30,000 48,000 49,000 50,000 55,000 583,000 Complaint Count 70 90 170 100 82 75 78 61 86 95 100 110 1,117 CPM 1,400 1,765 3,269 1,961 1,640 1,531 1,625 2,033 1,792 1,939 2,000 2,000 70 Complaints in January Divided By 50,000 Sales in January = 0.0014 0.0014 * 1,000,000 = 1,400 Complaints Per Million
Setting Thresholds One Method Set Up Complaint Threshold On Previous Year of Data Think of a Threshold as a Limit
Setting Thresholds One Method The 3 Sigma Limit In 2013 Can Be Your Threshold For 2014. Use This Number as Comparative Analysis Or As A Limit To Compare To Each Month In 2014.
Other Comparative Suggestions You Shouldn t Just Compare Each New Month in 2014 to Your 2013 Threshold. You Should Shift Your Data Every Month (rolling 12 months) You Should Also Compare the Current Monthly Complaint Data to the Previous 11 Months of Complaint Data Example: Compare the Count This Month to the Average of the Previous 11 Months
Complaints Use Your Data To Make A Line Graph 400 4,000 350 3,500 300 3,000 250 200 150 UCL (3 Sigma) 2,500 2,000 1,500 Complaints Per Million 100 Avg Complaints 1,000 50 500 0 Jun-10 Jul Aug Sep Oct Nov Dec Jan-11 Feb Mar Apr May Complaints 75 78 61 86 100 126 130 65 62 75 87 130 Avg Complaints 94 94 94 94 94 94 94 94 94 94 94 94 Sales 50,000 51,000 52,000 51,000 50,000 49,000 48,000 30,000 48,000 49,000 50,000 58,000 CPM 1,500 1,529 1,173 1,686 2,000 2,571 2,708 2,167 1,292 1,531 1,740 2,241 UCL (3 Sigma) 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 2,316 0
Complaint Scorecard Separate From Your Complaint System Identify Key Issues or Trends Just the Highlights No Emotion, Focus on the Data Senior Management Review You Are Not The Owner of the Issue
Practical Functions of a Complaint Scorecard Potential Warning of Emerging Problems Serves as an early warning system for significant trends Senior Management Attention Endorses extended detailed investigations of unique issues. Ensures that investigations receive adequate support. Ensures that timely updates are provided until closure. Endorses defect thresholds for identified product issues. Complaint Topic Awareness Has the issue been seen before? How often are we seeing the issue? Was there a previous CAPA for this same issue / same product? Is there a similar CAPA relative to a different product?
Basic Rules for Scorecard Co-Originators (Gatekeepers) Be independent of the complaint process. Have consistent contact with key associates. Don t ask permission to add an issue. Analyze data before an issue is added: Product information Customer information Internal information Preliminary investigation results
Scorecard Data Year Opened Month Opened Date Opened Status (New,Carryover, Closed) The Issue (Give short description of this problem) Product (Provide most commonly known product identifier) Complaint Class (How the complaint is coded) Details (Give more details but still keep it short ) Alert Triggers (Above 3 Sigma; Above Established Threshold ) Total Sales for Month Current DPM Average DPM Original DPM Complaint Threshold (Based on Previous Year Data) Type (Quality, Safety) Division (Device, Pharma) QA Group Owner (QA Plant Director) Owner's Update Owner s Closure Summary Owner's Closure Approval CAPA? ( Yes or No) (If No, the owner needs to explain why not) CAPA Title / Ref # (Provide reference for quick identification) CAPA Open Date CAPA Due Date CAPA Close Date Effectiveness Check? Date of Effectiveness Check
Closing A Complaint Scorecard Topic The Owner Decides If It Is Ready to Close The Owner Must Provide Closure Summary All Information Must Be In the Complaint File Senior Management May Request More Data
THANK YOU Ron Spivey RETIRED rspiveytx@gmail.com 817-939-4871