John Calleja Melbourne Pathology Services. Gold Coast 20th Sept 2013

Transcription

1 Sources of Targets John Calleja Melbourne Pathology Services AACB QC Satellite Meeting Gold Coast 20th Sept 2013

2 Content Definitions Selecting QC Materials Various Sources of Targets How effective they are / Pros & Cons How many Data Points if set your own What do we do when you have multiple instruments/methods t th Setting QC Limits What do we do when we have a Shift Summary & References J.Calleja - Melb. Path. - Sept

3 1. Definitions. J.Calleja - Melb. Path. - Sept

4 1. Definitions: Target A goal we are aiming to achieve. Target Value A value assigned to a reference material (eg) QC Material that we should aim to achieve when the material is analysed routinely. This provides us with a stable reference point against which to measure the stability of our process, over time. Allowable Limits are usually assigned around a Target Value, to denote an acceptable range we should aim to achieve. Target Material A QC Material with desirable attributes & characteristics. J.Calleja - Melb. Path. - Sept

5 2. Selecting QC Materials. J.Calleja - Melb. Path. - Sept

6 2.1 Selecting a Suitable QC material. - General Requirements: 1. Of similar matrix to the usual sample being measured. 2. Materials should have suitable analyte concentrations, which are a. Within the Analytical Measuring Range. b. At critical Clinical Decision Levels.. Normal & Pathological 3. Stable on Reconstitution / Opening. ( Bili, CK, ALP, HCO3 ) 4. Long Shelf Life. 5. Little Vial to Vial variation. 6. Suitable Vial size. - Given Consumption Rate / Stability 7. Safe for General Handling... HIV / HepB / Hep C ve 8. Come from a Reputable Manufacturer/ Distributor. - Prompt / Reliable Customer Service / Product Support 9. Economical... $$$$'s ( Standing Order..configurable ) 6

7 2.2 Selecting QC Materials Requirements/Eval Template General Demographics Analytes Analyser Measuring Range Patient Reference Interval Current QC Levs Level Profiling Results Appraisals J.Calleja - Melb. Path. - Sept

8 3. Sources of Targets. J.Calleja - Melb. Path. - Sept

9 3. Sources of Targets QC Package Inserts Consensus Values from a centralised QC DataBase eg. BioRad s QC-Net, Unity Real Time Set by Your Own Laboratory J.Calleja - Melb. Path. - Sept

10 3.1 Manufacturer QC Kit Insert Values An example of a commercially available QC Material This is an Assayed Material -> it comes with Assigned Values. J.Calleja - Melb. Path. - Sept

11 3.1 Manufacturer QC Kit Insert Values The insert states instrument / method specific values! FT4: +/- 3sd QC Limits pmol/l pmol/l pmol/l J.Calleja - Melb. Path. - Sept

12 3.1 Manufacturer QC Kit Insert Values PSA Abbott Architect Can t just use someone else s targets! Values differ between different Instruments & Methods! Disadv: Your Method Instrument may not be Listed! Vitros ECi Siemens Vista Roche e602 J.Calleja - Melb. Path. - Sept

13 3.2 Consensus Values from QC Database Labs using 3 rd Party QC Packages Mthly Uploads of labs QC Data Internet Log-in to QC-Net & Download Reports Bio-Rad Participant Lab BioRad complies the data from all participating labs & produces method specific summary stats! QC-Net Data- base! J.Calleja - Melb. Path. - Sept

14 3.2 Consensus Values from QC Database World-Wide Wide Report Method Specific Consensus Values for QC Lot! DisAdv: Lot must already be in use by other participants to be of use. J.Calleja - Melb. Path. - Sept

15 3.3 Establish Your Own Target Values ** Establish by - Running the QC Material lseveral times.. For all relevant tests.. On all relevant Instruments / Modules / Channels.. Over a number of days / runs / calibrations /operators.... Thereby exposing the process to as many possible sources of variation, as is practical. - Run the material passively, in parallel to your Existing QCs. - When sufficient Data has been accumulated; - Calculate Mean & SD - Remove any Outliers (Exclude values > 3sd) - Re-Determine the Mean - Set this as the... Target Value J.Calleja - Melb. Path. - Sept

16 3.3.1 How Many Data Points? ** Statistic: Standard Error of the Mean (SEM). This statistic ti ti gives us the Amount of Error, associated with a given Target Value.. Given the - SD of the data set & - the Number of QC observations. S where S = S.D. SEM = --- n = no. of control n observations. The Accuracy of a Target Value.. Increases... with an Increasing Number of Observations. J.Calleja - Melb. Path. - Sept

17 How Many Data Points? - Standard Error of the Mean (SEM) No. of Results 60...No Appreciable 50 Decrease in SEM for n > n = n = 20 n= Error of the Mean ( Arbitrary Units ) J.Calleja - Melb. Path. - Sept

18 4. Issues Relating to various Sources of Targets. J.Calleja - Melb. Path. - Sept

19 4.1 How reliable are Package Insert Values? J.Calleja - Melb. Path. - Sept

20 4.1 Reliability of Manufacturer Kit Insert Values - How appropriate are the targets & ranges provided? The values are.. Mean Values with +/- 3sd Ranges! They.. recommended that each laboratory establish their own acceptable ranges J.Calleja - Melb. Path. - Sept

21 4.1 Reliability of Manufacturer Kit Insert Values - How appropriate are the targets & ranges provided? The values are.. Mean Values with +/- 3sd Ranges! Values were produced by the various Inst. Manufacturers & Independent laboratories They.. recommended that each laboratory establish its own acceptable ranges Disclaimer.. Variations between lab ranges & established ranges, may be caused by.. Differences in; technique, reagents, instruments & by manufacturer test modifications J.Calleja - Melb. Path. - Sept

22 4.1 Reliability of Manufacturer Kit Insert Values - How appropriate are the targets provided? FT4 Reasonable! MPS Assigned Values Consensus Values Kit Insert FT4: Derived +/- 2sd QC Limits sd=0.62, cv=7% sd=2, cv=7% sd=4.66, cv=7% J.Calleja - Melb. Path. - Sept

23 4.1 Reliability of Manufacturer Kit Insert Values - How appropriate are the targets provided? Vit B12 Reasonable! MPS Assigned Values Consensus Values Kit Insert B12: Derived +/- 2sd QC Limits sd=30, cv=12.9% sd=38.5, cv=10% sd=53, cv=10% J.Calleja - Melb. Path. - Sept

24 4.1 Reliability of Manufacturer Kit Insert Values - However, Caution.. if you Don t Set Your Own identical sets of data but with different target values 1. Target= qc data 9 low lim target high lim 2. Target= qc data 9 low lim target high lim Inappropriately p set Targets will cause QC Rules to fail unnecessarily! J.Calleja - Melb. Path. - Sept

25 4.1 Reliability of Manufacturer Kit Insert Values - How appropriate are the ranges provided? FT4: +/- 3sd QC Limits pmol/l pmol/L pmol/l Derive, SD & CV -> sd=0.66 -> CV=7% -> sd= > CV=7% -> sd=4.68 -> CV=7% Compare to Biol. Goals CV i (FT4) = 7.6% Min: 0.75CV i = 5.7% Des: 0.5CV i = 3.8% Opt: 0.25CV i = 1.9% Ranges Given are significantly wider than the FT4 Biological Goals.. => May not be appropriate from Clinical Needs Basis! J.Calleja - Melb. Path. - Sept

26 4.2 What if you have Multiple Instruments measuring the same test? J.Calleja - Melb. Path. - Sept

27 4.2 Eg. MPS- Parallel Analysis on 18 IA Channels. Multiple Channels/ Instruments MPS: 2 Automation Lines 6 ISE Channels 10 Chem. Channels 1 Special Chem. 4 x e602 IA analysers 4 x e602 IA analysers 2 x ISE + 2 x c701 Chem. analysers 2 x ISE + 2 x c701 Chem. analysers Lots of Parallel Analysis Occuring! - ISE.. on 6 channels - Gen. Chem. on 3 chnls. - most IA on at least 2 chnls. - TSH on 5 chnls. - PSA, B12 on 4 chnls J.Calleja - Melb. Path. - Sept x ISE + c701 + c502 + e602 analysers + 1 Stat Line 27

28 4.2 Eg. MPS- Parallel Analysis on Multiple Channels/ Instruments Regional Labs 5 x Integra 400s 4 x Integra 800s J.Calleja - Melb. Path. - Sept

29 .. So should we ; Assign specific targets for every instrument t / channel measuring the same test or.. Use a common target? J.Calleja - Melb. Path. - Sept

30 Same Targets Between.. Different Instrument types? or.. Instruments of the same type? or.. On more than 1 measuring channel, of an instrument of the same type? Instrument 1, 2 & 3 Instrument 1, 2 & 3 Measure Cell 1 & 2 J.Calleja - Melb. Path. - Sept

31 4.21 QC Targets.. Between Different Instruments Types? Instrument 1, 2 & 3 J.Calleja - Melb. Path. - Sept

32 4.21 Recoveries differ for different instruments with different methodologies & calibration techniques PSA Abbott Architect You generally, Can t use the same targets, across different instrument t types! Vitros ECi Siemens Vista Roche e602 J.Calleja - Melb. Path. - Sept

33 Between.. Different Instrument types? Instrument 1, 2 & 3 Even if you have aligned the performances between analysers through Alignment Factors Recoveries for QC may still differ significantly!! - Albumin - BCP vs BCG / - if you have Bovine supplementation -> Albumin will be lower for BCP. - Creatinine - Integra vs. Cobas c701 - High Bilirubin (~ 80 umol/l) in Level-2 QC -> Lower creat. On Integra -> May need instrument specific Targets & Limits J.Calleja - Melb. Path. - Sept

34 4.22 QC Targets.. Between Same Instrument Types? Instrument 1, 2 & 3 J.Calleja - Melb. Path. - Sept

35 4.2 Parallel Analysis on Multiple Channels/ Instruments Callum Fraser states.... Quality specifications can be set.. for the allowable differences between two methods.. used to analyse the same analyte in the same laboratory; Allowable difference < 0.33 CV i J. Calleja - Melbourne Pathology 35

36 4.22 Eg. MPS Glucose Cobas c701 - Recoveries are quite consistent across 3 instruments Glucose Suggests - you can use the same targets, between instruments of the same type! Line 1 Line 1 Line 2 Line 2 Line 3 Line 3 Same: Tgt 4.75, 1sd 0.13 CV=2.7% Same: Tgt 15.4, 1sd 0.3 CV=1.9 % 0.5CVi (Gluc) =3.25 % J.Calleja - Melb. Path. - Sept

37 4.22 Eg. MPS Glucose Cobas c701 - Are the recoveries across the 3 different c701s.. < 0.33CVi? Max Bias L1: ( ) = /4.735 = 0.63% MaxBiasL2: ( ) 4) = / /15 = 0.52% Line 1 Yes! Inter-Instrument Bias is < 0.33CVi Line 2 Line 3 CV Goals, Opt, Des, Min = 1.63, 3.25, 4.9% / Bias Goal 0.33CVi = 2.15% J.Calleja - Melb. Path. - Sept

38 4.22 Eg. TSH Cobas e602 - Review of Performance for 5 Channels measuring TSH using common targets & Limits. You can use the same targets, between instruments of the same type! Chnl 1 Chnl 2 Chnl 3 Chnl 1 Chnl 2 Chnl 3 Chnl 1 Chnl 2 Chnl 3 Max Bias 2.4% Max Bias 2.17% Max Bias 2.0% Chnl 4 Chnl 5 Chnl 4 Chnl 5 Chnl 4 Chnl 5 Yes! Inter-Instrument Bias is < 0.33CVi Tgt 101sd 1.0,1sd CV=3 3.5% Tgt , 1sd 0.2 CV=3 3.33% 33% Tgt 26.7, 1sd 1.0 CV=3.7% CV Goals, Opt, Des, Min = 4.93, 9.8,14.7% / Bias Goal 0.33CVi = 6.5% J.Calleja - Melb. Path. - Sept

39 4.23 Other Advantages - of using common Targets across analysers. Common Targets & Limits mean.. Less of an Administrative Nightmare! Each Line item represents a channel on which Glucose can be measured. To capture the QC data from all the individual instruments/channels on the automation lines we have to define specific lab numbers (35 total) for each instrument/channel combination. J. Calleja - Melbourne Pathology We also must define a target & limit for each channel that an analyte can be measured on. 39

40 4.23 Further Advantage - It s Visually Easier to Keep Track of Inter- Instrument Alignment. Glucose SHBG Line 1 Line 1 Line 2 Line 2 Line 3 One channel shows a shift! Good Alignment! J.Calleja - Melb. Path. - Sept

41 4.23 However, Consider some of the Disadvantages! Common Targets & Limits it mean; The target is always a compromise b/w the mean values of all the analysers g Mean There can be a broadening of the overall variance; SD overall = (sd 2 m1 + sd 2 m2 + sd 2 m3) Can be a challenge to set an SD that sits within the Desirable Biological i l Goals & accommodates the spread of results across the analysers. Can t use Westgard s Bias Rules 10 mean or 4 x 1s rule - Otherwise these rules would be continually firing. J. Calleja - Melbourne Pathology Tn T Mod 1 Mod 2 Mod 3 Mod 1 Mod 2 Mod 3 B1-LOW B1-HI Mean B1-LOW B1-HI 41

42 4.23 Parallel Analysis on Multiple Channels/ Instruments Therefore... You need to Balance the Advantages / Disadvantages Common Targets across analysers Specific Targets on individual analysers J. Calleja - Melbourne Pathology 42

43 5.0 Setting Appropriate QC Limits? J.Calleja - Melb. Path. - Sept

44 Consider.. What guidelines or Quality Goals do we have.. to indicate... the level of quality required or.. the allowable error, we can permit.. to ensure that our results are medically useful? (eg) If we have a CV of 5% for HBA1c... is this Clinically Acceptable? J.Calleja - Melb. Path. - Sept

45 Quality Goals Hierarchy Provided by the Profession Evidence Based Studies DCCT Trial - HBA1c ( CV<2.5%) IFCC ISO AACC RCPA AACB Biological Goals - Based on CV I CVa < 0.5 CVi, CVa < 0.25 CVi, CVa < 0.75 CVi Clinician Survey Barnett.. et al Profession Defined By group of experts eg. RCPA-QAP Allowable Limits Proficiency Testing Schemes State of the Art Method.or. +/- 2sd of all results submitted Publication by a Lab or Group ISO Technical Committee 212 Task Force, 1999 J.Calleja - Melb. Path. - Sept

46 We can illustrate the importance of Analytical CV on result interpretation!.. Using inferences from the 1993 DCCT Trial.. J.Calleja - Melb. Path. - Sept

47 Remember... When we analyse a patient sample, there are 2 components of variation: Analytical l CV A & Biological CV i This is represented as CV T = (CV A 2 + CV i2 ) 1/2 J.Calleja - Melb. Path. - Sept

48 5.6 Effect of CV A on Result Interpretation CV T = (CV A2 + CV i2 ) 1/2 8.5 CV T = ( ) 1/ CV T = 3.6 % 9 HBA1c - Effect of CV on result interpretation -CVa=0 % Depiction of pure Biological Signal Only! low risk 7.5 High Risk In the DCCT Trial Intensive Treatment Cohorts had Mean HBA1cs of 7% Conventional Treatment Cohorts had Mean HBA1cs of 9%.. CVi (HBA1c) = 3.6%... We can use this study to illustrate the variation due to CVi alone, at these levels J.Calleja - Melb. Path. - Sept

49 Effect of CV A on Result Interpretation CV T = (CV A2 + CV i2 ) 1/2 HBA1c - Effect of CV A on result interpretation - CV A =2.5 % 11 Adequate Cva to distinguish 10.5 CV = /2 Low & High Risk cohorts. T ( ) 10 - Evidence Based Goal CV T = 4.38% At the Evidence Based HBA1c CV A Goal; CV A = 2.5% There is adequate distinction between cohorts! Low Risk High Risk J.Calleja - Melb. Path. - Sept

50 .. So.. minimising our analytical variation.. clearly l assists in patient result interpretation!.. But how much is enough?.. Given to us by Callum Fraser J.Calleja - Melb. Path. - Sept

51 5.7 Effects of Imprecision on Test Result Variability % increase in variability CVi adds 3% variability CVi adds 25% variability. Desirable Goal CVi adds 12% variability. Biological CV Goal. Ref. Biological Variation Principles to Practice Callum Fraser J.Calleja - Melb. Path. - Sept

52 6.0 What steps should we follow when we set our QC Limits. J.Calleja - Melb. Path. - April

53 6.1 Setting Appropriate p QC Ranges - Steps Involved.. Consider Quality Goals & Method Capability Don t just blindly use Mean +/- 2sd! Step 1 - Review the SDs & CVs achieved from The Target Setting Studies - Compare this to the Labs Ongoing CVs at similar concentrations Step 2 - Compare the labs achieved CV performance to the Instrument Manufacturer s.. stated CV specifications for Total CV - Refer to the manufacturers kit insert Step 3 - Consider the Relevant Quality Goals ( CV Desired), from the ISO TC212 Hierarchy ; Evidence Based / Biol. Var n / QAP Allowable Limits Step 4 - Consider the State-of-The-Art Performances - Compare CVs to QAP 20 th, 50 th & method Median CVs (E.O.C.Rs) Step 5 - Put all of this information together to determine what the allowable CV ( or Quality Goal ).. for your assay, should be. J.Calleja - Melb. Path. - Sept

54 6.2.1 Setting QC Ranges - Acceptable Limits Lab about to set targets & sd s for 2 qc s for Trigs on a Beckman analyser.. Based on achieved Evaluation Study performances QC Level-1 QC Level-2 Tgt = 1.00 Tgt = 2.2 Are these ranges appropriate? SD = CV = 5.1% SD = 0.1 CV = 4.5% Range: Range: Min = 0.9 Min = 2.0 Max = 1.1 Max = 2.4 J.Calleja - Melb. Path. - Sept

55 ? Appropriate QC Limits Lab Ongoing CVs J.Calleja - Melb. Path. - Sept

56 Labs Ongoing CVs Lab s Ongoing CVs Tgt = 0.95 Tgt = 2.0 SD = SD = 0.07 CV = 3.8% CV = 3.5% Lab s Evaln. CVs Tgt = 1.00 SD = Tgt = 2.2 SD = 0.1 CV = 5.1% The Lab s Eval. CVs Range: CV = 4.5% Range: are higher than routine CVs Min = 0.9 Min = 2.0 Max = 1.1 Max = 2.4 J.Calleja - Melb. Path. - Sept

57 ? Appropriate QC Limits Manufacturer Specifications J.Calleja - Melb. Path. - Sept

58 Trig - Expected Performance Note: Expected level Tgt = 1.00 SD = CV = 5.1% Range: Min = 0.9 Max = 1.1 Tgt = 2.2 SD = 0.1 CV = 4.5% Range: Min = 2.0 Max = 2.4 The Lab s QC CVs are J.Calleja - Melb. Path. - Sept

59 ? Appropriate QC Limits Biologically Based CVa Quality Goals J.Calleja - Melb. Path. - Sept

60 ? Appropriate QC Limits Refer to Westgard Web-site ( CVi = 20.9% J.Calleja - Melb. Path. - Sept

61 Labs Trig QC Goals vs. Biological Limits Biological i l Variability CV i (Triglyceride) = 20.9 % Lab s achieved CVs are Within Desirable & Optimal Biological Goals 1. Calculate the Analytical CV Goals Tgt = 1.00 Tgt = 2.2 CV a Optimal = 0.25 CV i = 5.25% CV a Desirable = 0.5 CV i = 10.5% CV a Minimal = 0.75 CV i = 15.75% SD = 0.51 CV = 5.1% Range: Min = 0.9 SD = 0.1 CV = 4.5% Range: Min = 2.0 Max = 1.1 Max = Compare Lab s CVs w Biological Goals J.Calleja - Melb. Path. - Sept

62 ? Appropriate QC Limits QAP Allowable Limits J.Calleja - Melb. Path. - Sept

63 Labs Trig QC Goals.vs. QAP Limits Labs Eval CVs Tgt = 1.00 Tgt = 2.2 SD = SD = 0.1 CV = 5.1% CV = 4.5% Range: Range: Min = 0.9 Min = 2.0 Max = 1.1 Max = 2.4? How do the Lab s CV goals Compare with QAP Allowable Limits J.Calleja - Melb. Path. - Sept

64 Labs Trig QC Goals.vs. QAP Limits What CV do we need to achieve 95.5% 5% of all results.. within the QAP Allowable Limits..? We can consider the QAP Target & Allowable Limits i as.. mean +/- 2 sd (95% C.I.) So.. 1sd = ( ) / 2 = 0.1 So.. CV required is.. ( 0.1 / 1.71 ) x 100 = 5.8 % J.Calleja - Melb. Path. - Sept

65 Labs Trig QC Goals.vs. QAP Limits What CV do we need to achieve 99.7% of all results.. within the QAP Allowable Limits.. Can consider Allowable range as mean +/- 3 sd (99.7% C.I.) So.. 1sd = ( ) / 3 = So.. CV required is.. ( 0.1 / 1.71 ) x 100 = 3.9 % J.Calleja - Melb. Path. - Sept

66 Labs Trig QC Goals.vs. QAP Limits Labs Eval. CVs are within the QAP Allowable Limits Labs Eval. CVs Tgt = 1.00 Tgt = 2.2 SD = SD = 0.1 CV = 5.1% CV = 4.5% Range: Range: Min = 0.88 Min = 2.0 Max = 1.12 Max = 2.4 Calculate CV goals for the Lab based on their mean values & QAP 2.2, 1SD=0.11 1SD=0.1 CV= 10% J.Calleja - Melb. Path. - Sept

67 ? Appropriate QC Limits State of the Art Performances J.Calleja - Melb. Path. - Sept

68 ? Appropriate QC Limits State-of-the-Art the 50 th percentile CV = 3.2% 20 th percentile CV = 2.4% Method Median CV = 3.5% J.Calleja - Melb. Path. - Sept

69 Put it All Together! Manufacturers CV Specs. Biological Goals CV a Optimal = 0.25 CV i = 5.25% CV a Desirable = 0.5 CV i = 10.5% CV a Minimal RCPA QAP ALEs = 0.75 CV i = 15.75% Lab s Evaluation CVs Tgt = Tgt = SD = SD = 0.1 CV = 5.1% CV = 4.5% Lab s Ongoing CVs Tgt = 0.95 Tgt = 2.0 SD = SD = CV = 3.8% CV = 3.5% ½ QAP ALE CV = 5.8% 1/3 QAP ALE Set Targets to: Lev 1: Tgt =1.0 1SD=0.037, CV= 3.7% Lev 2: Tgt=2.2, 1SD=0.075 CV=3.5% QAP State-of-the-Art CV = 3.9% 50 th %CV = 3.2% 20 th %CV = 2.4% Method CV = 3.5% J.Calleja - Melb. Path. - Sept

70 Also Consider Process Capability.. J.Calleja - Melb. Path. - Sept

71 Process Capability.. Cp Cp - is an index which relates the Allowable process spread.. to the.. Actual process spread Allowable Process Spread USL - LSL Cp = = Actual Process Spread 6sd Cp = 2 LSL Cp = 1 USL J.Calleja - Melb. Path. - Sept

72 Process Capability.. Cp - A High Cp Index is desirable... ie. A systematic shift can occur.. but.. your results will still be within the Allowable Limits. Consider : a) Cp = 2.. desirable b) Cp = 1.. Less desirable.. LSL USL LSL USL J.Calleja - Melb. Path. - Sept

73 Process Capability.. Cp - If you have a Method with excellent precision i... (eg) Better than Optimal Biological Variation Goals.. - Method can shift.. but results remain within the allowable Limits.. LSL USL Set allowable SD.. to match the Desirable Biol Goal J.Calleja - Melb. Path. - Sept

74 A Spread-Sheet Sheet Can Help Manage the Goals Lab QC Settings. Manufacturers CV Specs. J.Calleja - Melb. Path. - Sept

75 A Spread-Sheet Can Help Manage the Goals CV you need for SD = ½ ALE (95%) or SD = 1/3 ALE (99%) QAP Allowable Limits J.Calleja - Melb. Path. - Sept

76 A Spread-Sheet Can Help Manage the Goals CVI & CVG Opt, Des, Opt Biol Cva Goals Opt, Des, Opt Bias Goals J.Calleja - Melb. Path. - Sept

77 A Spread-Sheet Can Help Manage the Goals Your test Capabilities Which Biol. CV Goal achieving Whether achieving ½ QAP ALE (95%) or 1/3 QAP ALE (99%) Method Capability Whether achieving Manufacturer B/R CV J.Calleja - Melb. Path. - Sept

78 7.0 What do we do if we have ashift? Will not be discussed in the interests of time! J.Calleja - Melb. Path. - Sept

79 What level l of Bias is acceptable? Can be explained in the context of Ref. Intervals J.Calleja - Melb. Path. - Sept

80 7.1 What level of Bias is Acceptable? Callum Fraser.. discusses that the Reference Interval is made up of; With-in Subject & CV i Between Subject Variation CV G For all of us to use the same Reference Interval,.. the analytical Bias should be less than ¼ of the grouped Biological Variation. This is represented as: B < 0.25 (CV 2 A G + CV i2 ) 1/2 Becomes a Default Bias Goal! J.Calleja - Melb. Path. - Sept

81 Impact of a Shift or Bias Shift in Assay causing +ve Bias Results in <2.5% out of LRL Results in False Positives, > 2.5% out of U.R.L. +/-2sd = 95.5% Reference Interval! (c) John Calleja Melb Path 81

82 7.2 How much of the Population o is Displaced by Bias? % Out of each Ref Limit % of Results Outside of URL When Bias = 0 2.5% either side of Ref Limits % of Results Outside of LRL Desirable Bias B A < (CV I 2 + CV G2 ) 1/2 adds 2% outside of Ref Interval. B A < (CV I 2 + CV G2 ) 1/2 adds 16% B A < (CV I 2 + CV G2 ) 1/2 adds 34% Bias Goal. Ref. Biological Variation Principles to Practice Callum Fraser J.Calleja - Melb. Path. - Sept

83 Alternatively ti l We could use; Callum Fraser s.. Quality specifications.. for the allowable differences between two methods.. used to analyse a the same analyte a in the same laboratory; Allowable difference < 0.33 CV i J. Calleja - Melbourne Pathology 83

84 How should we Assess / Action a shift? J.Calleja - Melb. Path. - Sept

85 Assessing / Actioning a Shift Possible Actions: Assess Magnitude of Shift Percent deviation from QC Target or %Bias Pre & Post shift patient comparison studies Patient Data Extract movement in averages & percentiles Attribute a Cause Eg. Reagent or Cal Lot Change Reagent Reformulation Assess Clinical Relevance Clinical Consultation, 0.25 x (CV I2 +CV G2 ) 1/2, 0.33CVi Consultation with Manufacturer about a corrective action. New Lot Number of Calibrator / Rgt Calibrator Set Point Reassignment Apply a corrective Slope &/or Offset to results Derived from Pre & Post shift.. patient comparison studies Examination of influence of shift on patient averages Take Corrective Action Change QC Targets / Pt. Ref. Intervals, to compensate for the shift (c) John Calleja Melb Path 85

86 Attribute the Cause? eg. CEA Start w LJ Plots Draw in dates of reagent & calibrator lot changes etc.. Attribute the Cause, of the shift -> -> Shift due to Reagent Lot Change (Lot ) Estimate Bias Magnitude: Target Shift to Diff % Diff % J.Calleja - Melb. Path. - Sept

87 Estimate Magnitude of Shift eg. Pt. Comparisons for Ca++ Bias.. Post vs. Pre New Lot of Calibrator Y = x At URL: 2.6 mmol/l New Result = Bias = %Bias = 3.02% Assess against: 0.33CVi (1.9) = 0.63% -> 3.02% Worse Assess against: (CV 2 I + CV G2 ) 1/2 = 0.85% ->3.02% Worse (CV 2 I + CV G2 ) 1/2 = 1.27% -> 3.02% Worse (c) John Calleja Melb Path 87

88 What if we initially missed the shift.. &.. can t perform patient sample comparisons? J.Calleja - Melb. Path. - Sept

89 Perform Patient Data Extracts - Plot Patient Medians/Percentiles How Extract Data from LIS for a period including before & after the shift Calculate moving medians.. per 30.. or.. per 1000 sample results (if lots of data) Plot.. Moving Median vs. Date/Time Examine for a significant shifts / Estimate Magnitude Most useful.. when the detection of a shift has initially been missed. (c) John Calleja Melb Path 89

90 Patient Data Extracts - example: raw data Write Formulas to calculate the median for the preceding 30 samples =Median(F2:F31) =Median (F3:F32) =Median(F4:F33) Copy Formulas down spreadsheet.. to get rest of.. rolling 30 sample medians. (c) John Calleja Melb Path 90

91 - Example: Plot the Moving Median versus Date/Time Moving Median Sensitive to the Shift Bias ~ 0.15 mmol/l (c) John Calleja Melb Path 91

92 What if we decide we have a Clinically i ll Significant ifi Shift.. J.Calleja - Melb. Path. - Sept

93 Could Derive a Corrective Slope & Offset to realign the performances Y = x Re-Performed Comparisons with Slope & Offset installed Derived Corrective Slope & Offset (from Regression Eq): Slope = (1/1.0965) = 0.91 Offset = (0.1723/1.0965) = Method Alignment Improved (c) John Calleja Melb Path 93

94 What do we do with QC Targets?... J.Calleja - Melb. Path. - Sept

95 What do we do about QC Targets? If Shift assessed as Clinically Tolerable Calculate the mean values for all relevant QC levels including data after the Shift! Re-Assign the QC Target Values to these values If Shift assessed as Clinically i ll Significant ifi If Corrective Factors Implemented Should be no need to amend QC Targets Exceptions Where QC matrix behaves different to Patients If Reference Intervals modified.. (eg. due to assay Re-Standardisation) Calculate the mean values for all relevant QC levels including data after the Shift! Re-Assign the QC Target Values to these values (c) John Calleja Melb Path 95

96 What do we do about QC Limits? If you have already carefully calculated CV Goals.. -> Re-calculate your SDs.. to achieve equivalent CV goals, at the new target concentration. CV % = (SD/Mean ) x 100 If Former (Pre-Shift) Target = 100, SD = 5, CV Goal = 5% If Assay Shifted to Mean= 80 To maintain a CV Goal of 5% Re-arrange CV equation to solve for SD SD = (CV/100) x Mean SD = (5/100) x 80 SD = 4 (c) John Calleja Melb Path 96

97 8. Summary Kit Insert Target Values.. &.. Consensus Values from QC DBs are a reliable place to start. Kit Insert QC limits need closer scrutiny. Ideally we should establish our own targets & limits. We need Data points to ascribe a confident Target. You generally cannot use a common target across different instrument types You can use a common target between instruments/channels of the same type Consider Quality Goals hierarchy & method capability, when setting QC Limits If you have a persistent shift The magnitude should be assessed against Biologically Based Goals If insignificant, QC Targets & Limits should be revised, If significant - Corrective action should be taken. J.Calleja - Melb. Path. - Sept

98 References J.Calleja - Melb. Path. - Sept