AOAP eballot - Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B

Transcription

1 AOAP eballot - Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B The AOAP met on March 15, 2018 to review the Sequence IVB Development program. After the review a motion was made to Ballot the Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B. The motion detail is: Motion Motion by: Seconded by: Sequence IVB meets the defined needs in the GF-6 Needs Statement for measuring low temperature engine wear performance of an engine oil and is suitable for inclusion in ILSAC GF-6A and GF-6B, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter. Teri Kowalski, Toyota Ron Romano, Ford The AOAP Voted by Hand on accepting the Seq. IVB Motion. Hand Vote results: Affirmative = 6 Negative = 5 Abstain = 14 The Hand Vote results require that the Motion be resolved with a written eballot. AOAP members should Vote on the motion Sequence IVB is suitable for inclusion in ILSAC GF-6A and GF-6B using the eballot at The Sequence IVB GF-6 Motion and the supporting documentation are available the eballot Website. This eballot will close on Friday March 30, All Negative Votes must include comments which: a) Describes the section to which the negative ballot pertains b) Gives substantive reason(s) for negative vote. c) Proposes wording or action to resolve negative vote. All Abstentions/Waves to the AOAP Vote must include comments which: a) Gives substantive reason(s) for Abstain vote. b) Proposes wording or action to resolve negative vote. Any questions, please contact API.

2 AOAP Motion Sequence IVB meets the defined needs in the GF-6 Needs Statement for measuring low temperature engine wear performance of an engine oil and is suitable for inclusion in ILSAC GF-6A and GF-6B, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter. Motion by Teri Kowalski Seconded by Ron Romano 15

3 PCEOCP / AOAP March 15, 2018 Teri Kowalski William A Buscher III 1

4 Summary of Activity Since 12/7/17 Completed Precision Matrix 2 testing at Intertek, SwRI, Lubrizol and ExxonMobil Completed prove-out testing at Afton To-date a total of 45 prove-out and precision matrix tests have been completed Completed statistical analysis of Precision Matrix 2 Analysis performed for two data sets N = 28, includes independent and dependent labs N = 21, includes independent labs (official precision matrix design) Data supports the use of Sqrt(AVLI) transformation Significant oil difference: 1012 < 300 Lab differences are statistically different Surveillance panel addressing Stand within lab differences are not significantly different Reference oil targets established 2

5 Prove-out and Precision Matrix 2 Results PROVE-OUT TESTING Run Order Required Supplemental IAR Stand 3 IAR Stand 4 SwRI Stand 1 SwRI Stand 2 SwRI Stand 3 Lubrizol ExxonMobil Afton N/A 1.01 N/A N/A PRECISION MATRIX Run Order Precision Matrix Supplemental IAR Stand 1 IAR Stand 2 IAR Stand 3 SwRI Stand 1 SwRI Stand 2 Lubrizol ExxonMobil Afton N/A N/A N/A = 300 = 1011 = 1012 = Lobe Failure Oils 3

6 Average Intake Volume Loss by Oil (N = 21) 4

7 Reference Oil Targets (N = 21) Oil Number of Tests Target Mean Sqrt(AVLI) Target Mean AVLI Target Standard Deviation Sqrt(AVLI)

8 Summary of Activity Since 12/7/17 Sequence IV surveillance panel met on 1/11/18, 1/25/18, 3/1/18 and 3/7/18 Reviewed additional prove-out testing Operational data and test results Reviewed Precision Matrix 2 testing Extensive operational data analysis and review including: 1-hour operational data plots from start, mid and end of test 200-hour operational data plots Statistical analysis of operational ramp data Statistical analysis of operational data correlation to Sqrt(AVLI) N = 28 and N = 21 Precision Matrix 2 statistical analyses review Approved statistical analysis of Precision Matrix 2 Reviewed results from potential high wear candidate oils Voted that the test is ready for inclusion into GF-6 and to become an ASTM procedure Reviewing LTMS examples Next meeting planned for week of March 18 th 6

9 Sequence IV Surveillance Panel Motion The Sequence IV Surveillance Panel, having secured hardware supply, test fuel and reference oils for a test procedure that measures the performance of passenger car motor oil for low temperature engine wear, recommends to the Passenger Car Engine Oil Classification Panel, the Auto Oil Advisory Panel and the American Chemistry Council that the Sequence IVB test is ready for inclusion in ILSAC GF-6 and that the Sequence IVB procedure be published as an ASTM method. Realizing that the test parameters (AVLI and Fe content) need to be finalized and the LTMS still needs to be developed. Teri Kowalski / Ron Romano / Passed

10 Summary of Activity Since 12/7/17 Toyota solicited oil suppliers for Sequence IVB results from potential high wear oils Data from 3 oils from 3 suppliers was presented to Toyota and the Sequence IV surveillance panel 1 of the 3 oils produced high valve-train wear and very high Fe content at EOT, indicating high overall engine wear Much higher Fe versus valve-train wear than the typical valve-train wear to Fe correlation 1 of the 3 oils produced moderate valve-train wear and high Fe content at EOT, indicating high overall engine wear Much higher Fe versus valve-train wear than the typical valve-train wear to Fe correlation 1 of the 3 oils produced low valve-train wear and low Fe content at EOT, but is suspect of having formulation components that relaxed the oil degradation mechanism of the Sequence IVB test Surveillance panel action item for supplier to investigate and report back 8

11 Summary of Activity Since 12/7/17 Toyota solicited oil suppliers for Sequence IVB results from potential high wear oils Conclusions: Sequence IVB evaluates more than just the performance of a passenger car motor oil for low temperature valve-train wear, but evaluates the performance of a passenger car motor oil for low temperature engine wear Fe content is an indicator of engine wear, and is important, in addition to average intake volume loss Sequence IVB responds to a variety of passenger car motor oil formulation components 9

12 Average Intake Volume Loss by Oil (N = 30) X invalid 10

13 Summary of Activity Since 12/7/17 LTMS examples established and distributed to the Sequence IV surveillance panel for review and approval, with the following suggestions: Stand based system Calibration period: fifteen full length non-reference tests or 6 months Reference oils and assignment: 300 (40%), 1012 (40%) and 1011 (20%) A minimum of two reference tests will be required for each new stand Adopt the transform Sqrt(AVLI) for LTMS and severity adjustment calculations Select reference oil targets from presented N = 28 or N = 21 models Utilize limits on Zi (EWMA of severity), ei (prediction error), and the excessive influence calculation to determine acceptance and calculate severity adjustments The TMC will plot industry Zi charts to identify potential shifts in industry wide performance 11

14 Precision Matrix 2 LTMS Example N = 21 12

15 Precision Matrix 2 LTMS Example N = 21 13

16 PCEOCP Motion Sequence IVB is suitable for measuring low temperature engine wear performance of an engine oil and is recommended for adoption as an ASTM procedure, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter. Motion by Teri Kowalski Second by Ron Romano Discussion 14

17 AOAP Motion Sequence IVB meets the defined needs in the GF-6 Needs Statement for measuring low temperature engine wear performance of an engine oil and is suitable for inclusion in ILSAC GF-6A and GF-6B, realizing that the Sequence IV Surveillance Panel is evaluating Fe Content as an additional test parameter to complement the intake lifter average volume loss parameter. Motion by Teri Kowalski Seconded by Ron Romano 15

18 AOAP Motion 2 Include Seq. IVB in GF-6A and GF-6B for the purpose of demonstration of Low Temperature Engine Wear Phenomena using rate and report procedures. Motion by Matthew Ansari Seconded by TABLED w/o Second 16

19 ANNEX 17

20 Sequence IVB Precision Matrix Analysis (n=28) Statistics Group Jan. 25, 2018

21 Executive Summary Precision Matrix (PM) Analysis Highlights: This analysis includes the results of 28 valid precision matrix tests Data supports the use of Sqrt(AVLI) transformation Significant oil differences: 1012 < 300 Lab differences are statistically significant (A < B1) Stand within Lab differences are not statistically significant Estimated within a stand test precision (r; ASTM repeatability) Sqrt(AVLI) = Estimated test precision across labs and stands (R; ASTM reproducibility) Sqrt(AVLI) = Oil means and standard deviations Target Mean AVLI Target Standard Deviation Sqrt(AVLI) Number Target Mean Oil of Tests Sqrt(AVLI)

22 PM Analysis Concerns The two high results on Oil 300 at stands B1-2 and B1-3 have large influence on discrimination between oils 300 and Without these two tests, differences between oils are not statistically significant. Discrimination is not consistent among the stands. Labs F and G may not discriminate oils Stands rank oils differently This could be an issue if the same phenomenon is observed in candidate oils Test precision is large compared to the observed range of measurements; lab differences are larger than oil differences; the high and low oils diff by 1.4 standard deviations (lowest of any GF6 test). The resulting LTMS would likely allow calibration of stands that don t discriminate oils Discriminating future oils in the test will be difficult; especially with only one test result 20

23 PM Analysis Comments - 1 Statisticians chose to weight targets by lab (25% per lab) rather than by stand (approx. 14% per stand). The effect is that the average of a lab with 3 stands and the average of a lab with 1 stand will have the same 25% weighting on the targets. This was seen by stats group to better represent industry-wide performance, align with past analyses methodology, and does not affect any results other than the targets. Stand weighted targets could be pursued if the panel desires. Some belief amongst some stat group members that transforming individual lifter results before averaging may be more appropriate than transforming the average. Since the benefit of doing this new approach was minimal and time was short, this analysis is included in the appendix only. AVLI in the LTMS file is sometimes off in the hundredths place from the calculated average of the eight lifters shown in the same file. Impact is negligible, but the source of the AVLI in the LTMS file should be made clear. 21

24 PM Analysis Comments - 2 Based on analysis conducted, there is no additional benefit in using parameters other than AVLI Additional tests could help better understand discrimination and precision of the test. Statistical analyses have not yet been completed to assess the impact of operational differences on test severity. The outcome of such analyses and discussions could ultimately affect oil targets. Given the differences noted in the surveillance panel call on January 11 th, the panel may find it helpful to review the full datafiles for all tests. 22

25 Additional Comments - 1 A review of individual lifter measurements suggests some merit to the incorporation of an outlier screening methodology An initial review of the impact of outlier screening indicates minimal improvement in oil discrimination and precision It is unknown whether or not the number of outliers for candidate oil tests are more likely to occur as compared to reference oil tests. (Greater number of outliers in candidate oils would make a stronger case for outlier screening.) Lifter bias is observed and can be taken into account in outlier screening methods Some methodologies investigated included: Removal of the max and min lifter result of both non-transformed and meancentered lifter data Weighted average with higher weights for lifters that differ Similar approach to what is done for T12 and C13 for performance properties with lifter bias Evaluating several outlier screening methods listed in E178. Outlier screening can be pursued further if the surveillance panel deems it appropriate; the final methodology will likely impact oil targets that are established using nonscreened lifter 23 measurements

26 Additional Comments - 2 An initial review of the impact of lifter weighting indicates minimal improvement in oil discrimination and precision Lifter weighting can be pursued further if an engineering reason exists for the differences by position 24

27 Data Utilized Precision Matrix Data: 4 Labs {A, B1, F and G} 3 Reference Oils {300, 1012, and 1011} 7 Stands {A-1, A-2, B1-1, B1-2, B1-3, F-1 and G-1} Total number of tests = 28 Precision Matrix Data Table from Rich Grundza s IVB Matrix update. 25

28 Parameter Abbreviation AVLI - Average volume loss, Intake AMLI - Average mass loss, Intake AVLOSEXK - Average volume loss, Exhaust AMLOSEX - Average mass loss, Exhaust SumVLIE - Average volume loss, Intake + Exhaust SumMLIE - Average mass loss, Intake + Exhaust FEWMEOT Fe-Wear Metals at end of test 26

29 Data Calculation AMLI and AMLOSEX For Lab G data, multiplied individual lifter mass loss by 1000 and took the average of 8 lifters Remove test IVB s BL2EXHML = in calculating average which results to AMLOSEX=9.6 AVLOSEXK Remove Lab A test IVB s BL1EXKVL=-0.2 in calculating average which results to AVLOSEXK=0.85 Lab G did not measure AVLOSEXK for test IVB SumVLIE = AVLI + AVLOSEXK SumMLIE = AMLI + AMLOSEX 27

30 Summary of Model Results Model P-values Sqrt(AVLI) AMLI Sqrt(AVLOSEXK) Ln(AMLOSEX) Ln(SumVLIE) Sqrt(SumMLIE) Ln(FEWMEOT) Sqrt(AVLIS) Sqrt(AVLIOS) IND LTMSLAB LTMSAPP[LTMSLAB] Oil Discrimination, in standard deviation units, red means difference is statistically significant Precision RMSE, sr Repeatability, r Parameter Result No significant difference Most parameters except AMLI show that lab difference is greater than oil difference. The Volume Loss parameters showed no significant difference between stands within the lab. Note: n-size for these models is 28 except for SumVLIE and AVLOSEXK which has 27 28

31 Reference Oil Discrimination Comparison The table below compares the numbers of standard deviations of separation between the highest and lowest reference oil across GF-6 test types. The median is approx. 3.3 and the mean (without PHOS) is 3.4. Test Parameter Oil 1 Oil 2 Range Test SDs of Separation IIIH Ln(PVIS) IIIH WPD IIIHA Ln(MRV) IIIHB PHOS VIE FEI VIE FEI VIF FEI VIF FEI IX (LSPI) Sqrt(AvPIE + 0.5) VH AES VH Ln(10-RCS) VH AEV VH APV X (CW) Ln(CHST) IVB Sqrt(AVLI) *1: Oil 220 not used as a reference oil. Including this oil would yield approx. 12 SDs of separation between 220 and 222. *2: 271 vs

32 Average volume loss, Intake AVLI 30

33 Average Intake Volume Loss by Oil The below plot summarizes the AVLI test result data by reference oil. 31

34 Average Intake Volume Loss by Stand It appears that oil discrimination is not consistent among the stands; Labs F and G may not discriminate oils; Stands rank oils differently 32

35 Average Intake Volume Loss by Lab Below plot summarizes the AVLI test result data by test Lab and reference oil 33

36 Sqrt(AVLI) ANOVA Full Model Statistically significant differences: Oil Lab Not significantly different: Stands within Labs 34

37 Sqrt(AVLI) Oil Differences Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Oils significantly differ Oil 300 is statistically significantly different than oil 1012 Oil 1011 is not statistically significantly different than oils 300 and 1012 Plot shows Sqrt(AVLI) LSMeans by Oil, with 95% confidence intervals LSMeans by Oil Oil Sqrt(AVLI) LSMean AVLI LSMean LSMeans Differences Between Oils Oil1 Oil2 Sqrt(AVLI) LSMean Difference p-value

38 Sqrt(AVLI) Lab Differences Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Plot below of Sqrt(AVLI) LSMeans by Lab, with 95% confidence intervals Lab A is statistically significantly different than Lab B1. LSMeans by Lab Lab Sqrt(AVLI) LSMean AVLI LSMean A B F G LSMeans Differences Between Labs Lab1 Lab2 Sqrt(AVLI) LSMean Difference p-value B1 A G A B1 F G F F A B1 G

39 Sqrt(AVLI) Stand within Lab Differences Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Plot below of Sqrt(AVLI) LSMeans by Stand, with 95% confidence intervals Stands within labs are not statistically significantly different from each other LSMeans by Stand Stand Sqrt(AVLI) LSMean AVLI LSMean [ A] [ A] [ B1] [ B1] [ B1] [ F] [ G] LSMeans Differences Between Labs Stand1 Stand2 Sqrt(AVLI) LSMean Difference p-value [ A]1 [ A] [ B1]2 [ B1] [ B1]3 [ B1] [ B1]2 [ B1]

40 Sqrt(AVLI) Precision Repeatability Model: Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Reproducibility Model: Sqrt(AVLI) ~ Oil Model RMSE = Repeatability = r = Reproducibility = R = Based upon the AVLI pooled standard deviation ( ) and ASTM s repeatability (r), there is no significant difference between an AVLI of 2.00 and Note 1: An AVLI result of 2.00 was arbitrarily selected for comparison 38

41 Reference Oil Targets Model: Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Average Intake Volume Loss (AVLI) Unit of Measure: Sqrt(AVLI) Ref. Oil Target Mean Sqrt(AVLI) Target Mean AVLI St. Dev Target Means are the Oil LSMeans from the Model and Standard Deviations are calculated straight from Sqrt(AVLI). 39

42 PM Data Ranges Overlap 40

43 PM Data Ranges Overlap 41

44 Sequence IVB Precision Matrix Analysis (n=21) Statistics Group March 5, 2018

45 Executive Summary Precision Matrix (PM) Analysis Highlights: This analysis includes the results of 21 valid precision matrix tests from the independent labs Data supports the use of Sqrt(AVLI) transformation Significant oil differences: 1012 < 300 Lab differences are statistically significant (A < B1) Stand within Lab differences are not statistically significant Estimated within a stand test precision (r; ASTM repeatability) Sqrt(AVLI) = Estimated test precision across labs and stands (R; ASTM reproducibility) Sqrt(AVLI) = Oil means and standard deviations Oil Number of Tests Target Mean Sqrt(AVLI) Target Mean AVLI Target Standard Deviation Sqrt(AVLI)

46 PM Analysis Concerns The two high results on Oil 300 at stands B1-2 and B1-3 have large influence on discrimination between oils 300 and Without these two tests, differences between oils are not statistically significant. Discrimination is not consistent among the stands. Stands rank oils differently This could be an issue if the same phenomenon is observed in candidate oils Test precision is large compared to the observed range of measurements; the high and low oils differ by 1.9 standard deviations (lowest of any GF6 test). Discriminating future oils in the test will be difficult; especially with only one test result 44

47 Data Utilized Precision Matrix Data: 2 Labs {A, B1}, independent labs only 3 Reference Oils {300, 1012, and 1011} 5 Stands {A-1, A-2, B1-1, B1-2, B1-3} Total number of tests = 21 Precision Matrix Data Table from Rich Grundza s IVB Matrix update. 45

48 Reference Oil Discrimination Comparison The table below compares the numbers of standard deviations of separation between the highest and lowest reference oil across GF-6 test types. The median is approx. 3.3 and the mean (without PHOS) is 3.4. Test Parameter Oil 1 Oil 2 Range Test SDs of Separation IIIH Ln(PVIS) IIIH WPD IIIHA Ln(MRV) IIIHB PHOS VIE FEI VIE FEI VIF FEI VIF FEI IX (LSPI) Sqrt(AvPIE + 0.5) VH AES VH Ln(10-RCS) VH AEV VH APV X (CW) Ln(CHST) IVB Sqrt(AVLI) *1: Oil 220 not used as a reference oil. Including this oil would yield approx. 12 SDs of separation between 220 and 222. *2: 271 vs

49 Average volume loss, Intake ANALYSIS OF SQRT(AVLI) 47

50 Average Intake Volume Loss by Oil The below plot summarizes the AVLI test result data by reference oil. 48

51 Average Intake Volume Loss by Stand It appears that oil discrimination is not consistent among the stands; Stands rank oils differently 49

52 Average Intake Volume Loss by Lab Below plot summarizes the AVLI test result data by test Lab and reference oil 50

53 Sqrt(AVLI) ANOVA Full Model Statistically significant differences: Oil Lab Not significantly different: Stands within Labs 51

54 Sqrt(AVLI) Oil Differences Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Oils significantly differ Oil 300 is statistically significantly different than oil 1012 Oil 1011 is not statistically significantly different than oils 300 and 1012 Plot shows Sqrt(AVLI) LSMeans by Oil, with 95% confidence intervals LSMeans by Oil Oil Least Sq Mean AVLI LSMean LSMeans Differences Between Oils Oil1 Oil2 Sqrt(AVLI) LSMean Difference p-value

55 Sqrt(AVLI) Lab Differences Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Plot below of Sqrt(AVLI) LSMeans by Lab, with 95% confidence intervals Lab A is statistically significantly different than Lab B1. LSMeans by Lab Level Sqrt (AVLI) LSMean AVLI LSMean A B LSMeans Difference Between Labs Lab1 Lab2 Sqrt(AVLI) LSMean Difference p-value B1 A

56 Sqrt(AVLI) Stand within Lab Differences Model is Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Plot below of Sqrt(AVLI) LSMeans by Stand, with 95% confidence intervals Stands within labs are not statistically significantly different from each other LSMeans by Stand Stand Sqrt(AVLI) LSMean AVLI LSMean [ A] [ A] [ B1] [ B1] [ B1] LSMeans Differences Between Labs LSMean Stand1 Stand2 Difference [ A]1 [ A] [ B1]2 [ B1] [ B1]3 [ B1] [ B1]2 [ B1] p-value 54

57 Sqrt(AVLI) Precision Repeatability Model: Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Reproducibility Model: Sqrt(AVLI) ~ Oil Model RMSE = Repeatability = r = Reproducibility = R = Based upon the AVLI pooled standard deviation ( ) and ASTM s repeatability (r), there is no significant difference between an AVLI of 2.00 and Note 1: An AVLI result of 2.00 was arbitrarily selected for comparison 55

58 Reference Oil Targets Model: Sqrt(AVLI) ~ Oil, Lab, Stand(Lab) Ref. Oil Average Intake Volume Loss (AVLI) Unit of Measure: Sqrt(AVLI) Target Mean Sqrt(AVLI) Target Mean AVLI St. Dev 300 (n=7) (n=7) (n=7) Target Means are the Oil LSMeans from the Model and Standard Deviations are calculated straight from Sqrt(AVLI). 56

59 Precision Matrix 2 LTMS Example N = 28 57

60 Precision Matrix 2 LTMS Example N = 28 58

61 Precision Matrix 2 LTMS Example N = 21 59

62 Precision Matrix 2 LTMS Example N = 21 60