Reliability Coefficients Introductory notes That data used for these computations is the pre-treatment scores of all subjects. There are three items in the SIAS (5, 9 and 11) that require reverse-scoring. Those items are stored in the data frame in reverse-coded format. In other words, if a subject responded to one of those items with a score of 0, it has been coded as a 4; if the client responded with a score of 1 it has been coded as a 3, and so on. The following code computes categorical McDonald s omega using the MBESS::ci.reliability() function, classical Cronbach s alpha and the mean inter-item correlation using psych::alpha(). In general, we would expect McDonald s omega values to be higher than the alpha values, as (i) they are based on polychoric correlations, rather than Pearson correlations, and (ii) because it is well known in the psychometric literature that alpha tends to under-estimate reliability for congeneric measurement models. # Load libraries library(dplyr) Attaching package: 'dplyr' The following objects are masked from 'package:stats': filter, lag The following objects are masked from 'package:base': intersect, setdiff, setequal, union library(psych) library(mbess) Attaching package: 'MBESS' The following object is masked from 'package:psych': cor2cov library(lavaan) This is lavaan 0.5-23.1097 lavaan is BETA software! Please report any bugs. Attaching package: 'lavaan' 1
The following object is masked from 'package:mbess': cor2cov The following object is masked from 'package:psych': cor2cov # Load datafile load("dat.rda") # Setup dummy matrix for storing store reliabilty coefficients x <- matrix(na, nrow = 12, ncol = 3) rownames(x) <- c("sias20", "SIAS19", "SIAS17", "SIAS6", "SIPS-INT", "SPS", "SPS6", "SIPS-FOE", "SIPS-FAA", "SIPS-AGG", "Peters", "SIPS-Tot") colnames(x) <- c("omega", "alpha", "mean_r") # Function to compute omega, alpha, and mean inter-item correlation rel_stats <- function(data, items){ df <- data[, items] x <- c(mbess::ci.reliability(data = df, type = "categorical", interval.type = "none")$est, psych::alpha(df)[[1]][c(1,4)] %>% unlist() ) names(x) <- c("omega", "alpha", "mean_r") x } # Compute Omega, alpha and mean-inter-item correlation # Extract SIAS items at pre-treatment df <- dat %>% select(sias_0_1:sias_0_20) # Compute stats for SIAS measures x[1, ] <- rel_stats(df, 1:20) # sias20 x[2, ] <- rel_stats(df, c(1:4, 6:20)) # sias19 x[3, ] <- rel_stats(df, c(1:4, 6:8, 10, 12:20)) # sias17 x[4, ] <- rel_stats(df, c(2, 4, 6, 8, 10, 13)) # sias6 x[5, ] <- rel_stats(df, c(7, 10, 15, 16, 19)) # sips-int # Extract SPS items at pre-treatment df2 <- dat %>% select(sps_0_1:sps_0_20) 2
# Compute stats for SPS measures x[6, ] <- rel_stats(df2, 1:20) # sps20 x[7, ] <- rel_stats(df2, c(4, 7, 8, 15:17)) # sps6 x[8, ] <- rel_stats(df2, c(4, 6, 8, 13, 16, 17)) # sips-foe x[9, ] <- rel_stats(df2, c(12, 14, 15)) # sips-faa x[10, ] <- rel_stats(df2, c(4, 6, 8, 12:17)) # sips-agg # Peters total (SIAS6 + SPS6) df3 <- cbind(df[,c(2, 4, 6, 8, 10, 13)], df2[, c(4, 7, 8, 15:17)]) x[11, ] <- rel_stats(df3, 1:12) # SIPS total df4 <- cbind(df[,c(7, 10, 15, 16, 19)], df2[, c(4, 6, 8, 12:17)]) x[12, ] <- rel_stats(df4, 1:14) # sips total # Display coefficients round(x, 2) omega alpha mean_r SIAS20 0.91 0.83 0.21 SIAS19 0.90 0.83 0.23 SIAS17 0.89 0.85 0.26 SIAS6 0.71 0.69 0.27 SIPS-INT 0.81 0.79 0.45 SPS 0.96 0.91 0.34 SPS6 0.86 0.84 0.47 SIPS-FOE 0.88 0.86 0.51 SIPS-FAA 0.75 0.74 0.48 SIPS-AGG 0.91 0.87 0.44 Peters 0.87 0.83 0.29 SIPS-Tot 0.97 0.87 0.33 Note that in the above code, separate factor models were specified for each SIPS subscale. An alternate way of computing omega for the SIPS is to define the full two- or three-factor model, and use those solutions to compute omega. library(semtools) # This is semtools 0.4-14 All users of R (or SEM) are invited to submit functions or ideas for functions. # 3
Attaching package: 'semtools' The following object is masked from 'package:psych': skew # Rearrange SIPS items so that items 1-5 = interaction scale, 6-11 are foe and 12-14 faa df5 <- cbind(df[,c(7, 10, 15, 16, 19)], df2[, c(4, 6, 8, 13, 16, 17, 12, 14, 15)]) names(df5) <- paste0("v", 1:14) # SIPS 3 factor model m1 <- ' INT =~ v1 + v2 + v3 + v4 + v5 FOE =~ v6 + v7 + v8 + v9 +v10 + v11 FAA =~ v12 + v13 + v14 ' # Omega = Raykov's omega, omega2 = bentler's omega, omega3 = McDonald's omega # Note that alpha is model based on polychoric correlations and thus will differ # from traditional alpha (in the above table) computed using Pearson correlations fit_m1 <- cfa(m1, data = df5, estimator = "WLSMV", std.lv = TRUE, orthogonal = FALSE, missing = "pairwise") round(semtools::reliability(fit_m1),2) INT FOE FAA total alpha 0.79 0.86 0.74 0.87 omega 0.78 0.87 0.74 0.90 omega2 0.78 0.87 0.74 0.90 omega3 0.76 0.87 0.73 0.90 avevar 0.41 0.53 0.49 0.49 The code below compute omega for the SIPS two-factor model, where the two performance anxiety subscales (FOE and FAA) are aggregated. m2 <- ' INT =~ v1 + v2 + v3 + v4 + v5 AGG =~ v6 + v7 + v8 + v9 +v10 + v11 + v12 + v13 + v14 ' fit_m2 <- cfa(m2, data = df5, estimator = "WLSMV", std.lv = TRUE, orthogonal = FALSE, missing = "pairwise") round(semtools::reliability(fit_m2),2) INT AGG total 4
alpha 0.79 0.87 0.87 omega 0.78 0.88 0.89 omega2 0.78 0.88 0.89 omega3 0.76 0.88 0.89 avevar 0.41 0.45 0.45 SIAS-6 correlation matrix The reliability coefficients for the SIAS-6 is notably lower than for the other scales. We can probe the reason for this by examining the inter-item correlations which should be quite low (the mean inter-item correlation is.27). cor(df[, c(2, 4, 6, 8, 10, 13)]) %>% round(2) sias_0_2 sias_0_4 sias_0_6 sias_0_8 sias_0_10 sias_0_13 sias_0_2 1.00 NA 0.28 0.21 0.33 NA sias_0_4 NA 1 NA NA NA NA sias_0_6 0.28 NA 1.00 0.42 0.28 NA sias_0_8 0.21 NA 0.42 1.00 0.27 NA sias_0_10 0.33 NA 0.28 0.27 1.00 NA sias_0_13 NA NA NA NA NA 1 5