## Classical Test Theory Reliability and Items Analysis in SPSS and SAS

The data from this example come from Study 2 of Summers et al. (2005).

We are measuring the importance of various aspects of Parent-Professional partnerships for families who have children with disabilities. We begin with two subscales that were identified and refined through exploratory factor analysis in a previous study: Child (11 items) and Family (10 items). We will examine both scales using the psych package (Revelle 2017) in R (R Core Team 2017b).

First, we load the psych package. The following script checks if the package is installed first, if not it downloads and installs the package before loading the package.

  haspackage = require(psych)
if (!haspackage) install.packages("psych")
library(psych)

As the data were originally in SPSS format, we will import the data using the foreign package (R Core Team 2017a).

  haspackage = require(foreign)
if (!haspackage) install.packages("foreign")
library(foreign)

### Data Import

First, we import the data from the SPSS file (housed in the same directory as this Rmd file). Here, we save the data as a data frame (an R data set object; to.data.frame = TRUE) and we do not allow value labels (use.value.labels = FALSE) as these make the variables factors, which cannot be used with the psych package. Displayed are the names of the variables in the data file.

analysisData = read.spss(file = "FT2part.sav", to.data.frame = TRUE, use.value.labels = FALSE)
re-encoding from CP1252
names(analysisData)
  [1] "state"    "district" "familyid" "person"   "pagelnk"  "disgroup" "pi1"      "pi2"      "pi3"      "pi4"      "pi5"      "pi6"
[13] "pi7"      "pi8"      "pi9"      "pi10"     "pi11"     "pi12"     "pi13"     "pi14"     "pi15"     "pi16"     "pi17"     "pi18"
[25] "pi19"     "pi20"     "pi21"     "pi22"     "pi23"     "pi24"     "pi25"     "ps1"      "ps2"      "ps3"      "ps4"      "ps5"
[37] "ps6"      "ps7"      "ps8"      "ps9"      "ps10"     "ps11"     "ps12"     "ps13"     "ps14"     "ps15"     "ps16"     "ps17"
[49] "ps18"     "ps19"     "ps20"     "ps21"     "ps22"     "ps23"     "ps24"     "ps25"     "fi1"      "fi2"      "fi3"      "fi4"
[61] "fi5"      "fi6"      "fi7"      "fi8"      "fi9"      "fi10"     "fi11"     "fi12"     "fi13"     "fi14"     "fi15"     "fi16"
[73] "fi17"     "fi18"     "fi19"     "fi20"     "fi21"     "fi22"     "fi23"     "fi24"     "fi25"     "fi26"     "fi27"     "fi28"
[85] "fi29"     "fi30"     "fi31"     "fi32"     "fi33"     "fi34"     "fi35"     "fi36"     "fi37"     "fi38"     "fi39"     "fi40"
[97] "fi41"     "fs1"      "fs2"      "fs3"      "fs4"      "fs5"      "fs6"      "fs7"      "fs8"      "fs9"      "fs10"     "fs11"
[109] "fs12"     "fs13"     "fs14"     "fs15"     "fs16"     "fs17"     "fs18"     "fs19"     "fs20"     "fs21"     "fs22"     "fs23"
[121] "fs24"     "fs25"     "fs26"     "fs27"     "fs28"     "fs29"     "fs30"     "fs31"     "fs32"     "fs33"     "fs34"     "fs35"
[133] "fs36"     "fs37"     "fs38"     "fs39"     "fs40"     "fs41"     "ayr"      "amo"      "ada"      "bda"      "q1"       "q2_mm"
[145] "q2_yyyy"  "q2age"    "rq2age"   "q3"       "q4a"      "q4b"      "q4c"      "q4d"      "q4e"      "rhq4a"    "rhq4b"    "rhq4c"
[157] "rhq4d"    "rhq4e"    "rq4a"     "rq4b"     "rq4c"     "rq4d"     "rq4e"     "white"    "q5"       "q6a"      "q6b"      "q7"
[169] "q7r"      "q8"       "q9"       "q10"      "q11"      "q12"      "q13_mm"   "q13_yyyy" "q13age"   "q13agegp" "q14"      "q15a"
[181] "q15b"     "q15c"     "q15d"     "q15e"     "q15f"     "q15g"     "q15h"     "q15i"     "q15j"     "q15k"     "q15l"     "q15m"
[193] "q16"      "q17"      "q18a"     "q18b"     "q18c"     "pimis"    "psmis"    "pians"    "psans"    "usepart"  "pchim"    "pfamim"
[205] "pchsm"    "pfamsm"   "finmis"   "fsnmis"   "fidmis"   "fsdmis"   "fitmis"   "fstmis"   "fians"    "fsans"    "usefqol"  "fintim"
[217] "parim"    "gresim"   "disim"    "fintsm"   "parsm"    "gressm"   "dissm"    "filter_." "peruse"   "age2gp"   "pich"     "pifam"
[229] "pi"       "psch"     "psfam"    "ps"       "age3gp"   "usealt"   "nresp"   

### Analysis of 11-item Child Scale

We use complete observations to calculate our Alpha coefficient (to be consistent with the SAS and SPSS examples). In the item statistics portion of the output:

• raw.r is the correlation of each item with the total score, not corrected for item overlap
• std.r is the correlation of each item with the total score (not corrected for item overlap) if the items were all standardized
• r.cor is the item whole correlation corrected for item overlap and scale reliability
• r.drop is the item whole correlation for this item against the scale without this item
#create list of variables on CHILD11 Scale
child11Vars = c("pi5", "pi6", "pi7", "pi8", "pi9", "pi10", "pi11", "pi12", "pi13", "pi14", "pi15")
#Create a data frame with only these variables
child11Data = analysisData[child11Vars]
alpha(x = child11Data, use = "complete.obs")

Reliability analysis
Call: alpha(x = child11Data, use = "complete.obs")

raw_alpha std.alpha G6(smc) average_r S/N    ase mean   sd
0.91      0.92    0.93      0.51  11 0.0092  4.6 0.54

lower alpha upper     95% confidence boundaries
0.9 0.91 0.93

Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r  S/N alpha se
pi5       0.91      0.92    0.93      0.52 10.9   0.0096
pi6       0.91      0.91    0.92      0.51 10.6   0.0099
pi7       0.91      0.91    0.92      0.51 10.3   0.0101
pi8       0.91      0.91    0.92      0.51 10.5   0.0100
pi9       0.91      0.91    0.92      0.51 10.5   0.0100
pi10      0.91      0.91    0.92      0.51 10.5   0.0101
pi11      0.90      0.91    0.92      0.50 10.0   0.0104
pi12      0.90      0.91    0.92      0.49  9.6   0.0109
pi13      0.90      0.91    0.92      0.50 10.0   0.0104
pi14      0.91      0.92    0.93      0.52 11.0   0.0096
pi15      0.90      0.91    0.92      0.50  9.8   0.0103

Item statistics
n raw.r std.r r.cor r.drop mean   sd
pi5  187  0.68  0.67  0.62   0.59  4.4 0.88
pi6  187  0.69  0.71  0.69   0.63  4.7 0.66
pi7  186  0.73  0.75  0.73   0.68  4.7 0.63
pi8  177  0.73  0.72  0.68   0.65  4.4 0.92
pi9  185  0.73  0.72  0.69   0.66  4.4 0.86
pi10 188  0.74  0.72  0.69   0.66  4.3 0.82
pi11 187  0.78  0.79  0.76   0.73  4.7 0.65
pi12 186  0.85  0.84  0.84   0.80  4.6 0.74
pi13 187  0.78  0.79  0.78   0.73  4.6 0.71
pi14 184  0.66  0.66  0.61   0.58  4.6 0.74
pi15 184  0.79  0.81  0.81   0.75  4.8 0.52

Non missing response frequency for each item
1    2    3    4    5 miss
pi5  0.02 0.01 0.13 0.24 0.60 0.02
pi6  0.01 0.00 0.08 0.11 0.80 0.02
pi7  0.01 0.01 0.03 0.15 0.80 0.03
pi8  0.02 0.01 0.14 0.18 0.65 0.07
pi9  0.01 0.02 0.12 0.25 0.59 0.03
pi10 0.01 0.01 0.18 0.27 0.54 0.02
pi11 0.01 0.01 0.06 0.09 0.83 0.02
pi12 0.01 0.01 0.10 0.20 0.69 0.03
pi13 0.01 0.00 0.10 0.18 0.72 0.02
pi14 0.00 0.01 0.12 0.16 0.71 0.04
pi15 0.01 0.00 0.04 0.05 0.90 0.04

Here are the results if we drop item pi14:

#create list of variables on CHILD11 Scale
child10Vars = c("pi5", "pi6", "pi7", "pi8", "pi9", "pi10", "pi11", "pi12", "pi13", "pi15")
#Create a data frame with only these variables
child10Data = analysisData[child10Vars]
alpha(x = child10Data, use = "complete.obs")

Reliability analysis
Call: alpha(x = child10Data, use = "complete.obs")

raw_alpha std.alpha G6(smc) average_r S/N    ase mean   sd
0.91      0.92    0.93      0.52  11 0.0096  4.6 0.55

lower alpha upper     95% confidence boundaries
0.89 0.91 0.93

Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r  S/N alpha se
pi5       0.91      0.91    0.92      0.54 10.7   0.0099
pi6       0.90      0.91    0.91      0.53 10.2   0.0104
pi7       0.90      0.91    0.92      0.52  9.9   0.0106
pi8       0.90      0.91    0.92      0.53 10.1   0.0106
pi9       0.90      0.91    0.92      0.53 10.2   0.0105
pi10      0.90      0.91    0.92      0.53 10.3   0.0105
pi11      0.90      0.91    0.92      0.52  9.6   0.0110
pi12      0.89      0.90    0.91      0.51  9.2   0.0115
pi13      0.90      0.91    0.91      0.52  9.7   0.0109
pi15      0.90      0.90    0.91      0.51  9.5   0.0108

Item statistics
n raw.r std.r r.cor r.drop mean   sd
pi5  187  0.68  0.67  0.62   0.59  4.4 0.88
pi6  187  0.70  0.72  0.70   0.64  4.7 0.66
pi7  186  0.74  0.76  0.73   0.68  4.7 0.63
pi8  177  0.75  0.73  0.69   0.67  4.4 0.92
pi9  185  0.74  0.73  0.69   0.66  4.4 0.86
pi10 188  0.73  0.71  0.67   0.64  4.3 0.82
pi11 187  0.79  0.80  0.77   0.74  4.7 0.65
pi12 186  0.85  0.85  0.84   0.80  4.6 0.74
pi13 187  0.78  0.78  0.77   0.72  4.6 0.71
pi15 184  0.79  0.81  0.80   0.74  4.8 0.52

Non missing response frequency for each item
1    2    3    4    5 miss
pi5  0.02 0.01 0.13 0.24 0.60 0.02
pi6  0.01 0.00 0.08 0.11 0.80 0.02
pi7  0.01 0.01 0.03 0.15 0.80 0.03
pi8  0.02 0.01 0.14 0.18 0.65 0.07
pi9  0.01 0.02 0.12 0.25 0.59 0.03
pi10 0.01 0.01 0.18 0.27 0.54 0.02
pi11 0.01 0.01 0.06 0.09 0.83 0.02
pi12 0.01 0.01 0.10 0.20 0.69 0.03
pi13 0.01 0.00 0.10 0.18 0.72 0.02
pi15 0.01 0.00 0.04 0.05 0.90 0.04

Here are the results if we drop item pi10 and pi14:

#create list of variables on CHILD11 Scale
child9Vars = c("pi5", "pi6", "pi7", "pi8", "pi9", "pi11", "pi12", "pi13", "pi15")
#Create a data frame with only these variables
child9Data = analysisData[child9Vars]
alpha(x = child9Data, use = "complete.obs")

Reliability analysis
Call: alpha(x = child9Data, use = "complete.obs")

raw_alpha std.alpha G6(smc) average_r S/N  ase mean   sd
0.9      0.91    0.92      0.53  10 0.01  4.6 0.55

lower alpha upper     95% confidence boundaries
0.88 0.9 0.92

Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r  S/N alpha se
pi5       0.90      0.91    0.92      0.56 10.0    0.011
pi6       0.89      0.90    0.91      0.54  9.3    0.012
pi7       0.89      0.90    0.91      0.53  9.1    0.012
pi8       0.90      0.90    0.91      0.54  9.5    0.011
pi9       0.90      0.91    0.91      0.54  9.5    0.011
pi11      0.89      0.90    0.91      0.53  8.9    0.012
pi12      0.88      0.89    0.90      0.51  8.5    0.013
pi13      0.89      0.90    0.90      0.53  8.9    0.012
pi15      0.89      0.90    0.90      0.52  8.7    0.012

Item statistics
n raw.r std.r r.cor r.drop mean   sd
pi5  187  0.69  0.68  0.61   0.58  4.4 0.88
pi6  187  0.73  0.75  0.72   0.67  4.7 0.66
pi7  186  0.75  0.77  0.74   0.68  4.7 0.63
pi8  177  0.75  0.73  0.68   0.65  4.4 0.92
pi9  185  0.74  0.72  0.68   0.64  4.4 0.86
pi11 187  0.79  0.79  0.76   0.73  4.7 0.65
pi12 186  0.85  0.84  0.84   0.80  4.6 0.74
pi13 187  0.79  0.79  0.77   0.72  4.6 0.71
pi15 184  0.79  0.82  0.81   0.75  4.8 0.52

Non missing response frequency for each item
1    2    3    4    5 miss
pi5  0.02 0.01 0.13 0.24 0.60 0.02
pi6  0.01 0.00 0.08 0.11 0.80 0.02
pi7  0.01 0.01 0.03 0.15 0.80 0.03
pi8  0.02 0.01 0.14 0.18 0.65 0.07
pi9  0.01 0.02 0.12 0.25 0.59 0.03
pi11 0.01 0.01 0.06 0.09 0.83 0.02
pi12 0.01 0.01 0.10 0.20 0.69 0.03
pi13 0.01 0.00 0.10 0.18 0.72 0.02
pi15 0.01 0.00 0.04 0.05 0.90 0.04

### Analysis of 12-item Family Subscale

#create list of variables on CHILD11 Scale
family12Vars = c("pi10", "pi14", "pi16", "pi17", "pi18", "pi19", "pi20", "pi21", "pi22", "pi23", "pi24", "pi25")
#Create a data frame with only these variables
family12Data = analysisData[family12Vars]
alpha(x = family12Data, use = "complete.obs")

Reliability analysis
Call: alpha(x = family12Data, use = "complete.obs")

raw_alpha std.alpha G6(smc) average_r S/N    ase mean   sd
0.92      0.92    0.93       0.5  12 0.0082  4.4 0.59

lower alpha upper     95% confidence boundaries
0.91 0.92 0.94

Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r S/N alpha se
pi10      0.91      0.92    0.92      0.50  11   0.0091
pi14      0.92      0.92    0.93      0.52  12   0.0084
pi16      0.92      0.92    0.92      0.50  11   0.0088
pi17      0.91      0.91    0.92      0.49  11   0.0094
pi18      0.91      0.91    0.92      0.49  11   0.0092
pi19      0.91      0.91    0.92      0.49  11   0.0093
pi20      0.92      0.92    0.93      0.53  12   0.0082
pi21      0.91      0.91    0.92      0.49  11   0.0093
pi22      0.92      0.92    0.93      0.50  11   0.0089
pi23      0.91      0.92    0.92      0.50  11   0.0090
pi24      0.92      0.92    0.92      0.50  11   0.0090
pi25      0.92      0.92    0.93      0.51  11   0.0088

Item statistics
n raw.r std.r r.cor r.drop mean   sd
pi10 188  0.77  0.77  0.74   0.71  4.3 0.82
pi14 184  0.61  0.63  0.58   0.54  4.6 0.74
pi16 185  0.74  0.72  0.69   0.66  4.1 1.00
pi17 184  0.82  0.81  0.79   0.77  4.2 0.89
pi18 184  0.79  0.79  0.78   0.74  4.4 0.82
pi19 185  0.80  0.80  0.79   0.76  4.4 0.76
pi20 184  0.58  0.58  0.52   0.50  4.4 0.83
pi21 183  0.80  0.81  0.79   0.76  4.4 0.78
pi22 183  0.73  0.73  0.71   0.67  4.4 0.78
pi23 184  0.75  0.76  0.74   0.70  4.5 0.72
pi24 184  0.74  0.75  0.73   0.69  4.5 0.72
pi25 184  0.71  0.71  0.67   0.65  4.2 0.87

Non missing response frequency for each item
1    2    3    4    5 miss
pi10 0.01 0.01 0.18 0.27 0.54 0.02
pi14 0.00 0.01 0.12 0.16 0.71 0.04
pi16 0.02 0.03 0.24 0.25 0.46 0.03
pi17 0.01 0.01 0.23 0.28 0.47 0.04
pi18 0.01 0.01 0.15 0.26 0.58 0.04
pi19 0.00 0.00 0.17 0.29 0.54 0.03
pi20 0.01 0.02 0.14 0.22 0.61 0.04
pi21 0.01 0.00 0.15 0.24 0.61 0.04
pi22 0.01 0.00 0.15 0.28 0.56 0.04
pi23 0.01 0.01 0.08 0.27 0.64 0.04
pi24 0.01 0.00 0.10 0.23 0.66 0.04
pi25 0.02 0.01 0.17 0.33 0.48 0.04

Here are the results if we drop item 19:

#create list of variables on CHILD11 Scale
family11Vars = c("pi10", "pi14", "pi16", "pi17", "pi18", "pi20", "pi21", "pi22", "pi23", "pi24", "pi25")
#Create a data frame with only these variables
family11Data = analysisData[family11Vars]
alpha(x = family11Data, use = "complete.obs")

Reliability analysis
Call: alpha(x = family11Data, use = "complete.obs")

raw_alpha std.alpha G6(smc) average_r S/N    ase mean   sd
0.91      0.91    0.92      0.49  11 0.0093  4.4 0.59

lower alpha upper     95% confidence boundaries
0.89 0.91 0.93

Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r  S/N alpha se
pi10      0.90      0.90    0.91      0.48  9.4   0.0105
pi14      0.91      0.91    0.92      0.51 10.4   0.0096
pi16      0.91      0.91    0.91      0.50  9.8   0.0099
pi17      0.90      0.90    0.91      0.48  9.2   0.0109
pi18      0.90      0.90    0.91      0.48  9.3   0.0105
pi20      0.91      0.92    0.92      0.52 10.8   0.0093
pi21      0.90      0.90    0.91      0.48  9.1   0.0107
pi22      0.90      0.91    0.91      0.49  9.6   0.0102
pi23      0.90      0.90    0.91      0.48  9.4   0.0104
pi24      0.90      0.90    0.91      0.49  9.5   0.0103
pi25      0.91      0.91    0.91      0.50  9.8   0.0100

Item statistics
n raw.r std.r r.cor r.drop mean   sd
pi10 188  0.77  0.77  0.74   0.71  4.3 0.82
pi14 184  0.62  0.63  0.58   0.55  4.6 0.74
pi16 185  0.73  0.71  0.67   0.64  4.1 1.00
pi17 184  0.82  0.81  0.79   0.76  4.2 0.89
pi18 184  0.78  0.78  0.75   0.72  4.4 0.82
pi20 184  0.58  0.57  0.51   0.49  4.4 0.83
pi21 183  0.81  0.81  0.80   0.76  4.4 0.78
pi22 183  0.74  0.74  0.72   0.68  4.4 0.78
pi23 184  0.76  0.77  0.75   0.71  4.5 0.72
pi24 184  0.75  0.76  0.75   0.70  4.5 0.72
pi25 184  0.71  0.71  0.66   0.64  4.2 0.87

Non missing response frequency for each item
1    2    3    4    5 miss
pi10 0.01 0.01 0.18 0.27 0.54 0.02
pi14 0.00 0.01 0.12 0.16 0.71 0.04
pi16 0.02 0.03 0.24 0.25 0.46 0.03
pi17 0.01 0.01 0.23 0.28 0.47 0.04
pi18 0.01 0.01 0.15 0.26 0.58 0.04
pi20 0.01 0.02 0.14 0.22 0.61 0.04
pi21 0.01 0.00 0.15 0.24 0.61 0.04
pi22 0.01 0.00 0.15 0.28 0.56 0.04
pi23 0.01 0.01 0.08 0.27 0.64 0.04
pi24 0.01 0.00 0.10 0.23 0.66 0.04
pi25 0.02 0.01 0.17 0.33 0.48 0.04

Here are the results if we drop items 17, 18, and 19:

#create list of variables on CHILD11 Scale
family9Vars = c("pi10", "pi14", "pi16", "pi20", "pi21", "pi22", "pi23", "pi24", "pi25")
#Create a data frame with only these variables
family9Data = analysisData[family9Vars]
alpha(x = family9Data, use = "complete.obs")

Reliability analysis
Call: alpha(x = family9Data, use = "complete.obs")

raw_alpha std.alpha G6(smc) average_r S/N   ase mean   sd
0.88      0.89    0.89      0.46 7.8 0.013  4.4 0.58

lower alpha upper     95% confidence boundaries
0.86 0.88 0.91

Reliability if an item is dropped:
raw_alpha std.alpha G6(smc) average_r S/N alpha se
pi10      0.87      0.87    0.87      0.46 6.7    0.015
pi14      0.88      0.88    0.88      0.48 7.4    0.013
pi16      0.87      0.88    0.88      0.47 7.1    0.014
pi20      0.88      0.89    0.89      0.50 7.9    0.013
pi21      0.86      0.86    0.87      0.44 6.4    0.015
pi22      0.87      0.87    0.87      0.46 6.8    0.014
pi23      0.86      0.87    0.87      0.45 6.5    0.015
pi24      0.87      0.87    0.87      0.45 6.7    0.014
pi25      0.87      0.88    0.88      0.47 7.1    0.014

Item statistics
n raw.r std.r r.cor r.drop mean   sd
pi10 188  0.76  0.76  0.72   0.68  4.3 0.82
pi14 184  0.64  0.65  0.59   0.54  4.6 0.74
pi16 185  0.73  0.70  0.65   0.61  4.1 1.00
pi20 184  0.59  0.58  0.51   0.48  4.4 0.83
pi21 183  0.81  0.81  0.79   0.75  4.4 0.78
pi22 183  0.74  0.75  0.71   0.66  4.4 0.78
pi23 184  0.78  0.79  0.77   0.71  4.5 0.72
pi24 184  0.75  0.77  0.75   0.68  4.5 0.72
pi25 184  0.71  0.70  0.65   0.61  4.2 0.87

Non missing response frequency for each item
1    2    3    4    5 miss
pi10 0.01 0.01 0.18 0.27 0.54 0.02
pi14 0.00 0.01 0.12 0.16 0.71 0.04
pi16 0.02 0.03 0.24 0.25 0.46 0.03
pi20 0.01 0.02 0.14 0.22 0.61 0.04
pi21 0.01 0.00 0.15 0.24 0.61 0.04
pi22 0.01 0.00 0.15 0.28 0.56 0.04
pi23 0.01 0.01 0.08 0.27 0.64 0.04
pi24 0.01 0.00 0.10 0.23 0.66 0.04
pi25 0.02 0.01 0.17 0.33 0.48 0.04

## References

R Core Team. 2017a. Foreign: Read Data Stored by ’Minitab’, ’S’, ’Sas’, ’Spss’, ’Stata’, ’Systat’, ’Weka’, ’dBase’, ... https://CRAN.R-project.org/package=foreign.

———. 2017b. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Revelle, William. 2017. Psych: Procedures for Psychological, Psychometric, and Personality Research. Evanston, Illinois: Northwestern University. https://CRAN.R-project.org/package=psych.

Summers, J., L. Hoffman, J. Marquis, A. Turnbull, D. Postson, and L. Nelson. 2005. “Measuring the Quality of Family—Professional Partnerships in Special Education Services.” Except Children 72 (1): 65–81. doi:10.1177/001440290507200104.