Hostname: page-component-8448b6f56d-42gr6 Total loading time: 0 Render date: 2024-04-17T12:24:16.432Z Has data issue: false hasContentIssue false
  • English
  • Français

The myth of testing construct validity using factor analysis or correlations with normal or mixed clinical populations: Lessons from memory assessment

Published online by Cambridge University Press:  27 August 2003

Dean C. Delis ,
Mark Jacobson ,
Mark W. Bondi ,
Joanne M. Hamilton  and
David P. Salmon
Dean C. Delis*
Affiliation:
Department of Veterans Affairs Medical Center, San Diego, California University of California, San Diego, School of Medicine, San Diego, California
Mark Jacobson
Affiliation:
Department of Veterans Affairs Medical Center, San Diego, California University of California, San Diego, School of Medicine, San Diego, California
Mark W. Bondi
Affiliation:
Department of Veterans Affairs Medical Center, San Diego, California University of California, San Diego, School of Medicine, San Diego, California
Joanne M. Hamilton
Affiliation:
University of California, San Diego, School of Medicine, San Diego, California
David P. Salmon
Affiliation:
University of California, San Diego, School of Medicine, San Diego, California
*
Reprint requests to: Dean C. Delis, Ph.D., Psychology Service (116B), V.A. Medical Center, 3350 La Jolla Village Drive, San Diego, CA 92161. E-mail: ddelis@ucsd.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

For nearly a century, the primary method employed by psychologists to define and test the validity of constructs evaluated by assessment instruments has been shared-variance techniques such as intervariable correlations or factor analysis with large normative or mixed clinical samples. To illustrate the shortcomings of this approach, we conducted (1) correlational analyses of immediate- and delayed-memory measures separately in normal participants and in homogeneous samples of patients with either Alzheimer's disease or Huntington's disease; and (2) factor analysis of immediate and delayed-recall and recognition measures in a large, homogeneous sample of patients with Alzheimer's disease. The findings revealed that cognitive measures that share variance in the intact brain—thereby giving the facade of assessing a unitary construct—can dissociate and contribute to unique variance in the damaged brain, but only if the pathology occurs in brain regions known to disrupt vital cognitive processes tapped by those measures. The results illustrate that shared-variance procedures applied to normal or mixed clinical populations can mask some of the most vital cognitive constructs, such as the classic distinction between short- and long-term memory. Implications of these findings for research and clinical practice are discussed. (JINS, 2003, 9, 936–946.)

Keywords

Construct validityShared varianceFactor analysisCVLT
Type
Research Article
Information
Journal of the International Neuropsychological Society , Volume 9 , Issue 6 , September 2003 , pp. 936 - 946
Copyright
Copyright © The International Neuropsychological Society 2003

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aylward, A.E., Codori, A.M., Barta, P.E., Pearlson, G.D., Harris, G.J., & Brandt, J.B. (1996). Basal ganglia volume and proximity to onset in presymptomatic Huntington's disease. Archives of Neurology, 53, 1293–1296.10.1001/archneur.1996.00550120105023CrossRefGoogle Scholar
Boone, K.B., Pontón, M.O., Gorsuch, R.L., González, J.J., & Miller, B.L. (1998). Factor analysis of four measures of prefrontal lobe functioning. Archives of Clinical Neuropsychology, 13, 585–597.10.1093/arclin/13.7.585CrossRefGoogle ScholarPubMed
Bowden, S.C., Carstairs, J.R., & Shores, E.A. (1999). Confirmatory factor analysis of combined Wechsler Adult Intelligence Scale–Revised and Wechsler Memory Scale–Revised in a healthy community sample. Psychological Assessment, 11, 339–344.10.1037/1040-3590.11.3.339CrossRefGoogle Scholar
Braak, H. & Braak, E. (1991). Neuropathological staging of Alzheimer-related changes. Acta Neuropathologica, 82, 239–259.10.1007/BF00308809CrossRefGoogle Scholar
Buschke, H. (1973). Selective reminding for analysis of memory and learning. Journal of Verbal Learning and Verbal Behavior, 12, 543–550.10.1016/S0022-5371(73)80034-9CrossRefGoogle Scholar
Butters, N., Wolfe, J., Martone, M., Granholm, E., & Cermak, L.S. (1985). Memory disorders associated with Huntington's disease: Verbal recall, verbal recognition, and procedural memory. Neuropsychologia, 23, 729–743.10.1016/0028-3932(85)90080-6CrossRefGoogle ScholarPubMed
Butters, N., Delis, D.C., & Lucas, J.A. (1995). Clinical assessment of memory disorders in amnesia and dementia. Annual Review of Psychology, 46, 493–523.10.1146/annurev.ps.46.020195.002425CrossRefGoogle ScholarPubMed
Byrne, B.M. (1994). Structural equation modeling with EQS and EQS/Windows: Basic concepts, applications, and programming. Thousand Oaks, California: Sage Publications.Google Scholar
Cronback, L.J. & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 93, 179–197.Google Scholar
Delis, D.C. & Kaplan, E. (1983). Hazards of a standardized neuropsychological test with low content validity: Comment of the Luria-Nebraska Neuropsychological Battery. Journal of Consulting and Clinical Psychology, 51, 396–398.10.1037/0022-006X.51.3.396CrossRefGoogle ScholarPubMed
Delis, D.C. & Kramer, J.H. (2000). Advances in the neuropsychological assessment of memory disorders. In Boller, F. and Grafman, J. (Eds.), Handbook of neuropsychology, Volume 2 (pp. 25–48) (2nd ed.). Amsterdam, The Netherlands: Elsevier Science Publishers.Google Scholar
Delis, D.C., Kramer, J.H., Kaplan, E., & Ober, B.A. (1987). The California Verbal Learning Test. San Antonio, Texas: The Psychological Corporation.Google Scholar
Delis, D.C., Freeland, J., Kramer, J.H., & Kaplan, E. (1988). Integrating clinical assessment with cognitive neuroscience: Construct validation of the California Verbal Learning Test. Journal of Consulting and Clinical Psychology, 56, 123–130.10.1037/0022-006X.56.1.123CrossRefGoogle ScholarPubMed
Delis, D.C., Massman, P.J., Butters, N., Salmon, D.P., Cermak, L.S., & Kramer, J.H. (1991). Profiles of demented and amnesic patients on the California Verbal Learning Test: Implications for the assessment of memory disorders. Psychological Assessment, 3, 19–26.10.1037/1040-3590.3.1.19CrossRefGoogle Scholar
Delis, D.C., Kramer, J.H., Kaplan, E., & Ober, B.A. (1994). The California Verbal Learning Test–Children's Version. San Antonio, Texas: The Psychological Corporation.Google Scholar
Delis, D.C., Kramer, J.H., Kaplan, E., & Ober, B.A. (2000). The California Verbal Learning Test–Second Edition. San Antonio, Texas: The Psychological Corporation.Google Scholar
Delis, D.C., Kaplan, E., & Kramer, J.H. (2001). The Delis-Kaplan executive function system. San Antonio, Texas: The Psychological Corporation.Google Scholar
Della Sala, S., Gray, C., Spinnler, H., & Trivelli, C. (1998). Frontal lobe functioning in man: The riddle revisited. Archives of Clinical Neuropsychology, 13, 663–682.Google ScholarPubMed
DiPino, R.K., Kabat, M.H., & Kane, R.L. (2000). An exploration of the construct validity of the Heaton memory test. Archives of Clinical Neuropsychology, 15, 95–103.10.1093/arclin/15.2.95CrossRefGoogle Scholar
Fridlund, A.J. & Delis, D.C. (1987). Administration and scoring software program for the California Verbal Learning Test. San Antonio, Texas: The Psychological Corporation.Google Scholar
Graf, P., Squire, L.R., & Mandler, G. (1984). The information that amnesic patients do not forget. Journal of Experimental Psychology: Learning, Memory, & Cognition, 1, 164–178.Google Scholar
Heaton, R.K., Chelune, G.J., Talley, J.L., Kay, G.G., & Curtiss, G. (1993). Wisconsin Card Sorting Test Manual. Odessa, Florida: Psychological Assessment Resources.Google Scholar
Jernigan, T.L., Salmon, D.P., Butters, N., & Hesselink, J.F. (1991). Cerebral structure on MRI, Part II: Specific changes in Alzheimer's and Huntington's diseases. Biological Psychiatry, 29, 68–81.10.1016/0006-3223(91)90211-4CrossRefGoogle ScholarPubMed
Jurden, F.H., Franzen, M.D., Callahan, T., & Ledbetter, M. (1996). Factorial equivalence of the Wechsler Memory Scale–Revised across standardization and clinical samples. Applied Neuropsychology, 3, 65–74.10.1207/s15324826an0302_4CrossRefGoogle ScholarPubMed
Kaplan, R.M. & Saccuzzo, D.P. (1997). Psychological testing: Principles, applications, and issues (4th ed.). Pacific Grove, California: Brooks/Cole Publishing Co.Google Scholar
Kaplan, E., Fein, D., Kramer, J.H., Delis, D.C., & Morris, R. (1999). The Wechsler Intelligence Scale for Children–III as a Process Instrument (WISC–III–PI). San Antonio, Texas: The Psychological Corporation.Google Scholar
Kline, R.B. (1988). Principles and practice of structural equation modeling. New York: Guilford Press.Google Scholar
Larrabee, G.J. & Curtiss, G. (1995). Construct validity of various verbal and visual memory tests. Journal of Clinical and Experimental Neuropsychology, 17, 536–547.10.1080/01688639508405144CrossRefGoogle ScholarPubMed
Larrabee, G.J., Kane, R.L., Schuck, J.R., & Francis, D.J. (1985). Construct validity of various memory testing procedures. Journal of Clinical and Experimental Neuropsychology, 7, 239–250.10.1080/01688638508401257CrossRefGoogle ScholarPubMed
Leonberger, F.T., Nicks, S.D., Larrabee, G.J., & Goldfader, P.R. (1992). Facture structure of the Wechsler Memory Scale–Revised within a comprehensive neuropsychological battery. Neuropsychology, 6, 239–249.10.1037/0894-4105.6.3.239CrossRefGoogle Scholar
Macmann, G.M. & Barnett, D.W. (1994). Structural analysis of correlated factors: Lessons from the verbal-performance dichotomy of the Wechsler scales. School Psychology Quarterly, 9, 161–197.10.1037/h0088287CrossRefGoogle Scholar
Massman, P.J., Delis, D.C., Butters, N., Levin, B.E., & Salmon, D.P. (1990). Are all subcortical dementias alike?: Verbal learning and memory in Parkinson's and Huntington's disease patients. Journal of Clinical and Experimental Neuropsychology, 12, 729–744.10.1080/01688639008401015CrossRefGoogle ScholarPubMed
McKhann, G., Drachman, D., Folstein, M., Katzman, R., Price, D., & Stadlan, E.M. (1984). Clinical diagnosis of Alzheimer's disease: Report of the NINCDS–ADRDA Work Group under the auspices of Department of Health and Human Services Task Force on Alzheimer's Disease. Neurology, 34, 939–944.10.1212/WNL.34.7.939CrossRefGoogle ScholarPubMed
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.CrossRefGoogle Scholar
Millis, S.R., Malina, A.C., Bowers, D.A., & Ricker, J.H. (1999). Confirmatory factor analysis of the Wechsler Memory Scale–III. Journal of Clinical and Experimental Neuropsychology, 21, 87–93.10.1076/jcen.21.1.87.937CrossRefGoogle ScholarPubMed
Milner, B., Corkin, S., & Teuber, H.L. (1968). Further analysis of the hippocampal amnesic syndrome: 14-year follow-up study of H.M. Neuropsychologia, 6, 215–234.10.1016/0028-3932(68)90021-3CrossRefGoogle Scholar
Mirsky, A.F. & Duncan, C.C. (2001). A nosology of disorders of attention. Annals of the New York Academy of Sciences, 931, 17–32.10.1111/j.1749-6632.2001.tb05771.xCrossRefGoogle Scholar
Mirsky, A.F., Pascualvaca, D.M., Duncan, C.C., & French, L.M. (1999). A model of attention and its relation to ADHD. Mental Retardation and Developmental Disabilities Research Reviews, 5, 169–179.10.1002/(SICI)1098-2779(1999)5:3<169::AID-MRDD2>3.0.CO;2-K3.0.CO;2-K>CrossRefGoogle Scholar
Pearson, E.S. & Hartly, H.O. (1966). Biometrika Tables for Statisticians, Vol. 1. Cambridge, Massachusetts: Cambridge University Press.Google Scholar
Rey, A. (1964). Clinical exam in psychology. Paris, France: University of France Press.Google Scholar
Russell, E.W. (1975). A multiple scoring method for the assessment of complex memory functions. Journal of Consulting and Clinical Psychology, 43, 800–809.10.1037/0022-006X.43.6.800CrossRefGoogle Scholar
Ryan, J.J. & Paolo, A.M. (2001). Exploratory factor analysis of the WAIS–III in a mixed patient sample. Archives of Clinical Neuropsychology, 16, 151–156.10.1093/arclin/16.2.151CrossRefGoogle Scholar
Ryan, J.J., Rosenberg, S.J., & Mittenberg, W. (1984). Factor analysis of the Rey Auditory–Verbal Learning Test. International Journal of Clinical Neuropsychology, 6, 239–241.Google Scholar
Schmidt, M. (1997). Some cautions on interpreting qualitative indices for word-list learning tests. Clinical Neuropsychologist, 11, 81–86.CrossRefGoogle Scholar
Shear, J.M. & Craft, R.B. (1989). Examination of the concurrent validity of the California Verbal Learning Test. Clinical Neuropsychologist, 3, 162–168.10.1080/13854048908403289CrossRefGoogle Scholar
Smith, G.E., Malec, J.F., & Ivnik, R.J. (1992). Validity of the construct of nonverbal memory: A factor-analytic study in a normal elderly sample. Journal of Clinical and Experimental Neuropsychology, 14, 211–221.10.1080/01688639208402824CrossRefGoogle Scholar
Smith, G.E., Ivnik, R.J., Malec, J.F., Petersen, R.C., Kokmen, E., & Tangalos, E.G. (1994). Mayo cognitive factor scales: Derivation of a short battery and norms for factor scores. Neuropsychology, 8, 194–202.10.1037/0894-4105.8.2.194CrossRefGoogle Scholar
Squire, L.R. (1987). Memory and brain. New York: Oxford University Press.Google Scholar
Tabachnik, B. & Fidell, L. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins.Google Scholar
Ward, L.C., Ryan, J.J., & Axelrod, B.N. (2000). Confirmatory factor analyses of the WAIS–III standardization data. Psychological Assessment, 12, 341–345.10.1037/1040-3590.12.3.341CrossRefGoogle ScholarPubMed
Wechsler, D. (1945). A standardized memory scale for clinical use. Journal of Psychology, 19, 87–95.10.1080/00223980.1945.9917223CrossRefGoogle Scholar
Wechsler, D. (1987). Wechsler Memory Scale–Revised. San Antonio, Texas: The Psychological Corporation.Google Scholar
Wechsler, D. (1997). WAIS–III and WMS–III: Technical Manual. San Antonio, Texas: The Psychological Corporation.Google Scholar
Wiens, A.N., Tindall, A.G., & Crossen, J.R. (1994). California Verbal Learning Test: A normative data study. Clinical Neuropsychologist, 8, 75–90.10.1080/13854049408401545CrossRefGoogle Scholar