With broad training in both quantitative psychology and statistics and years of experience in educational measurement, my research interests span multiple disciplines, crossing theoretical advances with practical applications.I am primarily interested in methodological issues in psychometrics, with latent variable modeling, Bayesian statistics, and mixed-effects models being the main focus of my current quantitative research program.
Jonathan Templin
University of Iowa
Research
Current Research Funding
-
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2119912&HistoricalAwards=false
This doctoral dissertation research project will investigate the use of multidimensional tests in computerized adaptive testing. Multidimensional tests are used across many disciplines to make critical decisions on issues such as psychological health, educational ability, and vocational suitability. Some of these tests are administered as Computerized Adaptive Tests (CATs), a mode of testing that adapts the test in real time based on the examinee’s item responses. Generally, CATs are designed to work with item responses that are scored as 0 or 1, based on whether the examinee has chosen the correct response option or an incorrect response option. However, this dichotomous scoring scheme disregards information contained in incorrect response options, which can be modeled as nominal response data. Research on single-subject CATs has demonstrated that when its components are designed to work with nominal response data, the tests were shorter and more accurate than those designed for dichotomous responses. This project will examine the value of multidimensional adaptive tests that utilize item response options as nominal response data. Psychometric software will be developed and made publicly available. As a Doctoral Dissertation Research Improvement award, support is provided to enable a promising student to establish a strong, independent research career.
This research project will advance the use of multidimensional tests in computerized adaptive testing by incorporating item response options into the modeling, item selection, and estimation processes that constitute the adaptive test. The project will compare the accuracy, efficiency, and security of this approach to existing procedures. Adaptive test components and results will be simulated for Multidimensional Adaptive Tests (MATs) and Cognitive Diagnostic-Computerized Adaptive Tests (CD-CATs). Both MATs and CD-CATs can be used for either summative or formative assessment purposes. However, they differ in their theoretical assumptions and thus must be investigated separately. To compare the incorporation of item response options in multidimensional adaptive testing to current dichotomous scoring procedures, item banks and item selection methods will be manipulated. Common item selection methods for dichotomous response data will be generalized for use with nominal response data to ensure comparability.
This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.
-
The purpose of the project is to develop a computer adaptive, diagnostic assessment of teachable morphological skills for students in grades 3 to 5. The development of morphological skills is essential to students’ literacy growth. This is because knowledge of morphemes (for example, root words like nation, prefixes like inter, and suffixes like al) supports students’ reading achievement by influencing their ability to decode and access the meaning of multisyllabic words which then supports their reading comprehension.
-
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1813760&HistoricalAwards=false
One of the great challenges related to teachers and their knowledge is measuring their learning in ways that are both formative and meaningful in relation to their likely impact on students. This challenge persists despite efforts to define the knowledge teachers should have and despite previous innovative efforts to create good measures. This project tackles the challenge by specifically aiming to measure the kinds of knowledge developed in professional development (PD) programs that has been shown to matter for teachers’ classroom practices and their students’ learning. The project aims to develop an assessment that identifies patterns in the teachers’ learning in a way that helps drive subsequent professional development. The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.
The overall goal of this project is to pursue a potentially transformative approach to the assessment of teacher proportional knowledge by developing a measure that is well aligned with the content and skills taught in various PD programs. This instrument will be based on a new approach that builds on emerging psychometric models. Specifically, diagnostic classification models (DCMs) will be utilized to diagnose teachers’ learning during a PD program as well as employed to identify the progression in teachers’ learning. Statistical topic models (STMs) will be used to look for patterns of understanding that emerge from open-ended responses and provide natural-language insight into teachers’ reasoning. A final version of the assessment will be constructed for a national sample based on the results from the predictive validity stage, and this version will be tested with teachers who participate in various types of PD programs targeting proportional reasoning. This project has broad implications for the creation of assessments and for teacher education. It will provide insights about whether there is a clear learning progression for teachers. While much work has been done with students’ learning progression, much less is known about how teachers learn. Another implication is that the STM approach allows machine scoring of natural language in a way that highlights strengths and weaknesses in reasoning rather than simply returning a score. For formative use, this is information that is more helpful as it highlights areas for further instruction. A third implication is that DCMs will allow to assess teacher knowledge at a finer-grained understanding than is typically available, thus allowing for careful refinement of PD as well as a tool for showing overall growth in PD. A fourth implication is that a more systematic approach will be followed to capture the kinds of knowledge teachers need. Assessments developed using DCMs and STMs have the potential to serve as models for developing further instruments in other STEM content areas. Such assessments have the potential to not only help identify successful PD programs, but also to provide PD providers with rich data from which they can make instructional decisions.
This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.
-
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1544481&HistoricalAwards=false
The overall goal of this project will be to create and validate a “learning map” (Stat-LM) for the content of undergraduate introductory statistics. This learning map will be a graphical representation of statistics concepts with connections among the concepts suggesting effective learning sequences. Use of the learning map will improve undergraduate learning by providing diagnostic information to instructors about students in their statistics courses, informing professional development for undergraduate statistics instructors, and modeling how critical prerequisites taught in high school connect to postsecondary learning expectations. In the first phase of the project, researchers will collaborate with statistics instructors from five institutions to create a learning map representing the broadest possible content set for introductory statistics material and the typical learning patterns of students in statistics courses taught in high school, community college, and university settings. In the second phase of the project, the research team will study the accuracy of the learning map using data collected from the Comprehensive Assessment of Outcomes in a First Statistics Course (CAOS), a confirmed valid and reliable assessment of student learning in statistics. This analysis will provide evidence of the learning map”s accuracy as a model of how students develop knowledge of introductory statistics.
The project will build on research conducted at the University of Kansas in the development and application of learning maps as tools to support instruction and assessment by addressing two specific aims. First, the Stat-LM will be developed by expert statistics educators, educational psychologists, and statistics instructors from five varied types of institutions to model the broadest possible set of topics taught at the university level, thereby assuring its relevance for high school and undergraduate courses. Expert reviewers will provide feedback on the nodes and connections in the map in order to refine the content validity of the Stat-LM as a model of student learning. Second, the analysis using quantitative data collected from the CAOS will consist of an alignment study of the test items on the CAOS to the Stat-LM, identifying the nodes in the Stat-LM that students must have mastered to answer each item correctly. Along with data from the CAOS found to be aligned to the map, the map specification will be analyzed with diagnostic classification (also called Bayesian network methods; e.g., Rupp, Templin, & Henson, 2010). Based on results from these analyses, the researchers will evaluate the Stat-LM with respect to the ordered connections between nodes and the validity of nodes themselves as latent constructs. The Stat-LM will become a “living document”, subject to refinements throughout the proposed project and thereafter. As the goal of this project is to produce a tool for aiding statistics instruction, versions of the Stat-LM will be shared broadly for use within varied instructional settings.
Preprints
Observed Score Reliability Indices in Diagnostic Classification Models
Preprints
Publication year: 2020
Modification Indices for Diagnostic Classification Models
Preprints
Publication year: 2020
Awards
-
2017Robert L. Linn Memorial Lecture AwardUniversity of Colorado-Boulder and University of California-Los Angeles
-
2015Outstanding Contribution to Research in Cognition and AssessmentAmerican Educational Research Association, Cognition and Assessment SIG
-
2012Significant Contribution to Educational Measurement and Research MethodologyAmerican Educational Research Association, Division D