linkage INDEX
355
Index
A
Accountability with assessments for classroom learning, 283-287 versus instructional guidance, 223-224 ACT-R research group, 185 Adventures of Jasper Woodbury video series, 275 Alternative assessment practices, 29-30, 242248 AP Studio Art, 246-247 DIAGNOSER, 96, 247-248 Maryland State Performance Assessment Program, 245-246 National Assessment of Educational Progress (NAEP), 244-245 American Institutes for Research, 208 AmericaQuest, 267 Analysis of errors, 207 ethnographic, 101 microgenetic, 100-101 of protocols, 99-100, 207 of reasons, 207 AP Studio Art, 246-247 Aptitude, assessment of, 37 Aristotelian perspective, 206 Arithmetic keymath diagnostic test, 139 models of learning, 94-95 Assessment, 15-54. See also Alternative assessment practices; Classroom formative assessment; Educational assessment; Modern assessment practices; Purposes of assessment as an instrument of reform, 24-25 analyzing existing, 12, 303-304 of aptitude, 37 balanced system, 253-257 basing on contemporary foundations, 3032 contributions of measurement and statistical modeling to, 5-6, 110-172 of control-of-variables strategy, 216-217 curriculum-embedded, 13, 243 defined, 20 developmental, 136-137, 190-192, 250 fairness in, 214-218, 240-241 future of, 292-294 implications of brain research for, 107-109 implications of cognition for, 71-72 implications of expertise for, 90-92 implications of learning models for, 96-97 implications of observation methods for, 101-102 implications of subject-matter expertise for, 79 integrating models of cognition and learning with, 92-97 limitations of current, 14, 26-29, 310-312 linked with curriculum and instruction, 51-53
356
INDEX
matrix sampling, 13, 307 moderation, 117 potential future role of Bayes nets in, 164-165 in practice, 7-9, 220-259 precision and imprecision in, 42 publicizing its importance in improving learning, 14, 312-313 quality of feedback in, 234-236 and reasoning from evidence, 2-3, 42-43 reporting results of, 212-214 rethinking, 17-35 scientific foundations of, 55-172 static nature of current, 27-28 summative, 38 tasks, 116 using to assist learning, 7-8, 37-38 Assessment design, 173-288 enhancing overall process of, 270-271 funding research into improved, 11-12, 299-301 implications of new foundations for, 6-7, 176-219 inevitability of trade-offs in, 222-223 task-centered versus construct-centered approaches, 194 Assessment instruments. See also Large-scale assessment developers of, focusing on cognition, observation, and interpretation, 13, 305-306 task sets and assembly of, 200-202 Assessment systems, 252-257. See also BEAR assessment system balance between classroom and largescale assessment, 252-253 Assessment triangle, 19, 44-51, 263-271, 282, 296 cognition, 44-47 cognition-interpretation linkage, 51 cognition-observation linkage, 51, 263269 interpretation, 48-49 observation, 47-48 observation-interpretation linkage, 51, 269-270 relationships among the three vertices of, 49, 51 Associationist perspective. See Behaviorist perspective Australias Developmental Assessment program, 190-192
B
Balance-scale problems, solving, 49-50 Balanced assessment systems, 253-257 approximations of, 257 between classroom and large-scale, 252253 coherence of, 255-256 comprehensiveness of, 253-255 continuity of, 256-257 Base rate probabilities, 161 of subprocedure profile, 161 Bayes nets, 154-165 mixed-number subtraction, 156-164 potential future role in assessment, 164165 Bayes theorem, 155 BEAR. See Berkeley Evaluation and Assessment Research Center BEAR assessment system, 115-117 sample progress map from, 119 sample scoring guide for, 118 Behaviorist perspective, on knowing and learning, 61-62 Beliefs. See Student beliefs Berkeley Evaluation and Assessment Research (BEAR) Center, 115 Blueprints, 116 Brain research cognition and, 104-109 cognitive architecture and, 68-69 into enriched environments and brain development, 105-107 into hemispheric specialization, 104-105 implications for assessment, 107-109 Bridging research and practice, 294-296
C
CGI. See Cognitively Guided Instruction Change, models of, 128-134, 165-168 Changing expectations for learning, 21-25 higher standards and high-stakes tests, 23-25 societal, economic, and technological changes, 22-23 Chess experts, meaningful units as encoded by, 74-75 Children assessing problem-solving rules of, 46-47
INDEX
357
coming to understand the whole number system, 180-181 naive conceptions of, 83 Piagetian stages of development, 151 strategies for simple addition, 86 Classical test theory (CTT), 120-121 Classification, social context of, 88 Classroom assessment, 8-9, 225-241 balancing with large-scale assessment, 252-253 cognitively based approaches to, 230-233 fairness in, 240-241 learners role in, 236-240 new forms of, 12, 301-303 quality of feedback, 234-236 teachers role in, 234 transforming, 226-228 Classroom Connect, 267 Classroom formative assessment, 38 facilitating, 272-274 mandates and resources placing increasing emphasis on, 14, 310-312 CLP. See Computer as Learning Partner Cognition, 65-72 analyzing in existing assessments, 12, 303-304 and brain science, 104-109 developers of assessment instruments focusing on, 13, 305-306 implications for assessment, 71-72 importance of a model of, 6, 178-192, 229-230 as part of the assessment triangle, 44-47 Cognition-observation linkage, 51 complex problem solving, 265-269 concept organization, 265 enhancing, 263-269 theory-based item generation, 263-265 Cognitive architecture, 65-69 and brain research, 68-69 long-term memory, 67-68 working memory, 65-67 Cognitive coherence among curriculum, instruction, and assessment, 271-283 of balanced assessment, 255-256 Cognitive complexity, of science tasks, 210211 Cognitive elements in existing measurement models enhancement through diagnostics, 137, 142-147
incorporation of, 134-147 progress maps, 136-142 Cognitive models of learning of arithmetic, 94-95 for assessing childrens problem-solving rules, 46-47 and debugging of computer programs, 95-96 implications for assessment, 96-97 integrating with instruction and assessment, 92-97 and intelligent tutoring, 93 Cognitive perspective, on knowing and learning, 62-63 Cognitive sciences defined, 20 making advances available to educators, 11, 299 Cognitive structures adding to measurement models, 147-152 psychometric modeling of, 152-165 Cognitively Guided Instruction (CGI), 95, 234 and assessment, 230-231 Collaboration, recommendation for multidisciplinary, 11 Committee on Learning Research and Educational Practice, 294 Committee on the Foundations of Assessment, 1, 17, 291 Comparable validity, 214 Competencies. See Student competencies Complex problem solving, 265-269 Complex solution strategies, analysis of, 270 Comprehensiveness, of balanced assessment system, 253-255 Computational modeling and simulation, 99 Computer as Learning Partner (CLP), 278-279 Computer programs, debugging, 95-96 Concept organization, 265 Conceptual frameworks, 271 Conceptual scheme, to guide thinking and discussion, 11 Concurrent verbal protocols, 99 Conditional independence, 114 Conditional inference methods of, 218 versus unconditional, 215-216 Conditional probabilities, 159 Connecticut Common Core of Learning Assessment Project, 210
358
INDEX
Connectionist networks, model of long-term memory, 67-68 ConQuest software, 134 Construct-centered approach, to assessment design, 194 Construct variables, 112 Contemporary assessment. See Modern assessment practices Content-process space, 208 Continuity, of balanced assessment system, 256-257 Continuous latent variables, modeling of change in, 133-134 Control-of-variables strategy, assessment of, 216-217 Coordinated systems, of multiple assessments, 252 CoVis (Learning Through Collaborative Visualization) project, 279 Criterion-referenced testing, 214 Crystallized intelligence, 66 CTT. See Classical test theory Cultural norms, impact on expertise, 90 Curriculum assessment linked with, 51-53 developers of, creating tools to facilitate modern assessment practices, 13, 306307 formative assessment in, 228-229 Curriculum and Evaluation Standards for School Mathematics, 209 Curriculum-embedded assessment, 13, 243
Developmental continuua. See Progress maps DIAGNOSER, 96, 203, 247-248, 273 Diagnostic arithmetic test, keymath, 139 Diagnostic indices IEY diagnostic results, 146 incorporated into measurement models, 137, 142-147 KIDMAP, 144-145 DIF. See Differential item functioning Differential item functioning (DIF), 148, 215 Differential perspective, on knowing and learning, 60-61 DISC. See Dental Interactive Simulation Corporation Discrete latent variables, modeling of change in, 134 Discussion. See also Multidisciplinary discourse communities conceptual scheme and language to guide, 11 Distributed learning, 285 Domain-general knowledge, and problemsolving processes, 69-70 Dyslexia, neural bases of, 108-109
E
Economic change, and changing expectations for learning, 22-23 Educational assessment defined, 20 opportunities for advancing, 9-10, 260288 providing instruction in for teachers, 14, 309-310 purposes and contexts of use, 222-225 Educational curricula, developers of, creating tools to facilitate modern assessment practices, 13, 306-307 Educational decision making, studying how new forms of assessment affect, 12, 301-303 Educational reform assessment as an instrument of, 24-25 facilitating, 34 Educational Testing Service, 271 Educators. See Teacher education Environments. See also Learning environments enriched, and brain development, 105-107
D
Debugging of computer programs, 95-96 Dental Interactive Simulation Corporation (DISC), 266-267, 271 Department of Education, 299 Design. See Assessment design Designing and Conducting Investigations, sample performance map for, 125 Developers of assessment instruments. See also Large-scale assessment focusing on cognition, observation, and interpretation, 13, 305-306 Developers of educational curricula, creating tools to facilitate modern assessment practices, 13, 306-307 Developmental Assessment program (Australia), 136-137, 190-192, 250
INDEX
359
Equity issues, in linkage of assessments, 286287 Estimates, assessment results as, 42 Ethnographic analysis, 101 Evaluation. See Assessment Evidence, reasoning from, 2-3, 36-54, 112-117 Exemplars, 116 Expanding views of knowing and learning, 59-65 the behaviorist perspective, 61-62 the cognitive perspective, 62-63 the differential perspective, 60-61 points of convergence among, 64-65 the situative perspective, 63-64 Expectations for learning changing, 21-25, 33 in large-scale assessment, 249-250 Expertise, 79-92. See also Novices and experts impact of cultural norms and student beliefs, 90 implications for assessment, 90-92 multiple paths of learning, 81-83 practice and feedback, 84-87 predisposition to learn, 80 role of prior knowledge, 83-84 role of social context, 88-89 transfer of knowledge, 87-88 Eye-movement tracking, 96
Fluid intelligence, 66 fMRI. See Functional magnetic resonance imaging Formal measurement models as a form of reasoning from evidence, 112-117 reasoning principles and, 113-115 Formative assessment. See Classroom formative assessment Foundations, defined, 20 Fraction items, skill requirements for, 158 Functional magnetic resonance imaging (fMRI), 68, 108 Funding, needed for research into improved assessment design, 11-12, 299-301 Future of assessment, 292-294 bridging research and practice, 294-297
G
Generalizability theory (G-theory), 121-122 multivariate, 128 with raters and item type, 122 GenScope, 276, 278, 286-287 Goals, using large-scale assessment to signal, 248-249 GradeMap software, 119, 143 Growth, models of, 128-134 Guessing probability, 153
F
Facet clusters, 187 Facets-based instruction and learning, 186189, 202-206, 234, 273 separating medium effects from gravitational effects, 188-189 Facets DIAGNOSER, 96, 247-248 Facets of measurement, 121 Facets of student thinking, 187 Fairness in assessment, 7, 39, 214-218, 240241 conditional versus unconditional inferences, 215-216 Federal agencies, establishing multidisciplinary discourse communities, 12-13, 304-305 Feedback in assessment, 4, 84-87 and expertise, 84-87 large-scale, 249-250 quality of, 234-236
H
Hemispheric specialization, brain research into, 104-105 Hierarchical factor analysis, 149 Hierarchical linear modeling (HLM), 133 Hierarchical measurement models, 148-152 combining classes and continua, 151-152 High Stakes: Testing for Tracking, Promotion, and Graduation, 39, 253 High-stakes tests, and changing expectations for learning, 23-25 HLM. See Hierarchical linear modeling How Far Does Light Go? project, 280-281 How People Learn, 59 HYDRIVE intelligent tutoring system, 164165 inference networks for, 169-172
360
INDEX
I
IEY. See Issues, evidence and you curriculum IEY curriculum, target performance map for, 146 IMMEX (Interactive Multimedia Exercises) program, 270, 273-274 Inclined-plane schemas, novices and experts network representations of, 78-79 Individual achievement, using assessment to determine, 38-39 Inference conditional versus unconditional, 215-216 methods of, 97-102, 218 targets of, 45 Inference networks for the HYDRIVE student model, 170 structure of, 160 Information. See Publicity Information technology, 22 opportunities for advancing educational assessment, 9-10, 260-288 Instruction assessment linked with, 51-53 formative assessment in, 228-229 integrating models of cognition and learning with, 92-97 in learning assessment, providing for teachers, 14, 309-310 Instructional guidance, accountability versus, 223-224 Intelligent tutoring systems, 68, 231-233 application of Bayes nets in, 169-172 cognitive models of learning and, 93 effects on mathematics learning, 232-233 Interpretation analyzing in existing assessments, 12, 303-304 developers of assessment instruments focusing on, 13, 305-306 as part of the assessment triangle, 48-49 IRM. See Item response modeling Issues, evidence and you (IEY) curriculum, progress variables from, 116 Item generation, theory-based, 263-265 Item parameters, 113, 119, 154 Item response modeling (IRM), 123-126, 134 multidimensional, 128-129 with raters and item type, 126
J
James S. McDonnell Foundation, 304
K
KIDMAP, 142, 144-145 Knowing, expanding views of the nature of, 59-65 Knowledge domain-general, and problem-solving processes, 69-70 role of prior, 83-84 synthesis of existing, 299 transfer of, 87-88 Knowledge base expanding, 299-303 initial steps for building, 303-305 Knowledge-in-pieces perspective, versus theoretical perspective, 203-206 Knowledge Integration Environment (KIE), 278-281 Knowledge organization expert-novice differences in, 72-77 and schemas, 70-71 Knowledge tracing, 186
L
Language, guide to thinking and discussion, 11 Large-scale assessment, 8-9, 241-251 and advances in cognition and measurement, 241-242 alternative approaches to, 242-248 AP Studio Art, 246-247 balancing with classroom assessment, 252-253 feedback and expectations for learning, 249-250 increasing spending on, 24 Maryland State Performance System, 245246 National Assessment of Educational Progress, 244-245 New Standards Project, 250-251 sampling wider range of student competencies, 13, 307-308 using to signal worthy goals, 248-249
INDEX
361
Large-scale contexts, making new forms of assessment practical in, 12, 301-303 Latent class models, 126-127 ordered, 127 Latent semantic analysis (LSA), 269, 274 Latent variables continuous, modeling of change in, 133134 discrete, modeling of change in, 134 stage-sequential dynamic, 135 LCAG software, 134 Learners role, in classroom assessment, 236240 Learning. See also Expectations for learning; Student competencies advances in the science of, 3-5, 58-109 changing expectations for, 21-25 cognitive model of, 46-47 distinguished from development, 80 expanding views of the nature of, 59-65 impact of prior theories of, 25-30 impact of reflective inquiry on, 238-239 importance of a model of, 6-7, 178-192, 229-230 linkage of assessments for, 283-287 models of, integrating with instruction and assessment, 92-97 multiple paths of, 81-83 predisposition to, 80 principles for structuring, 87 problem-based, 276 publicizing the importance of assessment in improving, 14, 312-313 social context of, 89 using assessment to assist, 7-8, 37-38 Learning environments, technologyenhanced, 274-283 Learning Through Collaborative Visualization (CoVis) project, 279 Limitations of current assessment, 26-29 making policy makers aware of, 14, 310312 Linear models, families of, 131-132 Link tests, 116 Linkage of assessments for classroom learning and accountability, 283-287 equity issues in, 286-287 policy issues in, 285-286 pragmatic issues in, 286 privacy issues in, 287
LISP tutor, 164 Logistic regression, 148 LOGO language, 95 Long-term memory, 2, 67-68 production systems model of, 67-68 LSA. See Latent semantic analysis
M
Magic Number Seven argument, 66 Mandates, increasing emphasis on classroom formative assessment, 14, 310-312 Maryland State Performance Assessment Program (MSPAP), 245-246 MashpeeQuest performance task, 267-268 Mathematics effects of an intelligent tutoring system on learning, 232-233 student beliefs about the nature of, 91 Mathematics Test Creation Assistant, 263-264 Matrix sampling, 13, 248 Meaningful units, as encoded by chess experts, 74-75 Measurement models, 5-6, 112 adding cognitive structure to, 147-152 addition of new parameters, 148 formal, 112-117 hierarchization of, 148-152 incorporation of cognitive elements in existing, 134-147 standard, 117-127 Measurement science contributions to assessment, 5-6, 110-172 facets of, 121 impact of prior theories of, 25-30 making advances from available to educators, 11, 299 Media, publicizing the importance of assessment in improving learning, 14, 312-313 Mediated activity, 63 Memory contents of, 69-71 domain-general knowledge and problemsolving processes, 69-70 long-term, 67-68 schemas and the organization of knowledge, 70-71 working, 65-67 Metacognitive skills, importance of, 4, 78-79, 281
362
INDEX
Microgenetic analysis, 100-101 Middle School Mathematics Through Applications Project, 187-189 Mixed-number subtraction, Bayes net analysis of, 156-164 Mixture model approach, 151 Model tracing, 186 Model tracing, 167 Modeling computational, 99 psychometric, of cognitive structures, 152-165 statistical, 5-6, 110-172 of strategy changes, 165-168 Models of change and growth, 128-134 modeling of change in continuous latent variables, 133-134 modeling of change in discrete latent variables, 134 true-score modeling of change, 130-133 Models of cognition and learning, 185-192 Australias Developmental Assessment program, 190-192 Facets-based instruction and learning, 186-189, 202-206 Middle School Math Through Applications Project, 187-189 PAT algebra tutor, 185-186 Modern assessment practices, 30-32 developers of educational curricula creating tools to facilitate, 13, 306-307 publicizing, 14, 312-313 Monotonic development, cumulative, in a stage-sequential dynamic latent variable, 135 MOOSE Crossing, 279 Mozart effect, 105-107 M2RMCL, unified model and, 152-154 Multiattribute models, 128 Multidimensional item response model, 129 Multidisciplinary discourse communities, establishing, 11-13, 304-305 Multiple assessments coordinated systems of, 252 developing new systems of, 14, 310-312 Multiple-choice questions, to test for theoretical versus knowledge-inpieces perspective, 204-205 Multiple paths, of learning, 81-83 Multivariate G-theory, 128
N
NAEP. See National Assessment of Educational Progress National Academy of Education, 298 National Assessment of Educational Progress (NAEP), 40, 90, 124, 200, 224, 244-245 National Board of Medical Examiners, 270 National Council of Teachers of Mathematics, 23, 275 National Education Research Policies and Priorities Board, 298 National Institute of Child Health and Human Development, 299 National Institute of Mental Health, 108 National Institute on Aging, 108 National Research Council (NRC), 17, 23, 39, 59 Committee on Learning Research and Educational Practice, 294 National Science Foundation (NSF), 1, 17, 291, 299, 304 Neural bases of dyslexia, 108-109 New Standards Project, 23, 250-251 Newell-Dennett framework, 153 Newtonian perspective, 203 Norm-referenced results, 213 Novices and experts differences in, 4, 72-77 network representations of inclinedplane schemas, 78-79 NRC. See National Research Council NSF. See National Science Foundation Number Knowledge Test, 196-199
O
Object models, 271 Observation analysis of protocols for, 99-100 analyzing in existing assessments, 12, 303-304 and computational modeling and simulation, 99 developers of assessment instruments focusing on, 13, 305-306 ethnographic analysis, 101 implications for assessment, 101-102 methods of, 97-102 microgenetic analysis, 100-101 as part of the assessment triangle, 47-48
INDEX
363
of problem-solving rules in children, methods for, 49 and reaction-time studies, 98-99 of student performance, interpreting, 50 Observation-interpretation linkage, 51 analysis of complex solution strategies, 270 enhancing, 269-270 text analysis and scoring, 269 Observation models, 112 Office of Education Research and Improvement, National Education Research Policies and Priorities Board, 298 OLEA tutor, 164
P
PANMARK software, 134 Parallel distributed processing (PDP) systems, model of long-term memory, 68 Pasteurs Quadrant, 298 PAT algebra tutor, 185-186, 232 PDP. See Parallel distributed processing systems Performance maps for Designing and Conducting Investigations, 125 for the IEY curriculum, 146 PET. See Positron emission tomography Physics examination, components of A-level, 254 Physics problems, sorting of, 76 Piagetian stages, of child development, 151 Plan recognition, 167 Policy issues, in linkage of assessments, 285286 Policy makers recognizing limitations of current assessments, 14, 310-312 recommendations for, 305-312 Positron emission tomography (PET), 68, 108 Power law of practice, 85 Practice, in expertise, 84-87 Pragmatic issues, in linkage of assessments, 286 Precision and imprecision, in assessment, 42 Predictive tests, 18 Predisposition to learn, 80 Prior knowledge, role in expertise, 83-84
Privacy issues, in linkage of assessments, 287 Private-sector organizations, establishing multidisciplinary discourse communities, 12-13, 304-305 Probabilities base rate, 161 updated, 162 Problem-based learning, 276 Problem solving assessing in children, 46-47 complex, 265-269 domain-general knowledge, 69-70 methods for observing in children, 49 weak methods versus strong, 69-70 Production systems, 99 model of long-term memory, 67-68 Professional development programs, providing instruction in learning assessment, 14, 309-310 Profile strands. See Progress maps Progress maps, 117, 137, 190 BEAR assessment system, 119 cognitive elements in existing measurement models, 136-142 for counting and ordering, 191-193 keymath diagnostic arithmetic test, 139 of national writing achievement, 140-142 reporting individual achievement in spelling, 138 Progress variables, 115-117 from the issues, evidence and you curriculum, 116 Progressions of developing competence. See Progress maps Protocols analyzing, 207 concurrent verbal, 99 for observation and inference, 99-100 Psychological Tests and Personnel Decisions, 222 Psychometric modeling of cognitive structures, 152-165 Bayes nets, 154-165 unified model and M2RMCL, 152-154 Public opinion, recommendations regarding, 312-313 Publicity, on the importance of assessment in improving learning, 14...