Mentale belasting van contextopgaven in rekentoetsen

Cognitive load analytics of word problems

The trouble with PISA Math & similar test items is that the contexts are intended to be NEW, UNFAMILIAR ones: no mental representation yet! tweet

• The thing all mental representations have in common is that they make it possible to process large amounts of information quickly, despite the limitations of short-term memory. Indeed, one could define a mental representation as a conceptual structure designed to sidestep the usual restrictions that short-term memory places on mental processing.

  Ericsson & Pool, 2016 ‘Peak. Secrets from the new science of experience’ HMH (p. 61)

The literature on this topic is mainly in the English language. Therefore this webpage should be informative to English readers too. I will write mostly in Dutch, though; trying to write in English takes too much cognitive load ;-).

other pages on my website that contain relevant materials:

• Testing for ‘math in everyday situations’ http://benwilbrink.nl/projecten/rekenen_in_alledaagse_situaties.htm
• Situationisme, cognitieve psychologie, en rekentoets. Hoorzitting onderwijscommissie Tweede Kamer 4 december 2013. http://benwilbrink.nl/projecten/hoorzitting_4-12-2013_situationisme.htm
• Intellectuele capaciteiten in contexten: rekenopgaven. http://benwilbrink.nl/projecten/capaciteiten_in_contexten.htm
• Ben Wilbrink, Joost Hulshof en Henk Pfaltzgraff (2012). De rekentoetsen zijn niet valide. Dat wordt nog wat, met die rekentoetsen! Examens, Tijdschrift voor de Toetspraktijk. 9 #3, 26-31. paywalled voor € 1,95, en er is een alternatief: concepten
• Rekenproject: Woordproblemen http://www.benwilbrink.nl/projecten/woordproblemen.htm
• Word problems (literature) http://www.benwilbrink.nl/projecten/wordproblems.htm
• Rekenproject: Probleemoplossen http://www.benwilbrink.nl/projecten/probleemoplossen.htm
• Rekenproject: Contexten http://www.benwilbrink.nl/projecten/contexten.htm
• Rekenproject: hypothese 4. Contextopgaven testen intelligentie http://www.benwilbrink.nl/projecten/hypothese4.htm
• Rekenproject: Model opstellen http://www.benwilbrink.nl/projecten/model.htm
• Het belang van tekstbegrip in eindexamens: sterk overtrokken? http://www.benwilbrink.nl/projecten/tekstbegrip.htm
• Toetsvragen ontwerpen; Handreiking bij het maken van toetsvragen over de leerstof; 7. Problemen stellen http://www.benwilbrink.nl/projecten/toetsvragen.7.htm
• Literatuur over cognitie http://www.benwilbrink.nl/literature/cognitie.htm
• Zie ook, als voorbeeld van wat rekenen is: Jan van de Craats & Rob Bosch (2009). Basisboek rekenen. Pearson Education Benelux. http://staff.science.uva.nl/~craats/BR2HP.pdf

political context

Revisionist arithmetics education has pretty much destroyed the capability of Dutch youth to do any serious arithmetics on paper (PPON 2004, see for example Hickendorff’s 2011 dissertation) (‘Serious’? PPON exercises are rather simple; yet a fact that might have contributed to subjects trying to do them mentally instead of on paper). In an attempt to remedy this disastrous situation, the Dutch government is in the process of adding an arithmetics test to the exit examinations in secondary education and vocational education (mbo). For examples of the testing materials, see the website of CITO here. Even though the word problems have been written in Dutch, it should be immediately clear that the word problems are rather problematic in terms of cognitive load. Take the trouble to open one test (‘voorbeeldtoets’), for example the 2013 one for the high tracks in secondary education here.

A governmental commission-Bosker also concluded so (May 2014, the ministers of education have not yet revealed their position on the conclusions of this commission. A parliamentary debate is scheduled for the 18th of June). The main characteristic of these arithmetics tests is that they pretend to test the capability to use arithmetics in situations of daily life. This particular ideology is known as situationism, now part and parcel of constructivism even though both ideologies are somewhat antagonistic to each other. On the problematic sides of this educational ideology, see Anderson, Reder & Simon (1998). In Dutch, a short article on situationism Wilbrink, 2014; an annotated version has much references to the literature.

Of some importance is that PISA-math items are of the same type, are embeded in the same ideology (voiced by Andreas Schleicher, OECD head of PISA operations), and have an ancestry that goes back to Dutch revisionism in arithmatics (Hans Freudenthal and his co-workers). Cito analyzed the similarities and differences between items of PISA-Math and items on the arithmetics tests mentioned here; the report has not yet been made public (the results will be presented and discussed on Onderwijsresearchdagen ORD in Groningen, June 12, 2014, 9:00, bij Cito psychometrists)), the department of education under the direction of Sander Dekker is withholding it.

There is a kind of ‘Common Core’ controversy in the Netherlands concerning these arithmetics tests. To the outsider the debate looks like a conflict of opinions. To make any progress in this debate between protagonists of pseudo-scientific math education and protagonists of (cognitive) science as a foundation of math education and valid math testing, the next step should be to make a hard science analysis of cognitive load characteristics of revisionist education word problems. I will try to make a beginning in this webpage.

The problem

This paper is not about John Sweller’s cognitive load theory. It will be used, of course, whenever appropriate. In a way, this paper will be about the cognitive processes triggered by questioning, and how these processes impact on answering.

Keep in mind that much of the literature is about differences within subjects (experimental cognitive psychology), not between subjects (differential psychology, testing psychology). Ultimately, though, we will want to be able to predict differences between students on the basis of adequate theory of differences within subjects.

A key review publication to start with is Leighton_2013.

Mentioned by Leighton: a linguistic approach to question complexity by Herbert H. Clark (1969). Linguistic processes in deductive reasoning. Psychological Review.

This research is not simply about difficult words or the problems second language learners might have. Some items have made it into the canon of what to do or avoid in designing test items, such as the use of denials.

• The most important demonstration here has been that the principal difficulties inherent in many reasoning problems are not due to cognitive processes specific to these problems, but to the very language in which the problems are stated.

  From the Conclusions

Cognitive models: Halford, Wilson & Phillips, 1998. Yes, this is the way the word problems should be researched psychologically. The pdf also contains open peer commentaries (a.o. by Anderson, Lebiere, Lovett & Reder: ACT-R: A higher level account of processing capacity), and the reply by the authors.

A special topic is that of the (word) problem consisting of a combination of subproblems, each one necessary for the correct answer. That is: the probability of obtaining the correct answer is the product of the probabilities of obtaining the necessary information from the subproblems. See Wilbrink, 1998.

It is an important issue because the human assessor is not inclined to multiply the probabilities of obtaining the correct information from each of the subproblems, resulting in his underestimating the difficulty of the word problem.

Literature

Myrto-Foteini Ma Vilidi, Vincent Hoogerheid & Fred Paas (2014). A quick and easy strategy to reduce test anxiety and enhance test performance. Applied Cognitive Psychology, 28, 720-726. abstract

Robert C. Daniel & Susan E. Embretson (2010). Designing cognitive complexity in mathematical problem-solving items. Applied Psychological Measurement, 34, 348-364. abstract

Ik heb dit doorgenomen, en word er toch wel een beetje bedroefd van. Oé, ik begrijp die heel andere wereld van constructie van landelijke toetsen wel een beetje, maar wat ik niet begrijp is de totale desinteresse in de inhoud van die rekenopgaven en wat er cognitief voor nodig is om erop te kunnen scoren. Kokervisie. Dat is dan nog tot daaraantoe, maar kokervisie betekent ook dat er geen checks and balances zijn waar het om de inhoudelijke kwaliteit van toetsopgaven gaat (anders dan wat door de statistische wringer heen is gekomen), noch om validiteit van het gebruik van de toetsresultaten. Ik mis in dit artikel ook maar de kleinste aanwijzing dat de betreffende toetsvragen in naam en oppervlakkig geziene inhoud rekenopgaven zijn, maar dat zij samen in feite een onduidelijke soep van intellectuele capaciteiten testen. En dat laatste is toch echt geen geheim van de beroepsgroep, zie bijvoorbeeld een interview in 2014 met Intelligence editor Doug Detterman.

Uiteindelijk blijft er niet veel meer over dan een onderzoekje naar de bekende weg: opgaven met meer tussenstappen zijn moeilijker. Maar wat ik nu juist wil weten: hoe hangt de moeilijkheid exact af van cognitieve horden en bottlenecks: beschikbaar krijgen van nodige kennis, mogelijke overbelasting van het werkgeheugen, individuele verschillen hierbij en hoe die te verklaren, enzovoort. Niets van dat alles in Daniel & Embretson. Bijvoorbeeld: als voor een opgave zowel A als B en C nodig zijn, hoe waarschijnlijk is het dat A, B en C beschikbaar komen? En is het mogelijk dat het werkgeheugen A, B en C niet tegelijk aan kan?

Susan E. Embretson & Robert C. Daniel (2008). Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychology Science Quarterly, 50, 328-344. pdf

Anderson, J. R., Reder, L. M. & Simon, H. (1998). Radical constructivism and cognitive psychology. In D. Ravitch (Ed.) Brookings papers on education policy 1998 (227-278). Washington, DC: Brookings Institute Press. pdf [also available in JSTOR, registratie vereist om free online te kunnen lezen] A better version/scan in pdf : http://goo.gl/6ULfY4

Marian Hickendorff (2011). Explanatory latent variable modeling of mathematical ability in primary school : crossing the border between psychometrics and psychology. Proefschrift Universiteit Leiden. download

Jacqueline P. Leighton (2013). Learning Sciences, Cognitive Models, and Automatic Item Generation. In Mark J. Gierl & Thomas M. Haladyna (Eds) (2013). Automatic Item Generation. Theory and Practice (121-135). Routledge. [als eBook bij KB] info

This article is a key publication on item writing. Relevant references among others:

• Jacqueline P. Leighton & Robert J. Sternberg (in press). Reasoning and problem solving. In A. Healy & R. Proctor: Handbook of Psychology (631-659) 2nd edition vol. 4. Experimental Psychology. Wiley. info
• Jacqueline P. Leighton & Mark J. Gierl (2011). The learning sciences in educational assessment. Cambridge University Press. [als eBook bij KB te leen, niet in UB Wassweg] info
  • Ch. 5 Cognitive models of task performance for mathematical reasoning. 156-196
  • Ch. 6 Putting it all together: Cognitive models to inform the design and development of large-scale educational assessment. 197-233.
  • Herbert H. Clark (1969). Linguistic processes in deductive reasoning. Psychological Review, 76, 387-404. pdf and cited by 693 Google Scholar May 2014. A key-publication.
    In the past, deductive reasoning has often been studied as if it were an isolated process—even as if it were specific to a certain kind of task, such as the solution of three-term series problems. The processes described in the present paper, on the other hand, are quite general. They are not meant to explain the solution of two- and three-term series problems alone, but to account for certain linguistic processes in understanding statements and answering questions wherever they occur. The most important demonstration here has been that the principal difficulties inherent in many reasoning problems are not due to cognitive processes specific to these problems, but to the very language in which the problems are stated. Linguistic processes like these arise in every situation in which a problem is stated in linguistic terms.

    from the Conclusions

Joanna S. Gorin & Susan E. Embretsen (2013). Using Cognitive Psychology to Generate Items and Predict Item Characteristics. In Mark J. Gierl & Thomas M. Haladyna (Eds) (2013). Automatic Item Generation. Theory and Practice (136-). Routledge. [als eBook bij KB]

• We will attempt to persuade the reader that AIG endeavors are, at their core, analogous to carefully designed cognitive-experimental stimuli design.

  p. 137, http://goo.gl/DsYIVq

Edith Aurora Graf and James H. Fife (2013). Difficulty Modeling and Automatic Generation of Quantitative Items: Recent Advances and Possible Next Steps. In Mark J. Gierl & Thomas M. Haladyna (Eds) (2013). Automatic Item Generation. Theory and Practice (157-179). Routledge. [als eBook bij KB]

The text of the chapter itself turned out to be a disappointment for me. Almost nothing on cognitive modeling. Highly abstract in its verbal exposition. Most of the research mentioned turns out to be rather empiricist, using a black-box model of cognitive operations themselves. The testing industry is interested in constructing items with known difficulty and discrimination parameters. That is not why teachers and students should be interested in item models. It is that old story again: students get tested, often not even having the choice to opt out of testing. Students should have a voice in all of this. Society at large should have a voice. The educational community should have a voice. Not in this Graf and Fife chapter, though.

Nevertheless, a lot of research literature gets mentioned here, I will dig into that list. For example: the Graf papers (ETS) look interesting, but turn out to be grounded in NCTM's reform mathematics (constructivism).

• • James H. Fife, Edith Aurora Graf & Sarah Ohls (2011). Constructed-response mathematics tasks study. ETS Research Report. abstract and pdf download
  • Edith Aurora Graf, Karen R. Harris, Elizabeth Marquez, James H. Fife & Margaret Redman (2009). Cognitively Based Assessment of, for, and as Learning (CBAL) in Mathematics: A Design and First Steps Toward Implementation. ETS Research Report RM-09-07 abstract, no pdf available, request printed copy [did so, got pdf, thanks] Deze onderzoeklijn blijkt uit te gaan van de reformdidactiek van de NCTM; het is dus een beetje de vraag of Graf c.s. wel bezig zijn met rekenen en wiskunde, en misschien meer zijn geïnteresseerd in communiceren over rekenen en wiskunde of dingen die daar op lijken. Ik heb dus ernstige twijfels bij dit werk, maar moet er nog eens nauwkeuriger naar kijken. De eerste teleurstelling is dus al weer binnen)
  • Edith Aurora Graf (2009). Defining Mathematics Competency in the Service of Cognitively Based Assessment for Grades 6 Through 8. ETS RR-09-42. download pdf
  • Randy Elliot Bennett (2010). Cognitively Based Assessment of, for, and as Learning (CBAL): A Preliminary Theory of Action for Summative and Formative Assessment. Measurement, 8, 70-91. pdf [Does not address math assessment. Gaat niet over rekentoetsen. Interessant vanwege de ideologische/politieke achtergronden van CBAL.]
  • James H. Fife (2013). Automated Scoring of Mathematics Tasks in the Common Core Era: Enhancements to M-rater in Support of CBAL Mathematics and the Common Core Assessments. ETS Research Report RR-13-26 abstract and pdf download

Mark J. Gierl & Thomas M. Haladyna (Eds) (2013). Automatic Item Generation. Theory and Practice. Routledge. [als eBook bij KB] info

• 1. Automatic Item Generation: An Introduction Mark J. Gierl and Thomas M. Haladyna 3-12
  2. Automatic Item Generation: A Historical Perspective Thomas M. Haladyna 13-25 [behandelt Bormuth, 1970; Guttman facet theory 1953, 1959; item forms Hively 1974; concept formation Markle & Tiemann; 1998 ETS Invitational]
  3. Using Weak and Strong Theory to Create Item Models for Automatic Item Generation: Some Practical Guidelines with Examples Mark J. Gierl and Hollis Lai 26-39
  4. Item generation: Implications for a validity argument Isaac I. Bejar 40-55
  5. An Introduction to Assessment Engineering for Automatic Item Generation Richard M. Luecht
  6. Generating Items under the Assessment Engineering Framework Hollis Lai and Mark J. Gierl
  7. Using Evidence-Centered Design Task Models In Automatic Item Generation Kristen Huff, Cecilia B. Alves, James Pellegrino and Pamela Kaliski
  8. Learning Sciences, Cognitive Models, and Automatic Item Generation Jacqueline P. Leighton 121-135. This is a key publication on item writing. For annotations see here
  9. Using Cognitive Psychology to Generate Items and Predict Item Characteristics. Joanna S. Gorin & Susan E. Embretson 136-156
  10. Difficulty Modeling and Automatic Generation of Quantitative Items: Recent Advances and Possible Next Steps Edith Aurora Graf and James H. Fife 157-179
  11. Statistical Modeling of Automatically Generated Items Sandip Sinharay and Matthew S. Johnson
  12. Automatic Item Generation for Computerized Adaptive Testing Richard M. Luecht
  13. IGOR: a Web-Based Automatic Item Generation Tool Todd Mortimer, Eleni Stroulia, and Meisam Vosoughpour Yazdchi
  14. Obstacles for Automatic Item Generation Thomas M. Haladyna and Mark J. Gierl

Ben Wilbrink (2014). Al die teksten in onze eindexamens, waar komt dat fenomeen toch vandaan? Kan Sander Dekker precies 75 L zand in een kruiwagen scheppen? Van Twaalf Tot Achttien, Vakblad voor Voortgezet Onderwijs, 24, 28-30. concept

Kou Murayama, Reinhard Pekrun, Stephanie Lichtenfeld, Rudolf vom Hofe (accepted 2012). Predicting Long-Term Growth in Students' Mathematics Achievement: The Unique Contributions of Motivation and Cognitive Strategies. Child Development prepublication pdf

Establishes again the fact that every student is able to learn mathematics. I have yet to study the article. The data are from a German longitudinal study. I need to know what the authors' conception of ‘learning mathematics’ is. And I’d like to see the math items used in the research (not even one presented in the article itself).

David Bornstein (April 18, 2011). A Better Way to Teach Math The opninion pages of Opiniator webpage

• In particular, math teachers often fail to make sufficient allowances for the limitations of working memory and the fact that we all need extensive practice to gain mastery in just about anything. Children who struggle in math usually have difficulty remembering math facts, handling word problems and doing multi-step arithmetic. Despite the widespread support for ‘problem-based’ or ‘discovery-based’ learning, studies indicate that current teaching approaches underestimate the amount of explicit guidance, ‘scaffolding’ and practice children need to consolidate new concepts. Asking children to make their own discoveries before they solidify the basics is like asking them to compose songs on guitar before they can form a C chord.

Philip E. Ross (August 2006). The expert mind. Studies of the mental processes of chess grandmasters have revealed clues to how people become experts in other fields as well. Scientific American, 64-71 pdf

Ben Wilbrink (1998). Inzicht doorzichtig toetsen. In Theo H. Joostens en Gerard W. H. Heijnen (Red.). Beoordelen, toetsen en studeergedrag. Groningen: Rijksuniversiteit, GION - Afdeling COWOG Centrum voor Onderzoek en Ontwikkeling van Hoger Onderwijs, 13-29. html

This paper is of interest because it models the situation where the student has to be able to connect two or more separate bits of (knowledge) information in order to reach a correct answer. Retrieval from long-term declarative memory, for example. In terms of probabilities: the probability to answer the item (correctly) is the product of the separate probabilities of retrieving the fitting bits of information. Anyone knowledgeable about conjunctive chances will see that the difficulty of the multiple-bit item will be much, even very much, higher than the naive assessor might think it to be, taking the difficulty of retrieving one particular bit of information as a guide. Kind of a ‘framing’ effect, maybe? Well, the model might explain why naive assessors typically underestimate the difficulty of items in educational measurement, for example in Angoff-procedures to pinpoint the level of achievement that is just at the pass-fail point.

J.-A. LeFevre, D. DeStefano, B. Coleman & T. Shanahan (2005). Mathematical cognition and working memory. In Jamie I. D. Campbell: Handbook of Mathematical Cognition (361-378). abstract

No pdf available online; if you do not have access to the book, see instead Ashcraft & Krause 2007.

Mark H. Ashcraft & Jeremy A. Krause (2007). Working memory, math performance, and math anxiety. Psychonomic Bulletin & Review, 14, 243-248. pdf

Not a research article.

• We show how performance on a standardized math achievement test varies as a function of math anxiety, and that math anxiety compromises the functioning of working memory. High math anxiety works much like a dual task setting: Preoccupation with one’s math fears and anxieties functions like a resource-demanding secondary task. We comment on developmental and educational factors related to math and working memory, and on factors that may contribute to the development of math anxiety.

  from the abstract

• This article provides an overview of the relationships among working memory, math performance, and math anxiety. We provide examples from the mathematical cognition literature to show: the critical role of working memory in performing arithmetic and math; the relationship between math performance and math anxiety, especially on standardized math achievement tests; and finally, the way that math anxiety compromises the functioning of working memory when people do arithmetic and math.

  p. 243

Jamie I. D. Campbell (2005). Handbook of Mathematical Cognition(361-378). Psychology Press. contents

Technical Appendix A. Quantifying the Reading Demands of the TIMSS 2011 Fourth Grade Mathematics and Science Items. pdf

Helpful. A superficial approach; useful in empiricist/routine research on ‘perceptive load’. Right? Cognitive load is something very different. Risk: focus on reading demands to obscure attention for cogn load. PISA-Math intentionally is more situationist than TIMSS-Math; therefore higher cognitive load?

• “This document presents the coding guide that was used in order to quantify the readability information of TIMSS 2011 mathematics and science items in preparation for the reading demands analysis found in the Impact of Reading Ability on TIMSS Mathematics and Science Achievement at the Fourth Grade: An Analysis by Item Reading Demands (Mullis, Martin, & Foy, 2013). This information was used in order to consider reading achievement at the fourth grade in PIRLS 2011 in relation to the level of reading demands in the fourth grade TIMSS 2011 mathematics and science items. This document presents an overview of the methodology used in order to code items from the TIMSS 2011 mathematics and science assessments, followed by sample mathematics and science items and their coding.”

  Overview

Ina V.S. Mullis, Michael O. Martin, and Pierre Foy (2013). The Impact of Reading Ability on TIMSS Mathematics and Science Achievement at the Fourth Grade: An Analysis by Item Reading Demands. Prepared for IEA’s 4th International Research Conference in Singapore 2013. pdf

A lot of fuss, many countries involved. At the end of the day the results are as to be expected: grade 4 poor readers (as measured by PIRLS) are handicapped on the TIMSS-Math items because these items test the ‘construct’ solving math items in everyday contexts. Pseudoscience? I think so, but my thinking does not help all those kids around the world. Or the economies suffering from poor mathematical skills in the labor force (if that would be your criterion).

• “ At the fourth grade in mathematics, for example, topics in the TIMSS framework content domains of number, measurement, and data display specify that students should be able to solve routine and non-routine problems set in everyday contexts. Understanding the description of the everyday situations for these types of problems necessarily involves reading. Also, the data display area is based on ‘reading and interpreting’ tables, pictographs, bar graphs, and pie charts as well as creating such data displays.”

  from the Introduction

• “ Still, fourth grade students are likely to be at a disadvantage in learning mathematics and science as well as demonstrating high performance on the TIMSS assessment if they lack reading skills. The availability of PIRLS data on reading achievement at the fourth grade provides an ideal opportunity to investigate the relationship between reading ability and the reading demands of the TIMSS mathematics and science assessment items. This study capitalized on the unique availability of PIRLS and TIMSS achievement scores for the same fourth grade students across reading, mathematics and science for 34 countries and three benchmarking participants to examine the following overarching question: How does reading ability impact TIMSS mathematics and science achievement at the fourth grade?”

  from the Introduction

• “While the poorest readers consistently achieved at a lower level in mathematics than the best readers, they were additionally disadvantaged on the mathematics items that required more reading.”

  from the concluding section

Daeun Park, Gerardo Ramirez & Sian L. Beilock (2014) The role of expressive writing in math anxiety. Journal of Experimental Psychology: Applied (online first) [I have no access to a pdf] abstract

This research shows that math anxiety impacts on capacity of short term memory available for problem solving.

Gerdineke van Silfhout (2014). Fun to read or easy to understand? Establishing effective text features for educational texts on the basis of processing and comprehension research. Proefschrift Utrecht. Landelijke Onderzoekschool Taalwetenschap. pdf ophalen

• Gerdineke van Silfhout (2013). Daarom hebben we zoveel moeite met die studieteksten! Over tekstbegrip in het Voortgezet Onderwijs. webpagina

  ---

• Meer publicaties uit dit promotietraject hier

Erik de Corte & Lieven Verschaffel: De complexe relatie tussen redactie- en formule-opgaven. Enkele resultaten van een empirisch onderzoek in de aanvangsklas van het basisonderwijs. In G. de Zeeuw, W. Hofstee & J. Vastenhouw (Red.) (1983). Funderend onderzoek van het onderwijs en onderwijsleerprocessen. Onderwijs Research Dagen 1983. Swets & Zeitlinger. isbn 9026504969

Irwin Kirsch (2001). The International Adult Literacy Survey (IALS): Understanding What Was Measured. Research Report RR-01-25. Educational Testing Service. pdf

G. Greeno (1980). Some examples of cognitive task analysis with instructional implications. In Richard E. Snow, Pat-Anthony. Federico and William E. Montague (Eds.) (1980). Aptitude, learning and instruction. Volume 2: cognitive process analyses of learning and problem solving (1-21). Erlbaum. isbn 0898590469

An old cow?

Lucy Cragg & Camilla Gilmore (2014). Skills underlying mathematics: The role of executive function in the development of mathematics proficiency. Trends in Neuroscience and Education [online first] open access webpage or this pdf

• Children’s underachievement in mathematics is a consistent & significant problem (Dowker, 2009) with 21% of 11- year-olds leaving primary school without reaching the mathematics level expected of them, and 5% failing even to achieve the numeracy skills expected of a 7-year-old (Gross, 2007). These problems endure into adulthood, and it is estimated that a fifth of adults have numeracy skills below the basic level needed for everyday situations (Williams, Clemens, Oleinikova, & Tarvin, 2003). Mathematics ability is crucial for success in Western societies (Ancker & Kaufman, 2007) and poor mathematics skills have a bigger impact on life chances than poor literacy (Parsons & Bynner, 2005).

Kelly Trezise & Robert A. Reeve (2014). Working memory, worry, and algebraic ability. Journal of Experimental Child Psychology, 121, 120-136. abstract [geen pdf tot mijn beschikking]

Nash Unswortha, Keisuke Fukudab, Edward Awha, Edward K. Vogel (2014). Working memory and fluid intelligence: Capacity, attention control, and secondary memory retrieval. Cognitive Psychology, 71, 1-26. abstract [geen pdf tot mijn beschikking]

Zach Shipstead, Dakota R.B. Lindsey, Robyn L. Marshall, Randall W. Engle (2014). The mechanisms of working memory capacity: Primary memory, secondary memory, and attention control. Journal of Memory and Language, 72, 116-141. [geen pdf tot mijn beschikking] abstract [geen pdf tot mijn beschikking]

Nash Unsworth & Gregory J. Spillers (2010). Working memory capacity: Attention control, secondary memory, or both? A direct test of the dual-component model. Journal of Memory and Language, 62, 392-406. [geen pdf tot mijn beschikking] abstract

Cai-Ping Dang, Johan Braeken, Roberto Colom, Emilio Ferrer & Chang Liu (2014). Why is working memory related to intelligence? Different contributions from storage and processing. Memory, 22, 426-441. [geen pdf tot mijn beschikking] abstract

Klaus Oberauer & Rheinholg Kliegl (2006). A formal model of capacity limits in working memory Journal of Memory and Language, 55, 601-626. abstract, pdf open access

Lex Borghans & Trudie Schils (draft Otober 13, 2012). The Leaning Tower of Pisa Decomposing achievement test scores into cognitive and noncognitive components. Paper presented at the 2012 NBER Summer Institute in Education Economics, Cambridge. draft & powerpoint

Deze onderzoekers hebben wel een heeel bijzondere variabele te pakken: in de loop van de afname van PISA-Math blijkt er slechter te worden gescoord, terwijl de mate waarin dat het geval is kan verschillen van land tot land. Ik heb dan onmiddellijk het vermoeden dat Nederlandse leerlingen deels beter scoren dan anderen omdat ze die contextrekenopgaven al hun hele onderwijsloopbaan intensief hebben ontmoet.

Lex Borghans, Leo Kockelkorn, Trudie Schils (March 1, 2013). Low stakes, high stakes: The predictive power of math achievement tests. pdf

Tom Siegfried (April 22, 2014). Doctors flunk quiz on screening-test math.

ScienceNews, Magazine of the Society for Science & the Public abstract

Some sloppy thinking is going on here. The question does not mention anything about the false negative rate. The text, using baseball player drugs use testing, suggests both ates to be the same, allowing one to fuzzily speak of a test being 95 percent correct.

F. S. J. Riemersma & J. Meijer (1983). Leren oplossen van wiskundige problemen: analyse van hardopdenk-protocollen. SCO-rapport 26.

• We vroegen docenten om uit de door hen gebruikte leermethode enkele vraagstukken aan te geven die leerlingen in het algemeen als problematisch ondervonden. Deze vraag is voornamelijk bedoeld om onderwerpen voor protocol-analyse op het spoor te komen.

  Er was betrekkelijk weinig overlap, dat wil zeggen, leraren gaven voornamelijk andere opgaven op. Sommige vraagstukken werden wél door verschillende leraren opgegeven; een indicatie dat dit soort vraagstukken voor vele leerlingen nogal moeilijk is. Deze vraagstukken stoelden meestal op het gebruik van kennis in nieuwe situaties. Zo vereist bijvoorbeeld het berekenen van de lengte van een lichaamsdiagonaal in een kubus of balk het successief toepassen van Pythagoras in neen driedimensionale ruimte. Ook opgaven over inhoudsberekeningen (gebruik van derde nmachten) werden vaak genoemd. Notaties van puntverzamelingen in de vorm van ongelijkheden en vergelijkingen en het bepalen van doorsnede en vereniging van puntverzamelingen keerden ook regelmatig terug.

  3

Jacqueline Leighton & Mark Gierl (Eds.) (2007). Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge University Press. [als eBook in KB] info

• 1. Why Cognitive Diagnostic Assessment? 3 Jacqueline P. Leighton and Mark J. Gierl
  2. The Demand for Cognitive Diagnostic Assessment 19 Kristen Huff and Dean P. Goodman
  3. Cognitive Modeling of Performance on Diagnostic Achievement Tests: A Philosophical Analysis and Justification 61 Stephen P. Norris, John S. Macnab, and Linda M. Phillips
  4. Test Validity in Cognitive Assessment 85 Denny Borsboom and Gideon J. Mellenbergh
  5. Construct Validity and Cognitive Diagnostic Assessment 119 Xiangdong Yang and Susan E. Embretson
  6. Verbal Reports as Data for Cognitive Diagnostic Assessment 146 Jacqueline P. Leighton and Mark J. Gierl
  7. Test Construction and Diagnostic Testing 173 Joanna S. Gorin
  8. Cognitive Foundations of Structured Item Response Models 205 André A. Rupp and Robert J. Mislevy
  9. Using the Attribute Hierarchy Method to Make Diagnostic Inferences About Examinees’ Cognitive Skills 242 Mark J. Gierl, Jacqueline P. Leighton, and Stephen M. Hunka
  10. The Fusion Model Skills Diagnosis System 275 Louis A. Roussos, Louis V. DiBello, William Stout, Sarah M. Hartz, Robert A. Henson, and Jonathan L. Templin
  11. Using Information from Multiple-Choice Distractors to Enhance Cognitive-Diagnostic Score Reporting 319 Richard M. Luecht
  12. Directions for Future Research in Cognitive Diagnostic Assessment 341 Mark J. Gierl and Jacqueline P. Leighton

  Contents

Graeme S. Halford, William H. Wilson & Steven Phillips (1998). Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and Brain Sciences, 21, 803-831 abstract and pdf

The pdf also contains open peer commentaries (a.o. by Anderson, Lebiere, Lovett & Reder: ACT-R: A higher level account of processing capacity), and the reply by the authors.

Joanna S. Gorin & Susan E. Embretsen (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30, 394-411. abstract

Sidney H. Irvine and Patrick C. Kyllonen (Eds) (2002). Item generation for test development  Mahwah, NJ [etc.] : Lawrence Erlbaum Associates, 2002. -  [ UB Leiden magazijn 3 9672 D 40, nog niet opgezocht ] questia

• Since the mid-80s several laboratories around the world have been developing techniques for the operational use of tests derived from item-generation. According to the experts, the major thrust of test development in the next decade will be the harnessing of item generation technology to the production of computer developed tests. This is expected to revolutionize the way in which tests are constructed and delivered.

  In November 1998, the late Sam Messick, Sidney Irvine, and Patrick Kyllonen assembled a symposium, at ETS in Princeton, attended by the world's foremost experts in item-generation theory and practice. This book is a compilation of the papers presented at that meeting.

  This book's goal is to present the major applications of cognitive principles in the construction of ability, aptitude, and achievement tests. It is an intellectual contribution to test development that is unique, with great potential for changing the ways tests are generated. It will be a publishing landmark in the history of test development. The intended market includes professional educators and psychologists interested in test generation.

  
  Contents among others:

  • The Foundations of Item Generation for Mass Testing. S.H. Irvine
  • Using the Psychology of Reasoning to Predict the Difficulty of Analytical Reasoning Problems. S. Newstead, P. Bradon, S. Handley, J. Evans, I. Dennis
  • Approaches to Modeling Item-Generative Tests. I. Dennis, S. Handley, P. Bradon, J. Evans, S. Newstead
  • On the Roles of Task Model Variables in Assessment Design. R.J. Mislevy, L.S. Steinberg, R.G. Almond
  • Modeling the Difficulty of Quantitative Reasoning Items: Implications for Item Generation. M.K. Enright, K.M. Sheehan
  • Item-Generation Models for Higher Order Cognitive Functions. L.F. Hornke
  • Generative Testing: From Conception to Implementation. I.I. Bejar
    • zie ook A Feasibility Study of On-the-Fly Item Generation in Adaptive Testing pdf
  • Generating Abstract Reasoning Items With Cognitive Theory. S.E. Embretson
  • Item Generation for Repeated Testing of Human Performance. P.C. Kyllonen
  • Scoring Tests When Items Have Been Generated. D. Wright
  • On the Automatic Generation of Test Items: Some Whens, Whys, and Hows. H. Wainer
  • The MICROPAT Pilot Selection Battery: Applications of Generative Techniques for Item-Based and Task-Based Tests. D. Bartram
  • Item Generation and Beyond: Applications of Schema Theory to Mathematics Assessment. M.K. Singley, R.E. Bennett

Mary Kay Stein, Barbara W. Grover and Marjorie Henningsen (1996). Building Student Capacity for Mathematical Thinking and Reasoning: An Analysis of Mathematical Tasks Used in Reform Classrooms. American Educational Research Journal, 33, 455-488 abstract

Stein, M., Smith, M., Henningsen, M., & Silver, E. (2000). Implementing standards-based mathematics instruction: A casebook for professional development. New York: Teachers College Press. [niet gezien] info

I do not expect this book to be helpful in any way. Moreover, its ideology seems to be constructivism.

Jacqueline P. Leighton and Mark J. Gierl (2007). Defining and Evaluating Models of Cognition Used in Educational Measurement to Make Inferences About Examinees' Thinking Processes. Educational Measurement: Issues and Practice, 26 #2, 3-16. paywalled abstract

• For researchers and practitioners, awareness of different cognitive models may facilitate the evaluation of educational measures for the purpose of generating diagnostic inferences, especially about examinees' thinking processes, including misconceptions, strengths, and/or abilities. We think a discussion of the types of cognitive models underlying educational measures is useful not only for taxonomic ends, but also for becoming increasingly aware of evidentiary claims in educational assessment and for promoting the explicit identification of cognitive models in test development.

  from the abstract

Wim Van Dooren, Lieven Verschaffel, Brian Greer, and Dirk De Bock (2006). Modelling for Life: Developing Adaptive Expertise in Mathematical Modelling From an Early age. In Lieven Verschaffel, Filip Dochy, Monique Boekaerts and Stella Vosniadou (Eds) (2006). Instructional psychology: Past, present, and future trends. Sixteen essays in honour of Erik de Corte. Elsevier. [als eBook te leen in KB] info

Het boek is als eBook te leen in de KB, ik beschouw het dus maar als voor iedereen die dat wil voldoende toegankelijk. Het hoofdstuk begint met een soort beginselverklaring voor het situationisme dat we zo uitbundig terugvinden in onze rekentoetsen, in de PISA-rekentoetsen, en in het realistisch rekenen. Verschaffel heeft al wat langer onderzoek gedaan naar redactiesommen, en heeft een uitgesproken opvatting over contextopgaven in het rekenen. Ik ben zo vrij om een uitstekend voorbeeld in dit hoofdstuk, p. 92-93, volledig te citeren. Of misschien moet ik het in eigen woorden samenvatten? Eerst een stellingname die mij werd opgedrongen bij het bestuderen van de laatste sectie in dit hoofdstuk:

Stelling: ‘rekenen in contexten’ is een vorm van zelfontdekkend leren. Dat als doel voor het onderwijs stellen, zoals de auteurs hier doen unisono met de reform-gemeenschap, is het leren zelfontdekkend te leren als examineerbaar doel stellen.

Examens in de vorm van ‘rekenen in contexten’ testen ahw het vermogen tot zelfontdekkend leren (intellectuele capaciteiten)

Hoe werkt het brein? Enige dagen geleden vatte ik nog eens samen dat het probleem van transfer (volgens Stellan Ohlsson) geen afzonderlijk probleem is, maar dat hier sprake is van leren: een relatief algemene regel of set van regels aanpassen/verbijzonderen zodat ze passend zijn voor de nieuwe situatie. Dat is wat contextrekenopgaven vragen. Ik was daardoor onverwacht goed geprepareerd voor het volgende. In de op zich glasheldere — maar niet empirisch onderbouwde — visie van Verschaffel c.s. op het opstellen van rekenmodellen is het evident dat zij dat leren bij het oplossen van contextopgaven opvatten als zelfontdekkend leren, zonder dat zo te benoemen overigens. Ik had het al veel en veel eerder zelf moeten ontdekken (dat de visie van Verschaffel hierop neerkomt). Pro memorie: Lieven Verschaffel is in de tachtiger en negentiger jaren zo ongeveer de enige psycholoog geweest die Hans Freudenthal en zijn groep in Utrecht toejuichte, en daarmee in zekere zin legitimeerde wat de psychologie van het realistisch rekenen betreft. ‘Het nieuwe leren’. Verschaffel was vijf jaar geleden lid van de KNAW-commissie over het rekenonderwijs, en zal waarschijnlijk op 30 juni aanezig zijn in het Trippenhuis, bij het eerste jubileum van de commissie-Lenstra.

• The development of students’ disposition to apply mathematics to make sense of everyday-life situations and to solve problem situations in the real world — otherwise termed ‘mathematical modeling’ — gets a central place in contemporary mathematics curricula and reform documents worldwide. For example, in the Principles and Standards for School Mathematics by the National Council of Teachers of Mathematics (NCTM, 2000, p. 3) in the United States, one can read that:

  Quantitative information available to limited numbers of people a few years ago is now widely disseminated through popular media outlets. The need to understand and be able to use mathematics in everyday life and in the workplace has never been greater and will continue to increase.

  p. 91

• Neem het modelleren dat nodig is in de volgende situatie uit het werkelijke leven: Een reisleider wil het 509 meter hoge Taipei-gebouw bezoeken met een groep van 289. Hij vraagt de conciërge of het mogelijk is om iedereen binnen 45 minuten boven te krijgen. De conciërge weet weet dat er, door technische problemen met twee andere liften, maar een lift beschikbaar is, waar 20 mensen in kunnen, en dat de lift 86 seconden nodig heeft, in beide richtingen.

  Een goede modelontwerper stelt eerst een situatiemodel op dat bestaat uit een ‘quotitive’ verdelingsmodel (hoeveel groepen van 20 zijn er in de totale groep) en een verhoudingsrelatie tussen aantal reizen en benodigde tijd (ieder reis: heen, terug, tijd in- en uitstappen). Eerste deel van de oplossing is uit te vinden hoeveel reizen nodig zijn. Het rekenmodel afgeleid van het quotitieve verdelingsmodel is 289/20, resulterend in de afleiding in 14,45 ofwel 14 met rest 9. De interpretatie van dit resultaat moet zijn dat er 15 reizen nodig zijn. Dan moeten we de totale tijd mathematiserenals deze 15 maal de tijd voor iedere reis 2 × 86 s + in- en uitstaptijd, maar omdat de laatste reis alleen omhoog is, trekken we 86 sec van die som af. Voor een concreet resultaat moeten we wat aannamen maken: totale reistijd 3,5 minuut bijvoorbeeld, en rond de lift omlaag af op 1,5 minuut. Uit het rekenmodel 15 × 3.5 - 1,5 is de uitkomst 51. De interpretatie daarvan: het lunkt niet om 289 mensen in 45 minuten boven te krijgen. Maar het verschil is niet groot, kan de reisleider zijn plan misschien aanpassen? Od, als er kinderen in het gezelschap zijn, hoe komt de som er dan uit te zien?

  p. 92-93, in mijn woorden, b.w.

Van Dooren c.s. onderscheiden vier typen contextproblemen, in oplopende complexiteit en toenemend realisme, een voorstel dat zij ontleneen aan Galbraith & Stillman (2001, p. 301). Ik citeer het schema hierbeneden. Het vierde, meest ambitieuze niveau is kennelijk voor deze auteurs ook het ideaal voor het onderwijs (niet het promotieonderzoek, maar het basisonderwijs). Het lijkt mij gespeend van enig realistisch (pun intended) gevoel voor (verschillen in) menselijke intellectuele capaciteiten. Maar pluspuntje is: het maakt veel duidelijk over de ambities van de Leuvense hervormers.

• • Injudicious problems, wherein realistic constraints are seriously violated;
  • Context-separable problems, wherin the context plays no role in the solution and can be stripped away to expose a purely mathematical question;
  • Standard application problems, where the necessary mathematics is context-related and the situation is realistic, but where the procedure is still rather standard;
  • Genuine modelling problems, where no mathematics as such appears in the problem statement, and where the demarcation and formulation of the problem, in mathematical terms, must be at least partially supplied by the modeller.

  p. 104

• 

• 
  

  ---

  M. Niss (2001). Issues and problems of research on the teaching and learning of applications and modelling. In J. F. Matos, W. Blum, S. K. Houston & S. P. Carreira: Modelling and mathematics education. ICTMA9: Applications in science and technology (pp 72-89). Chichester, U.K.: Horwood. paywalled abstract [ik heb geen toegang] info on the book

  ---

  Naar dit hoofdstuk wordt verwezen door Van Dooren c.s. (2006, p. 103 ): “However, according to Niss (2001), in general international terms, genuine and extensive applications and modelling perspectives and activities continue to be scarce in everyday practices of mathematics education. Niss points to two important barriers, namely the difficulty of getting the modelling perspective into tests (in which it is easier to include tasks with a more closed character) and the extremely high demands that such a modelling approach puts on teachers (mathematically, pedagogically and personally). ” Ook het abstract van Niss is interessant: het wijst op een tekort aan empirisch onderzoek op deze punten.

  Welnu, we hebben hier mijns inziens te maken met een onderzoekersgemeenschap die weinig of geen contact heeft met wat er overigens in het onderwijsonderzoek en in de cognitieve psychologie gebeurt: daar is immers vanaf de zeventiger jaren intensief onderzoek gedaan naar probleemoplossen, ook in onderwijsomstandigheden, zoals de verschillen in probleemaanpak tussen experts en nieuwelingen. Het is mijns inziens ten onrechte dat Van Dooren, Verschaffel c.s., en hier Niss, telkens benadrukken dat er onderzoekmatig zo weinig bekend is. Er is voldoende bekend, maar daarvoor moet de algemene cognitief-psychologische literatuur worden geraadpleegd.

  Ik doe niet voor niets hier veel moeite om de boodschap van Niss helder te krijgen, want het verhaal is natuurlijk heel, heel erg bekend: reform-rekenonderwijs is ongelooflijk ambitieus, heeft torenhoge verwachtingen van niet alleen wat leerlingen aankunnen, maar vooral ook van waartoe leraren in staat zouden zijn. Niss goeit hier olie op die woelige baren, en dat is volkomen terecht. Wat is de conclusie hieruit: de ambities voor de rekentoetsen zijn al evenzeer torenhoog, Niss wijst erop dat een empirische basis voor de mogelijke juistheid van die ambities ontbreekt. Die empirische basis is bij de ontwikkeling van de rekentoetsen evenmin eerst aangebracht, dus heel die rekentoets is, wat Niss betreft, en wat Van Dooren c.s. betreft, dus een luchtkasteel. Moeten we aannemen. Op basis van het wel degelijk beschikbare onderzoek in de cognitieve psychologie weten we dat de ambities van de rekentoetsen zijn losgezongen van althans deze empirische basis.

  Beweren Niss, en Van Dooren, Verschaffel, Greer en De Bock, dat vooralsnog niet is aangetoond dat het mogelijk is om leerlingen de capaciteiten bij te brengen om de wat ingewikkelder contextrekenopgaven tot goede oplossingen te brengen? Mij verbaast het niet, want de opgave is bijna in beginsel onmogelijk; althans in deze termen: dat havisten op de rekentoets-3F gemiddeld hetzelfde resultaat kunnen halen als de vwo-ers doen. En dat is immers het enig juiste criterium voor een toets die pretendeert basisvaardigheden rekneen te toetsen!

Kenneth R. Koedinger, Albert T. Corbett & Charles Perfetti (2012). The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning. Conitive Science, 36, 757-798. abstract en/of full report

• Slotalinea uit het rapport:
  As we discussed above, other researchers have recognized the importance for effective instructional design of a detailed analysis of domain content into the components of knowledge that students bring to a course and those we would like them to take away. Nevertheless, Anderson and Schunn (2000) pdf expressed concern that “detailed cognitive analyses of various critical educational domains are largely ignored by psychologists, domain experts and educators”. They noted the tendency for psychologists to value domain-general results, domain experts (e.g., mathematicians and linguists) to value new results in the domain, and educational researchers to value more holistic explanations. Some progress has been made (e.g., Clark, Feldon, van Merriënboer, Yates, & Early, 2007; Lee, 2003), but careful cognitive task analysis of domain knowledge is not a standard research practice in any discipline. Whether it is done through the emergence of a new discipline or through interdisciplinary teams, we hope to see such analysis become a more routine part of instructional design for new instructional domains as well as for existing ones.

John R. Anderson & Christian D. Schunn (2000). Implications of the ACT-R learning theory: No magic bullets. In R. Glaser (Ed.), Advances in instructional psychology: 5 (pp. 1-34). Mahwah, NJ: Lawrence Erlbaum Associates. pdf

http://act-r.psy.cmu.edu/wordpress/wp-content/uploads/2012/12/716StoccoAnderson-JoCN-2007.pdf pdf

http://act-r.psy.cmu.edu/category/problem-solving-and-decision-making/mathematical-problem-solving/webpage

Arthur C. Graesser and Rolf A. Zwaan (1995). Inference generation and the construction of situation models. abstract In Charles A. Weaver III, Suzanne Mannes, Charles R. Fletcher and Walter Kintsch (Eds) (1995). Discourse Comprehension: Essays in Honor of Walter Kintsch [als eBook in KB]

Arnoud Verdwaald (1998). Relational transformations in the process of discourse representation of simple word problems. Dissertation Catholic University of Nijmegen. info

Highly relevant research. [I have seen the dissertation, copied it, I have yet to study it thoroughly; no online version available?] Het onderzoek gaat over eenvoudige redactiesommen. Bedrieglijk eenvoudig, want ze blijken lastig, ook voor volwassenen wanneer deze onder druk moeten werken. Het helpt dus enorm voor het begrijpen van de moeilijkheden met sommige rekenopgaven-3F wanneer het lukt om cognitieve modellen voor de onderstaande eenvoudige woordproblemen experimenteel te onderzoeken. Verdwaald zal met Nederlandse opgaven hebben gewerkt, ik zie zo gauw niet de exacte teksten daarvan; ik heb zelf maar even het Engels vertaald. Merk hoe bij het oplossen van deze vraagstukjes de aandacht heen en weer schiet tussen de gegevens, zoowel de getallen als de exacte woorden. Het gaat dus ook over mentale belasting, maar de ingang lijkt allereerst een linguïstische te zijn.

• Jan heeft 5 stuiters.
  Hij heeft drie stuiters meer dan Rob.
  Hoeveel stuiters heeft Rob?

  Jan had eerst enkele stuiters.
  Rob gaf toen 3 stuiters aan Jan.
  Jan heeft nu vijf stuiters.
  Hoeveel stuiters had Jan eerst?

  p. 5 [mijn vertaling, b.w.]

• M. D. LeBlanc & S. Weber-Russell (1996). Text integration and mathematical connections: A computer model of arithmetic word problem solving. Cognitive Science, 20, 357-407. abstract, if the abstract triggers your curiosity, go to pdf in researchgate.net
  LeBlanc and Weber-russell (1996) also take a stance that gives an indication of working memory demands. Actually, they are about the only researchers in this field who have taken working memory demands seriously. Note, for example, that there is nothing in Verschaffel and De Corte’s list [p.10 in Verdwaald, b.w.] to represent this ‘component’.

  Verdwaald p. 10

E. C. D. M. van Lieshout (2010). Enkele lijnen in het onderzoek van basale rekenvaardigheden. Rede uitgesproken ter gelegenheid van zijn afscheid als hoogleraar Orthopedagogiek met betrekking tot onderwijsleerproblemen aan de faculteit der Psychologie en Pedagogiek van de Vrije Universiteit Amsterdam op 5 maart 2010. rede

Stanley Woll (2002). Everyday Thinking. Memory, Reasoning, and Judgment in the Real World. Erlbaum. [als eBook in KB]info

Paul Cornell, MonicaRiordan, Mary Townsend-Gervis & Robin Mobley (2011). Barriers to critical thinking. Workflow interruptions and task switching among nurses. The Journal of Nursing Administration, 41, 407-414. [researchgate download] abstract

Waarom verwijs ik hier naar, en waarom is het abstract van belang voor onze rekentoetsproblematiek? Welnu, wanneer kandidaten onder examencondities twee of drie minuten tijd hebben voor telkens weer een andere contextopgave, dan hebben we een vergelijkbare stress-conditie als voor deze verpleegkundigen. Overigens heb ik zo’n vermoeden dat wat dit artikel beschrijft, direct relevant is voor de problemen die de ontwikkelaars van toetsen voor verpleegkundig rekenen ontmoeten (de meeste verpleegkundigen uit de beroepspraktijk scoren ‘onvoldoende’ op deze mede door het Cito ontwikkelde toetsen).

• Nurses are increasingly called upon to engage in critical thinking. However, current workflow inhibits this goal with frequent task switching and unpredictable demands. To assess workflow’s cognitive impact, nurses were observed at 2 hospitals with different patient loads and acuity levels. (..) Tasks were short with a mean duration of 62.4 and 81.6 seconds on the 2 units. More than 50% of the recorded tasks were less than 30 seconds in length. An analysis of task sequence revealed few patterns and little pairwise repetition. (..) The nonrepetitive flow and high amount of switching indicate nurses experience a heavy cognitive load with little uninterrupted time. This implies that nurses rarely have the conditions necessary for critical thinking.

  from the abstract

Gerd Gigerenzer, Ralph Hertwig & Thorsten Pachur (Eds.) (2011). Heuristcs. The Foundations of Adaptive Behavior. Oxford University Press. [niet gevonden in KB bij de eBooks, 2014] info

In de index geen arithmatics of mathematics. Interessante gedachte: bij beslissingen in het dagelijks leven speelt rekenen, laat staan wiskunde, dus geen rol van enige betekenis? Laat ik het eens anders formuleren: bij het aanpakken van contextopgaven volgen kandidaten waarschijnlijk niet de rationele strategie die de ontwerpers van deze vragen verwachten, maar strategieën die in het dagelijks leven vaak en met succes gebruikt worden. Een goed voorbeeld uit die laatste categorie is Herbert Simon’ satisficing strategie: als opvallende informatie in de opgave voldoende lijkt om tot een oplossing te komen, dan is dat de voorkeursaanpak voor de kandidaat. En dus niet het onthecht eerst zorgvuldig bestuderen van alle informatie, de vraagstelling, en dan nog weer eens de informatie, om dan een mogelijk plan van aanpak te bedenken, uit te werken, uit te voeren, en te controleren of het resultaat inderdaad een antwoord id op de gestelde vraag. Of die laatste rationele procedure nu uit ten treure is getraind, of niet: als die contextopgaven iedere keer een vooral toch heel nieuw probleem stellen, dan is voorgaande training immers niet zo geweldig relevant voor deze weer nieuwe opgave. Lees dus Chapter 2: Gerg Gigerenzer & Daniel G. Goldstein: Reasoning the fast and frugal way: models of bounded rationality. Of een andere publicatie uit de indertussen zeer uitgebreide literatuur op deze thematiek (waaronder de bestseller van Daniel Kahneman).

Helen C. Reed (30 juni 2014). Mathematical Thinking, Learning and Performance. Insights and Interventions for Primary and Secondary Education. Vrije Universiteit. http://dare.ubvu.vu.nl/handle/1871/51328

• Helen C. Reed, Michelle Gemmink, Marije Broebs-Paffen, Paul A. Kirschner & Jelle Jolles (submitted). Improving multiplication fact fluency by choosing between competing answers. Research in Mathematics Education (site). [dit hoofdstuk 3 is (nog) niet online beschikbaar]

L. N. Tronsky (2005). Strategy use, the development of automaticity, and working memory involvement in complex multiplication. Memory and Cognition, 33, 927-940. free access

• Most theories and models of arithmetic processing have been constructed with the implicit or explicit assumption that adults retrieve most, if not all, basic arithmetic facts from long-term memory (Ashcraft & Battaglia, 1978; Butterworth, Zorzi, Girelli, & Jonckheere, 2001; Campbell & Oliphant, 1992; De Rammelaere, Stuyven, & Vandierendonck, 1999, 2001; Koshmider & Ashcraft, 1991; Lemaire, Abdi, & Fayol, 1996). In recent years, there has been a proliferation of research in which strategy use and development in children’s and adults’ simple mental arithmetic problem solving have been examined (e.g., Geary, 1996; Hecht, 2002; Kirk & Ashcraft, 2001; LeFevre, Bisanz, et al., 1996; LeFevre, Sadesky, & Bisanz, 1996; Lemaire & Siegler, 1995; Seyler, Kirk, & Ashcraft, 2003; Siegler, 1988). One of the more interesting findings from this research is that a large percentage of adults continue to use strategies to solve basic addition, subtraction, multiplication, and division problems (e.g., Campbell & Xue, 2001; LeFevre, Bisanz, et al., 1996; LeFevre, Sadesky, & Bisanz, 1996; Seyler et al., 2003; Tronsky & Shneyer, 2004). This widespread finding has made it necessary to reexamine some of the empirical effects in basic arithmetic research in order to clarify and qualify some of those findings, so that more accurate models of arithmetical cognition can be developed; research related to this has just begun, at least in the domain of simple arithmetic (Campbell & Xue, 2001; Hecht, 2002; Seyler et al., 2003; Tronsky, Anderson, & McManus, 2005).

  Investigations of adults’ complex mental arithmetic skills also have become more numerous, especially recently (e.g., Ashcraft, Donley, Halas, & Vakali, 1992; Fürst & Hitch, 2000; Geary, Frensch, & Wiley, 1993; Logie, Gilhooly, & Wynn, 1994; Seitz & Schumann- Hengsteler, 2000; Trbovich & LeFevre, 2003). At present, few researchers have examined the strategies that adults use to solve these problems (with the exception of Geary et al., 1993), how the use of strategies impacts working memory (WM), and how the development of automaticity impacts WM involvement. The purpose of the present investigation is to examine adults’ initial strategy use in complex mental multiplication and corresponding WM involvement, to document how problemsolving processes and WM involvement change with practice, and to examine the factors that govern performance after retrieval from long-term memory (automaticity) has been established. In order to set the context for the present investigation, it is necessary to provide a review of theory and research related to strategy use and development in mental arithmetic, the structure of WM, and the implications that strategy use and automaticity have for WM’s role in arithmetic.

  p. 927

J.R. Hayes & H.A. Simon (1974). Understanding written problem instructions. In Lee W. Gregg (Ed.) (1974). Knowledge and Cognition. Erlbaum. pdf. Reprinted in Herbert A. Simon (Ed.) (1979). Models of thought. New Haven: Yale University Press.

Hui-Yu Hsu (2010). The study of Taiwanese students' experiences with geometric calculation with number (GCN) and their performance on GCN and geometric proff (GP). Dissertation University of Michigan (Edward Silver, chair of promotion committee) pdf

• Hui-Yu Hsu & Edward A. Silver (2014). Cognitive complexity of mathematics instructional tasks in a Taiwanese classroom: An examination of task sources. Journal for Research in Mathematics Education, 45, 460-496. abstract

  ---

  • Hui-Yu Hsu & Edward A. Silver: Detailed Analysis of the Problem-Solving Complexity of a GCN Task pdf

Gerdineke van Silfhout (2014). Fun to read or easy to understand? Establishing effective text features for educational texts on the basis of processing and comprehension research. Leuk om te lezen of makkelijk te begrijpen? Een oogbewegings- en begripsonderzoek naar effectieve tekstkenmerken in schoolboeken. Proefschrift Universiteit Utrecht. pdf

Ruth C. Clark & Richard E. Mayer (2011). e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Second edition. Wiley. info [als eBook te leen in KB]

• Amon other chapters:
  3. Applying the Multimedia Principle: Use Words and Graphics, Rather Than Words Alone.
  4. Applying the Contiguity Principle: Align Words to Corresponding Graphics.
  5. Applying the Modality Principle: Present Words as Audio Narration, Rather Than On-Screen Text.
  6. Applying the Redundancy Principle: Explain Visuals with Words in Audio or Text: Not Both.
  7. Applying the Coherence Principle: Adding Interesting Material Can Hurt Learning.
  8. Applying the Personalization Principle: Use Conversational Style and Virtual Coaches.

  Wiley site

Leen VanBeek, Pol Ghesquière, Bert DeSmedt & Lieven Lagae (2014). The arithmetic problem size effect in children: an event-related potential study. Frontiers in Neuroscience 25 September 2014 ResearchGate or free access

I do not know what to make of this, weak models compared to those of ACT-R. Are there any cross-references tot the work of John Anderson and his colleagues?

• One of the most robust phenomena in the field of mathematical cognition is the problem size effect, which indicates that reaction time (RT) and error rate increase as the magnitude of the operands in an arithmetic problem increases (e.g., Stazyk et al., 1982; Campbell and Graham, 1985; Dehaene, 1992; Ashcraft and Guillaume,2009). Numerous behavioral studies have reported this problem size effect in adults and children (see for a review Zbrodoff and Logan,2005). The electrophysiological correlates of this problem size effect have been well documented in adults (Jost et al.,2004a,b; Núñez-Peñaetal.,2005, 2006, 2011; Núñez-Peña, 2008), which makes this effect an excellent paradigm to investigate mental arithmetic.

• Stazyk, E.H., Ashcraft, M.H.,and Hamann, M.S.(1982). A network approach to mental multiplication. J. Exp.Psychol.Learn.Mem.Cogn. 8, 320-335. doi: 10.1037/0278-7393.8.4.320
• Campbell, J.I.D.,andGraham,D.J.(1985).Mental multiplication skill: structure, process, and acquisition. Can. J.Psychol. 39,338-366. doi:10.1037/h0080065
• Dehaene, S.(1992). Varieties of numerical abilities. Cognition 44, 1-42.doi: 10.1016/0010-0277(92)90049-N
• Ashcraft, M.H., and Guillaume, M.M.(2009). Mathematical cognition and the problem size effect. Psychol.Learn.Motiv. 51, 121-151. doi:10.1016/S0079-7421(09)51004-3 abstract: http://www.sciencedirect.com/science/article/pii/S0079742109510043 Also as chapter 4 in Brian H. Ross (Ed.) (2009). The Psychology of Learning and Motivation. [als eBook in KB]

Sian L. Beilock (2008). Math performance in stressful situations. Current Directions n Psycholgical Science, 17, 339-343. [Available from: https://www.researchgate.net/publication/228924765_Math_performance_in_stressful_situations [accessed Apr 5, 2015]]

• REFERENCES

  • Ashcraft, M.H., & Kirk, E.P. (2001). The relationships among working memory, math anxiety, and performance. Journal of Experimental Psychology: General, 130, 224-237.
  • Beilock, S.L., & Carr, T.H. (2001). On the fragility of skilled performance: What governs choking under pressure? Journal of Experimental Psychology: General, 130, 701-725.
  • Beilock, S.L., & Carr, T.H. (2005). When high-powered people fail: Working memory and ‘‘choking under pressure’’ in math. Psychological Science, 16, 101-105.
  • Beilock, S.L., & DeCaro, M.S. (2007). From poor performance to success under stress: Working memory, strategy selection, and mathematical problem solving under pressure. Journal of Experimental Psychology: Learning, Memory, & Cognition, 33, 983- 998. abstract
  • Beilock, S.L., Jellison, W.A., Rydell, R.J., McConnell, A.R., & Carr, T.H. (2006). On the causal mechanisms of stereotype threat: Can skills that don’t rely heavily on working memory still be threatened? Personality & Social Psychology Bulletin, 38, 1059-1071.
  • Beilock, S.L., Kulp, C.A., Holt, L.E., & Carr, T.H. (2004). More on the fragility of performance: Choking under pressure in mathematical problem solving. Journal of Experimental Psychology: General, 133, 584-600.
  • Beilock, S.L., Rydell, R.J., & McConnell, A.R.(2007). Stereotype threat and working memory: Mechanisms, alleviation, and spillover. Journal of Experimental Psychology: General, 136, 256-276.
  • Conway, A.R.A., Kane, M.J., Bunting, M.F., Hambrick, D.Z., Wilhelm, O., & Engle, R.W. (2005). Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review, 12, 769-778.
  • Engle, R.W. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11, 19-23.
  • Gimmig, D., Huguet,P., Caverni,J.P., & Cury,F.(2006). Choking under pressure and working memory capacity: When performance pressure reduces fluid intelligence (Gf). Psychonomic Bulletin & Review, 13, 1005-1010.
  • Hyde, J.S., Fennema, E., & Lamon, S.J. (1990). Gender differences in mathematics performance: A meta-analysis. Psychological Bulletin, 107, 139-155.
  • Imbo, I., Vandierendonck, A., & Verguewe, E. (2007). The role of working memory in carrying and borrowing. Psychological Research, 71, 467-483.
  • Miyake, A., & Shah, P. (1999). Models of working memory: Mechanisms of active maintenance and executive control. New York: University Press. [PSYCHO C3.0.-159]
  • Steele, C.M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613-629.
  • Trbovich, P.L., & LeFevre, J. (2003). Phonological and visual working memory in mental addition. Memory & Cognition, 31, 738-745.

Interesting prediction: students might learn nothing from exercise context math problems (PISA-like). From John Sweller (). Story of a research program. Education Review webpage tweet

  http://www.benwilbrink.nl/projecten/14contextopgaven_mentale_belasting.htm