Testing for ‘math in everyday situations’

red lines: Yet better to be formulated:

PISA) Wait a moment—PISA Math is not a high stakes test. It definitely is not an exit exam for students! Not for students, indeed. For politicians it is, however. PISA Math is rapidly becoming the international ‘standard’ for what math education should be about, what it should ‘deliver’. The items in the Dutch rekentoets are in many respects identical to those in PISA Math—there is a research report by Cito on this question, it is kept a secret by the Department of Education. The problem stil not explained by Cito is: how is it possible for Dutch youth to score in the world top on PISA Math, yet do so poorly on the Dutch rekentoets that it has become a political problem of huge proportions?

LO) It is not only this kind of questioning itself, but also the kinds of questioning left out (LO) of testing: testing for mastery of algorithms of arithmetic, or for mental arithmetic. ‘Algorithm’ in the mathematical sense, not the fuzzy one like this (Everyday Math). The Dutch rekentoets explicitly does not test mastery of algorithmic arithmetic, and does not seem to take mental arithmetic seriously (no speed limit imposed, small number of items).

IC) The accusation that testing for ‘math use in situations of daily life’ really is testing for differences in intellectual capabilities (IC) is not the same as claiming that these tests will correlate highly with standardized intelligence tests. Let us say instead that, f.e., PISA Math is a funny IQ-test. One example: working memory capacity as an intellectual capability is not directly tested in most standardized IQ tests, yet it is evidently the case that differences in working memory capacity are influential in results on contextual math questions of the PISA-type.

IH) Is this a special class: the context presented may inhibit (IH) inference to information in the possession of the student. Not exactly the same as the transfer problem. And what is the relation to strategy training on improving access to knowledge?

• Kenneth L. Watson & Michael J. Lawson (1995). Improving access to knowledge: the effect of strategy training for question answering in high school geography. British Journal of Educational Psychology, 65, 97-112. abstract

WP). Of course, word problems (WP) have always been used in math education. Lots of research (see here), yet strangely irrelevant to using word problems in high stakes testing (exit exams) situations. How is that?

BUS). The busing problem (BUS). Highly popular among reform educationalists, especially so in the Netherlands. 36 passenger places in every bus. How many buses are needed for 1128 passengers to Mickey Mouse Land? What do you count as a correct answer? In Dutch: see here. [It is merely an authoritarian convention, and not mathematics, to only count the integer number ‘32’ as a correct answer]. In NAEP it is not damaging to students. In the Netherlands, students will have to answer questions like this one in exit exams, very high stakes to them.

• Alan H. Schoenfeld (2007). What Is Mathematical Proficiency and How Can It Be Assessed? In Alan H. Schoenfeld (Ed.) (2007). Assessing mathematical proficiency (59-73). Cambridge University Press. p. 69-70. pdf

PP). Are there any positive points (PP) at all?

D1) What is the demarcation between funny math problems and true math problems? Barry Garelick gives a hint I will have to follow up. I call this #1, because there are more questions of demarcation (such as Thijssen De Examenidioot on denksommen (what might be an English equivalent?) [Treffers, last note in his last book, on denksommen in rekentoets],

• Barry Garelick (2010). The separate path and the well traveled road. webpage

TC) True constructs (TC) are important in psychological testing as well as in examinations [Dutch: constructzuiverheid] (do not use calculators in arithmetics tests; distinguish between math modeling - math calculation - interpretation of results; beware of cognitive overload)

• f.e. this random find (interesting for its references mainly): Benjamin Euen, Heike Wendt & Wilfried Bos (2015). Do TIMSS and PIRLS measure only a single general cognitive ability? ECER EERA web page

LC). Using new kinds of problem situations in fact will test students on learning capabilities (LC) (among other things). This observation follows from Stellan Ohlsson’s 2011 theory of learning: starting with the abstract rule, specializing that rule to fit concrete situations. On the difference between known and new kinds of problem situations see also the dissertation by Anton J. H. Boonen (2015). Comprehend, visualize & calculate. Solving mathematical word problems in contemporary math education. Amsterdam: Vrije Universiteit. I take issue with Boonen on the difference he makes.

GP). Dutch educators, f.e. Anne van Streun and the math test-3F team (Victor Schmidt, pres.), define ‘math in everyday situations’ as problem solving requiring a problem solving scheme attributed to George Polya (GP). This might be a huge misconception. It is not at all clear to Artificial Intelligence scientists like Allen Newell, student of Polya, what exactly is the theory of Polya. See Newell 1983 Technical Report version is free access; scan 21 Mb of the book chapter.

RT). Retrieval (RT) of relevant information, even if well known, might be problematic.

• Greg A. Perfetto, John D. Bransford and Jeffery J. Franks (1983). Constraints on access in a problem-solving context. Memory & Cognition, 11, 24-31. pdf. Cited 207 times.

The general point is: testing for problem solving in examinations is tricky business if the relevant productions [for this technical term see the literature based on the work of Allen Newell & Herbert A. Simon] have not been specifically exercised. For Dutch readers: Wilbrink Toetsvragen ontwerpen, hoofdstuk Problemen stellen

• Finally, the finding that access to relevant information is strongly influenced by whether one is informed or uninformed has important implications for educational practice. Many classroom tests designed to assess learning seem to be episodic in nature. That is, students know that the correct answers to problems were provided by a particular professor, set of readings, and so forth. They are therefore informed about possible sources of information that are relevant to the test question they receive. Imagine that a student performs very well on a test such as this yet, when leaving the classroom, confronts everyday problems in which information from the course would be useful. Will the student spontaneously utilize this information? The results of the present experiments suggest that the answer may weIl be no. It seems clear that theories of learning and instruction must address the question of how information can become accessible under conditions in which students are not explicitly informed about the particular acquisition context that is relevant to the problems they confront.

  Perfetto, Bransford, Franks, 1983, cs p. 31

Bransford and Franks here suggest that transfer is an important problem in the world as we know it, as well as in learning theory. I do not think so: in vocational situations the problems typically are of very well known kinds. The same will be true of problems in daily life (grocery shopping etcetera). The Bransford and Franks argument might well be false, as it has always been (see the Spencer quote at the top of this web page). Then the question becomes: what might the demarcation be between typical and a-typical problem situations?

WA). There is somewhat more than a familiy resemblance with applied math as implemented in, f.e., the Dutch courses on Wiskunde A (WA). The ideology behind them is roughly the same, in the Netherlands identified with the Freudenthal Institute (Realistic Math Education also called Reform Math Education, RME). The program for secondary education started in 1985. An important but today mostly forgotten document is: Cito-werkgroep (not dated, about 1987). Wiskunde A: doelgericht toetsen. Leerdoelen en voorbeeldopgaven verzorgd door het Cito. isbn 9001186319 [not online, in Dutch only]. It explains how Cito [the Dutch Educational Testing Service] expects to handle the many problems involved in designing exam questions on this ‘applied math’.

PA). How can students master their ‘math in stituations of daily life’? By practising on a lot of context math problems? That is probably a very inefficient way. The parallel here is with learning to write. Scardamalia 1981 ‘How children cope with the cognitive demands of writing’, p. 100-101:

• Marlene Scardamalia: 'How Children Cope with the Cognitive Demands of Writing' in Carl H. Frederiksen and Joseph F. Dominic (Eds.) (1981). Writing: The nature, development, and teaching of written communication. Volume 2: Process, development and communication. Erlbaum. isbn 0898591589.

AMB). Paper tests are highly artificial. How could they ever be valid tests of the propensity or capability of using math in everyday situations? The information available to the candidates is wildly different from that what they encounter in those supposedly mathematical daily life situations. One of the problems is: in daily life situations come without necessarily being labelled as problems to be solved mathematically. It is difficult to say of situations in daily life that the information available is ambiguous: after all, it is the only information there is, one has to cope with that. The test situation is totally different: it is artificial, and made up by item writers wanting you, the testee, to see the available information in a particular way, not in other ways. A good working hypothesis now is: context problems (such as PISA Math items) inherently are ambiguous (AMB) to testees (and analysts such as your reporter ;-). Can one fill a 70 L wheelbarrow with exactly 70 L sand? The Dutch education secretary Sander Dekker told our Parliament he can! Not just once, but even 65 times in a row! The prediction therefore is: ambiguities will be revealed by letting testees report (in one way or another, from verbal report, scratch paper, eye tracking (Gerdineke van Silfhout, dissertation; site), to fMRI-scan) on their attack on individual problems. Not every possible ambiguity is fatal for the valid use of the item in exams or other high stakes tests, of course. Failing an exam on a score just below the cutting point, however, is exactly the kind of situation where any ambiguity at all is unacceptable. The criterion: Court of Law (Dutch study by Job Cohen, on the rights of students in higher education in the Netherlands. By the way, Job Cohen also was mayor of Amsterdam).

AC). The testpsychology of ‘Assessment Centers’ (AC) is exactly the model that is relevant to the development of math tests (such as PISA Math) that pretend to be valid representations or at least valid predictors of behaviour in situations in daily life being in one way or another characteristically mathematical.

• chapter 7 in the (Dutch) report http://benwilbrink.nl/publicaties/90SelectieNPA.htm. English info: https://en.wikipedia.org/wiki/Assessment_center; the literature is tricky: most of it is biased towards money making with AC’s, almost no AC’s being developed according to valid methodological rules.

SIT). There are situations, and situations. Situations one of a kind, and situations of different kinds. A situation here is, of course, a problem situation. Don’t worry, the semantics of reform math is full of surprises. A characteristic of valid examinations is that they test for mastery of the kinds of problem situations that students have been exercising in the past few weeks, months or years. A characteristic of psychological tests, such as intelligence tests, is that they confront the testees with problem situations that supposedly are entirely new to them — or thoroughly and equally well known to all testees. See also Job Cohen 1981 on the crucial distinction between same kind and new kind of problem situation. Where does PISA Math stand? Or the exams based on the idea of testing for use of math in situations of daily life? At issue is that the reform math literature does not recognize any distinction here, at least as far as I know that literature, or it must be that the ideal is to present exclusively new kinds op problem situations.

• In Dutch: the distinction between summative tests in education, and psychological tests Wilbrink 1986.

  See also the dissertation of Anton Boonen on the theme of new kinds of contexts in math problems.

QL). Surely knowing one’s math must impact on quality of life (QL). This requires longitudinal cohort research, and it has been done!

• Reyna and others (2009). How Numeracy Influences Risk Comprehension and Medical Decision Making. Psychological Bulletin, 135, 943-973. Open access http://goo.gl/JuJ1zD ]

EC). Education itself is an important part of daily life. Educational careers (EC) surely will depend on the quality of math education in the early years.

• Siegler and others http://www.psy.cmu.edu/~siegler/Siegler-etal-inpressPsySci.pdf

S). People typically do not tend to solve their problems in daily life in rational ways, but so as to get preliminary answers that are perfectly satisfactory to them. Herbert Simon called it ‘satisficing’ (S).

• Herbert A. Simon, in: Rational choice and the structure of the environment. Psychological Review, 1956, 63, 129-138 (reprinted in his Models of thought and in Models of man, social and rational. Wiley, 1957.

DM). In general: daily decision making (DM) is not rational [Daniel Kahneman Thinking fast and slow. ] Expect also school children to behave ‘not rational’ in the sense of Simon, or of Kahneman. F.e., do not expect students to use shortcuts in solving arithmetics problems if straight calculation surely will give the right answer [a.o. research by Joke Torbeyns]

EXP). Expertness (EXP). Typically the math needed on a daily basis in vocational and daily life will eventually be mastered to a high degree of expertise.

• Anders K. Ericsson, part of the Anderson, Reder, Simon chapter http://goo.gl/6ULfY4 ; but also Jean Lave info favorite of situationist thinkers about math education.

VT). vocational training (VT) includes training in the specific math needed in that vocation. Pick up any course book for technical vocations (I favour the old ones, for flight mechanics in the early forties, or the paper printing industry). In reform math countries/provinces vocational institutions (nursing, teaching, administration) nowadays will have a very difficult time bringing their students’ arithmetics fluency up to the level that minimally is required for the responsible professional.

H). Historical (H): Dutch master van Pelt systematically took inventories of the kind of math being used by housewives, masons, carpenters, etcetera, exercised those with his grade six pupils, and took care every pupil left school with a copybook of those exercises, for later consultation. [see the first pages in Van Pelt 1903 (in Dutch) pdf] What can we, a century later, still learn from him?

CM). Cost of mistakes (CM). Lack of number sense may cost one dearly. In the Netherlands a nurse was convicted to prison sentence for wrongful death. She mistook the prescription of 0.30 mg for ten times 0.3 mg; the mistake proved fatal to her patient. Also in the Netherlands prime minister Mark Rutte did not remember correctly, after returning from Brussels, a critical amount of 90 billion euro, instead reporting it to be 50 billion. The question now becoming: are there systematic surveys of number sense failures and their incurred costs? For example: the number of mistakes made by nurses in their daily routine: according to a Dutch investigation that number is very high, and it can be reduced significantly by pharmacists taking over critical calculations in medication preparation. Nursing is typically the kind of vocation that education should adequately prepare the students for, regarding number sense and language mastery.

PC). ‘Math in daily life’ is a poor concept (PC). At the very least one should distinguish 1) daily use such as calculations/estimations of time, distances, amounts; 2) the rare occasion some significant decision has to be made, based on calculations: booking a vacation, buying insurance, renting a house; 3) the routine but possibly complex math in vocational practice. The first can be researched, f.e., with techniques used by Mihaly Csikszentmihalyi in his research of flow experience. The second can be researched in the psychological lab, or simply in real life situations. The third should be quite interesting to the math researcher: in many vocational situations correct calculations are crucial, mistakes might come at a cost, so there must an extensive literature covering math use in vocational situations. I’ll have to search for that literature (ergonomics, for example?).

MT). Allocating students to different math tracks (MT) on the premise of the kind of vocation they will probably have in ten to twenty years is highly problematic morally. This is at issue in the Netherlands, where time and again different tracks have been created for the mathematically strong versus weak students. The weaker tracks emphasize using math in problem situations of daily life, the stronger tracks emphasizing using math is math type problems.

SC). Instead of seeking the fundaments in what is known scientifically, protagonists of ‘math in daily situations’ rely on two somewhat antagonistic ideologies: constructivism, and the more recent situationism. [ Anderson, Reder and Simon sounded alarm bells on this development at the end of last century, f.e. http://goo.gl/12hnpD ]

TR). The idea of transfer (TR) lies at the basis of much (fuzzy psychological) thinking in reform math. The basic (but naive) observation is that many people do not seem to use their knowledge of math in situations where it evidently would be applicable. We know already, from the above, that human decision making need not be rational in the school sense; it is no wonder people do not take the trouble to calculate their options if a satisficing approach already solved their decision problem. In psychology itself transfer is not quite well understood. Stellan Ohlsson (2011, Deep Learning. How the Mind Overrides Experience) thinks transfer is nothing else but learning itself, and I think he has strong arguments for his position.

AM). Anecdotal materials (AM), such as this Albertan piece, the kind of stuff one hears at birthday parties, from employers, from desperate parents, about their Johnny not knowing any more how to add. Surely Johnny, if he doesn’t now know how to mentally add, won’t be able to mentally add in daily life, during his lifetime (what would be the cost of that?). Even assuming these anecdotes to be valid, they have a signaling value only. Some cases might count as critical incidents: if valid, they are proof of something being seriously amiss. For example: a teacher declaring, for national television (the Netherlands: Brandpunt), ‘In this school I am forbidden to teach long division’.

RE). Reverse Engineering (RE): given test or exam questions presumably testing for the use of math in situations of daily life, is it possible that the situation imagined truly is a situation that occurs in real life daily, or once in a lifetime, as well as a situation that education should prepare its students for? Most items from PISA Math and comparable tests such as the Dutch rekentoets are fantastic, contrived, trivial, not mathematic at all; meaning there is no real world corresponding to most of these contrived math test items.

CLT). Another approach will be along the lines of psychological theories such as Sweller’s Cognitive Load Theory (CLT) or simply using Miller’s 7 plus or minus chunks capacity of working memory — for children even (substantially) less than 7 chunks. Research literature: see here. It will be immediately evident that confronting students with situations they have no experience with, can not possibly be valid for the use of math in that kind of situation later in life. Etcetera.

E). The E of Exercise. The amazing observation of the outsider in the math education debate: reform protagonists really do think investing in fluency in the basic algorithms of arithmetics is wasting time and money. I can’t remember even one protagonist acknowledging that fluency in multiplication and division of large numbers necessarily implies fluency in the basic operations on single digit numbers, and probably also otherwise a high degree of number sense. It is this number sense that is of lifetime importance regarding one’s health in the realms of life that really count: health, also financial health. The research mentioned, by Reyna and others, already hinted in that direction. More research would be very welcome.

CA). That number sense can be made less abstract by operationalising it in models of cognitive architecture (CA), such as Anderson’ ACT-R model, or Newell’s SOAR. The next step then is meticulously researching what is happening in the brain while people tackle their everyday problems involving number sense in one way or another. This kind of research already has been done by Lebiere and Anderson (a cognitive specification for solving simple algebraic equations, subsequently using fMRI-scanning to follow the brain-events from the young problem solvers), it only has to be specialised to number sense implicated in daily functioning of people.