Questions

Annotations

Ben Wilbrink

Questions always are posed in specific contexts, situations that add meaning to whatever the student interprets the question to be about, and what kind of answer she is expected to give. A good example of the enormous power of whatever is tacit, implicit, in questioning is the abuse of word problems in arithmetics courses, as researched by, among others, Verschaffel, Greer and De Corte (2000).

The sophisticated designer of achievement tests items is knowledgeable of at least the biggest dangers of completely derailed questioning.

For myself, I will be looking for possibilities to identify and characterize the most important threats to validity stemming from whatever it is that is tacit or implicit, staying tacit and implict, instead of being brought out into the open and being subjected to empirical testing.

direct hits

Robert J. Sternberg (2008b). Assessing what matters. Educational Leadership, 65, 20-26. html

Dillon, J. T. Dillon (1982). The multidisciplinary study of questioning. Journal of Educational Psychology, 74, 147-165.

Vooral interessant waar het gaat om vragen van de leraar, dus vragen waarop het antwoord al bekend is, vragen van een ander in plaats van van jezelf. Dus: Hoeveel botjes heeft een leeuw?

Hintikka, Jaakko Hintikka (2007). Socratic epistemology. Explorations of knowledge-seeking by questioning. Cambridge University Press.

regulars

Richard C. Anderson (1972). How to construct achievement tests to assess comprehension. Review of Educational Research, 42, 145-170. jstor

Was used in ch. 3 in my 1983 Toetsvragen schrijven
Nog eens teruglezen in 2010 geeft mij aanleiding tot twee opmerkingen: 1) Op p. 145 spreekt Anderson over ‘ideas get off a printed page into a person’head’. Dat is een primitieve opvatting, waarschijnlijk ook al wel in 1972. Zie voor actuele ontwikkelingen zoals in het werk van Scott Kelso (1995) (situated cognition), of Anna Sfard (2008). En 2): Er is bij Anderson geen besef dat de meeste toetsvragen, of het nu om begrijpen gaat of niet, analytisch van karakter zijn, dus uitgesproken eenzijdig (Robert Sternberg zet naast de analytische capaciteiten de creatieve, praktische, en wijze)
p. 160: Which questions to ask? Goeie vraag.
p. 163 Effects of tests on learning. "Answering test questions during or shortly after instruction has a pronounced effect on learning and retention." Rothkopf: mathemagenic behavior. "Most of the questions employed in mathemagenic research have entailed verbatim recall [my accentuation, bw]."
p. 164: Psychometric considerations. "Manipulating tests to control difficulty level and discriminating power tortures validity." Zo is het.
p. 168: "Procedures currently in use for constructing and describing achievement tests are a mess. Conclusions about methods, variables, or procedures can hardly be taken seriously when you don't know what the test measures. Drastic action is indicated. Journal editors are admonished forthwith to reject papers unless they contain 1) a documented rationale for selecting the questions to be asked and 2) a fully-explicated analysis of the relationship between the questions and the preceeding instruction."

Lieven Verschaffel, Brian Greer and Erik de Corte (2000). Making sense of word problems. Lisse: Swets & Zeitlinger.

Zie de wordproblems.htm pagina voor aantekeningen

L. M. Reder (1988). Strategic control of retrieval strategies. In G. H. Bower: The psychology of learning and motivation. volume 22, 227-259. Academic Press. pdf

The designer of achievement test questions should be aware of the peculiarities of retrieval strategies for special kinds of questions, such as multiplechoce, true/false, etcetera
Of special interest here: true/false questions or yes/no questions: questioning non-existent knowledge. What happens in questions that ask whether false statements are true or false?
See also John R. Anderson, Christian Lebiere, and others (1998). The atomic components of thought. Erlbaum. p. 87 ff.

Gün R. Semin (2000). Language as a cognitive and behavioral structuring resource: question-answer exchanges. In Wolfgang Stroebe and Miles Hewstone (2000). European review of social psychology. Volume 11 pp. 75-104. Wiley.

Semin does not specialize in question-answering issues. This chapter is useful for its signalling of possible relevant publications.
Semin, G. R. (2007d). Language and social cognition. F. Strack & J. Förster (Ed.). Social Cognition – the Basis of Human Interaction. Psychology Press. preprint pdf
Semin, G. R. & Smith, E. R. (2002). Interfaces of social psychology with situated and embodied cognition. Cognitive Systems Research, 3, 385-396. preprint pdf

Carrier, & Fautsch-Patridge (1981). Levels of questions: a framework for the exploration of processing activities. ContEdPs, 6, 365-382.

Castle, W.M. (1976). Med.Educ., 10, 97-104. (Over negatieve formuleringen. p. 104: "... in conversation Africans and Europeans intuitively answer negative questions differently. If you ask a European patient 'Are you not well' he would answer 'Yes', meaning 'you are right, I am not well.' A sick African patient asked 'Are you not well?' would answer 'No', beacuse he feels sick. One African student, before we became aware of the problem, scored 43% in a paper where thirteen questions were phrased negatively."

Stephen I. Brown and Marion I. Walter (2005³). The Art of Problem Posing. Erlbaum. questia

Turning the tables on designing questions: shift control to the students. Problem posing as learning. The field here is mathematics.

Gall, M.D. (1971). The use of questions in teaching. RER, 41, 707-721. fc p. 718: This survey of research on questions over a fifty-year period reveals that the main trend has been the development of techniques to describe questions used by teachers in classroom practice. .... whereas in the past researchers have developed taxonomies to describe questions which teachers ask, they need now to develop taxonomies based on types of questions which teachers should ask.

Garcia-Mila, M. (1992). Cross-domain development of scientific reasoning. Cognition and Instruction, 9, 285-328.

Arthur C. Graesser and John B. Black (Eds) (1985). The psychology of questions. Hillsdale, New Jersey: Lawrence Erlbaum. (ao.: Arthur C. Graesser and Tamar Murachver: Symbolic procedures of question answering - Tmar Murachver, Kelly E. Murray and Arthur C. Graesser: Answering some questions about a model of question answering - Murray Singer: Mental processes of question answering - Gary M. Olson, Susan A. Duffy and Robert L. Mack: Question-asking as a component of text comprehension - Susan Kemper, Robert Estill, Nelson Otalvaro and Margaret Schadler: Questions of facts and questions of inferences - Susan R. Goldman: Inferential reasoning in and about narrative texts - Don Nix: Notes on the efficacy of questioning - Steven P. Shwartz and Wendy G. Lehnert: Data base querying by computer)

Don Nix (1985). Notes on the efficacy of questioning. In Arthur C. Graesser and John B. Black (Eds) (1985). The psychology of questions. Hillsdale, New Jersey: Lawrence Erlbaum. p. 297: "This chapter focuses on the use of questioning techniques for the purpose of directly teaching inferential reading comprehension and meta-comprehension skills to children in classroom settings. (...) It is assumed that inferential comprehension is a complex process: the child must activily transform what, on the page, is a string of symbols into an inferentially integrated network of meaning. The nature of classroom questioning is viewed in terms of what impact it can have on a child's ability to perform this complex process." (...) A pervasive quality of questions is that the ones spontaneously asked by teachers tend to be 'detail' questions (e.g. Guzak, 1968). That is, in order to answer the question it is not necessary to inferentially integrate various parts of the text. Op p. 298 een interessante uiteenzetting over vragen op een (te) hoog abstractieniveau (Nix noemt het niet zo). If a teacher is asked to generate a good comprehension question about this text, and is not allowed to generate a question which simply requires an answer overtly stated therein, the teacher may generate a 'main idea' question, that is, 'What is the main iedea in this passage?' Clearly, a student can not simply retell a specific proposition. Instead, it is necessary to inferentially integrate several propositions along with what one already knows about the world. The difficulty is that the procedure for doing such an integration is ineffable, the concept of a main idea is not itself defined in a teachably explicit manner. The teacher does not have access to a set of components which can be explained and taught in a step-by-step manner, and which together constitute main 'ideaness.' Thus, regardless of whether or not the question is answered adequately; the evaluatio of the answer can not be made in terms that explicitly teach the student how to answer such questions in the future. The same difficulty exists with other typical classroom conceptualizations for inferential questions. Most of the concepts are 'specific skills' (Davis, 1968) such as: main idea, sequence, author's intent, characterization, and predicting outcome. These notions are not explicitly defined so that given text instances can be designated as one or the other (Nix, 1983a). Teachable rules for answering such questions are difficult to specify, in part, because the question types are not actually defined, and in aprt, because given text instances can often be classified simultaneously as more than one type. Nix beschrijft dan zijn techniek voor het genereren van inferential questions, gebaseerd op wat hij 'links' noemt, maar dat komt neer op de relaties in een schema van de betreffende tekst.

Robert M. Gagné (1976). The content analysis of subject-matter: a dialogue between Robert M. Gangé and M. David merrill. The computer as an aid in the design of crietrion-referenced tests.Instructional Science, 5, 1-28.

Basically the terrain covered here is the same as in Klausmeier, Markle and Tiemann, Merrill and Tennyson.

The article does not seem to be referred to in more recent literature.

I should study this article again, there are a lot of rather naive ideas in it, ideas that on a big scale are being made use of until this day. Such as designing items using a list of verbs. And more of this dangerous nonsense. More friendly formulated: Gagné tries to develop design rules that do not depend—not too much, at least—on the essential content of what has been taught (and hopefully also learned by students).

The stuff is probably the same as that in the book by Gagné and Briggs.

Some letters between Gagné and Merrill are printed here also. It is a delight to read the one by Merrill.

Glass, A. L., Holyoak, K. J., & Santa, J. L. (1979). Cognition. Amsterdam: Addison-Wesley.

Chapter 10: The structure of categories. I used this in 1980 to help design and formulate the contents of chapter 3 of 'Test item design.'
How categories are defined - Category representation: The structure of color categories, Defining attributes versus prototypes, How categories are learned - Natural categorization: Basic level categories, Abstraction and categorization, How to represent category structure

Graesser, A. C., & Franklin, S. P. (1990). QUEST: a cognitive model for question answering. Discourse Processes, 13, 279-304.

Arthur C. Graesser and natalie K. Person (1994). Question asking during tutoring. American Educational research Journal, 31, 104-137.

from the abstract: "Students partially self-regulated their learning by identifying knowledge deficits and asking questions to repair them, but they need training to improve these skills. We identified some ways that tutors and teachers might improve their question-asking skills."

Arthur C. Graesser, Murray Singer and Tom Trabasso (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371-395.

from the abstract: "The distinctive assumptions of the constructionist theory embrace a principle of search (or effort) after meaning. According to this principle, readers attempt to construct a meaning representation that addresses the reader's goals, that is coherent at both local and global levels, and that explains why actions, events, and states are mentioned in the text. This study reviews empirical evidence that addresses this theory and contrasts it with alternative theoretical frameworks."

Arthur C. Graesser, Peter Wiemer-Hastings, and Katja Wiemer-Hastings (2001). Constructing Inferences and Relations during Text Comprehension. In Sanders, Schilperoord, Spooren: Text representation: linguistic and psycholinguistic aspects. http://mnemosyne.csl.psyc.memphis.edu/trg/papers/artspdfs/gwhwh2.pdf [dead link? 1-2009]

Brian Greer (1997). Modelling reality in mathematics classrooms: the case of word problems. Learning and Instruction, 7, 293-307.

H. Groen en H. Kreeft (1992). Een voorstel voor kwaliteitscriteria voor centrale examens in het voortgezet onderwijs. Arnhem: Cito, interne examennotitie.

Een soort interne standards. Is over bijvoorbeeld bias heel kort, veel te kort gezien het enorme belang ervan (in 2007 een proefschrift erover, Tamara van Schilt-Mol). Korte literatuurlijst, vooral Cito-stukken, en Wilbrink's Toetsvragen schrijven.

Christiaan Hamaker (1986). The effect of adjunct questions on prose learning. Review of Educational Research, 56, 212-242.

from the abstract: "When compared to factual adjunct questions, higher order adjunct questions lead to improved performance on repeated, related, and unrelated higher order test questions, and possibly also on unrelated factual test questions. This indicates that higher order adjunct questions may have a more general facilitative effect than factual adjunct questions. "
Dit is m.i. toch een heel mooi resultaat uit een onderzoekliteratuur die verwant is aan het thema dat het laten uitleggen en verklaren leidt tot betere onderwijsresultaten vergeleken met docenten die hier geen gewoonte van maken. Ik zal het review-artikel van Christiaan toch nog eens op de relevante details na moeten lezen. Zoekend op de titel van het artikel, geeft Google eigenlijk niets relevants aan. Dat is teleurstellend.
Christiaan Hamaker (1984). The use of adjunct questions in educational texts. A review of research literature and six experiments on the effects of adjunct application questions. Proefschrift UvA

Hamilton, E.R.H. (1929). The art of interrogation. (Austin, 1949, verwijst daarnaar, op p. 160; ik ken het boek niet).

Hiz, H. (ed.) (1979). Questions. Dordrecht: Reidel.

Hoffman, R. R., Shadbolt, N. R., Burton, A. M., & Klein, G. (1995). Eliciting knowledge from experts: a methodological analysis. OrgBehHumDecProc 62, 129-158. fc

Jardine, N. (20002). The scenes of inquiry. On the reality of questions in the sciences. Oxford: Clarendon Press.

Jennison, B., & Ogborn, J. (eds.) (1994). Wonder and delight. Essays in science education in honour of the life and work of Eric Rogers 1902-1990. Bristol: Institute of Physics publishing. '96 A wondeful book with anecdotes and exemplars, also of exam questions and exam questioning practice. N. Joel ‘Shredders’ 183-186. p. 184: typically, a working session [in a two week seminar with teachers] would open with Eric Rogers inviting one of the participating teachers to propose a physics question. Other members of the group were then encouraged to discuss or challenge its validity or its usefulness. What is the purpose of this question? Does it really do what you intend it to do? How could this question or problem be rephrased to make it more useful? What learning objectives would it contribute to? How would you use it as part of your teacing? What broader teaching strategy would this imply? Perhaps an experimental investigation on the part of the student? And so on. The author of the problem would generally first defend it against points raised by others, but would gradually come to accept criticisms, modificatons and further developments of it by the group. It was this tearing apart of the questions that led Eric Rogers to give the name ‘shredder’ to this sor of workshop.

Johnson-Laird, P. N., & Byrne, R. J. M. (1991). Deduction. London: Lawrence Erlbaum. '97 Laat zien hoe moeilijk deductieve opgaven zijn waarbij de inhoud irrelevant is. Ipso: toetsvragen die deze vorm kiezen, meten dus heel sterk ook iets anders dan alleen inhoud.

Jos Kessels, Ad van der Kam en Jan Tollenaar (1989). De zaak Arlet; inleiding in de kennistheorie + Handleiding voor de docent. Meppel: Boom.

King, Alison (1994). Guiding knowledge construction in the classroom: effects of teaching children how to question how to explain. AERJ, 31, 338-369. gezien, aardig artikel

Knowles, D. (Ed.) (1990). Explanation and its limits. Cambridge: Cambridge University Press.

Kotovsky, K., & H.A. Simon, What makes some problems really hard: explorations in the problem space of difficulty. Cognitive Psychology, 1990, 22, 143-183.

Ladas (1973). Mathemagenic effects of factual review questions on learning of incidental information. RER, 43, 71-82.

Wendy G. Lehnert (1978). The process of queston answering. A computer simulation of cognition. Erlbaum.

Lundeberg, M. A., & Fox, P. W. (1991). Do laboratory findings on test expectancy generalize to classroom outcomes? RER 61, 94-106. "We conducted a meta-analysis of both classroom and laboratory studies of the effects of expecting a recall, recognition, essay, multiple-choice or true-false test on students' subsequent achievement. In laboratory studies, studying with a recall set produced strong positive effect sizes for both discrete and prose materials. However, studying with a recognition set produced no effects with discrete materials and small negative effects with prose materials. In contrast, results from classroom studies indicated that students achieved most when preparing for the test they received. (...)"

Geoff Martz, John Katzman and Adam Robinson (1993). Cracking the GMAT. Princeton Review. New York: Villard Books.

This is one of a multitude of books from the industry 'How to beat the test by becoming test-wise?'

Medin, D. L., Lynch, E. B., & Coley, J. D. (1997). Categorization and reasoning among three experts: Do all roads lead to Rome? Cognitive Psychology, 32, 49-96. geen kopie. "To what degree do conceptual systems reflect universal patterns of featural covariation in de world (similarity) or universal organizing principles of mind, and to what degree do they reflect specific goals, theories and beliefs of the categorizer?" Leuke vraag: wanneer de stof waar de leerling verondersteld wordt greep op te krijgen, voor de leerling een heel andere betekenis heeft dan voor de leraar, zou er wel eens een behoorlijk probleem kunnen zijn.

Hans van der Meij (1994). Student questioning: a componential analysis. Learning and individual differences, 6 (2). pp. 137-161. pdf

abstract The focus of this review is on spontaneous student questioning. Questioning is depicted from start to finish, from the onset of perplexity until the learning that may result. In addition, the characteristic features of questions are described, the assumptions that underlie them, and the answers that follow. Special attention will be given to individual differences in question asking per se and to individual differences in general constructs such as verbal ability and self-esteem that are related to question asking. All of these ideas will be presented within the general framework of the (slightly modified) componential model of questioning of Dillon (1988a, 1990).

Janine Swaak and Ton de Jong (1996). Measuring intuitive knowledge in science: The development of the what-if test. Studies in Educational Evaluation, 22, 341-362. pdf

King, Alison (1994). Guiding knowledge construction in the classroom: effects of teaching children how to question how to explain. AERJ, 31, 338-369. gezien, aardig artikel

logic and linguistics

Nuel D. Belnap Jr. and Thomas B. Steel Jr. (1976). The logic of questions and answers. London: Yale University Press. (question logic - question-answering systems - formal languages - linguistics - psychology and education - erotetic semantics) (Contains: Bibliography of the theory of questions and answers, by Urs Egli and Hubert Schleichert (annotated)) ("This book takes as its goal the creation of a formal language that can be used to ask and answer questions in an orderly, fruitful way." "A work of interest to logicians and philosophers of language, The logic of questions and answers has its broadest potential use in the applications of the theory to computer question-answering systems")

R. D. Hull: A semantics for superficial and embedded questions in natural language. In Edward L. Keenan: Formal semantics of natural language (p. 35-45). Cambridge University Press.

Refers C. L. Baker (1968). Indirect questions in English. thesis University of Illinois.

related on my website

Designing achievement test items [in Dutch, examples pages in English] chapter 1

Validity (of tests, of test items, therefore also validity of questions) html See also the sitemap