Ben Wilbrink

Questions always are posed in specific contexts, situations that add meaning to whatever the student interprets the question to be about, and what kind of answer she is expected to give. A good example of the enormous power of whatever is tacit, implicit, in questioning is the abuse of word problems in arithmetics courses, as researched by, among others, Verschaffel, Greer and De Corte (2000).

The sophisticated designer of achievement tests items is knowledgeable of at least the biggest dangers of completely derailed questioning.

For myself, I will be looking for possibilities to identify and characterize the most important threats to validity stemming from whatever it is that is tacit or implicit, staying tacit and implict, instead of being brought out into the open and being subjected to empirical testing.

direct hits

Robert J. Sternberg (2008b). Assessing what matters. Educational Leadership, 65, 20-26. html

Dillon, J. T. Dillon (1982). The multidisciplinary study of questioning. Journal of Educational Psychology, 74, 147-165.

Hintikka, Jaakko Hintikka (2007). Socratic epistemology. Explorations of knowledge-seeking by questioning. Cambridge University Press.


Richard C. Anderson (1972). How to construct achievement tests to assess comprehension. Review of Educational Research, 42, 145-170. jstor

Lieven Verschaffel, Brian Greer and Erik de Corte (2000). Making sense of word problems. Lisse: Swets & Zeitlinger.

L. M. Reder (1988). Strategic control of retrieval strategies. In G. H. Bower: The psychology of learning and motivation. volume 22, 227-259. Academic Press. pdf

Gün R. Semin (2000). Language as a cognitive and behavioral structuring resource: question-answer exchanges. In Wolfgang Stroebe and Miles Hewstone (2000). European review of social psychology. Volume 11 pp. 75-104. Wiley.

Carrier, & Fautsch-Patridge (1981). Levels of questions: a framework for the exploration of processing activities. ContEdPs, 6, 365-382.

Castle, W.M. (1976). Med.Educ., 10, 97-104. (Over negatieve formuleringen. p. 104: "... in conversation Africans and Europeans intuitively answer negative questions differently. If you ask a European patient 'Are you not well' he would answer 'Yes', meaning 'you are right, I am not well.' A sick African patient asked 'Are you not well?' would answer 'No', beacuse he feels sick. One African student, before we became aware of the problem, scored 43% in a paper where thirteen questions were phrased negatively."

Stephen I. Brown and Marion I. Walter (20053). The Art of Problem Posing. Erlbaum. questia

Gall, M.D. (1971). The use of questions in teaching. RER, 41, 707-721. fc p. 718: This survey of research on questions over a fifty-year period reveals that the main trend has been the development of techniques to describe questions used by teachers in classroom practice. .... whereas in the past researchers have developed taxonomies to describe questions which teachers ask, they need now to develop taxonomies based on types of questions which teachers should ask.

Garcia-Mila, M. (1992). Cross-domain development of scientific reasoning. Cognition and Instruction, 9, 285-328.

Arthur C. Graesser and John B. Black (Eds) (1985). The psychology of questions. Hillsdale, New Jersey: Lawrence Erlbaum. (ao.: Arthur C. Graesser and Tamar Murachver: Symbolic procedures of question answering - Tmar Murachver, Kelly E. Murray and Arthur C. Graesser: Answering some questions about a model of question answering - Murray Singer: Mental processes of question answering - Gary M. Olson, Susan A. Duffy and Robert L. Mack: Question-asking as a component of text comprehension - Susan Kemper, Robert Estill, Nelson Otalvaro and Margaret Schadler: Questions of facts and questions of inferences - Susan R. Goldman: Inferential reasoning in and about narrative texts - Don Nix: Notes on the efficacy of questioning - Steven P. Shwartz and Wendy G. Lehnert: Data base querying by computer)

Don Nix (1985). Notes on the efficacy of questioning. In Arthur C. Graesser and John B. Black (Eds) (1985). The psychology of questions. Hillsdale, New Jersey: Lawrence Erlbaum. p. 297: "This chapter focuses on the use of questioning techniques for the purpose of directly teaching inferential reading comprehension and meta-comprehension skills to children in classroom settings. (...) It is assumed that inferential comprehension is a complex process: the child must activily transform what, on the page, is a string of symbols into an inferentially integrated network of meaning. The nature of classroom questioning is viewed in terms of what impact it can have on a child's ability to perform this complex process." (...) A pervasive quality of questions is that the ones spontaneously asked by teachers tend to be 'detail' questions (e.g. Guzak, 1968). That is, in order to answer the question it is not necessary to inferentially integrate various parts of the text. Op p. 298 een interessante uiteenzetting over vragen op een (te) hoog abstractieniveau (Nix noemt het niet zo). If a teacher is asked to generate a good comprehension question about this text, and is not allowed to generate a question which simply requires an answer overtly stated therein, the teacher may generate a 'main idea' question, that is, 'What is the main iedea in this passage?' Clearly, a student can not simply retell a specific proposition. Instead, it is necessary to inferentially integrate several propositions along with what one already knows about the world. The difficulty is that the procedure for doing such an integration is ineffable, the concept of a main idea is not itself defined in a teachably explicit manner. The teacher does not have access to a set of components which can be explained and taught in a step-by-step manner, and which together constitute main 'ideaness.' Thus, regardless of whether or not the question is answered adequately; the evaluatio of the answer can not be made in terms that explicitly teach the student how to answer such questions in the future. The same difficulty exists with other typical classroom conceptualizations for inferential questions. Most of the concepts are 'specific skills' (Davis, 1968) such as: main idea, sequence, author's intent, characterization, and predicting outcome. These notions are not explicitly defined so that given text instances can be designated as one or the other (Nix, 1983a). Teachable rules for answering such questions are difficult to specify, in part, because the question types are not actually defined, and in aprt, because given text instances can often be classified simultaneously as more than one type. Nix beschrijft dan zijn techniek voor het genereren van inferential questions, gebaseerd op wat hij 'links' noemt, maar dat komt neer op de relaties in een schema van de betreffende tekst.

Robert M. Gagné (1976). The content analysis of subject-matter: a dialogue between Robert M. Gangé and M. David merrill. The computer as an aid in the design of crietrion-referenced tests.Instructional Science, 5, 1-28.

  • Basically the terrain covered here is the same as in Klausmeier, Markle and Tiemann, Merrill and Tennyson.
  • The article does not seem to be referred to in more recent literature.
  • I should study this article again, there are a lot of rather naive ideas in it, ideas that on a big scale are being made use of until this day. Such as designing items using a list of verbs. And more of this dangerous nonsense. More friendly formulated: Gagné tries to develop design rules that do not depend—not too much, at least—on the essential content of what has been taught (and hopefully also learned by students).
  • The stuff is probably the same as that in the book by Gagné and Briggs.
  • Some letters between Gagné and Merrill are printed here also. It is a delight to read the one by Merrill.
  • Glass, A. L., Holyoak, K. J., & Santa, J. L. (1979). Cognition. Amsterdam: Addison-Wesley.

    Graesser, A. C., & Franklin, S. P. (1990). QUEST: a cognitive model for question answering. Discourse Processes, 13, 279-304.

    Arthur C. Graesser and natalie K. Person (1994). Question asking during tutoring. American Educational research Journal, 31, 104-137.

    Arthur C. Graesser, Murray Singer and Tom Trabasso (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371-395.

    Arthur C. Graesser, Peter Wiemer-Hastings, and Katja Wiemer-Hastings (2001). Constructing Inferences and Relations during Text Comprehension. In Sanders, Schilperoord, Spooren: Text representation: linguistic and psycholinguistic aspects. [dead link? 1-2009]

    Brian Greer (1997). Modelling reality in mathematics classrooms: the case of word problems. Learning and Instruction, 7, 293-307.

    H. Groen en H. Kreeft (1992). Een voorstel voor kwaliteitscriteria voor centrale examens in het voortgezet onderwijs. Arnhem: Cito, interne examennotitie.

    Christiaan Hamaker (1986). The effect of adjunct questions on prose learning. Review of Educational Research, 56, 212-242.

    Hamilton, E.R.H. (1929). The art of interrogation. (Austin, 1949, verwijst daarnaar, op p. 160; ik ken het boek niet).

    Hiz, H. (ed.) (1979). Questions. Dordrecht: Reidel.

    Hoffman, R. R., Shadbolt, N. R., Burton, A. M., & Klein, G. (1995). Eliciting knowledge from experts: a methodological analysis. OrgBehHumDecProc 62, 129-158. fc

    Jardine, N. (20002). The scenes of inquiry. On the reality of questions in the sciences. Oxford: Clarendon Press.

    Jennison, B., & Ogborn, J. (eds.) (1994). Wonder and delight. Essays in science education in honour of the life and work of Eric Rogers 1902-1990. Bristol: Institute of Physics publishing. '96 A wondeful book with anecdotes and exemplars, also of exam questions and exam questioning practice. N. Joel ‘Shredders’ 183-186. p. 184: typically, a working session [in a two week seminar with teachers] would open with Eric Rogers inviting one of the participating teachers to propose a physics question. Other members of the group were then encouraged to discuss or challenge its validity or its usefulness. What is the purpose of this question? Does it really do what you intend it to do? How could this question or problem be rephrased to make it more useful? What learning objectives would it contribute to? How would you use it as part of your teacing? What broader teaching strategy would this imply? Perhaps an experimental investigation on the part of the student? And so on. The author of the problem would generally first defend it against points raised by others, but would gradually come to accept criticisms, modificatons and further developments of it by the group. It was this tearing apart of the questions that led Eric Rogers to give the name ‘shredder’ to this sor of workshop.

    Johnson-Laird, P. N., & Byrne, R. J. M. (1991). Deduction. London: Lawrence Erlbaum. '97 Laat zien hoe moeilijk deductieve opgaven zijn waarbij de inhoud irrelevant is. Ipso: toetsvragen die deze vorm kiezen, meten dus heel sterk ook iets anders dan alleen inhoud.

    Jos Kessels, Ad van der Kam en Jan Tollenaar (1989). De zaak Arlet; inleiding in de kennistheorie + Handleiding voor de docent. Meppel: Boom.

    King, Alison (1994). Guiding knowledge construction in the classroom: effects of teaching children how to question how to explain. AERJ, 31, 338-369. gezien, aardig artikel

    Knowles, D. (Ed.) (1990). Explanation and its limits. Cambridge: Cambridge University Press.

    Kotovsky, K., & H.A. Simon, What makes some problems really hard: explorations in the problem space of difficulty. Cognitive Psychology, 1990, 22, 143-183.

    Ladas (1973). Mathemagenic effects of factual review questions on learning of incidental information. RER, 43, 71-82.

    Wendy G. Lehnert (1978). The process of queston answering. A computer simulation of cognition. Erlbaum.

    Lundeberg, M. A., & Fox, P. W. (1991). Do laboratory findings on test expectancy generalize to classroom outcomes? RER 61, 94-106. "We conducted a meta-analysis of both classroom and laboratory studies of the effects of expecting a recall, recognition, essay, multiple-choice or true-false test on students' subsequent achievement. In laboratory studies, studying with a recall set produced strong positive effect sizes for both discrete and prose materials. However, studying with a recognition set produced no effects with discrete materials and small negative effects with prose materials. In contrast, results from classroom studies indicated that students achieved most when preparing for the test they received. (...)"

    Geoff Martz, John Katzman and Adam Robinson (1993). Cracking the GMAT. Princeton Review. New York: Villard Books.

    Medin, D. L., Lynch, E. B., & Coley, J. D. (1997). Categorization and reasoning among three experts: Do all roads lead to Rome? Cognitive Psychology, 32, 49-96. geen kopie. "To what degree do conceptual systems reflect universal patterns of featural covariation in de world (similarity) or universal organizing principles of mind, and to what degree do they reflect specific goals, theories and beliefs of the categorizer?" Leuke vraag: wanneer de stof waar de leerling verondersteld wordt greep op te krijgen, voor de leerling een heel andere betekenis heeft dan voor de leraar, zou er wel eens een behoorlijk probleem kunnen zijn.

    Hans van der Meij (1994). Student questioning: a componential analysis. Learning and individual differences, 6 (2). pp. 137-161. pdf

    Janine Swaak and Ton de Jong (1996). Measuring intuitive knowledge in science: The development of the what-if test. Studies in Educational Evaluation, 22, 341-362. pdf

    King, Alison (1994). Guiding knowledge construction in the classroom: effects of teaching children how to question how to explain. AERJ, 31, 338-369. gezien, aardig artikel

    logic and linguistics

    Nuel D. Belnap Jr. and Thomas B. Steel Jr. (1976). The logic of questions and answers. London: Yale University Press. (question logic - question-answering systems - formal languages - linguistics - psychology and education - erotetic semantics) (Contains: Bibliography of the theory of questions and answers, by Urs Egli and Hubert Schleichert (annotated)) ("This book takes as its goal the creation of a formal language that can be used to ask and answer questions in an orderly, fruitful way." "A work of interest to logicians and philosophers of language, The logic of questions and answers has its broadest potential use in the applications of the theory to computer question-answering systems")

    R. D. Hull: A semantics for superficial and embedded questions in natural language. In Edward L. Keenan: Formal semantics of natural language (p. 35-45). Cambridge University Press.

    related on my website

    Designing achievement test items [in Dutch, examples pages in English] chapter 1

    Validity (of tests, of test items, therefore also validity of questions) html See also the sitemap


    Questioning Exchange: A Multidisciplinary Review.

    August 1, 2010 \ contact ben apenstaartje

    Valid HTML 4.01!