• thuis
• sitemap
• publicaties
• projecten
• mail

Literatuur over toetsvragen (ontwerpen)

Ben Wilbrink

Ben Wilbrink (1983/in bewerking). Toetsvragen ontwerpen html

André A. Rupp, Jonathan Templin & Robert A. Henson (2010). Diagnostic Measurement. Theory, Methods, and Applications. info [als eBook in KB] supplemental materials website

Sean H. K. Kang, Harold Pashler, Nicolas J. Cepeda, Doug Rohrer, Shana K. Carpenter & Michael C. Mozer (2011). Does incorrect guessing impair fact learning? Journal of Educational Psychology, 103, 48-59 pdf

Robert V. Lindsey, Jeff D. Shroyer, Harold Pashler & Michael C. Mozer (accepted for publication 2013). Improving students' long-term knowledge retention through personalized review. Psychological Science Hal Pashler website

Goes further than the title suggests: might present a model for adaptive/formative testing too. Might be a nice technique to study effects of changes in rules for combination of grades into and-of-course grades [Dutch: examenreglingen die meer of minder compensatorisch zijn met al dan niet vrijstellende deeltoetsen]

Yoonhee Jang, Hal Pashler & David E. Huber (submitted 2014). Manipulations of choice familiarity in multiple-choice testing support a retrieval practice account of the testing effect. Journal of Educational Psychology Hal Pashler website

• The benefit of testing upon subsequent testing is called the ‘testing effect’ — an item that is retrieved on the initial test is found to be strengthened as seen in performance on subsequent tests (e.g., Gates, 1917; Glover, 1989; Spitzer, 1939; for reviews, see Bjork, 1975; Dempster, 1996; Roediger & Karpicke, 2006).

  p. 1

  • Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, 6 (40).
  • Glover, J. A. (1989). The "testing" phenomenon: Not gone but nearly forgotten. Journal of Educational Psychology, 81, 392-399.
  • Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology, 30, 641-656.
  • Bjork, R. A. (1975). Retrieval as a memory modifier: An interpretation of negative recency and related phenomena. In R. L. Solso (Ed.), Information processing and cognition: The Loyola symposium (pp. 123-144). Hillsdale, NJ: Erlbaum.
  • Dempster, F. N. (1996). Distributing and managing the conditions of encoding and practice. In E. L. Bjork & R. A. Bjork (Eds.), Memory (pp.317-344). San Diego: Academic Press.
  • Roediger, H. L., III, & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181-210.

Stephen E. Newstead, Peter Bradon, Simon J. Handley, Ian Dennis & Jonathan St. B. T. Evans (2006). Predicting the difficulty of complex logical reasoning problems. Thinking & Reasoning, 12, 62-90. abstract

Lijkt me de moeite waard. Wie heeft een pdf voor me? [Aangevraagd via Research Gate]

David Bartholomae (1986). Words from afar. Introduction; in Anthony R. Petrosky & David Bartholomae The Teaching of Writing (1-7). NSSE, University of Chicago Press

Theo Thijssen (1929). De examen-idioot of De kinderexamens van 1928. Overdruk uit De Bode. orgaan van de Bond van Ned. Onderwijzers. Bondsdrukkerij "De Volharding". scan 24 Mb.

De examenidioot: in de titel een verbindingsstreepje.

David R. Krathwohl (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41, 212-264. pdf

Hanneke Geerlings, Wim J. van der Linden and Cees A. W. Glas (2013). Optimal Test Design With Rule-Based Item Generation. Applied Psychological Measurement, 37, 140-161. abstract

• Arendasy, M., & Sommer, M. (2007). Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items. Learning and Individual Differences, 17, 366-383. abstract
• Geerlings, H., Glas, C. A. W., & van der Linden, W. J. (2011). Modeling rule-based item generation. Psychometrika, 76, 337-359.

Hee-Sun Lee, Ou Lydia Liu & Marcia C. Linn (2011). Validating Measurement of Knowledge Integration in Science Using Multiple-Choice and Explanation Items. Applied Measurement in Education, 24, 115-136. abstract

• Knowledge integration theory reflects a constructivist view of learning and builds on extensive research on science instruction (Bransford, Brown, & Cocking, 2000; Linn, 2006; Linn, Davis, & Bell, 2004). It captures how well students generate and use evidence to connect scientifically relevant and normative ideas when explaining a scientific phenomenon or solving a scientific problem (Linn & Hsi, 2000).

  
  ( . . . ) We distinguish knowledge integration items from those designed to measure content knowledge in large scale assessments including the National Assessment of Educational Progress (NAEP, National Assessment Governing Board, 2004) and the Trends in International Mathematics and Science Study (TIMSS, see Schmidt, Raizen, Britton, Bianchi, & Wolfe, 1997). For these tests, designers use item specification tables derived from curriculum frameworks that delineate the science concepts and process skills. This approach focuses on content validity because it shows how well items represent the domain of science for assessment. Knowledge integration assessment goes beyond measuring recall of ideas or processes and focuses on how well students use ideas to explain complex phenomena.This may occur when students apply ideas learned in one context to problems in another novel context.

Thomas P. Hogan and Gavin Murphy (2007). Recommendations for Preparing and Scoring Constructed-Response Items: What the Experts Say. Applied Measurement in Education, 20, 427-441. abstract

Mark J. Gierl & Thomas M. Haladyna (Eds.) (2013). Automatic Item Generation. Theory and Practice. Routledge. site

Dorothea P. Simon (1976). Spelling - a task analysis. Instructional Science, 5, 277-302. abstract

Dit artikel is misschien historisch interessant, maar verder toch niet.

J.R. Hayes & H.A. Simon (1974). Understanding written problem instructions. In Lee W. Gregg (Ed.) (1974). Knowledge and Cognition. Erlbaum. pdf. Reprinted in Herbert A. Simon (Ed.) (1979). Models of thought. New Haven: Yale University Press.

Louis M. Hsu (1979). Ordering Power of Separate versus Grouped True-False Tests: Interaction of Type of Test with Knowledge Levels of Examinees. Applied Psychological Measurement 3, 529-536. abstract

Robert B. Frary (1980). The Effect of Misinformation, Partial Information, and Guessing on Expected Multiple-Choice Test Item Scores. Applied Psychological Measurement 4, 79-90. abstract

Henk Blok & Wim E. Saris (1983). Using Longitudinal Data to Estimate Reliability. Applied Psychological Measurement 7, 295-301.abstract

Isaac I. Bejar (1983). Subject Matter Experts' Assessment of Item Statistics. Applied Psychological Measurement, 7, 303-310.abstract

Albert C. Oosterhof & Pamela K. Coats (1984). Comparison of Difficulties and Reliabilities of Quantitative Word Problems in Completion and Multiple-Choice Item Formats. Applied Psychological Measurement, 8, 287-294.abstract

David V. Budescu (1987). Open-Ended Versus Multiple-Choice Response Formats—It Does Make a Difference for Diagnostic Purposes. Applied Psychological Measurement, 11, 385-395.abstract

• “This study was conducted to determine the degree to which subject matter experts could predict the difficulty and discrimination of items on the Test of Standard Written English. It was concluded that despite an extended training period the raters did not approach a high level of accuracy, nor were they able to pinpoint the factors that contribute to item difficulty and discrimination.”

David V. Budescu (1988). On the Feasibility of Multiple Matching Tests — Variations on a Theme by Guiliksen. Applied Psychological Measurement, 12, 5-14.abstract

Terry A. Ackerman & Philip L. Smith (1988). A Comparison of the Information Provided by Essay, Multiple-Choice, and Free-Response Writing Tests. Applied Psychological Measurement, 12, 117-128.abstract

A.H.G.S. van der Ven & F.M. Gremmen (1992). The Knowledge or Random Guessing Model for Matching Tests. Applied Psychological Measurement,16, 177-194.abstract

Menucha Birenbaum, Kikumi K. Tatsuoka & Yaffa Gutvirtz (1992). Effects of Response Format on Diagnostic Assessment of Scholastic Achievement. Applied Psychological Measurement,16, 353-363.abstract

In het geval van opgaven algebra.

Klaas Sijtsma & Anton C. Verweij (1999). Knowledge of Solution Strategies and IRT Modeling of Items for Transitive Reasoning. Applied Psychological Measurement, 23, 55-68. abstract

Onderzoek waarbij de leerlingen hun antwoorden op de toets hebben moeten motiveren. Zie ook hoofdstuk 2 van Toetsvragen ontwerpen hfdst 2.

Randy Elliot Bennett, Mary Morley & Dennis Quardt (2000). Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized. Applied Psychological Measurement, 24, 294-309. abstract

Joanna S. Gorin & Susan E. Embretson (2006). Item Difficulty Modeling of Paragraph Comprehension Items. Applied Psychological Measurement, 30, 394-411. abstract [vraagvormen, item forms]

Jimmy de la Torre (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33, 163-183. abstract

Het gebruikte voorbeeld is een breukenopgave 2 4/7 - 7/12. Het lijkt me wel interessant en van belang om te zien hoe dit ‘cognitief-psychologisch’ wordt geanalyseerd. De psychometrie van een en ander is meer arcaan van karakter, wat mij betreft met weinig realiteitswaarde. WILLIAM R. CgAWFOgD

Sarah-Jane Leslie, Sangeet Khemlani & Sam Glucksberg (2011). Do all ducks lay eggs? The generic overgeneralization effect. Journal of Memory and Language 65, 15-31. pdf

P. J. Cosijn (1915). Oefeningen bij de Nederlandsche spraakkunst. Bezorgd door R. Zuidema. De Erven F. Bohn. — negende druk 120 blz. ingenaaid Met nog een voorwoord van Cosijn, uit 1994, bij gelegenheid van een nieuwe druk (welke) die door Zuidema werd verzorgd. Opmerkelijk aan dit boek is dat het een soort cloze-test opgaven zijn, met meest aangeboden tweekeuzen (zoals tussen lange ij en korte ei). Ik zal het meteen in hoofdstuk 2 opnemen.

Ruurd Taconis, Monica Ferguson-Hessler & Gerrit Verkerk (1997). Physics problem solving and the transition from general secondary education to higher education. Tijdschrift voor Onderwijsresearch, 123-144. Bij dit onderzoek valt de vraag te stellen of problem solving skills wel bestaan, los van intelligentieverschillen, en zo ja, of deze domeinspecifiek zijn, of generiek. Een klein onderzoekje op dat laatste vraagstuk is gedaan door Veenman & Verhey 2001 Learning and Individual Differences. Maar ik moet niet zeuren. Dit artikel bevat een schat aan informatie over vaardigheden in het prbleemoplossen rond overgangsproblematiek vo-ho.

Susan E. Embretson (1998). A Cognitive Design System Approach to Generating Valid Tests. Psychological Methods, 3, 380-396. abstract

Raymond S. Nickerson (1996). Ambiguities and Unstated Assumptions in Probabilistic Reasoning. Psychological Methods, 3, 380-396.

• abstract Ostensibly simple probabilistic reasoning problems are sometimes surprisingly difficult. One source of difficulty is the omission from a problem description of information essential to an unambiguous understanding of the situation. When this is so, assumptions must be made to permit the computation of probabilities. Different assumptions lead to different conclusions, and if the assumptions are not stated, disagreements regarding problem solutions can occur, even among experts. The author reviews several such problems and their treatment in the literature. He proposes accounts of why problems can be difficult even when not ambiguous and discusses some approaches that can make solutions easier to find or understand.

Amos Tversky (1964). On the optimal number of alternatives at a choice point. Journal of Mathematical Psychology, 1, 386-391. abstract

• Optimaal aantal alternatieven bij meerkeuzevragen

Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2, 51-78.abstract

Janine Swaak & Ton de Jong, T. (1996). Measuring intuitive knowledge in science: the development of the what-if test. Studies in Educational Evaluation, 22, 341-362. pdf

Learning in discovery environments, such as simulations. Inzicht?

Robert D. Tennyson & Ok-Choon Park (1980). The teaching of concepts: a review of instructional design research literature. Review of Educational Research, 50, 55-70. abstract

N. G. Verhoeven & C. H. Beuk (mei 1983). Het construeren van meerkeuzevragen. Arnhem: Cito; 56 blz. (Algemene publikatie Nr 29)

Niet meer verkrijgbaar?

Howard Wainer, J. R. Jefferson Wadkins & Alfred Rogers (1984). Was there one distractor too many? Journal of Educational Statistics, 9, 5-24 abstract & read online free JSTOR

This incident is known as "the rolling circle problem". [Ik heb er een tweet aan gewijd.] Wainer c.s. tried to find out more about this question. Would adding the true answer to the alternatives make a difference? It did not. Meaning: this SAT item is way too difficult because of its wording. In the correct version it would not therefore be acceptable as SAT item.

• youtube
• web page Published Online: June 2, 1982 EdWeek June 2, 1982. Students Outsmart S.A.T. Again; 300,000 Tests Will Be Rescored. The College Board has begun recalculating the mathematics scores of 300,000 students who took the Scholastic Aptitude Test (sat) on May 1, after three students proved that the correct answer to one question was not among the possible answers listed on the answer sheet.

  As a result of the discovery--made separately by three New York high-school students--the College Board, the sponsor of the sat, said last week that it will rescore the mathematics section of the examination without the flawed question.

  It has also advised 3,000 colleges, which have already received some of the May 1 test scores, that new results will be sent out within 10 days.

  Officials of the College Board, which sponsors the aptitude examination that is taken by about one million high-school juniors and seniors each year, said that most scores will remain unchanged.

  But in some cases, they said, students' scores may change by up to 10 points on the tests' scale of 200 to 800.

  The question--the fourth error discovered in the sat test within the last two years--is shown below.

  In the figure above, the radius of circle A is one third the radius of circle B. Starting from the position shown in the figure, circle A will roll around circle B. At the end of how many revolutions of circle A will the center of circle A first reach its starting point?

  (A) 3/2 (B) 3 (C) 6 (D) 9/2 (E) 9

  The College Board's explanation of the error said: "The circumference of the large circle is three times the circumference of the small circle. If the small circle were to rotate along a straight line segment equal in length to the circumference of the large circle, it would make three revolutions. So the intended answer to this problem was choice (B) 3.

  "However, the motion of the small circle is not in a straight line, but rather around the large circle. This revolving action around the large circle contributes an extra revolution as circle A rolls around circle B. Thus, the answer to this question should have been 4, not 3.''

• http://mathpluspublishing.com
• nytimes

  Mathematics scores are being recalculated for 300,000 students who took Scholastic Aptitude Tests around the nation on May 1 because three students proved there was a faulty answer, the College Board said yesterday.
  
  
  The correct answer for the problem was not among the choices given. All 300,000 tests have to be rescored and recalculated, the College Board said.
  
  
  It was the third case of a flawed question on the tests taken by more than one million high school juniors and seniors each year and scored on a scale of 200 to 800. Each time, students caught the mistake.
  
  
  ''It was a human error,'' said Barrie Kelly, the College Board's executive director of communication. The board said it would not identify the students involved in reporting the mistake until permission was granted. ''The problem came to light Friday and today we sent mailgrams to 3,000 colleges advising them recalculated scores would go out to them within the next 10 days,'' Miss. Kelly said. Adjustments of About 10 Points
  
  
  Daniel B. Taylor, the College Board's executive vice president for operations, said as a result of the flawed question, he anticipated score adjustments of 10 points, up or down.
  
  
  The disputed question shows a large circle, B, and to the left of it a small circle, A, touching B. ''In the figure above,'' the problem states, ''the radius of circle A is one-third the radius of circle B. Starting from position shown in figure, circle A rolls around circle B. At the end of how many revolutions of circle A will the center of circle A first reach its starting point?''
  
  
  The choice was given among these answers: (a) 3 over 2. (b) 3. (c) 6. (d) 9 over 2. (e) 9. ''The answer to this question should have been 4, not 3, as was indicated on the answer key,'' Miss Kelly said. This explanation, proving the students right, was given: ''The circumference of the large circle is three times the circumference of the small circle. If the small circle were to rotate among a straight line segment equal in length to the circumference of the large circle, it would make three revolutions.
  
  
  ''So the intended answer to this problem was choice B, 3. However, the motion of the small circle is not in a straight line, but rather around the large circle. This revolving action around the large circle contributes an extra revolution as circle A rolls around circle B. Thus, the answer to this question should have been 4, not 3.''
  
  
  Mr. Taylor said the Educational Testing Service, which develops and administers the tests under contract to the board, would delete the flawed question and recalculate scores for the remainder of the match section of each test.

• nytimes The next time a SAT question is found wanting is 1997.
• nytimes Opinion on the rolling circle problem.

William P. Wallace (1980). On the use of distractors for testing recognition memory. Psychological Bulletin, 88, 696-704.

• abstract Procedures for testing recognition memory have almost invariably required that Ss discriminate old items from new items. At the process level, recognition refers to the capacity of knowing or feeling that a given event has been experienced before. Several previous studies have used a discrimination procedure to test this capacity as a way to prevent Ss from indiscriminately claiming to recognize all test events. It is argued that valid objective tests of recognition memory can be conducted without formal discrimination-testing procedures. Testing recognition memory without distractor items on the test is a viable laboratory procedure, and it is more relevant to the basic concept of recognition memory than the standard discrimination-testing procedures.

Kenneth L. Watson & Michael J. Lawson (1995). Improving access to knowledge: the effect of strategy training for question answering in high school geography. British Journal of Educational Psychology, 65, 97-112. abstract

toetstraining probleemoplossen

Howard I. Weinberg, John Wadsworth & Robert S. Baron (1983). Demand and the impact of leading questions on eyewitness testimony. Memory & Cognition, 11, 101-104. abstract

Voor het ontwerpen van toetsvragen is dit relevant onderzoek, want het voorschotelen van suggestieve vragen en zeker van suggestieve alternatieven in keuzevragen komt in de toetspraktijk nogal eens voor. Het is de vraag wat daarmee wordt getoetst. En of dat dus wel valide is. De leerling als ooggetuige van het eigen geheugen, zeg maar.

Richard T. White (1974). The validation of a learning hierarchy. American Educational Research Journal, 1, 121-136. abstract

Richard T. White (1973). Research Into Learning Hierarchies. Review of Educational Research, 43, 361-375.abstract

P. H. Winne (1979). Experiments relating teachers' use of higher cognitive questions to student achievement. Review of Educational Research, 49, 13-50.

Ton de Jong (1980). Correctievoorschriften en interbeoordelaarsbetrouwbaarheid. Cito Bulletin 6.

Ton de Jong (1981). Correctievoorschriften voor de beoordeling van open vragen en interbeoordelaarsbetrouwbaarheid. Tijdschrift voor Onderwijsresearch, 6, 217-229. pdf

Guda Oly (2002). Meer over meer hoofdstukken vermogensrecht voor bedrijfseconomen. Faculteit der Rechtsgeleerdheid UvA.

Kevin Crehan & Thomas M. Haladyna (1991). The validity of two item-writing rules. Journal of Experimental Education, 59, 183-192.

• Geen overtuigend empirisch bewijs over de onwenselijkheid van de onvolledige stam, en over het alternatief 'geen van deze'. Maar ja, op basis van statistische analyses; niet op basis van cognitief-psychologisch onderzoek, of ervaringen van leerlingen/studenten.

Diamond, J., & Evans, W.J. (1973). The correction for guessing. Review of Educational Research, 43(2), 181-191.

• Nuttig review
• statistische benadering
• geen aandacht voor het maken van fouten
• Interessant: correctie voor raden toegepast onder de instructie niet te raden. Wonderlijk, of de mogelijkheid open te laten (omits).
  The derivation of this formula [ S = R - W/(k-1) ] is based upon the assumptions that all wrong answers are guessed wrong and that all correct answers are obtained either by knowledge or guessing. The presence of misinformation and partial information is not considered.

  p. 181

  In two non-empirical works, both Mattson (1965) and Magnusson (1967) pointed out that random guessing reduces reliability by increasing error variance without a corresponding increase in true score variation.

  p. 183

  Lord (1964) cautioned against instructing pupils to answer every item, since these instructions may introduce errors of measurement through forced random guessing with a resulting drop in validity.

  p. 184

  In fact Cureton (1966) indicated that, unless examinees are explicitly instructed to mark every item, it is unethical not to use the correction. [zoals ook De Groot, 1969. b.w.]

  p. 188

Christa S. C. Asterhan and Baruch B. Schwarz (2007). The effects of monological and dialogical argumentation on concept learning in evolutionary theory. Journal of Educational Psychology, 99, 626-639.

J. T. Dillon (1984). The classification of research questions. Review of Educational Research, 54, 327-361.

• Lees toetsvragen waar researchvragen staat!

Dillon (1982). Cognitive correspondence between question statement and response. AERJ, 19, 540-551.

Dillon, J. T. Dillon (1982). The multidisciplinary study of questioning. Journal of Educational Psychology, 74, 147-165. abstract

Vooral interessant waar het gaat om vragen van de leraar, dus vragen waarop het antwoord al bekend is, vragen van een ander in plaats van van jezelf. Dus: Hoeveel botjes heeft een leeuw?

• Gebruikt in 2.6 van Toetsvragen ontwerpen. Zie vooral p. 158 e.v.
  Teacher questioning constitutes the reverse of the logivian’s standard question situation. For example, whereas a questioner would be presumed not to know the answer and to think the respondent to know it, a teacher is presumed to know the answer and to think that tthe student might not.
  ( . . . )
  In favor of much the same purposes, teachers use questions as a predominant technique and other practitioners avoid them. Purpose is similar, practices are opposite.
  In general, the educational literature contains little theory of questions but many presumptions about their nature, function, and effect. These presumptions are put into question by the theoretical analyses of erotetic logicians. For example, it is not clear how, as presumed in education, teacher questions are to stimulate student thought and discussion. Educational research has not been able to demonstrate these effects (Dillon, in press; Winne, 1979). question is functional in A's thought but not on that account in B's thought. Thus, from the premise that thinking begins with a question, logical analysis would conclude that student questions stimulate student thinking; the educational literature cites the same premise for concluding that teacher questions stimulate student thinking (e.g., Bellack, Kliebard, Hyman, & Smith, 1966, p. 249).

  Logical theory can further be helpful in defining and classifying teacher questions, as into higher-cognitive types, and also in relating them to possible answers. For example, it is not clear how, as presumed in pedagogical manuals, higher teacher questions cause higher student thought and higher answers (e.g., ISCS, 1972, p. 2-4). This view appears to rest on defining a question by its presumed causative effect rather than its expressive function and confounding the cognition of the questioner with that of the respondent. Logical theory would suggest that a high-level question would represent the talk, perhaps the thought, of the teacher, but not cause similar thought or intended response in the student. Moreover, because teacher questions do not express perplexity or seek unknown information, they serve little function in the teacher's own thinking, and likely do not stimulate higher-cognitive processes in students. Logicians and philosophers would say that the question "fails to arise" and is "meaningless for purposes of inquiry" (Belnap, 1969; Knight, 1967). There is little inquiry involved in asking these questions and little in answering them. One implication for practice, especially during discussion classes, might be for the teacher to pose questions about which the teacher experiences perplexity and wonders after the answer. Research could examine the effect such inquiry questions have on student thinking processes and also on student questioning behavior. Other functions that teacher questions serve are revealed by sociolinguistic perspectives. For example, questions establish relationships of power and authority, and are effective means for social control and discourse management (e.g., Mishler, 1975a).

  p 160

Dillon, J.T., The remedial status of student questioning. J.of Curr. Studies 20, 1988, 197-210

J. A. Fodor, M. F. Garrett, E. C. T. Walker & C. H. Parkes (1980). Against definitions. Cognition, 8, 1-105. pdf

Edward J. Furst (1981). Bloom's taxonomy of educational objectives for the cognitive domain: philosophical and educational issues. Review of Educational Research, 51, 441-453. abstract

• Furst was a member of the Bloom commission.

• Further difficulties of a philosophical nature stem from the possible distinction between behavioral and substantive (content) elements in statements of educational objectives. This distinction on which the taxonomy rests came from Tyler’s model of curriculum planning, itself an example of the “ideal model of the Rational Curriculum Plan” (Sockett, 1971, p. 17). Though Tyler had held that a statement of objectives should indicate both the kind of behavior and the area of content or of life to be useful for planning instruction, the authors chose to classify objectives on the basis of intended behaviors, more or less disregarding the particular content (Bloom et al., 1956, p. 12).

  p. 443

Robert M. Gagné(1976). The content analysis of subject matter. a dialogue between Robert M. Gagné and M. David Merill. The computer as an aid in the design of criterion-referenced tests. Instructional Science, 5, 1-28.abstract

Basically is dit dezelfde stuff als Klausmeier, Markle & Tiemann, Merrill & Tennyson. Ik denk niet dat dit een nuttige bijdrage is, behalve dan in historische zin.

• . . . the most important and frequently-occurring relational concepts (“operations”) number only twelve.

A. C. Graesser, M. Singer and T. Trabasso (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371-395. pdf

Mary Hegarty, Marcel Adam just & Ian R. Morrison (1988). Mental models of mechanical systems: Individual differences in qualitative and quantitative reasoning. Cognitive Science, 20, 191-236. abstract

Robert R Hoffman, Nigel R. Shadbolt, A. Mike Burton & Gary Klein (1995). Eliciting knowledge from experts: a methodological analysis. Organizational Behavior and Humans Decision Processes, 62, 129-158. pdf

Jill H. Larkin & Herbert A. Simon (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65-99. schema, probleemoplossen abstract [oudere jaargangen vrij te downloaden op http://csjarchive.cogsci.rpi.edu/contents.html (behalve de allereerste)]

Wendy Lehnert (1977). Human and computational question answering. Cognitive Science, 1, 47-73. pdf

Vragen naar tekstbegrip, wat is dat?

• The question answering problems that arise in a story-understanding system are discussed. Some solutions are described and illustrated by a conputer program which reads stories and answers questions.

Frederic M. Lord (1977).  Optimal number of choices per item: a comparison of four approaches.  Journal of Educational Measurement, 14, 33-38. preview

Jill H. Larkin & F. Reif (1976). Analysis and teaching of a general skill for studying scientific text. Journal of Educational Psychology, 68, 431-440. abstract

Richard Meisner, Richard Luecht & Mark Reckase (1993). The comparability of the statistical characteristics of test items generated by computer algorithms. ACT Research Report Series 93-9. pdf

Michener, E.R. (1978). Understanding understanding mathematics. Cognitive Science, 2, 361-383. [free download http://csjarchive.cogsci.rpi.edu/contents.htm ] Geeft een mooie reeks heuristieken voor inzicht, en dus ook voor inzichtvragen. abstract

Gregory L. Murphy & Douglas L. Medin (1985). The role of theories in conceptual coherence. Psychological Review 92, 289- pdf

Gregory L. Murphy (1988). Comprehending complex concepts. Cognitive Science, 529-562. [free download http://csjarchive.cogsci.rpi.edu/contents.html]

Michael C. Rodriguez (2005). Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement: Issues and Practice, 24 summer, 3-13. abstract

Steven M. Downing and Thomas M. Haladyna (Eds) (2006). Handbook of test development. Erlbaum. site [heel kostbaar] [Editie 2011, editors: Suzanne Lane, Mark Raymond, Steven M. Downing & Thomas S. Haladyna. eBook in KB]

J. C. Powell (1968). The interpretation of wrong answers from a multiple choice test. Educational and psychological Measurement, 28, 403-412. [geen abstract]

De stelling van Powell is eigenlijk deze; de formule voor corrigeren voor raadkansen gaat ten onrechte uit van weten of raden; zoals ik in mijn hoofdstuk 2 aangeef: leerlingen kunnen het antwoord ook gewoon fout weten.

• A commercially prepared multiple choice test, The Proverbs Test, was analyzed to determine the possible information content of the wrong answers. It was demonstrated that the assumption that wrong answers are randomly distributed does not hold in this case. Wrong answers clearly have psychological and statistical meaning. The implications of these findings could have great importance for test interpretation. Further research is necessary to define these implications.

  Summary

Henry A. Alker, Julia A. Carlson & Margaret G. Hermann (1969). Multiple-choice questions and student characteristics. JEP, 60, 231-243. abstract

• Though limited, this study has tried to deal scientifically with an issue too often reacted to defensively. Perhaps doing so has opened a Pandora's box. Such a result may be necessary to motivate both proponents and critics of multiple-choice tests to settle their differences in the laboratory.

  Laatste zinnen

Michael C. Rodriguez & Thomas M. Haladyna (2013). Writing selected-response items for classroom assessment. In James H. McMillan (Ed.) (2013). SAGE Handbook of Research on Classroom Assessment (293-311). SAGE. [UB Leiden PEDAG. 51.e.85] [3 juni 2013 nog niet als eBook beschikbaar, dus ook niet in KB] site

Weinig nieuws?

Anthony P. van Antwerpen (1981). Evalueren met meerkeuzevragen. Een studie over het logisch analyseren van meerkeuzevragen. Proefschrift Leiden.

Een merkwaardig onderzoek, kennelijk als medewerker van het Cito gedaan. De meerkeuzevragen zijn toetsvragen scheikunde mavo. Dat logisch analyseren moet letterlijk worden opgevat. Ik zie geen analyses waar psychologie aan te pas komt. Afijn, er zijn hier empirische data gegeven met de bijbehorende meerkeuzevragen, best interessant materiaal. Ik kreeg dit boek pas 2013 in handen.

Joseph Raymond Gerberich (1956). Specimen objective test items. A guide to achievement test construction. New York: Longmans, Green and Co. [Niet in UB Leiden]

Frank Kok (1988). Vraagpartijdigheid. Methodologische verkenningen. Proefschrift UvA (Mellenbergh, co: Van der Flier).

Norman Frederiksen (1984). The real test bias. Influences of testing on teaching and learning. American Psychologist, 39, 193-202. abstract

• It is suggested that the greater costs of tests in other formats [dan meerkeuze, b.w.] might be justified by their value for instruction — to encourage the teaching of higher level cognitive skills and to provide practice with feedback.

  lead

Brent Bridgeman and Rick Morgan (1996). Success in College for Students With Discrepancies Between Performance on Multiple-Choice and Essay Tests. Journal of Educational Psychology, 88, 333-340. abstract

Richard E. Mayer, J. Larkin & J. B. Kadane (1984). A cognitive analysis of mathematical problem solving ability. In Robert J. Sternberg: Advances in the psychology of human intelligence, volume 2 (231-273). Erlbaum

Fons Vernooij (1993). Het leren oplossen van bedrijfseconomische problemen. Didactisch onderzoek naar kostprijs- en nettowinstvraagstukken in het voortgezet onderwijs. Proefschrift Erasmus Universiteit Rotterdam. free access

Een belangrijk thema is dat inconsistenties in de leerstof doorwerken op de pogingen van leerlingen om inconsistente vraagstellingen te beantwoorden (dat alles terwijl de beoordelaars de inconsistenties niet signaleren).

Debra Dhillon (). Predictive models of question difficulty. A critical review of the literature. Centre for Education Research and Policy. pdf

Katz, I. R., Bennett, R. E., & Berger, A. (2000). Effects of response format on difficulty of SAT mathematics items: It’s not the strategy. Journal of Educational Measurement, 37(1), 39-57.

Katz, I. R. & Berger, A. (1995). Strategies underlying score differences on SAT mathematical items. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA., April.

Katz, I. R., Lipps, A. W., & Trafton, J. G. (2002). Factors Affecting Difficulty in the Generating Examples Item Type (GRE Board Professional Report No. 97-18P). Princeton, NJ: Educational Testing Service.

Malpas, A. J. & Brown, M. (1974). Cognitive demand and difficulty of GCE O-level mathematics pretest items. British Journal of Educational Psychology, 44(2), 155-162.

Nhouyvanisvong, A. & Katz, I. R., & Singley, M. K. (1997). Toward a unified model of problem solving in well determined and underdetermined algebra word problems. Paper presented at the annual meeting of the American Educational Research Association, Chicago, March.

Pollitt, A., Entwistle, N., Hutchinson, C., & de Luca, C. (1985). What Makes Exam Questions Difficult? Edinburgh: Scottish Academic Press.

Tabachneck, H. J. M., Koedinger, K. R., & Nathan, M. J. (1995). A cognitive analysis of the task demands of early algebra. In Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society. Mahwah, NJ: Lawrence Erlbaum Associates.

Peter van Bart, Johan Kerstens & Arie Sturm (1998). Een grammatica van het Nederlands. Een inleiding. Amsterdam University Press.

Is deze zin (vraagstelling) grammaticaal? Hoe complex? Is hij mis te verstaan? Natuurlijk niet geschreven om toetsvragen te analyseren. http://www.lotpublications.nl/fun-to-read-or-easy-to-understand-fun-to-read-or-easy-to-understand-establishing-effective-text-features-for-educational-texts-on-the-basis-of-processing-and-comprehension-research

Gerdineke van Silfhout (2014). Fun to read or easy to understand? Establishing effective text features for educational texts on the basis of processing and comprehension research. Proefschrift Utrecht. Landelijke Onderzoekschool Taalwetenschap. pdf ophalen

• Gerdineke van Silfhout (2013). Daarom hebben we zoveel moeite met die studieteksten! Over tekstbegrip in het Voortgezet Onderwijs. webpagina

  ---

• Meer publicaties uit dit promotietraject hier

Janet Gail Donald (2002). Learning to think. Disciplinary perspectives. Jossey-Bass info and review by William Peirce; Another review by Perry Klein

Lijkt me een heel aardig boek. Niet vanwege de diepgang (die is er geloof ik niet), maar vanwege de breedte over disciplines heen (maar juist wel mèt de disciplines, inhoudelijk disciplinair dus). Ik heb er geen tijd voor. Het boek is voor mij eigenlijk te oppervlakkig: een enorme bak details, verbonden in oppervlakkige categorieën. Het meeste weet ik toch al wel, het zou mij dus gaan om de diepere lagen; en die lijken hier te ontbreken. De auteur is geen Stellan Ohlsson, zal ik maar zeggen.

Dan Dimaggio (2010). The loneliness of the long-distance test scorer. web page

Jeanette Lubbers & Jan Muthert (1991). Cijferen of ontcijferen. Wiskunde A of tekstverklaring? Euclides, 66, 275-284. pdf

Steven Rasmussen (March 2015). The Smarter Balanced Common Core Mathematics Tests Are Fatally Flawed and Should Not Be Used. pdf

Laura S. Hamilton, E. Michael Nussbaum & Richard E. Snow (1997). Interview procedures for validating science assessments. Applied measurment in Education, 10, 181-200. researchgate.net

• This study investigates the usefulness of small-scale interview studies as a means to explore the validity of science achievement tests in several formats. Interviews and observations were conducted with 41 high school students taking multiple-choice and constructed-response science items, and with 49 fifth- and sixth-grade students completing hands-on science tasks. The procedure provided support for interpretations of subscales of achievement, identified some of the reasoning processes and sources of knowledge that students applied to the tasks, and revealed ways in which the strategies applied by successful and unsuccessful students differed. The importance of conducting small-scale, qualitative interview studies as part of the test validation process is discussed.

  abstract

Greg A. Perfetto, John D. Bransford and Jeffery J. Franks (1983). Constraints on acces in a problem-solving context. Memory & Cognition, 11, 24-31. fc in map tvr pdf

(oude aantekening:)Transer. Leuke experimentjes. Vooral ook: zonder informatie over elevantie van ander materiaal, worden inadequate oplossingen gegenereed, die blijken later alsnog als blokkade te werken.

A. P. van Antwerpen (1981). Evalueren met meerkeuzevragen. Een studie over het logisch analyseren van meerkeuzevragen. Proefschrift RU Leiden.

Merkwaardig betoog over keuevragen. Lijkt me geen promotieonderzoek, maar is toch zo afgestempeld door promotor Len de Klerk, die zal Van Antwerpen dan wel opdracht hebben gegeven ook nog wat citaten uit een stapel toetsliteratuur te plukken. Van Antwerpen maakt het allemaal wel een beetje ingewikkeld, zonder ergens enige kern aan te raken. Dat is bij doorbladeren mijn indruk, en dan stopt het: want waarom zou ik dit gaan bestuderen?

Seymour Sudman & Norman M. Bradburn (1982 8th). Asking questions. A practical guide to questionnaire design. San Francisco: Jossey-Bass. isbn 0875895468

G. M. Ruch (1929). The objective or new-type examination: an introduction to educational measurement. Chicago : Scott, Foresman. [not online]

• 1. The argument for objective examinations 3-148
  2. How to construct an objective examination 149-280
  3. Experimental and theoretical considerations 281-404
  4. Statistical treatment and interpretation of objective test results 405-471

Brenda Jennison and Jon Ogborn (Eds) (1994). Wonder and delight. Essays in science education in honour of the life and work of Eric Rogers 1902-1990. Bristol: Institute of Physics publishing. isbn 0750303158

A wonderful book with anecdotes and exemplars, also of exam questions and exam questioning practice. Rogers had strong opinions on achievement test items and the way to design them. He is the only one, or at least one of the very few, item designers that I probably would call a real expert. (see my 'Designing test items' 2.6)

• . . . typically, a working session [in a two week seminar with teachers] would open with Eric Rogers inviting one of the participating teachers to propose a physics question. Other members of the group were then encouraged to discuss or challenge its validity or its usefulness. What is the purpose of this question? Does it really do what you intend it to do? How could this question or problem be rephrased to make it more useful? What learning objectives would it contribute to? How would you use it as part of your teacing? What broader teaching strategy would this imply? Perhaps an experimental investigation on the part of the student? And so on. The author of the problem would generally first defend it against points raised by others, but would gradually come to accept criticisms, modificatons and further developments of it by the group. It was this tearing apart of the questions that led Eric Rogers to give the name ‘shredder’ to this sor of workshop.

  N. Joel, 183-186. p. 184

M. Gall, B. Dunning en R. Weathersby (1971). Minikursus Denkvragen stellen. Groningen: Wolters-Noordhoff. Ned. bewerking: P. L. v. d. Plas en W. J. M. de Roos, 1977. isbn 9001588905

Ideologisch boek. Ik heb dat destijds niet onderkend (heb dit boek destijds ook niet echt bekeken). De lezer wordt op geen enkele manier egwaarschuwd voor de ideologische basis. Het lijkt dan een boek te zijn dat berust op het werk van Bloom c.s. over een cognitieve taxonomie, maar ook dat is een ideologisch werkstuk, op zich weer gericht tegen eenzijdig behaviorisme in bepaalde geschriften over onderwijs. Nou ja. Ik volsta met de allereerste zin te citeren, dan is het wel duidelijk welk bezwaar er tegen dit boek is. Hier is evident sprake van een vakgroep onderwijskunde (aan de Rijksuniversiteit te Leiden) die de kluit aan het belazeren is, natuurlijk netjes in de termen van de taxonomie. Maar zoiets als het denkvermogen ontwikkelen bestaat dus niet, tenzij het bestaan wordt bewezen. En dat laatste is tot op heden nog niet gebeurd. Wie verbaasd is over mijn stelling: merk op dat het hier gaat om denkvermogen in generieke zin, losgekoppeld van domeinkennis. Onderwisjkunde Nijmegen heeft vertaling van bijbehorend deel 2 verzorgd: Effektief vragen stellen.

• Het doel van deze minikusus is om u te leren gebruik te maken van het type vragen dat het denkvermogen van de leerlingen helpt ontwikkelen.

  blz. 1

G. J. Mellenbergh (1971). Studies in studietoetsen. Amsterdam: Psychologisch Laboratorium van de Universiteit van Amsterdam. Proefschrift (promotor A. . de Groot). (offset)

• 1. Een vergelijking tussen verschillende soorten toetsen
  2. Een onderzoek naar het beoordelen van open vragen
  3. Het beoordelen en klassificeren van items
  4. Een model van asch toegepast bij meerkeuzeitems uit studietoetsen (4.4. Toepassing van het model op een Wiskunde-toets (brugklas) 141-146) (nee, geen zinnig woord over de stof), daar gaan psychometrici niet over ;-)

  G. J. Mellenbergh (1971). Het beoordelen en klassificeren van meerkeuze-items uit studietoetsen. Nederlands Tijdschrift voor de Psychologie, 26, 518-551.

  G. J. Mellenbergh (1971). Een onderzoek naar het beoordelen van open tvr.rtfvragen. Nederlands Tijdschrift voor de Psychologie, 26, 102-120.

H. F. M. Crombag (1970). Vergelijking van een examen met open vragen en een meerkeuze-toets. Nederlands Tijdschrift voor de Psychologie, 25, 349-358.

R. F. van Naerssen (1970). Tweekeuze-items in studietoetsen. Nederlands Tijdschrift voor de Psychologie, 25, 393-403.

Teelen Kennismanagement (2012 5e). Toetsontwkkeling in de praktijk. Hoe maak ik goede vragen en toetsen? isbn 9789081679404 info

Martin Nuy (1994). Toetsen voor cijfers: proefwerken, schoolonderzoeken, tentamens. Nijkerk: Intro. isbn 9022661916

Mooi cursusboek, kort, helder, overzichtelijk, veel voorbeelden, vooral over het ontwerpen van toetsvragen. Ik heb na doorbladeren het vermoeden dat hij mijn Toetsvragen schrijven goed heeft gelezen, wat heel mooi is.

Arthur C. Graesser & John B. Black [Eds] (1985). The psychology of questions. Lawrence Erlbaum. isbn 0898594448

• • ao.:
  • Arthur C. Graesser and Tamar Murachver: Symbolic procedures of question answering 15-88
  • Tmar Murachver, Kelly E. Murray and Arthur C. Graesser: Answering some questions about a model of question answering 89-120
  • Murray Singer: Mental processes of question answering 121-156
  • Gary M. Olson, Susan A. Duffy and Robert L. Mack: Question-asking as a component of text comprehension 219-226
  • Susan Kemper, Robert Estill, Nelson Otalvaro and Margaret Schadler: Questions of facts and questions of inferences 227-246
  • Susan R. Goldman: Inferential reasoning in and about narrative texts 247-276
  • Don Nix: Notes on the efficacy of questioning 297-334
  • Steven P. Shwartz and Wendy G. Lehnert: Data base querying by computer 359-373
• Arthur C. Graesser (Guest editor) (1990). Psychological research on question answering and question asking. Special Issue, Discourse Processes, 13, 259-3.. .

Don Nix (1985). Notes on the efficacy of questioning. In Arthur C. Graesser and John B. Black (Eds) (1985). The psychology of questions. Hillsdale, New Jersey: Lawrence Erlbaum.

• p. 297: "This chapter focuses on the use of questioning techniques for the purpose of directly teaching inferential reading comprehension and meta-comprehension skills to children in classroom settings. (...) It is assumed that inferential comprehension is a complex process: the child must activily transform what, on the page, is a string of symbols into an inferentially integrated network of meaning. The nature of classroom questioning is viewed in terms of what impact it can have on a child's ability to perform this complex process." (...) A pervasive quality of questions is that the ones spontaneously asked by teachers tend to be 'detail' questions (e.g. Guzak, 1968). That is, in order to answer the question it is not necessary to inferentially integrate various parts of the text.

  Op p. 298 een interessante uiteenzetting over vragen op een (te) hoog abstractieniveau (Nix noemt het niet zo).

  p. 298: "If a teacher is asked to generate a good comprehension question about this text, and is not allowed to generate a question which simply requires an answer overtly stated therein, the teacher may generate a 'main idea' question, that is, 'What is the main idea in this passage?' Clearly, a student can not simply retell a specific proposition. Instead, it is necessary to inferentially integrate several propositions along with what one already knows about the world. The difficulty is that the procedure for doing such an integration is ineffable, the concept of a main idea is not itself defined in a teachably explicit manner. The teacher does not have access to a set of components which can be explained and taught in a step-by-step manner, and which together constitute main 'ideaness.' Thus, regardless of whether or not the question is answered adequately; the evaluation of the answer can not be made in terms that explicitly teach the student how to answer such questions in the future. The same difficulty exists with other typical classroom conceptualizations for inferential questions. Most of the concepts are 'specific skills' (Davis, 1968) such as: main idea, sequence, author's intent, characterization, and predicting outcome. These notions are not explicitly defined so that given text instances can be designated as one or the other (Nix, 1983a). Teachable rules for answering such questions are difficult to specify, in part, because the question types are not actually defined, and in part, because given text instances can often be classified simultaneously as more than one type.

Paul van den Broek, Yuhtsuen Tzeng, Kirsten Risden, Tom Trabasso & Patricia Basche (2001). Inferential questioning: Effects on comprehension of narrative texts as a function of grade and timing. Journal of Educational Psychology, 93, 521-529. researchgate.net

• In addition, inferential questioning challenges the processing capacities of younger or less skilled readers and, hence, may interfere with comprehension.

  from the abstract

H. De Neve en P. J. Janssen (1992). Succesvol examineren in het hoger onderwijs. Leuven: Acco. isbn 9033426668

• Een overzichtelijk boekje voor docenten. Neemt enkele principiële inzichten uit 'Toetsvragen schrijven' over: het ontwerpen van toetsvragen is een kunde, geen kunst; het doel van toetsen is primair didactisch van aard, niet om te selecteren, en in die zin moeten itemstatistieken worden gebruikt.

Rob van Hattum en Hein Meijers (1998). De Nationale Wetenschapsquiz. Vragen & antwoorden. Antwerpen: Kosmos. isbn 9021530848

F. M. Austin (1949). The art of questioning in the classroom. London: University of London Press.

Leuk boekje, gaat over vragen van kleine kinderen, meer dan over vragen aan kinderen. Een paar heel andere manieren om tegen vragenstellen aan te kijken.

C. W. Odell (1928). Traditional examinations and new-type tests. New York: The Century Co. (Selected and annotated bibliography p. 439-469)

• 1. The pastand present status of examinations 3
  2. What are good examinations? 32
  3. How to make and give examinations 59
  4. Scoring pupils&squo; responses 81
  5. The marking system and its meaning 109
  6. The distribution of marks 141
  7. Meits and limitations of traditional and new-type examinations 175
  8. Examples of traditional or essay examinations 205
  9. The constrution and use of new-type examinations 237
  10. Single-answer ecall tests 255
  11. Multiple-answer tests 281
  12. alternative tests 334
  13. completion tests 358
  14. matching tests 376
  15. incorrect-statement tests 388
  16. Miscellaneous types of the new examination 395
  17. Objective tests in institutions of higher learning 422

G. M. Ruch (1929). The objective or new-type examination: an introduction to educational measurement. Chicago : Scott, Foresman. review

• Chapters: Points of view - The criteria of a good test or examination - Objections to the traditional examination - Advantages and limitations of objective examinations - Students' attitudes toward examinations (a rather short chapter) - Relative values of standardized and non-standardized tests - The building of an objective test or examination - Illustrative types of objective tests - Selected complete examinations (p. 213-264) - Rules for drafting objective test items - Experimental studies of new-type examinations - Chance and guessing in recognition tests - The negative and other suggestion effects in the true-false tests - Examinations, marks, and marking systems - Statistical problems related to measurement - General bibliography (p. 447-471, 377 items in all)
• No mention of the Starch 1916 monograph, nor - in the first section - any other older than 1921.

Benjamin S. Bloom, J. Thomas Hastings & George F. Madaus (1971). Handbook on formative and summative evaluation of student learning. isbn 070061149

Nuel D. Belnap Jr. and Thomas B. Steel Jr. (1976). The logic of questions and answers. Yale University Press. isbn 0300019629

• (question logic - question-answering systems - formal languages - linguistics - psychology and education - erotetic semantics) (Contains: Bibliography of the theory of questions and answers, by Urs Egli and Hubert Schleichert (annotated)) ("This book takes as its goal the creation of a formal language that can be used to ask and answer questions in an orderly, fruitful way." "A work of interest to logicians and philosophers of language, The logic of questions and answers has its broadest potential use in the applications of the theory to computer question-answering systems")

John Nolt, Dennis Rohatyn & Achille Varzi (1998). Schaum's outline of theory and problems of logic. 2nd edition. Schaum's Outline Series, McGraw-Hill. isbn 0070466491

For example, refutation trees p. 68-79.