Original publication 'Toetsvragen schrijven' 1983 Utrecht: Het Spectrum, Aula 809, Onderwijskundige Reeks voor het Hoger Onderwijs ISBN 90-274-6674-0. The 2006 text is a revised text.

Item writing

Techniques for the design of items for teacher-made tests

3. Course content inventory

Ben Wilbrink

this database of examples has yet to be constructed. Suggestions? Mail me.


"For the most part, it behooves a critical epistemologist to construct a theory of knowledge explaining how we know the things we do, but, in a few instances, a theory may explain why w ethink we know when we do not."

Keith Lehrer (1990). Theory of knowledge. Routledge. p. 2.

This chapter is about what a body of knowledge is, and how to characterize it. It should also tackle the question what it is to have one, get one in the first place, or change one's wrong one. Strictly taken, it is not necessary first to describe the course content in its constituting terms, relations between terms, and descriptions giving otherwise meaning to certain of its terms. But it will surely furnish an excellent starting point for the design of test items.

Usually a 'body of knowledge' is understood to be a discupline such as English, mathematics, or chemistry. Think, however, also of the unusual interpretation, that it is 'critical thinking,' or Deanna Kuhn's 'inquiry and argument,' or even her (2005) proposal to organise a curriculum around the concept of 'causality,' instead of the traditional content nobody can convingly tell you the reasons for (which reminds me of the way assessment typically is regarded, and in its turn that is what made me research the history of educational assessment html).

A useful resource for this description of course content is the technique of schematizing, see the chapter 3 text, or the CmapTools site. A chapter on the concept of intelligence might be characterized by the following scheme, using terms that are on different levels of abstraction. Distinguishing this kind of levels of abstraction is in its turn a powerful resource in designing questions on course content.
schema verknooptheid

Figure 1. Scheme of terms related to the concept of intelligence. The scheme was constructed using CmapTools, software to picture knowledge structures http://cmap.ihmc.us/.

Dominic G. B. Edelen (1962). The structure of field space. An axiomatic formulation of field physics. University of California Press.


What is this thing called 'course content'? The immedeate answer to this question is obvious: it is whatever is presented as course content to the student. And obviously this answer is hopelessly inadequate. It is in the same league as 'intelligence is whatever it is the intelligence test measures.'

Usually, in order to help the student solve this puzzle, some meta-statements are added in the course book, expressing what its content is about, and/or what kind of mastery the student should be able to demonstrate at the close of the course or at the end of course achievement test. Lacking this, it will be shown what kind of questions the student should be able to answer at the summative test. If these questions have beeen designed by teachers believing item writing is an art, students will be shown art and will therefore not get well informed about what it is that is considered mastery of the course content.

Take the course that should result in a definite skill: teaching, combat, singing. The point is not that skills somehow are simpler than mastery of intellectual content, they probably are not. The point is that there is unity between the skill as taught, the skill as needed in the profession, and the skill as assessed in the exit test. In the first centuries of their existence, European university examinations were demonstrations of teaching skill, and the examinees earned the 'licence to teach' everywhere in the realm of the catholic church. Students exercised/learned whatever was deemed necessary to acquire this skill of teaching. In the medieval guilds the training was wholly analogous to that in the university guild of masters.
Since the twelfth century educational institutions and processes have become somewhat more complicated, it is not now self evident what it is that particular courses or text books are about, even if it is announced on their page one.

A fundamental approach, such as needed by the educational technologist automating item design, would probably suffocate itself in the large number of translations that has to made: intentions of text book writers, content of text books, constructs of this course contents made by the students, a conception of what it is to test for knowledge of course content, developing the kind of questions that match this, etcetera. The definitely simpler task of developing computerized courseware had to struggle with such a cascade of 'knowledge maps.'

By now you will be susceptible to the idea of a shortcut, to avoid the mess of so many different models that have to be made explicit, and then must be related to each other. The shortcut is to highlight, schematize, or whatever other technique you as teacher would like to use, what it is that is important in your course, and whatever else is supportive of this core content. The second step of the shortcut is to design a small number of typical test items - at the formative or summative level - that students should be able to handle and answer. And that is it. Do not forget to evaluate the success of your efforts, and learn from your mistakes and successes.

Be aware of your own secretly harbored models of what it is students might have learned in your course, what it is failed students have failed to master, what it is to 'assess' student mastery, what your own 'expertness' in teaching and assessment is. Models abound. Get rid of the junk. Searching the internet for Clancey's book title teaches us that this kind of modeling approach is not used abundantly any longer in AI research and course design (but see http://monet.aber.ac.uk:8080/monet/index.html, or edutechwiki.unige.ch).

"The greatest enemy of understanding is coverage" (Howard Gardner, Educational Leadership 50/7 April, 1993) [I must look this up]

K. Barker, B. Porter, and P. Clark. (2001). A Library of Generic Concepts for Composing Knowledge Bases. First International Conference on Knowledge Capture, October 21-23. abstract pdf

P. Clark and B. Porter (1997). Building Concept Representations from Reusable Components. AAAI'97, 369-376, CA:AAAI Press. pdf

mental models - erroneous beliefs

This book uses extensively the naive conception - mental model - of the laws of movement. Much research has been done on the character, pervasiveness and robustness of this particular mental model, roughly the Aristotelian view of physics, versus the classical, Newtonian, view. It is to be expected that there are countless mental models, in physics as well as in other disciplines, that play havoc with instruction and assessment if going unheeded. The challenge for instruction as well as assessment is to effect the necessary conceptual change, and test for it. Searching the internet for 'conceptual change' will result in many relevant sources, however, a short review of the literature is presented by Jonassen (2006), not online.

Our first joint article examined systematic errors in the casual statistical judgments of statistically sophisticated researchers (Tversky & Kahneman, 1971). Remarkably, the intuitive judgments of these experts did not conform to statistical principles with which they were thoroughly familiar. In particular, their intuitive statistical inferences and their estimates of statistical power showed a striking lack of sensitivity to the effects of sample size. We were impressed by the persistence of discrepancies between statistical intuition and statistical knowledge, which we observed both in ourselves and in our colleagues. We were also impressed by the fact that significant research decisions, such as the choice of sample size for an experiment, are routinely guided by the flawed intuitions of people who know better. In the terminology that became accepted much later, we held a two-system view, which distinguished intuition from reasoning. Our research focused on errors of intuition, which we studied both for their intrinsic interest and for their value as diagnostic indicators of cognitive mechanisms. (Kahneman, 2002 pdf)

It was not the intention of Kahneman to make a mockery of intuitive expert judgments in general in this - his Nobel laureate prize lecture. Quite to the contrary, as his lecture testifies. The systematic erroneous judgment on this kind of statistical events therefore is a quite specific mental model.
In due time I will collect the best known mental models. Undoubtedly the belief in the law of small numbers belongs to that category, it was researched by Tversky and Kahneman (1971 html); this one is able to make fools of statisticians and researchers knowledgeable in statistics, beware.

mental models - conceptual change

David R. Kaufman, Stella Vosniadou, Andy diSessa and paul Thagard (2000). Scientific explanation, systematicity, and conceptual change. Symposium. html

Extending the conceptual change approach to mathematics learning and teaching. Learning and Instruction, 14, 2004, 445 and following: special issue. editorial
Stamatia Stafylidou and Stella Vosniadou (2004). The development of students' understanding of the numerical value of fractions. Learning and Instruction, 14, 503-518. pdf
Xenia Vamvakoussi and Stella Vosniadou (2004). Understanding the structure of the set of rational numbers: a conceptual change approach. Learning and Instruction, 14, 453-467. pdf
Stella Vosniadou en Lieven Verschaffel (2004). Extending the conceptual change approach to mathematics learning and teaching. Learning and Instruction, 14 pdf

Stacy Rebich and Catherine Gautier (2005). Journal of Geoscience Education, v. 53, n. 4, 355-365. pdf

Susan Carey (1985). Conceptual change in childhood. MIT Press.

3.1 Observable terms

At a down-to-Earth level, there are concrete things and events, some of which are the subject of the discipline the course content is about. Let us start by gazing in the skies, however. They have always been somewhat mysterious, it nog being quite clear what exactly it is one sees happening there. A photograph of a small part of the sky will show us things and events in somewhat the same way we are used to observe using our eyes without technical resources. Easily the most spectacular photograph I have ever seen in my life, is the following, containing a lot of 'observables,' and leaving a lot to explain about what exactly it is we think to see here. It also illustrates an unusual kind of scheme, abstract terms superposed on an observable level picture. A 'lensed quasar' is a quasar that is represented in our observation in multiple images, in this case a 'direct' one in the middle, and four 'mirrored' ones. The phenomenon is caused by a cluster of galaxies between us and the quasar, this cluster by its sheer mass 'bending' the curvature of space, bending the path of light. Knowing the relativistic 'mechanism,' makes us sure the five images 'really' belong to one and the same bright galaxy. Wow, what a theory, and what a picture! Looking at the quasar is looking back ten billion years in time, lightyears in distance. Look at the yet more detailed photograph one at hubblesite.org

Galaxy Cluster SDSS J1004+4112

Figure 2. The ultimate example of interpretation of observations? The 12 billion years old lensed galaxy's image gets magnified and tripled (red circle/ovals) by a cluster of galaxies functioning as a lens by force of its gravitational field. More detailed pictures on html or direct as jpg. Credit: ESA, NASA, K. Sharon (Tel Aviv University) and E. Ofek (Caltech). For the scientific interpretation see Lamer e.a. (2006) pdf

The Hubble telescope is a super high tech instrument orbiting the Earth, making it possible to construct the most fantastic 'images' of phenomena in the Universe. Interestingly, for the very first telescope known, the same could be said. What would one see if it were pointed to the sky, to Jupiter for example? And how to interpret what one saw? Here is another example where it is far from self-evident that what you 'seeing' is self-evident.

Jupiter's moons

Figure 3. Looks less spectacular, but was definitely more so at the time: AD 1610. From Galileo Galilei's letter to the Prince of Venice, describing the discovery of four big moons of Jupiter (html; Galilei's complete works in Italian available as pdf on http://gallica.bnf.fr/, this letter in volume III part II, p. 427). Another argument that the Earth is not the middlepoint of the Universe. Some of his contemporaries could not 'see' Jupiter's moons through Galileo's own telescope. The meaning of what there is to see, can not be 'seen.' See also the Voyager images of the moons html, and this page from The Galileo Project html.

And yes, using new technologies, it is possible to construct images of 'events' that do not happen at all, they exist only in the 'conctructed eye' of the beholder. Look at Jupiter's atmosphere whirling. The illusion is that you see Jupiter rotating, but that is not so. The film is made of images taken exactly one 'Jupiter-day' apart. Start the film.
Animatie van
bewegingen in de Jupiterbewolking, zoals waargenomen door Voyager 1.

Figure 4. Making movement visible by compressing time, or showing something which does not exist as such (producing an artifact): movements in Jupiter's atmosphere as observed by the approaching Voyager 1. For the film, click the picture. The film does not show Jupiter's rotation! The instantaneous black spots on this still are shadows of two of the four big Jupiter moons. The animation is taken from the beautiful site of the Dutch Volkssterrenwacht Urania html, see there also for Jupiter in color.

observation failure

It is possible not to see what evidently and plainly is before one's eyes. The 1988 Downing of Iran Air Flight 655 is a tragic example of what has been known for a long time from the psychological laboratory: under pressure people can see things that just are not there, and do not see what is plainly before their eyes. One of the crucial factors in the downing of this civilian plane - 290 casualties - was the interpretation that this plane was diving towards the Vincennes while in observable fact in was continuously climbing. The 'observable' item was not the plane itself - it was seventeen miles away from the Vincennes - but the instrument reading of the plane's flight. Fear aboard the Vincennes was so intense, the expectation of an attack was so strong, that the instrument was plainly misread by the crew. The commanding officer in his turn believed the verbal report and not the data he himself also could see plainly. The ship's commander believed his commanding officer.

The documentary makes a point of the Vincennes quite unprofessionally bringing itself in a position where this kind of mistake is a much bigger risk than it otherwise would be: at the moment of the shooting the Vincennes was miles in Iranian territorial waters chasing Iranian gunboats that had taken shots at a Vincennes helicopter. Not in the documentary: the helicopter(s) might have provoked the Iranian boats inside Iranian waters.

National Geographic 'Air crash investigation: Mistaken identity.'
Wikipedia Iran Air Flight 655 (IR655) has another story on this.

The Iran Air Flight 655 case is a pretty tough one. Professionals should however be prepared for this kind of mistake; strong expectations coloring what they 'see' happening. Ordinary people should be prepared also: driving your car inattentively may get you into trouble if you are not in fact at the point you think you are (the end of this road is still further on ....). In assessment: be aware of the possibility that your expectations might determine your grading, instead of the facts. We are getting into psychological territory here.


learning categories

W. Ahn, R. L. Goldstone, B. C. Love, A. B. Markman, & P. Wolff (Eds.), Categorization inside and outside the lab: Festschrift in honor of Douglas L. Medin. Washington, DC: APA. [nog opzoeken]

Andrea L. Patalano, Seth Chin-Parker, and Brian H. Ross (2003). The Role of Coherence in Category-Based Explanation. 910-915. pdf

observational concept learning. Introductory: Wickelgren (1977 p. 274-76). Learning concepts in the natural world, or in the laboratory world of the psychologist (Hull, 1922, is mentioned by Wickelgren, using Chinese characters).

inferential concept learning. Introductory: Wickelgren (1977 p. 276). The classic study here is Bruner, Goodnow and Austin (1956) using Harvard students as subjects. Studies such as these will suggests instructional methods, or explain why they are effective, and therefore also how to design test items.

Chin-Parker, S., & Ross, B. H. (2004). Diagnosticity and prototypicality in category learning: A comparison of inference learning and classification learning. Journal of Experimental Psychology: Learning, Memory and Cognition, 30, 216-226.

types of concepts and attributes. Introductory: Wickelgren (1977 p. 278). The regular concept is of the conjunctive type: examples share one or more defining characteristics. The disjunctive concept comprises examples having either characteristic A or characteristic B, and ar called inclusive-or if having both characteristics also defines an example, and exclisive-or if examples do not have both characteristics. Implicational concepts: an example belongs to the concept if its having characteristic A implies it also has characteristic B. In this way the playing field rapidly can become quite complex, without even talking about more the fuzzier kinds of concepts such as defined by family resemblances etcetera. [Ulrich Neisser and P. Weene (1962). Hierarchies in concept attainment. Journal of Experimental psychology, 64, 644-645.]

nonexamples. Introductory: Wickelgren (1977 p. 278-9). [Hovland and Weiss (1953). Transmission of information concerning concepts through positive and negative instances. Journal of Experimental psychology, 45, 165-182.] [W. Freibergs and Endel Tulving (1961). The effect of practice on utilization of information from positive and negative instances in concept formation. Canadian Journal of Psychology, 15, 101-106. M. J. Fryatt and Endel Tulving (1963). Interproblem transfer in identification of concepts involving positive and negative instances. Canadian Journal of Psychology, 17, 106-117. ]

typicality Gregory L. Murphy and Brian H. Ross (2005). The two faces of typicality in category-based induction. Cognition, 95, 175-200. pdf

Educational Psychologist (2004), volume 39 nr 1. Thematic on personal epistemology. Not directed specifically at assessment, however.


3.2 Abstract terms and constructs

De betekenis van een woord is zijn gebruik in een taal; zijn gebruik in de praktijk; zijn rol in het taalspel; zijn plaats in de grammatica. De betekenis van een woord is datgene wat uitgelegd wordt in de uitleg van zijn betekenis. De betekenis van een woord is zijn doel. De betekenis van een zin is zijn gebruik; de methode waarop hij kan worden geverifieerd; wat wordt uitgelegd in de uitleg van zijn betekenis. De rechtvaardiging of de grond voor een bewering vormt zijn betekenis.
[Voor de bronvermelding voor ieder van deze uitspraken, zie Baker en Hacker 1980, blz. 685]



Erickson, J. E., Chin-Parker, S., & Ross, B. H. (in press). Inference and Classification Learning of Abstract Coherent Categories. Journal of Experimental Psychology: Learning, Memory and Cognition.

Erickson, Jane E.; Chin-Parker, Seth; Ross, Brian H. (2005). Inference and Classification Learning of Abstract Coherent Categories. Journal of Experimental Psychology: Learning, Memory, and Cognition. 31(1), Jan 2005, 86-99. http://www.apa.org/journals/xlm.html.

Michael F. Verde, Gregory L. Mutphy and Brian H. Ross (2005). Influence of multiple categories on the prediction of unknown properties. Memory & Cognition, 33, 479-487. pdf

3.3 Theoretical terms

time and simultaneity

To talk about time, about simultaneity at a distance, you have to synchronize your clocks. And is you want to synchronize your clocks, you have to start with one, flash a signal to the other, and adjust for the time that the flash takes to arrive.
What could be simpler? Yet with this procedural definition of time, the last piece of the relativity puzzle fell into place, changing physics forever.

Peter Galison (2003). Empires of time. Einstein's clocks, poincaré's maps. Norton. p. 13-14.

Why Galison's book? It is about the clock-coordinating procedure. It should contain everything on all levels of concreteness-abstractness: clocks, what they are supposed to do, coordinating clocks, the maps necessary to be able to do so, the philosophical puzzles this coordination poses, their solution by positing the most theoretical of theoretical terms. Would you believe Paris in the 1880's had a pneumatic pipe system to coordinate clocks in the city? (p. 93: the control room at the Rue du Télégraph)

3.4 Connectedness of terms


Figure 1. Vertical connectedness of terms in the 'intelligence' concept.

For the fundamental theory behind schemes like that in Figure 1, in the social sciences, see Borsboom, Mellenbergh and Van Heerden (2003) pdf

schema theorie

Figure 2. Scheme of the theory of falling bodies; ; v = speed reached, g = acceleration, t time passed, h = vertical distance passed.


Figure 3. Group of Dutch intelligence tests.

3.4 more literature

Entwistle, N. (1995). Frameworks for understanding as experienced in essay writing and in preparing for examinations. Educational Psychologist, 30, 47-54. abstract

3.5 Variants of definitions

schema definities

Figure 1. Scheme of possible ways of defining terms.

  1. The quantity of matter is the measure of the same thing, arising from its density and bulk conjointly.
  2. The quantity of motion is the measure of the same, arising from the velocity and quantity of matter conjointly.
  3. The vis insita , or innate force of matter, is a power of resisting, by which every body, as much as in it lies, continues in its present state, whether it be of rest, or of moving uniformly forwards in a right line.
  4. An impressed force is an action exerted upon a body , in order to change its state, either of rest, or of uniform motion in a right line.
  5. A centripetal force is that by which bodies are drawn or impelled, or any way tend, towards a point as to a centre.
  6. The absolute quantity of a centripetal force is the measure of the the same, proportional to the efficacy of the cause that propagates it from the centre, through the spaces round about
  7. The accelerative quantity of a centripetal force is the measure of the same, proportional to the velocity which it generates in a given time.
  8. The motive quantity of a centripetal force is the measure of the same, proportional to the motion which it generates in a given time.

Isaac Newton (1686/1729/1934). Principia: Sir Isaac Newton's Mathematical Principles of Natural Philosophy & His System of the World. p. 1.

.... we mean by any concept nothing more than a set of operations; the concept is synonymous with the corresponding set of operations. [p. 5, as cited in Joel 1999, p. 169]

P. W. Bridgman (1927). The logic of modern physics. New York: Macmillan
See also Joel Michell 1999, p. 169 ff on the operationism made popular by this book by Nobel Prize winner Bridgman. "For example, according to Bridgman, the length of an object is not a property it possesses independently of us (such as its extension in space) but is, instead, constituted by our operations of length measurement (such as bringing the object into the appropriate relation to a tape measure)." [p. 169]

This possibility of the operational definition was abused later by psychologists in claiming scientific status for concepts such as 'intelligence' as measured by 'intelligence tests.' On this theme see the work of Joel Michell.


how to use definitions: the case of mathematics

Miguel R. Wilhelmi, Juan D. Godino and Eduardo Lacasta (2007). Didactic effectiveness of mathematical definitions. The case of absolute value. International Electronic Journal of Mathematics Education, 2, numer 2. pdf

family resemblance

Aaron B. Hoffman and Gregory L. Murphy (2006). Category Dimensionality and Feature Knowledge: When More Features Are Learned as Easily as Fewer. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, No. 3, 301-315. pdf.

3.6 Literature

Denny Borsboom, Gideon J. Mellenbergh and Jaap van Heerden (2003). The theoretical status of latent variables. Psychological Review, 110, 203-219. pdf

Nancy Cartwright (1983) How the laws of physics lie. Open University Press.

David H. Jonassen (2006). On the role of concepts in learning and instructional design. Educational Technology: Research & Development, 54, 177-196.

Daniel Kahneman. A perspective on judgment and choice: Mapping bounded rationality. Or his December 8 2002 Nobel prize lecture: Maps of bounded rationality: A perspective on intuitive judgment and choice. pdf. Daniel Kahneman (2003). Experiences of collaborative research. American Psychologist. 58, 723-730. Zie ook zijn 2002 autobiografische artikel (een korte versie van het AP-artikel?): pdf

Deanna Kuhn (2005). Education for thinking. Harvard University Press. (first pages available as html)

G. Lamer, A. Schwope, L. Wisotzki, and L. Christensen (May 2, 2006). Strange magnification pattern in the large separation lens SDSS J1004+4112 from optical to X-rays. Article published by EDP Sciences and available at http://www.edpsciences.org/aa; A&A preprint doi http://dx.doi.org/10.1051/0004-6361:20064934. pdf

Isaac Newton (1686/1729/1934). Principia: Sir Isaac Newton's Mathematical Principles of Natural Philosophy & His System of the World. translation Andrew Motte; revision Florian Cajori. Dover-edition eighth printing 1974.

Amos Tversky and Daniel Kahneman (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110. html

Wayne A. Wickelgren (1977). Learning and memory. Englewood Cliffs: Prentice-Hall.

Wayne A. Wickelgren (1979). Cognitive psychology. Prentice Hall.

More literature

Holger Becker (2005). Semantische und lexikalische Aspekte der mathematischen Fachsprache des 19. Jahrhunderts. Dissertation. Carl von Ossietzky Universität Oldenburg. http://docserver.bis.uni-oldenburg.de/publikationen/dissertation/2006/becsem05/pdf/

Peter Achinstein (1968). Concepts of science. A philosophical analysis. Baltimore: The Johns Hopkins Press.

Ken Barker, Vinay K. Chaudhri, Shaw Yi Chaw, Peter E. Clark, James Fan, David Israel, Sunil Mishra, Bruce Porter, Pedro Romero, Dan Tecuci, Peter Yeh (2004). A Question-Answering System for AP Chemistry: Assessing KR&R Technologies. The Ninth International Conference on the Principles of Knowledge Representation and Reasoning (KR2004) pdf

Benjamin S. Bloom, J. Thomas Hastings and George F. Madaus (Eds) (1971). Handbook on formative and summative evaluation of student learning. London: McGraw-Hill.

Milic Capek (Ed.) (1976). The concepts of space and time. Their structure and their development. Dordrecht/Boston: Reidel.

John B. Carroll (????). Words, meanings, and concepts. Chapter 12 in Lorin W. Anderson (1985): Perspectives on School Learning: Selected Writings of John B. Carroll. Erlbaum. This might be an original text for the book, no publication data have been given on this item. II have yet to read this chapter (and get the book first, UB Leiden magazijn 3A, code 1990 C 8).

Nancy Cartwright's Philosophy of Science An International Workshop December 16-17, 2002 html on this page the contribution's abstracts, and download links.

Audrey B. Champagne, Richard F. Gunstone and Leopold E. Klopfer (1985). Instructional consequences of students' knowledge about physical phenomena. In Leo H. T. West and A. Leon Lines: Cognitive structure and conceptual change (pp. 61-90). Academic Press.

Eduard Jan Dijksterhuis (1950/1969). The mechanization of the world picture. London: Oxford University Press.

Susan E. Embretson (1996). Cognitive Design Principles and the Successful Performer: A Study on Spatial Ability. Journal of Educational Measurement, 33, p. 29 ff.

Avron Barr and Edward A. Feigenbaum (Eds) (1981). The handbook of artificial intelligence. Volume 1. Stanford: HeurisTech Press; Los Altos: Kaufmann. isbn 0865760055. Section III: Avron Barr and James Davidson (Eds) knowledge representation, articles: Robert Fillman on Logic - Douglas Appelt on semantic nets - Anne Gardner on semantic primitives - James Bennett on frames

Robert Glaser (1997). Assessment and education: Access and achievement. National Center for Research on Evaluation, Standards, and Student Testing (CRESST) Center for the Study of Evaluation (CSE) Graduate School of Education & Information Studies University of California, Los Angeles. http://www.cse.ucla.edu/Reports/TECH435.pdf [dead link? 2-2008]

Clark Glymour (1998). What went wrong? Reflections on science by observation and The bell curve. Philosophy of Science, 65, 1-32. pdf

Clark Glymour (2003). Learning, prediction and causal Bayes nets. TRENDS in Cognitive Sciences, 7, 43-48 pdf

Gopnik, Alison, Clark Glymour, David M. Sobel, Laura E. Schulz, Tamar Kushnir, and David Danks. A theory of causal learning in children: Causal maps and Bayes nets," Psychological Review, Vol. 111, No. 1 (2004). pdf

Gopnik, A. and Nazzi, T. (2001) Words, kinds and causal powers: a theory theory perspective on early naming and categorization. Early Categorization (Rakison, D., Oakes, L. eds), Oxford University Press. pdf

Wallace Hannum (1988). Designing courseware to fit subject matter structure. In David H. Jonassen (Ed.) (1988). Instructional designs for microcomputer courseware (pp. 275-296). Erlbaum.

Guershon Harel and Larry Sowder (2005). Advanced mathematical-thinking at any age: Its nature and its development. Mathematical Thinking and Learning, 7, 27-50.

Barbara Koslowski (1996). Theory and evidence. The development of scientific reasoning. MIT Press.

Chwee Beng Lee (2005). Modeling conceptual change in problem solving. Paper at Redesigning pedagogy, research, policy, practice. Singapore 2005 http://conference.nie.edu.sg/paper/New%20Converted/ab00452.pdf [dead link? 2-2008]

Krittaya Leelawong, Joan Davis, Nancy Vye and Gautam Biswas (2002). The effects of feedback in supporting learning by teaching in a teachable agent environment pdf.

Wendy G. Lehnert (1978). The process of queston answering. A computer simulation of cognition. Erlbaum.

Po-Hung Liu (2002). Developing college students' views on mathematical thinking in a historical approach, problem-based calculus course. http://www.math.uoc.gr/~ictm2/Proceedings/pap191.pdf 2nd INTERNATIONAL CONFERENCE ON THETEACHING OF MATHEMATICS  paper

Wolfram Meyerhöfer (2005). Was misst TIMSS? Einige Überlegungen zum Problem der Interpretierbarkeit der erhobenen Daten. pdf

E. R. Michener (1978). Understanding understanding mathematics. Cognitive Science, 2, 361-383.

Clayton T. Morrison, Tim Oates, and Gary King (2001). Grounding the Unobservable in the Observable: The Role and Representation of Hidden State in Concept Formation and Refinement. In Working Notes of AAAI Spring Symposium Workshop: Learning Grounded Representations. http://eksl.cs.umass.edu/papers/morrison-sss01.pdf [dead link? 2-2008]

Dwayne H. Mulder (1998). Explanation, Understanding, and Subjectivity. html

Herman E. Stark (1998). Expertise and Rationality html

A selective biography of the philosophy of science. Compiled by Ward E. Jones and Samir Okasha with W. H. Newton-Smith. Oxford: Sub-Faculty of Philosophy, 1998. http://www.herts.ac.uk/philosophy/scibib.html [dead link, Dec. 2008]

Joseph D. Novak and Alberto J. Canas (2006). The Theory Underlying Concept Maps and How To Construct Them. Technical report IHMC Cmap Tools 2006-01, Florida Institute for Human and Machine Cognition

R. Nozick (1981). Philosophical Explanations. Cambridge, MA: Harvard University Press.

Nicos Papadouris and Costas Constantinou (2003). Epistemological and Reasoning Difficulties Associated with Scientific Experimentation. Paper The 4th ESERA Conference, Noordwijkerhout, The Netherlands.pdf

W. Schnotz, S. Vosiadou and M. carretero (Eds) (1999). New perspectives on conceptual change. Amstrdam: Pergamon. [nog niet gezien, niet in UB Leiden]

E. E. Smith and D. L. Medin (1981). Categories and concepts. Cambridge, Massachusetts, Harvard University Press.

John L. Pollock (1974). Knowledge and justification. Princeton University Press.

Rehder, B. (2001) A causal model theory of categorization Proceedings of the 21st Annual Conference of the Cognitive Science Society (Hahn, M., Stoness, S.C. eds), pp. 595–600, Vancouver. pdf

Ernest Sosa and Jaegwon Kim (Eds) (2002). Epistemology. An anthology. Blackwell. isbn 0631197249

Monika Walczak (1998). The Classical Conception Of Rationality html

T. Williamson (2000). Knowledge and its limits. Oxford: Oxford University Press.

Paul Brna (n.d.). Conceptual change and mental models. A bibliography. html


Examples of cognitive maps

Albert L. Stevens and Allan Collins: Multiple conceptual models of a complex system. In Richard E. Snow, Pat-Anthony. Federico and William E. Montague (Eds.) (1980). Aptitude, learning and instruction. Volume 2: cognitive process analyses of learning and problem solving (p. 177-198). Erlbaum.

James C. Lester, Bruce W. Porter (1995). Developing and Empirically Evaluating Robust Explanation Generators: The KNIGHT Experiments. Also in: Computational Linguistics, 23, 65-101.

Byron Marshall, Hsinchun Chen, and Therani Madhusudan (accepted 2005). Matching Knowledge Elements in Concept Maps using a Similarity Flooding Algorithm. Decision Support Systems. http://oregonstate.edu/~marshaby/Papers/MatchKnowledgeElements_PrePrintVersion.pdf

Diana C. Rice, Joseph M. Ryan and Sara M. Samson (1998). Using Concept Maps to Assess Student Learning in the Science Classroom: Must Different Methods Compete? Journal of Research in Science Teaching, 35, 1103-1127. http://www.clab.edc.uoc.gr/hy302/papers%5Cconcept_map%20assesment.pdf

Karoline Afamasaga-Fuata'i (2004). An undergraduate student's understanding of differential equations through concept maps and vee diagrams. A. J. Canas, J. D. Novak, F. M. González, Eds. Proc. of the First Int. Conference on Concept Mapping Pamplona, Spain 2004 pdf

Maria Araceli Ruiz-Primo, Richard J. Shavelson and Susan Elise Schultz (1997). On The Validity Of Concept Map-Base Assessment Interpretations: An Experiment Testing The Assumption Of Hierarchical Concept Maps In Science. CSE Technical Report 455. pdf

Maria Araceli Ruiz-Primo, Susan Elise Schultz and Richard J. Shavelson (1997). Concept Map-Based Assessment in Science: Two Exploratory Studies. CSE Technical Report 436. pdf

John H. Milam, Jr., Susan A. Santo and Lisa A. Heaton (2000?). Concept Maps for Web-Based Applications. ERIC Technical Report. pdf Terhi Mäntylä and Ismo T Koponen (2003). Understanding the character of physical laws through the intertwined epistemic roles of experiments and models: An example from physics teacher education. The ESERA 2003 http://www1.phys.uu.nl/esera2003/program.shtml conference at Noordwijkerhout on Research and the quality of science education. pdf

Druzdzel, Marek and Clark Glymour. "What Do College Ranking Data Tell Us About Student Retention?" Proceedings of the 1994 AAAI Workshop on Knowledge Discovery in Databases. pdf

conceptual graphs

David J. Cox (2005). Explanation by pattern means massive simplication. How to use Patterns to Explain Complex Subjects, or Symbolic Logic Outridden. A Public Service site. The FLIPP Diagram technique.

Robyn M. Dawes (1988). Rational choice in an uncertain world. London: Harcourt Brace Jovanovich. (a new edition has appeared 2001?) (could be a source of inspiration for Deanna Kuhn) (Interesting possibility: rational choice of the student in an uncertain world of being assessed; rational choice of the teacher in an uncertain world of trying to fathom the individual student's mastery. Nothing of the sort has been done by Robyn Dawes, it is home work for you and me)

Robert J. Mislevy (1993). A framework for studying differences between multiple-choice and free-response test items. In Randy Elliot Bennett and William C. Ward Construction versus choice in cognitive measurement (p. 75-106). Erlbaum.


Common sense / Folk science

I would like to bring together publications on the theme of the distance between folk science and science, and how education can bridge it (if it sees the problem at all). As an example of what I am after, see Rakic, below. If you have any suggestions, mail them.

en.wikipedia folk science

Natasa Rakic (1997). Common sense, time and special relativity. Dissertation University of Amsterdam.


The Archimedes Project http://archimedes2.mpiwg-berlin.mpg.de/archimedes_templates/project.htm

Concept Inventories site. This work is inspired by earlier work of D. Hestenes and others, see here for downloadable publications.

Paul Hewitt Conceptual physics website (Picture 'Gravity at Play' on his website)

Gregory K. W. K. Chung and Eva L. Baker (1997). Year 1 Technology studies: Implications for technology in assessment. CSE Technical Report 459. http://www.cse.ucla.edu/Reports/TECH459.pdf [dead link? 2-2008] (oa concept mapper)

IHMC Cmap Tools site

authentic testing

Authentic testing is somewhat of an issue in the literature on achievement testing. Originally, it is very much an American issue, a reaction to the overwhelming use of standardized tests using a limited choice of testing formats. As such, it is much less an issue in European continental countries where national examinations are used, narrowly defined on course content. Dwyer (1993, p. 269) cites Mitchell (1989, 1990, see Dwyer): "Authentic assessment is continuous with instruction: It complements and expands the curriculum, and preparation for the assessment should literally be the curriculum."

Carol Anne Dwyer (1993). Innovation and reform: Examples from teacher assessment. In Randy Elliot Bennett, William C. Ward: Construction versus choice in cognitive measurement: Issues in Constructed Response, Performance Testing, and Portfolio Assessment (p. 265-290). Erlbaum.

May 4, 2007 \ contact ben at at at benwilbrink.nl     Valid HTML 4.01!   http://www.benwilbrink.nl/projecten/06examples3.htm