Original publication 'Toetsvragen schrijven' 1983 Utrecht: Het Spectrum, Aula 809, Onderwijskundige Reeks voor het Hoger Onderwijs ISBN 90-274-6674-0. The 2006 text is a revised text.

Item writing

Techniques for the design of items for teacher-made tests

3. Course content inventory

Ben Wilbrink

this database of examples has yet to be constructed. Suggestions? Mail me.

This chapter is about what a body of knowledge is, and how to characterize it. It should also tackle the question what it is to have one, get one in the first place, or change one's wrong one. Strictly taken, it is not necessary first to describe the course content in its constituting terms, relations between terms, and descriptions giving otherwise meaning to certain of its terms. But it will surely furnish an excellent starting point for the design of test items.

Usually a 'body of knowledge' is understood to be a discupline such as English, mathematics, or chemistry. Think, however, also of the unusual interpretation, that it is 'critical thinking,' or Deanna Kuhn's 'inquiry and argument,' or even her (2005) proposal to organise a curriculum around the concept of 'causality,' instead of the traditional content nobody can convingly tell you the reasons for (which reminds me of the way assessment typically is regarded, and in its turn that is what made me research the history of educational assessment html).

A useful resource for this description of course content is the technique of schematizing, see the chapter 3 text, or the CmapTools site. A chapter on the concept of intelligence might be characterized by the following scheme, using terms that are on different levels of abstraction. Distinguishing this kind of levels of abstraction is in its turn a powerful resource in designing questions on course content.

Figure 1. Scheme of terms related to the concept of intelligence. The scheme was constructed using CmapTools, software to picture knowledge structures http://cmap.ihmc.us/.

Dominic G. B. Edelen (1962). The structure of field space. An axiomatic formulation of field physics. University of California Press.

• p. 36 description of the concept of field: "Matter, per se, is a priori observable on a macroscopic level, in that it forms the basis of our known facts of experience, and we know how to make observations of macroscopic matter in the laboratory. Such observations are represented mathematically by describing the fibers of the amtter continuum (see Secs. 2 and 3 of Chap. VI for a discussion of the fibers of the matter continuum). When we say that we observe fields, however, what we mean is that we observe matter being acted upon by something that we term a field. We do not, in fact, observe the fields themselves, but only their effects on 'test particles.' Our basi meaning of field - the way in which we fundamentally define a field - is given in terms of interactions with test particles. When we write down a system of equations to 'describe' a field, we mean that these equations describe quantities determining the magnitude, direction, and so forth, of the interactions of a something termed a field, the presence of which results in certain characteristic changes in the fibers of the matter continuum. From the 'observable facts of experience' criterion, a field thus has no a priori existential properties in itself, but rather is an abstraction [my emphasis] arrived at by a classification of the types and natures of interactions between matter and the something that we term a field."

reflection

What is this thing called 'course content'? The immedeate answer to this question is obvious: it is whatever is presented as course content to the student. And obviously this answer is hopelessly inadequate. It is in the same league as 'intelligence is whatever it is the intelligence test measures.'

Usually, in order to help the student solve this puzzle, some meta-statements are added in the course book, expressing what its content is about, and/or what kind of mastery the student should be able to demonstrate at the close of the course or at the end of course achievement test. Lacking this, it will be shown what kind of questions the student should be able to answer at the summative test. If these questions have beeen designed by teachers believing item writing is an art, students will be shown art and will therefore not get well informed about what it is that is considered mastery of the course content.

Take the course that should result in a definite skill: teaching, combat, singing. The point is not that skills somehow are simpler than mastery of intellectual content, they probably are not. The point is that there is unity between the skill as taught, the skill as needed in the profession, and the skill as assessed in the exit test. In the first centuries of their existence, European university examinations were demonstrations of teaching skill, and the examinees earned the 'licence to teach' everywhere in the realm of the catholic church. Students exercised/learned whatever was deemed necessary to acquire this skill of teaching. In the medieval guilds the training was wholly analogous to that in the university guild of masters.
Since the twelfth century educational institutions and processes have become somewhat more complicated, it is not now self evident what it is that particular courses or text books are about, even if it is announced on their page one.

A fundamental approach, such as needed by the educational technologist automating item design, would probably suffocate itself in the large number of translations that has to made: intentions of text book writers, content of text books, constructs of this course contents made by the students, a conception of what it is to test for knowledge of course content, developing the kind of questions that match this, etcetera. The definitely simpler task of developing computerized courseware had to struggle with such a cascade of 'knowledge maps.'

• William J. Clancey, (1987). Knowledge based tutoring. The GUIDON Program. Cambridge, Mass.: MIT.
  
• Joseph Beck, Mia Stern, and Erik Haugsjaa (1996). Applications of AI in Education, ACM Crossroads Student Magazine, Fall 1996, 3.1, Issue on Artificial Intelligence. html
  • Five models, see their figure
    
  
  
• Sophiana Chua Abdullah (2003). Student modelling by adaptive testing - a knowledge-based approach. A thesis submitted to the University of Kent at Canterbury in the subject of computer science. pdf
  • from the abstract Several experiments in knowledge acquisition for the construction of an adaptive test are described, in particular, experiments to elicit information for the domain knowledge, the student model and the problem progression strategy. It shows how a description of a particular problem domain may be captured using traditional techniques that are supported by software developed in the constraint logic extension to Prolog. It also discusses knowledge acquisition techniques for determining the sequence in which questions should be asked.
• Laura Lising and Andrew Elby (2004?). The impact of epistemology on learning: a case study from introductory physics. American Journal of Physics. pdf
  • summary Summary. This paper presents a case study of the influence of epistemology on learning for a student (Jan) in a reformed, introductory college physics course. Analysis of videotaped class work, written work, and interviews shows that many of Jan's difficulties in class are epistemological in nature. Our primary goal is to show instructors and curriculum developers that a student's epistemological stance - her tacit or explicit views about knowledge and learning - have a direct, causal influence on her physics learning.

  
  

By now you will be susceptible to the idea of a shortcut, to avoid the mess of so many different models that have to be made explicit, and then must be related to each other. The shortcut is to highlight, schematize, or whatever other technique you as teacher would like to use, what it is that is important in your course, and whatever else is supportive of this core content. The second step of the shortcut is to design a small number of typical test items - at the formative or summative level - that students should be able to handle and answer. And that is it. Do not forget to evaluate the success of your efforts, and learn from your mistakes and successes.

Be aware of your own secretly harbored models of what it is students might have learned in your course, what it is failed students have failed to master, what it is to 'assess' student mastery, what your own 'expertness' in teaching and assessment is. Models abound. Get rid of the junk. Searching the internet for Clancey's book title teaches us that this kind of modeling approach is not used abundantly any longer in AI research and course design (but see http://monet.aber.ac.uk:8080/monet/index.html, or edutechwiki.unige.ch).

"The greatest enemy of understanding is coverage" (Howard Gardner, Educational Leadership 50/7 April, 1993) [I must look this up]

K. Barker, B. Porter, and P. Clark. (2001). A Library of Generic Concepts for Composing Knowledge Bases. First International Conference on Knowledge Capture, October 21-23. abstract pdf

P. Clark and B. Porter (1997). Building Concept Representations from Reusable Components. AAAI'97, 369-376, CA:AAAI Press. pdf

• abstract Our goal is to build knowledge-based systems capable of answering a wide variety of questions, including questions that are unanticipated when the knowledge base is built. For systems to achieve this level of competence and generality, they require the ability to dynamically construct new concept representations, and to do so in response to the questions and tasks posed to them. Our approach to meeting this requirement is to build knowledge bases of generalized, representational components, and to develop methods for automatically composing components on demand. This work extends the normal inheritance approach used in frame-based systems, and imports ideas from several different areas of AI, in particular compositional modeling, terminological reasoning, and ontological engineering. The contribution of this work is a novel integration of these methods that improves the efficiency of building knowledge bases and the robustness of using them.

mental models - erroneous beliefs

This book uses extensively the naive conception - mental model - of the laws of movement. Much research has been done on the character, pervasiveness and robustness of this particular mental model, roughly the Aristotelian view of physics, versus the classical, Newtonian, view. It is to be expected that there are countless mental models, in physics as well as in other disciplines, that play havoc with instruction and assessment if going unheeded. The challenge for instruction as well as assessment is to effect the necessary conceptual change, and test for it. Searching the internet for 'conceptual change' will result in many relevant sources, however, a short review of the literature is presented by Jonassen (2006), not online.

It was not the intention of Kahneman to make a mockery of intuitive expert judgments in general in this - his Nobel laureate prize lecture. Quite to the contrary, as his lecture testifies. The systematic erroneous judgment on this kind of statistical events therefore is a quite specific mental model.
In due time I will collect the best known mental models. Undoubtedly the belief in the law of small numbers belongs to that category, it was researched by Tversky and Kahneman (1971 html); this one is able to make fools of statisticians and researchers knowledgeable in statistics, beware.

mental models - conceptual change

David R. Kaufman, Stella Vosniadou, Andy diSessa and paul Thagard (2000). Scientific explanation, systematicity, and conceptual change. Symposium. html

• "Conceptual change is the subject of considerable research across all of the cognitive sciences. In particular, it is central to investigations in the philosophy of science, cognitive development, and science education. "

Extending the conceptual change approach to mathematics learning and teaching. Learning and Instruction, 14, 2004, 445 and following: special issue. editorial
Stamatia Stafylidou and Stella Vosniadou (2004). The development of students' understanding of the numerical value of fractions. Learning and Instruction, 14, 503-518. pdf
Xenia Vamvakoussi and Stella Vosniadou (2004). Understanding the structure of the set of rational numbers: a conceptual change approach. Learning and Instruction, 14, 453-467. pdf
Stella Vosniadou en Lieven Verschaffel (2004). Extending the conceptual change approach to mathematics learning and teaching. Learning and Instruction, 14 pdf

Stacy Rebich and Catherine Gautier (2005). Journal of Geoscience Education, v. 53, n. 4, 355-365. pdf

Susan Carey (1985). Conceptual change in childhood. MIT Press.

• " ... a case study of children's acquisition of biological knowledge between ages 4-10. Drawing on evidence from a variety of sources, Carey analyzes the ways that knowledge is restructured during this development, comparing them to the ways that knowledge is restructured by an adult learner, and to the ways that conceptual frameworks have shifted in the history of science."
• UB Leiden: biblitheek CURIUM [nog niet gezien]; KB
• Susan C. Johnson and Susan Carey (1998). Knowledge Enrichment and Conceptual Change in Folkbiology: Evidence from Williams Syndrome. Cognitive Psychology, 37, 156-200. pdf
• Susan Carey (2004?). Science Education as Conceptual Change. House Committee on Science. html

3.1 Observable terms

At a down-to-Earth level, there are concrete things and events, some of which are the subject of the discipline the course content is about. Let us start by gazing in the skies, however. They have always been somewhat mysterious, it nog being quite clear what exactly it is one sees happening there. A photograph of a small part of the sky will show us things and events in somewhat the same way we are used to observe using our eyes without technical resources. Easily the most spectacular photograph I have ever seen in my life, is the following, containing a lot of 'observables,' and leaving a lot to explain about what exactly it is we think to see here. It also illustrates an unusual kind of scheme, abstract terms superposed on an observable level picture. A 'lensed quasar' is a quasar that is represented in our observation in multiple images, in this case a 'direct' one in the middle, and four 'mirrored' ones. The phenomenon is caused by a cluster of galaxies between us and the quasar, this cluster by its sheer mass 'bending' the curvature of space, bending the path of light. Knowing the relativistic 'mechanism,' makes us sure the five images 'really' belong to one and the same bright galaxy. Wow, what a theory, and what a picture! Looking at the quasar is looking back ten billion years in time, lightyears in distance. Look at the yet more detailed photograph one at hubblesite.org

Figure 2. The ultimate example of interpretation of observations? The 12 billion years old lensed galaxy's image gets magnified and tripled (red circle/ovals) by a cluster of galaxies functioning as a lens by force of its gravitational field. More detailed pictures on html or direct as jpg. Credit: ESA, NASA, K. Sharon (Tel Aviv University) and E. Ofek (Caltech). For the scientific interpretation see Lamer e.a. (2006) pdf

The Hubble telescope is a super high tech instrument orbiting the Earth, making it possible to construct the most fantastic 'images' of phenomena in the Universe. Interestingly, for the very first telescope known, the same could be said. What would one see if it were pointed to the sky, to Jupiter for example? And how to interpret what one saw? Here is another example where it is far from self-evident that what you 'seeing' is self-evident.

Figure 3. Looks less spectacular, but was definitely more so at the time: AD 1610. From Galileo Galilei's letter to the Prince of Venice, describing the discovery of four big moons of Jupiter (html; Galilei's complete works in Italian available as pdf on http://gallica.bnf.fr/, this letter in volume III part II, p. 427). Another argument that the Earth is not the middlepoint of the Universe. Some of his contemporaries could not 'see' Jupiter's moons through Galileo's own telescope. The meaning of what there is to see, can not be 'seen.' See also the Voyager images of the moons html, and this page from The Galileo Project html.

And yes, using new technologies, it is possible to construct images of 'events' that do not happen at all, they exist only in the 'conctructed eye' of the beholder. Look at Jupiter's atmosphere whirling. The illusion is that you see Jupiter rotating, but that is not so. The film is made of images taken exactly one 'Jupiter-day' apart. Start the film.

Figure 4. Making movement visible by compressing time, or showing something which does not exist as such (producing an artifact): movements in Jupiter's atmosphere as observed by the approaching Voyager 1. For the film, click the picture. The film does not show Jupiter's rotation! The instantaneous black spots on this still are shadows of two of the four big Jupiter moons. The animation is taken from the beautiful site of the Dutch Volkssterrenwacht Urania html, see there also for Jupiter in color.

observation failure

The Iran Air Flight 655 case is a pretty tough one. Professionals should however be prepared for this kind of mistake; strong expectations coloring what they 'see' happening. Ordinary people should be prepared also: driving your car inattentively may get you into trouble if you are not in fact at the point you think you are (the end of this road is still further on ....). In assessment: be aware of the possibility that your expectations might determine your grading, instead of the facts. We are getting into psychological territory here.

psychology

learning categories

W. Ahn, R. L. Goldstone, B. C. Love, A. B. Markman, & P. Wolff (Eds.), Categorization inside and outside the lab: Festschrift in honor of Douglas L. Medin. Washington, DC: APA. [nog opzoeken]

Andrea L. Patalano, Seth Chin-Parker, and Brian H. Ross (2003). The Role of Coherence in Category-Based Explanation. 910-915. pdf

• from the abstract Category coherence refers to the extent to which a category is perceived to be a meaningful whole (Patalano & Ross, 2002; Ross & Patalano, 2002). from the general discussion The experiments described here provide an initial understanding of one aspect of category structure - category coherence - that influences category selection in the service of category-based causal explanation.
• The introduction offers an overview of the field.

observational concept learning. Introductory: Wickelgren (1977 p. 274-76). Learning concepts in the natural world, or in the laboratory world of the psychologist (Hull, 1922, is mentioned by Wickelgren, using Chinese characters).

inferential concept learning. Introductory: Wickelgren (1977 p. 276). The classic study here is Bruner, Goodnow and Austin (1956) using Harvard students as subjects. Studies such as these will suggests instructional methods, or explain why they are effective, and therefore also how to design test items.

Chin-Parker, S., & Ross, B. H. (2004). Diagnosticity and prototypicality in category learning: A comparison of inference learning and classification learning. Journal of Experimental Psychology: Learning, Memory and Cognition, 30, 216-226.

types of concepts and attributes. Introductory: Wickelgren (1977 p. 278). The regular concept is of the conjunctive type: examples share one or more defining characteristics. The disjunctive concept comprises examples having either characteristic A or characteristic B, and ar called inclusive-or if having both characteristics also defines an example, and exclisive-or if examples do not have both characteristics. Implicational concepts: an example belongs to the concept if its having characteristic A implies it also has characteristic B. In this way the playing field rapidly can become quite complex, without even talking about more the fuzzier kinds of concepts such as defined by family resemblances etcetera. [Ulrich Neisser and P. Weene (1962). Hierarchies in concept attainment. Journal of Experimental psychology, 64, 644-645.]

nonexamples. Introductory: Wickelgren (1977 p. 278-9). [Hovland and Weiss (1953). Transmission of information concerning concepts through positive and negative instances. Journal of Experimental psychology, 45, 165-182.] [W. Freibergs and Endel Tulving (1961). The effect of practice on utilization of information from positive and negative instances in concept formation. Canadian Journal of Psychology, 15, 101-106. M. J. Fryatt and Endel Tulving (1963). Interproblem transfer in identification of concepts involving positive and negative instances. Canadian Journal of Psychology, 17, 106-117. ]

typicality Gregory L. Murphy and Brian H. Ross (2005). The two faces of typicality in category-based induction. Cognition, 95, 175-200. pdf

• from the abstract Studies of category-based induction using different methods have found somewhat contradictory results for whether typical items are a stronger basis for induction.
• p. 175 [induction] "A central function of categories is their ability to tell us properties of novel entities. When going to a new restaurant, we know what to expect and what to do; when ordering the chicken, we have some idea of how it will taste and what its nutritional qualities will be; when picking up the fork and knife, we know what they are to be used for. Even though we may never have been in the restaurant before and almost certainly have never encountered that chicken or that silverware before, we can use our knowledge of the categories they are members of to make fairly accurate predictions about them. Thus, category-based induction is an important part of intelligent behavior."

Educational Psychologist (2004), volume 39 nr 1. Thematic on personal epistemology. Not directed specifically at assessment, however.

philosophy

3.2 Abstract terms and constructs

psychology

inference/induction

Erickson, J. E., Chin-Parker, S., & Ross, B. H. (in press). Inference and Classification Learning of Abstract Coherent Categories. Journal of Experimental Psychology: Learning, Memory and Cognition.

Erickson, Jane E.; Chin-Parker, Seth; Ross, Brian H. (2005). Inference and Classification Learning of Abstract Coherent Categories. Journal of Experimental Psychology: Learning, Memory, and Cognition. 31(1), Jan 2005, 86-99. http://www.apa.org/journals/xlm.html.

• abstract Category learning research has primarily focused on how people learn to classify items using simple observable features. However, classification is only 1 way to learn categories. In addition, many concepts have an underlying coherence that explains the featural similarity among exemplars, such as abstract coherent concepts whose instances differ greatly on their observable features. In 3 experiments, the authors investigated how abstract coherent categories are acquired through 2 common means of category learning, classification and inference. Because inference promotes more focus on within-category information than does classification, they hypothesized that inference learning would lead to a better understanding of the underlying coherence of abstract coherent categories. All 3 experiments support this prediction.

Michael F. Verde, Gregory L. Mutphy and Brian H. Ross (2005). Influence of multiple categories on the prediction of unknown properties. Memory & Cognition, 33, 479-487. pdf

• "Knowing an item's category helps us predict its unknown properties."

3.3 Theoretical terms

time and simultaneity

Why Galison's book? It is about the clock-coordinating procedure. It should contain everything on all levels of concreteness-abstractness: clocks, what they are supposed to do, coordinating clocks, the maps necessary to be able to do so, the philosophical puzzles this coordination poses, their solution by positing the most theoretical of theoretical terms. Would you believe Paris in the 1880's had a pneumatic pipe system to coordinate clocks in the city? (p. 93: the control room at the Rue du Télégraph)

3.4 Connectedness of terms

Figure 1. Vertical connectedness of terms in the 'intelligence' concept.

For the fundamental theory behind schemes like that in Figure 1, in the social sciences, see Borsboom, Mellenbergh and Van Heerden (2003) pdf

Figure 2. Scheme of the theory of falling bodies; ; v = speed reached, g = acceleration, t time passed, h = vertical distance passed.

Figure 3. Group of Dutch intelligence tests.

3.4 more literature

Entwistle, N. (1995). Frameworks for understanding as experienced in essay writing and in preparing for examinations. Educational Psychologist, 30, 47-54. abstract

• The understanding as described by these students has much to do with coherence between the bits and pieces of information in the course content, the connectedness
• from the abstract: "Those who had reached a deep level of understanding also reported being able to perceive and review frameworks for their understandings in ways suggesting an awareness of knowledge objects experienced in a quasi-sensory way." (This 'knowledge object' concept is somewhat peculiar; yet it is, in my opnion as well as experience, a 'critical incident' phenomenon in learning in educational contexts. It is not a phenomenon of knowing many facts and their details; it is a deeper understanding of the particular field that comes together with a firm grip on its cognitive contents; a kind of perfect understanding allowing the adequate solution of any problem posed (assuming it allows for an adequate solution, of course).

3.5 Variants of definitions

Figure 1. Scheme of possible ways of defining terms.

This possibility of the operational definition was abused later by psychologists in claiming scientific status for concepts such as 'intelligence' as measured by 'intelligence tests.' On this theme see the work of Joel Michell.

psychology

how to use definitions: the case of mathematics

Miguel R. Wilhelmi, Juan D. Godino and Eduardo Lacasta (2007). Didactic effectiveness of mathematical definitions. The case of absolute value. International Electronic Journal of Mathematics Education, 2, numer 2. pdf

family resemblance

Aaron B. Hoffman and Gregory L. Murphy (2006). Category Dimensionality and Feature Knowledge: When More Features Are Learned as Easily as Fewer. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, No. 3, 301-315. pdf.

• from the abstract These results show that rather than interfering with learning, categories with more dimensions cause individuals to learn more. The authors contrast the learning of family resemblance categories with learning in classical conditioning and probability learning paradigms, in which competition among features is well documented.

3.6 Literature

Denny Borsboom, Gideon J. Mellenbergh and Jaap van Heerden (2003). The theoretical status of latent variables. Psychological Review, 110, 203-219. pdf

• This article examines the theoretical status of latent variables as used in modern test theory models. First, it is argued that a consistent interpretation of such models requires a realist ontology for latent variables. Second, the relation between latent variables and their indicators is discussed. It is maintained that this relation can be interpreted as a causal one but that in measurement models for interindividual differences the relation does not apply to the level of the individual person. To substantiate intraindividual causal conclusions, one must explicitly represent individual level processes in the measurement model. Several research strategies that may be useful in this respect are discussed, and a typology of constructs is proposed on the basis of this analysis. The need to link individual processes to latent variable models for interindividual differences is emphasized.

Nancy Cartwright (1983) How the laws of physics lie. Open University Press.

David H. Jonassen (2006). On the role of concepts in learning and instructional design. Educational Technology: Research & Development, 54, 177-196.

Daniel Kahneman. A perspective on judgment and choice: Mapping bounded rationality. Or his December 8 2002 Nobel prize lecture: Maps of bounded rationality: A perspective on intuitive judgment and choice. pdf. Daniel Kahneman (2003). Experiences of collaborative research. American Psychologist. 58, 723-730. Zie ook zijn 2002 autobiografische artikel (een korte versie van het AP-artikel?): pdf

Deanna Kuhn (2005). Education for thinking. Harvard University Press. (first pages available as html)

• See also the Education for Thinking Project home page
• Kuhn's articles are not freely available on the internet, however, see
• Corinne Zimmerman (2005) The Development of Scientific Reasoning Skills: What Psychologists Contribute to an Understanding of Elementary Science Learning. Final Draft of a Report to the National Research Council Committee on Science Learning Kindergarten through Eighth Grade. pdf
• Jacqueline P. Leighton, Rebecca J. Gokiert and Ying Cui (2005). Investigating the Statistical and Cognitive Dimensions in Large-Scale Science Assessments: Causal and Categorical Reasoning in Science. AERA paper pdf
  • This second objective is focused on providing increasing support for the view of dynamic performance in science achievement. The results of the present study indicate that science assessments involve at least two substantive dimensions to which students reactÑcausal reasoning and categorical reasoningÑdescribed in the scientific reasoning literature (Kuhn & Deane, 2004).

G. Lamer, A. Schwope, L. Wisotzki, and L. Christensen (May 2, 2006). Strange magnification pattern in the large separation lens SDSS J1004+4112 from optical to X-rays. Article published by EDP Sciences and available at http://www.edpsciences.org/aa; A&A preprint doi http://dx.doi.org/10.1051/0004-6361:20064934. pdf

Isaac Newton (1686/1729/1934). Principia: Sir Isaac Newton's Mathematical Principles of Natural Philosophy & His System of the World. translation Andrew Motte; revision Florian Cajori. Dover-edition eighth printing 1974.

Amos Tversky and Daniel Kahneman (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110. html

• abstract People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psychological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions.

Wayne A. Wickelgren (1977). Learning and memory. Englewood Cliffs: Prentice-Hall.

Wayne A. Wickelgren (1979). Cognitive psychology. Prentice Hall.

More literature

Holger Becker (2005). Semantische und lexikalische Aspekte der mathematischen Fachsprache des 19. Jahrhunderts. Dissertation. Carl von Ossietzky Universität Oldenburg. http://docserver.bis.uni-oldenburg.de/publikationen/dissertation/2006/becsem05/pdf/

• "In dieser Arbeit geht es um die Frage nach der Entstehung und Entwicklung einer Wissenschaftssprache."

Peter Achinstein (1968). Concepts of science. A philosophical analysis. Baltimore: The Johns Hopkins Press.

• Definitions, The interpretation of terms, Theories, Observational terms, Theoretical terms, Analogies and models, On a semantical theory of models
• Bertrand Russell's entry on the theory of knowledge for The Encyclopaedia Britannica html
  The question how knowledge should be defined is perhaps the most important and difficult of the three [the others: data; methods of inference] with which we shall deal. This may seem surprising: at first sight it might be thought that knowledge might be defined as belief which is in agreement with the facts. The trouble is that no one knows what a belief is, no one knows what a fact is, and no one knows what sort of agreement between them would make a belief true.

Ken Barker, Vinay K. Chaudhri, Shaw Yi Chaw, Peter E. Clark, James Fan, David Israel, Sunil Mishra, Bruce Porter, Pedro Romero, Dan Tecuci, Peter Yeh (2004). A Question-Answering System for AP Chemistry: Assessing KR&R Technologies. The Ninth International Conference on the Principles of Knowledge Representation and Reasoning (KR2004) pdf

• abstract

Benjamin S. Bloom, J. Thomas Hastings and George F. Madaus (Eds) (1971). Handbook on formative and summative evaluation of student learning. London: McGraw-Hill.

• The educational philosophy is given in the very first chapter, and best summarized as (p. 17) "We take the position that the main task of the education process is to change the learners in desirable ways, and that it is the primary task of teachers and curriculum makers to specify in precise terms the ways in which students will be altered by the learning process." With due respect, but this is a rather authoritarian position, just a bit amazing given the year of publication (1971) and the authors' idealism in educational matters. In 2006 it is perfectly clear that American youth is not served well under this philosophy. For the alternative, see Deanna Kuhn (2005).
• This fantastic handbook is built on the cognitive taxonomy system, and the learning for mastery idealism. Both, of course, are rather obsolete in 2006, the first for lack of empirical evidence, the second for its rather sectarial character.
• Nevertheless, the book is a fountain of examples of descriptions/schemes of course material, and as such it might be useful to its proud possessors as a rich heuristic device.

Milic Capek (Ed.) (1976). The concepts of space and time. Their structure and their development. Dordrecht/Boston: Reidel.

• Philip Catton (). The most measured understanding of spacetime. doc

John B. Carroll (????). Words, meanings, and concepts. Chapter 12 in Lorin W. Anderson (1985): Perspectives on School Learning: Selected Writings of John B. Carroll. Erlbaum. This might be an original text for the book, no publication data have been given on this item. II have yet to read this chapter (and get the book first, UB Leiden magazijn 3A, code 1990 C 8).

• first paragraph: " The teaching of words, and of the meanings and concepts they designate or convey, is one of the principal tasks of teachers at all levels of education. It is a concern of textbook writers and programmers of self-instructional materials as well. Students must be taught the meanings of unfamiliar words and idioms; they must be helped in recognizing unfamiliar ways in which familiar words may be used; and they must be made generally aware of the possibility of ambiguity in meaning and the role of context in resolving it. Often the task that presents itself to the teacher is not merely to explain a new word in familiar terms, but to shape an entirely new concept in the mind of the student."

Nancy Cartwright's Philosophy of Science An International Workshop December 16-17, 2002 html on this page the contribution's abstracts, and download links.

Audrey B. Champagne, Richard F. Gunstone and Leopold E. Klopfer (1985). Instructional consequences of students' knowledge about physical phenomena. In Leo H. T. West and A. Leon Lines: Cognitive structure and conceptual change (pp. 61-90). Academic Press.

Eduard Jan Dijksterhuis (1950/1969). The mechanization of the world picture. London: Oxford University Press.

• p. 30 (cited in Champagne et al p. 62): to this day every student of elementary physics has to struggle with the same errors and misconceptions which then had to be overcome, and on a reduced scale, in the teaching of this branch of knowledge in schools, history repeats itself every year. The reason is obvious: Aristotle merely formulated the most commonplace experiences in the matter of motion as universal scientific propositions, whereas classical mechanics, with its principle of inertia and its proportionality of force and acceleration, makes assertions which not only are never confirmed by everyday experience, but whose direct experimental verification is fundamentally impossible .... .
• Maurice Clavelin (1968/1974). The natural philosophy of Galileo. Essay on the origins and formation of classical mechanics. MIT Press. Clavelin was not aware of the Dijksterhuis study.

Susan E. Embretson (1996). Cognitive Design Principles and the Successful Performer: A Study on Spatial Ability. Journal of Educational Measurement, 33, p. 29 ff.

• abstract An important trend in educational measurement is the use of principles of cognitive psychology to design achievement and ability test items. Many studies show that manipulating the stimulus features of items influences the processes, strategies, and knowledge structures that are involved in solution. However, little is known about how cognitive design influences individual differences. That is, does applying cognitive design principles change the background skills and abilities that are associated with successful performance? This study compared the correlates of two spatial ability tests that used the same item type but different test design principles (cognitive design versus psychometric design). The results indicated differences in factorial complexity in the two tests; specifically, the impact of verbal abilities was substantially reduced by applying the cognitive design principles.

Avron Barr and Edward A. Feigenbaum (Eds) (1981). The handbook of artificial intelligence. Volume 1. Stanford: HeurisTech Press; Los Altos: Kaufmann. isbn 0865760055. Section III: Avron Barr and James Davidson (Eds) knowledge representation, articles: Robert Fillman on Logic - Douglas Appelt on semantic nets - Anne Gardner on semantic primitives - James Bennett on frames

• Dated, but probably a very nice summary of the relevant possibilities

Robert Glaser (1997). Assessment and education: Access and achievement. National Center for Research on Evaluation, Standards, and Student Testing (CRESST) Center for the Study of Evaluation (CSE) Graduate School of Education & Information Studies University of California, Los Angeles. http://www.cse.ucla.edu/Reports/TECH435.pdf [dead link? 2-2008]

• final comment To conclude, assessments in education of the abilities and achievements of learners must be designed and used in ways that take account of present knowledge of human learning and the goals of modern society. Reaching these aspirations will be impossible if we continue to carry the ballast of practices that were designed for a time gone by. New perspectives are now offered on the nature of knowledge and abilities that are brought to learning and on the nature of competent achievement that results from instruction. The innovative merger of learning, instruction, and assessment can now drive the design of educational environments that support and respect human cognitive ability.

Clark Glymour (1998). What went wrong? Reflections on science by observation and The bell curve. Philosophy of Science, 65, 1-32. pdf

Clark Glymour (2003). Learning, prediction and causal Bayes nets. TRENDS in Cognitive Sciences, 7, 43-48 pdf

• Recent research in cognitive and developmental psychology on acquiring and using causal knowledge uses the causal Bayes net formalism, which simultaneously represents hypotheses about causal relations, probability relations, and effects of interventions. The formalism provides new normative standards for reinterpreting experiments on human judgment, offers a precise interpretation of mechanisms, and allows generalizations of existing theories of causal learning. Combined with hypotheses about learning algorithms, the formalism makes predictions about inferences in many experimental designs beyond the classical, Pavlovian cue -> effect design.

Gopnik, Alison, Clark Glymour, David M. Sobel, Laura E. Schulz, Tamar Kushnir, and David Danks. A theory of causal learning in children: Causal maps and Bayes nets," Psychological Review, Vol. 111, No. 1 (2004). pdf

• Abstract We outline a cognitive and computational account of causal learning in children. We propose that children employ specialized cognitive systems that allow them to recover an accurate "causal map" of the world: an abstract, coherent, learned representation of the causal relations among events. This kind of knowledge can be perspicuously understood in terms of the formalism of directed graphical causal models, or "Bayes nets". Children's causal learning and inference may involve computations similar to those for learning causal Bayes nets and for predicting with them. Experimental results suggest that 2- to 4-year-old children construct new causal maps and that their learning is consistent with the Bayes net formalism.

Gopnik, A. and Nazzi, T. (2001) Words, kinds and causal powers: a theory theory perspective on early naming and categorization. Early Categorization (Rakison, D., Oakes, L. eds), Oxford University Press. pdf

• Conclusion We have argued that "kinds" can be usefully understood as groups of objects with common current and future causal powers. This conception stems from our general conception of intuitive theories as "causal maps": abstract representations of the causal structure of the world. In a series of empirical studies, we have shown that children do categorize and name objects based on their causal powers, from at least 30 months of age. Even younger children, around 18 months old, already show the ability to sort objects into multiple, exhaustive categories and to sort entirely perceptually dissimilar objects together when they are given a common name. These abilities do not appear to be present in younger infants. They also seem to be highly correlated with the acquisition of names and particularly with a naming spurt.
  There also seem to be important changes in children's kind understanding. We suggest that the youngest infants may not relate perceptual similarities among objects and causal powers at all. At a later stage, from about a year until about three or four, children do seem to predict strong correlations between perceptual similarities and causal powers. However, with the acquisition of language, children may have the further realization that names are correlated with causal powers in a way that may bypass perceptual similarities. Eventually, this may be what leads them to a deeper conception of underlying essences.
  Causal inference, induction and learning seem to us particularly promising avenues of investigation for several reasons. Empirically, we have discovered that causal understanding plays an important and deep role in a great deal of children's early knowledge, including their knowledge of categories. From a functional and evolutionary perspective, causal understanding is profoundly adaptive. And methodologically, the problem of causal inference is an area where powerful formal and computational tools and techniques are available. Such tools may help us make real progress towards a developmental science that is both genuinely developmental and genuinely scientific.

Wallace Hannum (1988). Designing courseware to fit subject matter structure. In David H. Jonassen (Ed.) (1988). Instructional designs for microcomputer courseware (pp. 275-296). Erlbaum.

• Does not seem to be cited much. What struck me in its contents: the students are supposed to be tabula rasa; all that the instructor (the program) needs to do, is get the students to learn 'the material.' This is not particularly Wallace's fault: he simply uses the known literature. So much the worse for the known literature.

Guershon Harel and Larry Sowder (2005). Advanced mathematical-thinking at any age: Its nature and its development. Mathematical Thinking and Learning, 7, 27-50.

• abstract This article argues that advanced mathematical thinking, usually conceived as thinking in advanced mathematics, might profitably be viewed as advanced thinking in mathematics (advanced mathematical-thinking). Hence, advanced mathematical- thinking can properly be viewed as potentially starting in elementary school. The definition of mathematical thinking entails considering the epistemological and didactical obstacles to a particular way of thinking. The interplay between ways of thinking and ways of understanding gives a contrast between the two, to make clearer the broader view of mathematical thinking and to suggest implications for instructional practices. The latter are summarized with a description of the DNR system (Duality, Necessity, and Repeated Reasoning). Certain common assumptions about instruction are criticized (in an effort to be provocative) by suggesting that they can interfere with growth in mathematical thinking.

Barbara Koslowski (1996). Theory and evidence. The development of scientific reasoning. MIT Press.

• This work falls in the same category as that of Deanna Kuhn (below).

Chwee Beng Lee (2005). Modeling conceptual change in problem solving. Paper at Redesigning pedagogy, research, policy, practice. Singapore 2005 http://conference.nie.edu.sg/paper/New%20Converted/ab00452.pdf [dead link? 2-2008]

• from the abstract This paper proposes engaging students in a series of modeling activities to solve real world problem such as water shortage. It is expected that during the modeling activity, learners will go through a series of iterative model development cycles in which descriptions, explanations, and predictions are gradually refined and revised based on feedback from trail testing (Lesh & Doerr, 2003). This highly engaging problem solving process will thus enable the learners to develop a well conceptualized model.

Krittaya Leelawong, Joan Davis, Nancy Vye and Gautam Biswas (2002). The effects of feedback in supporting learning by teaching in a teachable agent environment pdf.

• abstract The idea that teaching others is a powerful way to learn is both intuitively compelling, and one that has garnered support in the research literature. The present study investigates aspects of the "learning by teaching" process that contribute to enhanced learning outcomes for students. We developed a computer-based teachable "agent" that students explicitly teach using concept maps. Results indicate that providing students with opportunities to quiz their agent decreases the amount of irrelevant information and increases the proportion of causal information in students' maps, whereas having opportunities to query their agent increases the interconnectedness of concepts in students' maps. The results point to the importance of including various forms of feedback in designing teachable agent environments that promote learning.
• See also Daniel L. Schwartz, John D. Bransford, and David Sears (2005). Efficiency and Innovation in Transfer, in Jose P. Mestre: Transfer of learning: from a modern multidisciplinary perspective (pp 1-51). San Francisco: Sage. pdf.

Wendy G. Lehnert (1978). The process of queston answering. A computer simulation of cognition. Erlbaum.

Po-Hung Liu (2002). Developing college students' views on mathematical thinking in a historical approach, problem-based calculus course. http://www.math.uoc.gr/~ictm2/Proceedings/pap191.pdf 2nd INTERNATIONALÊCONFERENCE ON THETEACHING OF MATHEMATICSÊ paper

• from the abstract It has been held that heuristic training alone is not enough for developing oneÕs mathematical thinking. One missing component is a mathematical point of view. Many educational researchers propose problem-based curricula to improve studentsÕ views of mathematical thinking. Meanwhile, scholars in different areas advocate using historical problems to attain this end. This paper reports findings regarding effects of a historical approach, problem-based curriculum to foster Taiwanese college studentsÕ views of mathematical thinking.

Wolfram Meyerhöfer (2005). Was misst TIMSS? Einige Überlegungen zum Problem der Interpretierbarkeit der erhobenen Daten. pdf

• abstract The design and interpretation of aptitude tests in mathematics provoke questions as to what each of the set tasks actually measures. With structural or objective hermeneutics, this article introduces a methodology capable of discerning the various dimensions of skills required for a particular task. Not only does this approach allow for the recognition of the technical requirements of the task, its off-putting factors and the image of the subject conveyed. The methodology is also able to locate the elements addressing the kind of skill that can more accurately be classified as ãtest ability". Focusing on an example selected from a TIMSS aptitude test, the discussion seeks to demonstrate that the theoretical construction employed in setting the test is hardly suited to define with any sense of permanence what is measured by each task.

E. R. Michener (1978). Understanding understanding mathematics. Cognitive Science, 2, 361-383.

Clayton T. Morrison, Tim Oates, and Gary King (2001). Grounding the Unobservable in the Observable: The Role and Representation of Hidden State in Concept Formation and Refinement. In Working Notes of AAAI Spring Symposium Workshop: Learning Grounded Representations. http://eksl.cs.umass.edu/papers/morrison-sss01.pdf [dead link? 2-2008]

Dwayne H. Mulder (1998). Explanation, Understanding, and Subjectivity. html

Herman E. Stark (1998). Expertise and Rationality html

• abstract I explore the connection between expertise and rationality. I first make explicit the philosophically dominant view on this connection, i.e., the Ôexpert-consultation' view. This view captures the rather obvious idea that a rational way of proceeding on a matter of importance when one lacks knowledge is to consult experts. Next, I enumerate the difficulties which beset this view, locating them to some extent in the current philosophical literature on expertise and rationality. I then propose that different lessons should be drawn for rationality from the fact of expertise. One is that some empirical and phenomenological studies of the nature of expertise can be fruitfully applied by analogy to theories of the rational agent.other:

A selective biography of the philosophy of science. Compiled by Ward E. Jones and Samir Okasha with W. H. Newton-Smith. Oxford: Sub-Faculty of Philosophy, 1998. http://www.herts.ac.uk/philosophy/scibib.html [dead link, Dec. 2008]

Joseph D. Novak and Alberto J. Canas (2006). The Theory Underlying Concept Maps and How To Construct Them. Technical report IHMC Cmap Tools 2006-01, Florida Institute for Human and Machine Cognition

• available at html pdf 1M or high quality 31 pages pdf 5.4Mb

• Beautiful examples of schematizing.

R. Nozick (1981). Philosophical Explanations. Cambridge, MA: Harvard University Press.

Nicos Papadouris and Costas Constantinou (2003). Epistemological and Reasoning Difficulties Associated with Scientific Experimentation. Paper The 4th ESERA Conference, Noordwijkerhout, The Netherlands.pdf

W. Schnotz, S. Vosiadou and M. carretero (Eds) (1999). New perspectives on conceptual change. Amstrdam: Pergamon. [nog niet gezien, niet in UB Leiden]

E. E. Smith and D. L. Medin (1981). Categories and concepts. Cambridge, Massachusetts, Harvard University Press.

John L. Pollock (1974). Knowledge and justification. Princeton University Press.

Rehder, B. (2001) A causal model theory of categorization Proceedings of the 21st Annual Conference of the Cognitive Science Society (Hahn, M., Stoness, S.C. eds), pp. 595Ð600, Vancouver. pdf

• In this article I propose that categorization decisions are often made relative to causal models of categories that people possess. According to this causal-model theory of categorization, evidence of an exemplar's membership in a category consists of the likelihood that such an exemplar can be generated by the category's causal model. Bayesian networks are proposed as a representation of these causal models. Causal-model theory was fit to categorization data from a recent study, and yielded better fits than either the prototype model or the exemplar-based context model, byaccounting, for example, for the confirmation and violation of causal relationships and the asymmetries inherent in such relationships.

Ernest Sosa and Jaegwon Kim (Eds) (2002). Epistemology. An anthology. Blackwell. isbn 0631197249

Monika Walczak (1998). The Classical Conception Of Rationality html

• ABSTRACT: The aim of this paper is to describe the classical conception of rationality, i.e., to indicate the theses traditionally associated with this conception. I do not intend to discuss these theses in detail. Rather, I focus on the question regarding the main elements of the classical conception of rationality. I am interested in the rationality of cognition and of knowledge (epistemic/epistemological rationality), especially in the rationality of science (scientific rationality).

T. Williamson (2000). Knowledge and its limits. Oxford: Oxford University Press.

Paul Brna (n.d.). Conceptual change and mental models. A bibliography. html

ontology

Examples of cognitive maps

Albert L. Stevens and Allan Collins: Multiple conceptual models of a complex system. In Richard E. Snow, Pat-Anthony. Federico and William E. Montague (Eds.) (1980). Aptitude, learning and instruction. Volume 2: cognitive process analyses of learning and problem solving (p. 177-198). Erlbaum.

• p. 196: "representations of expert knowledge must be further removed from the surface forms in which people talk than most current [CAI] systems contemplate. Unfortunately, this makes many aspects of building expert systems more difficult."
• p. 196: "The implications for education are equally profound. This view suggests that multiple models should be taught explicitly as alternative points of view about a topic. The emphasis should be on the kinds of situations and problems for which each model is aplicable, and on how to apply them to solve different types of novel problems."

James C. Lester, Bruce W. Porter (1995). Developing and Empirically Evaluating Robust Explanation Generators: The KNIGHT Experiments. Also in: Computational Linguistics, 23, 65-101.

• abstract To explain complex phenomena, an explanation system must be able to select information from a formal representation of domain knowledge, organize the selected information into multisentential discourse plans, and realize the discourse plans in text. Although recent years have witnessed significant progress in the development of sophisticated computational mechanisms for explanation, empirical results have been limited. This paper reports on a seven year effort to empirically study explanation generation from semantically rich, large-scale knowledge bases. In particular, it describes Knight, a robust explanation system that constructs multi-sentential and multi-paragraph explanations from the Biology Knowledge Base, a large-scale knowledge base in the domain of botanical anatomy, physiology, and development. We introduce the Two Panel evaluation methodology and describe how Knight's performance was assessed with this methodology in the most extensive empirical evaluation conducted on an explanation system. In this evaluation, KNIGHT scored within 'half a grade' of domain experts, and its performance exceeded that of one of the domain experts.

Byron Marshall, Hsinchun Chen, and Therani Madhusudan (accepted 2005). Matching Knowledge Elements in Concept Maps using a Similarity Flooding Algorithm. Decision Support Systems. http://oregonstate.edu/~marshaby/Papers/MatchKnowledgeElements_PrePrintVersion.pdf

• abstract Concept mapping systems used in education and knowledge management emphasize flexibility of representation to enhance learning and facilitate knowledge capture. Collections of concept maps exhibit terminology variance, informality, and organizational variation. These factors make it difficult to match elements between maps in comparison, retrieval, and merging processes. In this work, we add an element anchoring mechanism to a similarity flooding (SF) algorithm to match nodes and substructures between pairs of simulated maps and student-drawn concept maps. Experimental results show significant improvement over simple string matching with combined recall accuracy of 91% for conceptual nodes and concept to link to concept propositions in student-drawn maps.

Diana C. Rice, Joseph M. Ryan and Sara M. Samson (1998). Using Concept Maps to Assess Student Learning in the Science Classroom: Must Different Methods Compete? Journal of Research in Science Teaching, 35, 1103-1127. http://www.clab.edc.uoc.gr/hy302/papers%5Cconcept_map%20assesment.pdf

• from the abstract The results of the study suggest that a concept map might be used in assessing declarative and procedural knowledge, both of which have a place in the science classroom. One important implication of these results is that science curriculum and its corresponding assessment need not be dichotomized into knowledge/comprehension versus higher-order outcomes.

Karoline Afamasaga-Fuata'i (2004). An undergraduate student's understanding of differential equations through concept maps and vee diagrams. A. J. Canas, J. D. Novak, F. M. González, Eds. Proc. of the First Int. Conference on Concept Mapping Pamplona, Spain 2004 pdf

• abstract The paper presents the case of a student (Nat) who participated in a semester long study which investigated the impact of using concept maps and vee diagrams on studentsÕ understanding of advanced mathematics topics. Through the construction of concept maps and vee diagrams, Nat realized that there was a need for him to deeply reflect on what he really knows, determine how to use what he knows, identify when to use which knowledge, and be able to justify why using valid mathematical arguments. He found that simply knowing formal definitions and mathematical principles verbatim did not necessarily guarantee an in-depth understanding of the complexity of inter-connections and inter-linkages between mathematical concepts and procedures. During seminar presentations and one-on-one consultations, Nat found that using the constructed concept maps and vee diagrams greatly facilitated discussions, critiques, dialogues and communication. The paper discusses the results from this case study and some implications for teaching mathematics.

Maria Araceli Ruiz-Primo, Richard J. Shavelson and Susan Elise Schultz (1997). On The Validity Of Concept Map-Base Assessment Interpretations: An Experiment Testing The Assumption Of Hierarchical Concept Maps In Science. CSE Technical Report 455. pdf

Maria Araceli Ruiz-Primo, Susan Elise Schultz and Richard J. Shavelson (1997). Concept Map-Based Assessment in Science: Two Exploratory Studies. CSE Technical Report 436. pdf

John H. Milam, Jr., Susan A. Santo and Lisa A. Heaton (2000?). Concept Maps for Web-Based Applications. ERIC Technical Report. pdf Terhi Mäntylä and Ismo T Koponen (2003). Understanding the character of physical laws through the intertwined epistemic roles of experiments and models: An example from physics teacher education. The ESERA 2003 http://www1.phys.uu.nl/esera2003/program.shtml conference at Noordwijkerhout on Research and the quality of science education. pdf

• The schematic representation of the role of experiments and models in establishing the physical law.

• " As a simple example of scientific imagination, the case of free fall is discussed, as "seen" by Aristotle, Galileo, Newton, and Einstein. It is argued that a fully developed theory like Newton's theory of gravitation, for example, does not come easily and immediately; the question-and-answer procedure necessarily involves experiments, generates problems that must be solved, often using data that is selected on the basis of an incomplete theoretical background. To illustrate the activity and scientific thinking involved in building big theories in physics, the scientific thinking of Aristotle, Galileo, Newton, Maxwell and Einstein's are discussed."

• The part of interest is the paragraph on Siegler's Balance Beam Tasks, a Piagetian paradigm . The domain of interest here are four different levels of understanding - four different student models - in children of different age levels. A very small domain indeed. Using tests, Mislevy wnats to estimate the student models. The item design is based on the possible student models, and the possibilities of balance and imbalance of a balance beam. This, by the way, is an interesting technique of designing test items; it was worked out by Siegler, not by Mislevy. The balance been has been used also in Wilson 1989.
• R. S. Siegler (1981). Developmental sequences within and between concepts. Monographs of the Society for Research in Child Development, 46 (serial # 189)(Siegler's homepage contains (Oct. 2006) much work that he has made available online, except this 1981 monograph)
  • abstract Describes and discusses the rule-assessment approach, a new research strategy for studying developmental sequences in children's acquisition of knowledge. Four experiments were conducted to illustrate the utility of this approach across a variety of concepts and a wide range of ages (three-year-olds to college students). (Author/CM [ERIC])

  
  

• M. R. Wilson (1989). Saltus. A psychometric model of discontinuity in cognitive development. Psychological Bulletin, 105, 276-289.
• Peter Pirolli and Mark Wilson (). A Theory of the Measurement of Knowledge Content, Access, and Learning. http://citeseer.ist.psu.edu/cache/papers/cs/8049/http:zSzzSzwww.parc.xerox.comzSzistlzSzprojectszSzuirzSzpubszSzpdfzSzUIR-R-1998-11-Pirolli-PsychReview-cogm.pdf/a-theory-of-the.pdf (via html)
  • abstract We develop an approach to the measurement of knowledge content, knowledge access and knowledge learning. This approach has two elements: First we describe a theoretical view of cognition, called the Newell-Dennett framework, which we see as being particularly favourable to the development of a measurement aproach. Then, we describe a class of measurement models, based on Rasch modeling, which we see as being particularly favourable to the development of cognitive theories. Knowledge content and access are viewed as determining the observable actions selected by an agent in order to achieve desired goals in observable situations. To the degree that models within the theory fit the data at hand, one considers measures of observed behavior to be manifestations of intelligent agents having specific classes of knowledge content and varying degrees of access to that knowledge. Although agents, environment, and knowledge are constitutively defined (in terms of one another), successful application of our theory affords separation of parameters associated with the person from those associated with the environment. We present and discuss two examples of measurement models developed within our approach that address the evolution of cognitive skill, strategy choice and application, and developmental changes in mixtures of strategy use.
  • Uses the Siegler Balance Beam again [NB I have yet to study this one]
  
  

Common sense / Folk science

en.wikipedia folk science

Natasa Rakic (1997). Common sense, time and special relativity. Dissertation University of Amsterdam.

• not available online
• p. 1: "It has often been argued that the common sense concept of time is incompatible with the STR [Special Theory of Relativity]. However, the aim of this thesis is to counter these arguments and to show how the common sense iew of time can be reconciled with special relativity.
  Commons sense is the usual name for the ordinary conceptual schemes which guide the activities of our daily life. It manifests itself in learning, reasoning, concept formation, and is usually considered to be encoded in our natural language.
• Natasa Rakic (1997). Past, Present, Future, and Special Relativity. British Journal for the Philosophy of Science, 251-280.

Links

The Archimedes Project http://archimedes2.mpiwg-berlin.mpg.de/archimedes_templates/project.htm

• From the home page: " Numerous treatises on mechanics as well as other forms of documentation of mechanical knowledge and practices constitute the project corpus. Ongoing research at the MPIWG on the long-term development of mental models of mechanical thinking and their manifestation in technical terminologies, inferences of practitioners, engineers, and scientists plays an important role in the testbed design.": a.o. Jürgen Renn: The Relation of Practical Experience and Conceptual Structures in the Emergence of Science: Mental Models in the History of Mechanics. http://www.mpiwg-berlin.mpg.de/en/forschung/projects/mentalModels/index.html [dead link? 12-2008]

Concept Inventories site. This work is inspired by earlier work of D. Hestenes and others, see here for downloadable publications.

Paul Hewitt Conceptual physics website (Picture 'Gravity at Play' on his website)

Gregory K. W. K. Chung and Eva L. Baker (1997). Year 1 Technology studies: Implications for technology in assessment. CSE Technical Report 459. http://www.cse.ucla.edu/Reports/TECH459.pdf [dead link? 2-2008] (oa concept mapper)

IHMC Cmap Tools site

• Open source software for the construction of concept maps
  • List of publications using concept mapping html

  The late Jan Lanzing Concept Mapping Homepage html

  http://cmap.ihmc.us/

  WISE The web-based inquiry science environment http://wise.berkeley.edu/

  • "To help students connect science instruction to their own understanding, their thinking needs to made visible. WISE provides students with opportunities and software to represent their own thinking, as well as that of other students and even scientists, and to respond to one another's ideas. Thus, student receive feedback about their current models and can compare their models with others'."
  • Marcia C. Linn, Douglas Clark and James D. Slotta (2002). WISE design for knowledge integration. p. 517-538. http://m1.cust.educ.ubc.ca:8200/learningandtecharticles/WISEdesignforknowledgeintegration.pdf [dead link? 12-2008]
    • from the abstract This paper describes WISE design team practices, features of the WISE learning environment, and patterns of feature use in current library projects. The success of WISE in classrooms illustrates how flexibly adaptive projects can meet the needs of diverse teachers. Variation amongst library projects shows that designers can support inquiry with a wide variety of activities.
    • The WISE project does not seem to be related in any way to the Deanna Kuhn Center's activities or to the literature on mental models, therefore I suspect its instructional approach to be rather traditional, while using the wonderful techniques of the internet. It might just be the case, however, that the instructional shells are able to accommodate Kuhnian inquiry- and argumentation based activities as well. These authors are interested in whether a WISE project has made science accessible, which does not exactly sound like a student friendly educational philosophy. I have yet to study the WISE project's question design approach, b.w.
  
  

authentic testing

Authentic testing is somewhat of an issue in the literature on achievement testing. Originally, it is very much an American issue, a reaction to the overwhelming use of standardized tests using a limited choice of testing formats. As such, it is much less an issue in European continental countries where national examinations are used, narrowly defined on course content. Dwyer (1993, p. 269) cites Mitchell (1989, 1990, see Dwyer): "Authentic assessment is continuous with instruction: It complements and expands the curriculum, and preparation for the assessment should literally be the curriculum."

Carol Anne Dwyer (1993). Innovation and reform: Examples from teacher assessment. In Randy Elliot Bennett, William C. Ward: Construction versus choice in cognitive measurement: Issues in Constructed Response, Performance Testing, and Portfolio Assessment (p. 265-290). Erlbaum.

May 4, 2007 \ contact ben at at at benwilbrink.nl       http://www.benwilbrink.nl/projecten/06examples3.htm

---