Original publication 'Toetsvragen schrijven' 1983 Utrecht: Het Spectrum, Aula 809, Onderwijskundige Reeks voor het Hoger Onderwijs ISBN 90-274-6674-0. The 2006 text is a revised text.

Item writing

Techniques for the design of items for teacher-made tests

7. Posing problems

Ben Wilbrink

this database of examples has yet to be constructed. Suggestions? Mail me.

7.1 Speaking of problems


Figuur 1 Scheme of problem approach

At a fair, players throw coins onto a board checkered with squares. If a coin touches a boundary, it's lost. If it rolls off the board, it's returned. But if it lies wholly within a square, the player wins his coin back plus a prize. Copyright ( 1988) by National Council of Teachers of Mathematics. Used [by Romberg] with permission. What is the probability of winning this game?

Consult my source for the details on why this is a great problem, and how students go about solving it

Thomas A. Romberg (1994). Classroom instruction that fosters mathematical thinking and problem solving: connections between theory and practice. In Alan H. Schoenfeld (Ed.) (1994). Mathematical thinking and problem solving. Erlbaum. questia

Wason's 2 4 6 problem

An experimenter tells you she has a rule in mind about triples of whole numbers. She also tells you that the triple '2 4 6' conforms to that rule. Your task is to discover the rule by thinking of new triples, the experimenter will tell you whether a new triplet conforms to the rule or not.

P. C. Wason (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12, 129-140. The original article does not seem to be available on the www. Try cuil.com, search: wason problem. Note: it is not the same as the Wason selection task, another famous problem in the same line of research.

Jonathan St. B. T. Evans (1989). Bias in human reasoning: Causes and consequences. Erlbaum. See p. 45 ff. This book is not available in questia.com. However, searching questia on its title resuts in a number of related books that are.

7.2 Inventory

7.3 Heuristics

Figuur 1 Figure 1. Construction of a problem statement.

kenmerk = characteristic

plan = plan

toets = test

This is figure 1 from the Dutch chapter 7

Figuur 2 Figure 2. Analysis of a designed problem statement.

plan = plan

kenmerk = characteristic

concrete situatie = specific situation = test

This is figure 2 from the Dutch chapter 7


Block Drawn by Rope
A 3.57 kg block is drawn at constant speed 4.06 m along a horizontal floor by a rope. The force on the block from the rope has a magnitude of 7.68 N and is directed 15.0° above the horizontal. What are (a) the work done by the rope's force, (b) the increase in thermal energy of the block floor system, and (c) the coefficient of kinetic friction between the block and floor?

Construct a complete Model of the following situation: A 3.57 kg block is drawn at constant speed 4.06 m along a horizontal floor by a rope. The force on the block from the rope has a magnitude of 7.68 N and is directed 15.0° above the horizontal.

Student responses to these two different problems will vary greatly. The standard problem will have responses that are numeric answers and will be accompanied by varying degrees of work and likely little justification on how, or why the answer was attained. The response to the modeling problem is a constant velocity model, adapted to the situation described. A complete model for this situation would include kinematic graphs, motion maps, a system schema, a force diagram, and energy pie charts, as well as applications of Newton's Second Law and the First Law of Thermodynamics.

Eric Brewe (????). Modeling theory applied; modeling instruction in university physics. pdf retrieved july 2006

Tim Stelzer and Gary Gladding (2001). The Evolution of Web-Based Activities in Physics at Illinois. Forum on Education, Fall. html

Gladding (2005) presentation:


What we used to do:

What we do now:

an impossibility

Prove that every number greater than 4 is the sum of two odd primes.

The conjecture is known as Goldbach's conjecture, no one has been able to prove it. It is, of course, easy to construct lots and lots of examples showing the conjecture to be true in those particular cases: 8 = 3 + 5; 10 = 3 + 7 = 5 + 5; etcetera. This kind of problem is very special indeed. Fermat's Last Theorem used to be another example, it has now been proved by Andrew Wiles and Richard Taylor Wiki.

George Pólya (1954/68). Mathematics and plausible reasoning. Volume I: Induction and analogy in mathematics. Volume II: Patterns of plausible inference. Princeton University Press. pp. 4-7.

7.4 History

The Cambridge mathematical tripos in the nineteenth century is an extremely competitive examination spread out over some eight full days, sometimes in bitter cold as well. There are a few publications giving the full examinations, as well as the intended answers. Almost every problem is for almost every examinee a difficult one, a very difficult one. It is possible to see for yourself, as some of these oldies have been reprinted in the Elibron Classic series.

William Walton, Charles Mackenzie. Solutions of the Problems and Riders Proposed in the Senate-House Examination for 1854. By the Moderators and Examiners. With an Appendix, Containing the Examination Papers in Full.
Elibron Classics, 2002, 238 pages.
ISBN 140216131X paperback
ISBN 1402128185 hardcover
Replica of 1854 edition by Macmillan and Co., Cambridge.

Exact numbers on the difficulty of the problems are extremely scarce, but two series of marks of the best 100 honours students were published by James Galton in his book 'Hereditary genius,' see here. Galton also explains the examination procedure. Reproduced here, using the original source code:

Scale of merit among the men who obtain mathematical honours at Cambridge.
The results of two years are thrown into a single table. 
The total number of marks obtainable in each year was 17,000.
Number of marks obtained by
Number of candidates in the two
years, taken together, who obtained
those marks.
Under 500
500 to 1,000
1,000 to 1,500
1,500 to 2,000
2,000 to 2,500
2,500 to 3,000
3,000 to 3,500
3,500 to 4,000
4,000 to 4,500
4,500 to 5,000
5,000 to 5,500
5,500 to 6,000
6,000 to 6,500
6,500 to 7,000
7,000 to 7,500
7,500 to 8,000
The precise number of marks obtained by the senior wrangler in the more
remarkable of these two years was 7,634; by the second wrangler in the
same year, 4,123; and by the lowest man in the list of honours, only 237.
Consequently, the senior wrangler obtained nearly twice as many marks as
the second wrangler, and more than thirty-two times as many as the lowest
man. I have received from another examiner the marks of a year in which
the senior wrangler was conspicuously eminent.
I have included in this table only the first 100 men in each year. The omitted residue is too
small to be important. I have omitted it lest, if the precise numbers of honour men were stated,
those numbers would have served to identify the years. For reasons already given, I desire to
afford no data to serve that purpose.

7.5 Literature

more literature

Many items are as yet on my 'to do' list: they are mentioned here, but not used in the above text yet.

David McMath, Marianna Rozenfeld, and Richard Sommer (www accessed 2006). A Computer Environment for Writing Ordinary Mathematical Proofs. (Education Program for Gifted Youth, Stanford University) pdf

Hanne ten Berge, Stephan Ramaekers en Albert Pilot (2004). The design of authentic tasks that promote higher-order learning. Paper presented at the EARLI-SIG Higher Education/IKIT-conference, June 18-21, 2004. pdf

Bruce G. Buchanan and Richard O. Duda (1982). Principles of rule-based expert systems. Heuristic Programming Projec Report no. HPP-82-14. pdf

Michelene T. H. Chi, Robert Glaser, and Marshall J. Farr (Eds) (1989). The nature of expertise. Hillsdale: Erlbaum. (ao: Beth Adelson and Elliot Soloway: A model of software design - Jeanette A. Lawrence: Expertise on the bench: Modeling magistrates' judicional decision-making - James F. Voss and Timothy A. Post: On the solving of ill-structured problems - Guy J. Groen and Vimla L. Patel: The relationship between comprehension and reasoning in medical expertise - Alan Lesgold, Harriet Rubinson, Paul Feltovich, Robert Glaser, Dale Klopfer, and Yen Wang: Expertise in complex skill: diagnosing X-ray pictures - William J. Clancey: Acquiring, representing, and evaluating a competence model of diagnostic strategy)

A. S. Elstein, L. S. Shulman en S. A. Sprafkan (1978). Medical problem solving: An analysis of clinial reasoning. Harvard University Press.

Peter Gerjets, Katharina Scheiter and Richard Catrambone (2004). Designing Instructional Examples to Reduce Intrinsic Cognitive Load: Molar versus Modular Presentation of Solution Procedures. Instructional Science 32, 33-58,

Peter Gerjets, Richard Catrambone and Katharina Scheiter (2006). Reducing Cognitive Load and Fostering Cognitive Skill Acquisition: Benefits of Category-Avoiding Instructional Examples. http://www.ccm.ua.edu/pdfs/99.pdf [Dead link? May 3, 2009]

Adriaan D. Groot (1978). Thought and choice in chess. Den Haag: Mouton, 1978.

Guershon Harel and Larry Sowder (1998). Students' proof schemes. Research on Collegiate Mathematics Education, Vol. III. In E. Dubinsky, A. Schoenfeld, & J. Kaput (Eds.), AMS, 234-283. pdf

Guershon Harel and Larry Sowder (In Press). Toward a comprehensive perspective on proof, In F. Lester (Ed.), Second Handbook of Research on Mathematics Teaching and Learning, National Council of Teachers of Mathematics. http://math.ucsd.edu/~harel/downloadablepapers/TCPOLTOP.pdf [Dead link? May 3, 2009]

Ross Honsberger (1978). Mathematical morsels. The Mathematical Association of America.

David H. Jonassen (1997). Instructional design model for well-structured and ill-structured problem-solving learning outcomes. Educational Technology: Research and Development, 45, 65-95.

D. Jonassen (2000). Toward a Design Theory of Problem Solving. Educational Technology Research and Development, 2000, 48, 63-85. http://www.coe.missouri.edu/~jonassen/PSPaper%20final.pdf [Dead link? May 3, 2009]

David H. Jonassen (2003). Using cognitive tools to represent problems. Journal of Research in Technology in Education, 35, 362-381. http://tiger.coe.missouri.edu/~jonassen/ProbRep.zip [Dead link? May 3, 2009]

David H. Jonassen (2004?). The future of learning: Learning to solve problems. pdf

David H. Jonassen and Woei Hung (under review). Learning to Troubleshoot: A New Theory-Based Design Architecture. pdf

Ton de Jong (Ed.) (2003). Knowledge management Interactive Training System. KITS consortium. pdf

J. H. Larkin, J. H. (1981). Enriching formal knowledge: a model for learning to solve textbook physics problems. In J. R. Anderson Cognitive skills and their acquisition. Hillsdale, New Jersey, Erlbaum, 1981.

Peter Lehman (1996). Will that be on the exam? Schema theory and testing in sociology. Teaching Sociology, October. pdf

Lesh, R., & Harel, G. (2003). Problem solving, modeling, and local conceptual development. International Journal of Mathematics Thinking and Learning, 5, 157-189. http://math.ucsd.edu/~harel/downloadablepapers/Problem%20solving,%20modeling,%20and%20local%20conceptual%20development.pdf [ead link? May 3, 2009]

Alan Newell and Herbert A. Simon (1972). Human problem solving. Englewood Cliffs, New Jersey, Prentice Hall, 1972.

G. Polya (1957/1971). How to solve it. A new aspect of mathematical method. Princeton, New Jersey, Princeton University Press.

Joseph Psotka, L. Dan Massey and Sharon A. Mutter. (Ed.) (1988). Intelligent tutoring systems. Lessons learned. Hillsdale/NJ: Lawrence Erlbaum. 552 pp., isbn 0805801928 paperback (a.o.: ) (contents) (http://www.cs.cmu.edu/~mazda/ITS/literature.htm Textbooks on intelligent tutoring systems)

L. W. T. Schuwirth, D.E. Blackmore , E. Mom , F. van den Wildenberg , H.E.J.H. Stoffers en C.P.M. van der Vleuten (1999). How to write short cases for assessing problem-solving skills. Medical Teacher, 21, 144-150.

Daniel L. Schwartz and Taylor Martin (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22, 129-184. as paper

Herbert A. Simon (1980). Problem solving and education. In D. T. Tuma and F. Reif (1980).

Herbert A. Simon (1973). The structure of ill-structured problems. Artificial Intelligence, 4, 181-201. Reprinted in Herbert A. Simon (1977). Models of discovery and other topics in the methods of science. Dordrecht: Reidel.

L. Sowder and G. Harel (2003). Case Studies of Mathematics Majors' Proof Understanding, Production, and Appreciation. Canadian Journal of Science, Mathematics and Technology Education. 3, 251-26. http://math.ucsd.edu/~harel/downloadablepapers/CJSMTE.pdf [Dead link? May 3, 2009]

D. T. Tuma and F. Reif (Eds). (1980). Problem solving and education: issues in teaching and research. Hillsdale, New Jersey, Erlbaum.

Renée E. Weiss (2003). Designing Problems to Promote Higher-Order Thinking. New Directions for Teaching and Learning 95, 25-31. pdf

W. A. Wickelgren (1974). How to solve problems, Elements of a theory of problems and problem solving, San Francisco, Freeman, 1974.



"Soar is a general cognitive architecture for developing systems that exhibit intelligent behavior. Researchers all over the world are using Soar."

IHMC Cmap Tools site

Weisstein, Eric W. "Archimedes' Cattle Problem." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/ArchimedesCattleProblem.html

August 15, 2008 \ contact ben at at at benwilbrink.nl    

Valid HTML 4.01!   http://www.benwilbrink.nl/projecten/06examples7.htm