1 1a Generator — 1m. 3-valued — 1o. 1oa advanced 2 2a Mastery Envelope 3 3a Predictor 4 4a Ruling 5 5a Learning 6 6a Expectation 6b. special 6.1a. advanced 7 7a Last Test 8 8a Strategy 9 9a True Utility
Clicking the Go button will start a new simulation and/or analysis using the values specified in the menu.
exception: using option 1 to leave the analysis out, and next another option, the analysis will take affect ony at the second ‘Go.’
Because the applet is in a scrollable window, scrolling will cause the applet to execute again. Remember this whenever you declare a larger number of runs or observations. Eventually I might make the applets available in stand-alone format. Such as spa_applet1.sa.htm
The simulation simulates test scores for the specified number of observations or cases, assuming mastery having the value specified; the resulting distribution of test scores will be plotted as a solid green bar diagram. For larger numbers of observations the simulation will will be closer to the binomial distribution plotted in blue. If you want the analysis only, turn the simulation offf by putting the number of observations = 0.
The analysis evaluates a probability distribution function called the binomial distribution; the distribution plotted is this mathematical distribution. The binomial distribution is the theoretical distribution of test scores given a particular mastery, for the number of observations growing very large. If you want the simulation only, put option = 1 to turn the analysis off.
Mastery may be chosen between 0 and 1; mastery as a concept is defined on the set of test items that every concrete test is supposed to have been sampled from: it is the proportion of items the candidate would answer correctly if given the opportunity to try alle items in the domain.
The number of test items is the length of the test; this number is not limited.
The number of observations or runs is the number of runs for the simulation, or, equivalently, the number of cases or persons that is represented in the simulated distribution.
The reference point is a particular test score that is of interest to the user of the applet. It might be the cutoff score in pass-fail testing, the minimum score that is still considered to be a level of sufficient mastery, etcetera. It is marked by a solid vertical in the plot. Its value may be set at 0.
The horizontal scale is that of test scores obtained, the lowest score being 0.
The vertical scale as a default will be automatically chosen and depends on the highest value of the analytical distribution, this point is tabmarked. The vertical values are frequencies in the case of simulations, and probabilities in the case of analyses. Their values may be plotted using option 203 and 204, respectively. Of course frequencies and probabilities will be comparable. In fact, option 2 will plot the vertical differences separately.
The statistics reported are the mean and standard deviation for every distribution. Because the theoretical value of the binomial statistics assumes the number of observations to be infinite, there will be small differences between the actual statistics and the statistics obtained by evaluating their formulas. Simulation and analytical statistics will differ from each other, of course; the differences wil be smaller the larger the number of observations is chosen.
The three-valued generator implements a model allowing the specification of the parameter ‘misunderstanding’ and the parameter ‘bonus’,
October 10 2009. The applet does not function appropriately. I will correct this problem in a few days time. Do not put ‘wrong’ = 0 or 1 - ‘wrong’ = ‘mastery.’
The model is not binomial. Of course, it is possible to approximate this model’ theoretical and simulated distribution using a binomial on the parameter p = mastery + bonus * ( 1 - mastery - wrong ). Comparing the 3-valued model results to those of the approximation, shwos the results of the 3-valued model to be a fraction too low. This is the other problem that I will have to solve.
The explanation of the approximation formula is rather simple. The expectation of the score to be earned on the very first item is E( score ) = mastery + bonus * ( 1 - mastery - wrong ). Translate this expectation into a probability to get a score of 1 point. Voila.
In order to manipulate the plot, one should be free to choose the horizontal and vertical scaling.
Be aware that mastery is defined as the proportion of items known in the domain. In multiple choice testing, it is not the case that the other items will be items that the candidate will guess: surely there will be a certain proportion of items that the candidate will answer wrongly. As of august 2009 the applet does not have the option to specify this misunderstanding. I will update the applet soon, however. In the meantime the probability to correctly guess the answer should absorb the probability to wrongly know the answer. Regrettably, the situation of allowing bonus points for items left unanswered, cannot be modeled correctly by the applet as it stands now. Allowing a bonus effectively makes mc-items three-valued: correct, wrong, or bonus. Just like mastery, misunderstanding (propotion wrong in the domain) should be specified. Most of the literature on guessing does not recognize this simple fact, implicitly but falsely assuming that all wrong answers on mc-items are the result of guessing.
Here you will find another, older, version of the module 1 applet (also its advanced version here below), based on the class DoBinomial dedicated to the binomial / the generator only. DoBinomial is the class that takes care of (most) everything that is input or output, including plotting of distributions. DoBinomial uses the class Basics: the methods Basics.getSimulatedBinomial and Basics.Binomial. The class Basics always is the most recent version, the class is used by most or all other applets of the SPA model. The methods mentioned come in two flavors: a five resp. three parameter method is the standard, a nine (Basics.Simulate) resp. eight parameter method to take care of two tests cases and/or subdomain divisions.
misunderstanding is the proportion of all possible items, satisfying the specific criteria for a particular test or subject matter, that the student would answer incorrectly if given the opportunity. See the spa_generator text here. This parameter takes effect only in combination with a positive value for the quessing parameter. Guessing then occurs only on the items that have not been answered either correctly or incorrectly, i.e. either on the basis of mastery, or on the basis of misunderstanding. The testee need not be aware of the difference, of course. How does it work: mastery .6 and misunderstanding .2 means that the probability for an item stay unanswered (either correctly or incorrectly) is 1 - .6 - .2 = .2. A change to guess an unknown item correct of .5 will mean in this particular case that the chance that the next item will be an unknown item that will be guessed ‘correctly’ is .5 × .2 = .1. The simulation in this particular case will use as success parameter, therefore, .6 (mastery) + .1 (lucky guesses) = .7. As will the analytical plot of the binomial distribution: the binomial parameter is .7.
option 100 will plot a 2 * n2 scheme of an item pool: every item a small rectangle, filled if it would be answered correct. The scheme is what the item pool might look like for a particular student having known mastery. In fact this scheme will be generated as one big simulated test; therefore the number corret will most of the times not exactly match the assumed mastery. This option is not available in the regular applet # 1 above.
option 101 will plot 2 * n schemes of simulated tests of n items, mastery assumed known. This option is not available in the regular applet # 1 above.
option 102 will plot 2 * n schemes of simulated tests of n items, mastery assumed known, items ordered. This option is not available in the regular applet # 1 above.
option 103 Same as 102, except given mastery is changed randomly for every new item simulated either +0.05 or -0.05.
With options 100 to 103 the width parameter determines the number of pixels width/height of every items. The minimum is 2 pixels, Java's fillpolygon will not plot rectangles on less then 2 pixels width.
This kind of scheme has been used in projects and publications since the early eighties (first year in dentistry and in law, University of Amsterdam, to inform students on the chance aspects of the tests they would have to sit that year, so they would be able to better prepare themselves or to better interpret their own results compared to those of their fellow students, see 4.3 in Voorthuis and Wilbrink (1987 html)
Here you will find another, older, version of the module 1 applet’ advanced version, based on the class SPA_DoBinomial dedicated to the binomial / the generator only. SPA_DoBinomial is the class that takes care of (most) everything that is input or output, including plotting of distributions. SPA_DoBinomial uses the class SPA_Basics
If you come across an applet that is not functioning properly, please mail me. It is not possible always to check all applets for unintended consequences of changes in classes. As this is a project in progress, such changes are made on a routine basis.
Applets are known to work correctly under:
Internet Explorer under Windows XP
Firefox 1.0.7 under Windows XP
Safari 1.2 under MacOSX 10.3.9
FireFox 220.127.116.11 under MacOSX 10.3.9
It might be the case that the applets do not open properly in browsers under Windows, or in browsers other than Safari under MacOS X: the applet field remains gray or blank.
In module chapters original applets have been replaced with screenshots; therefore applet problems should not hinder readers of the SPA project. Readers not able to use the applets in their browser, and yet willing to do so, may contact me, if preferences of the browser pertaining to Java do not seem to be the problem.
Information about Java, and applets in particular:
MacOS X: There is a problem with Java versions 1.4 for browsers other than Safari. See http://javaplugin.sourceforge.net/Readme.html; http://developer.apple.com/documentation/Java/Conceptual/Java131Development/deploying/chapter_3_section_5.html; simile.mit.edu/repository/ misc/java_embedding_plugin/readme.rtf
MacOS X: Opera, version 8.5, produces 'java.lang.UnsupportedClassVersionError: Spa_BinomialApplet (Unsupported major.minor version 48.0)
MacOS X: Internet Explorer 5.2 for Mac, [preferences: enable Java on; cookies: never ask; web content: enable plug-ins on] produces 'java.lang.UnsupportedClassVersionError: Spa_BinomialApplet (Unsupported major.minor version 48.0)
Windows: Java Applet plug-in makes it possible for your computer (including Windows¨ XP, Me, NT, 2000, 98, or 95) to run applets in your browser. http://www.mcdonalds.com/search/help/plug_play/sunmicro.html
Mail your opinion, suggestions, critique, experience on/with the SPA