Strategic preparation for achievement testing

Applets

Ben Wilbrink

For the text on the SPA model see the project.

applets     advanced applets         other applets

    1         1a    Generator        — 1m. 3-valued — 1o. 1oa advanced
    2         2a    Mastery Envelope
    3         3a    Predictor
    4         4a    Ruling
    5         5a    Learning
    6         6a    Expectation     6b. special 6.1a.  advanced
    7         7a    Last Test
    8         8a    Strategy
    9         9a True Utility

applet 1

• If the applet bar chart above does not show up - it should look like the (scaled down) picture at the left because this applet will plot only after clicking the ‘Go’ button - the cause might be a faulty program. In that case the older applet (see below) might still function appropriately. If the applet itself is OK, then check the preferences of your browser as far as Java is concerned (JAVA 1.4.2), try a newer version of your browser, or download the new browser Firefox by Mozilla. Firefox will offer you the opportunity to download the appropriate Java software from the Sun site. Under Windows Firefox probably will run applets smoothly. On the Mac OS X platform the applet will run smoothly under Safari and Firefox, while Internet Explorer does not quite correctly plot the bar chart but is acceptable. Currently (April 2005) the JVM used is 1.4.2.
  
  

menu items

• Clicking the Go button will start a new simulation and/or analysis using the values specified in the menu.

  exception: using option 1 to leave the analysis out, and next another option, the analysis will take affect ony at the second ‘Go.’

  Because the applet is in a scrollable window, scrolling will cause the applet to execute again. Remember this whenever you declare a larger number of runs or observations. Eventually I might make the applets available in stand-alone format. Such as spa_applet1.sa.htm

• Be aware that all parameters, as well as the option, must hae a specific value. The applet will not run on either a blanc, or anything else that is not a discrete or real number.

• The simulation simulates test scores for the specified number of observations or cases, assuming mastery having the value specified; the resulting distribution of test scores will be plotted as a solid green bar diagram. For larger numbers of observations the simulation will will be closer to the binomial distribution plotted in blue. If you want the analysis only, turn the simulation offf by putting the number of observations = 0.

• The analysis evaluates a probability distribution function called the binomial distribution; the distribution plotted is this mathematical distribution. The binomial distribution is the theoretical distribution of test scores given a particular mastery, for the number of observations growing very large. If you want the simulation only, put option = 1 to turn the analysis off.

• Mastery may be chosen between 0 and 1; mastery as a concept is defined on the set of test items that every concrete test is supposed to have been sampled from: it is the proportion of items the candidate would answer correctly if given the opportunity to try alle items in the domain.

• The number of test items is the length of the test; this number is not limited.

• The number of observations or runs is the number of runs for the simulation, or, equivalently, the number of cases or persons that is represented in the simulated distribution.

• The reference point is a particular test score that is of interest to the user of the applet. It might be the cutoff score in pass-fail testing, the minimum score that is still considered to be a level of sufficient mastery, etcetera. It is marked by a solid vertical in the plot. Its value may be set at 0.

• The reported probability is the proportion of scores equal to or higher than the reference point.
• The option option will make it possible to run special analyses without cluttering the menu of the applet; available options will be mentioned in the applet window if option = 0.

• The horizontal scale is that of test scores obtained, the lowest score being 0.

• The vertical scale as a default will be automatically chosen and depends on the highest value of the analytical distribution, this point is tabmarked. The vertical values are frequencies in the case of simulations, and probabilities in the case of analyses. Their values may be plotted using option 203 and 204, respectively. Of course frequencies and probabilities will be comparable. In fact, option 2 will plot the vertical differences separately.

• The statistics reported are the mean and standard deviation for every distribution. Because the theoretical value of the binomial statistics assumes the number of observations to be infinite, there will be small differences between the actual statistics and the statistics obtained by evaluating their formulas. Simulation and analytical statistics will differ from each other, of course; the differences wil be smaller the larger the number of observations is chosen.

• NOT specifying a value for any of the parameters makes it impossible for the applet to run: at least specify a default value 0 or 0.0 (zero). Becareful to use decimal dots, not periods. Anything that is not a number, will obstruct the applet.
• Illegal specifications will be changed to default values that will enable to applet to run. That one of the values has been changed, will be reported in the plot, including thevariable number thatthe program itself uses. The numbers here are: 40: runs, 1: items, 13: mastery and 0: option. If other numbers are reported to have been changed, the fault is in the program itself: contact me, mentioning the number.
• Simulating a binomial process will result in a distribution that approaches the binomial distribution, the larger the number of runs. Of course it is possible to test for the fit between simulation and the binomial distribution by evaluating the chi-square statistic. But of what use would this be? There is no question about whether the binomial simulation will approximate the binomial distribution. It will do so. Different simulations of the same magnitude will result in different values in chi-square. So what? There is nothing here that turns about having available the chi-squarestatistic, so it is not made available. It is a big nuisance to program because of low cell frequencies that must be handled in one way or another. Another way to argue the irrelevance of testing for fit is that the statistical process generating the simulated scores is known perfectly well: it has been programmed. Testing for ‘fit’ would be totally out of place here. Of course, the program might be at fault; because simulation and analysis are strictly independent in the program, except for their input and output procedures, it is highly unlikely that a mistake in the simulation program would not be revealed by plotting simulation results against the theoretical results, for a number of runs that is large enough.

Generator applet 1m: three-valued

The three-valued generator implements a model allowing the specification of the parameter ‘misunderstanding’ and the parameter ‘bonus’,

October 10 2009. The applet does not function appropriately. I will correct this problem in a few days time. Do not put ‘wrong’ = 0 or 1 - ‘wrong’ = ‘mastery.’
The model is not binomial. Of course, it is possible to approximate this model’ theoretical and simulated distribution using a binomial on the parameter p = mastery + bonus * ( 1 - mastery - wrong ). Comparing the 3-valued model results to those of the approximation, shwos the results of the 3-valued model to be a fraction too low. This is the other problem that I will have to solve.
The explanation of the approximation formula is rather simple. The expectation of the score to be earned on the very first item is E( score ) = mastery + bonus * ( 1 - mastery - wrong ). Translate this expectation into a probability to get a score of 1 point. Voila.

new menu items in this three-valued generator

• Mastery still is the proportion of items in the domain that is known correctly. Correct answers still will be assumed to get the score 1.
• The proportion of items attempted and answered typically will be larger than the proportion of mastered items. Therefore, the parameter misunderstanding is the proportion of items in the domain that will be attempted and answered wrongly. Wrong answers will be assumed to get the score 0. There is no option to give wrong answers the score -1, but the user who wants to do so, will be able to translate the results on the 0 — 1 scale linearly to the -1 — +1 scale.
• Of course, candidates must not be forced to provide answers on items they do not know. Therefore, questions left unanswered will be honored with a bonus score of .5, or any other proportion between 0 and 1 that the user will have reason to use instead. In the case of a multiple-choice test, one may set the bonus as the inverse of the number of alternatives, or somewhat higher.
• This applet does provide option 111 to specify a guessing probability on multiple-choice items (forced guessing). If candidates are forced to guess on items they do not know (it certainly is not recommended to use forced guessing!), the mastery parameter will be replaced by the result of the formula [mastery + ( 1 - mastery - misunderstanding ) * bonus], ‘bonus’ now being the probability to correctly guess the answer on a multiple-choice item from this specific domain that the candidate does not know and therefore does not attempt to answer. Option is 111 effectively results in a binomial distribution on the re-evaluated parameter. Be aware of the changed use of the ‘bonus’ parameter: under option 111 it is the probability to guess the correct answer on items from the domain that the candidate does not (think to) know the answer on. [September 3, I still have to check on one or two things]
• Note that demanding a proper justification of the answers on multiple choice items, will remove the problem of guessing on these items. Any guessing that is left possible is the kind of guessing that candidates might try on open questions also.
• Because the bonus is a proportion, and not a probability (except under option 111), the distribution of test scores no longer will be binomial. The mastery part itself still is binomial, but the bonus points must then be added to the scores earned by answering correctly. There probably is not an analytical expression for the resulting theoretical distribution, so I call it simply the three-valued distribution, the three values being: mastery, misunderstanding, and not knowing. There being no formula for the theoretical distribution does not make it less ‘theoretical&rsquo: it is a theoretical construction, for the details see the module 1 chapter: 1. generator
• For the other menu items, see above, or the generator applet 1.

Generator applet 1msc: free scaling of the plot

In order to manipulate the plot, one should be free to choose the horizontal and vertical scaling.

new menu items: scaling

• The height parameter h will determine the height of the plot of the theoretical distribution. Setting h = 1 will plot the theoretical distribution in the full height of the available space. The plot of the simulation will be in the same scale; simulations using a not so large number of runs will plot better by reducing the height parameter. Zero height will result in a default value that depends on the height of the theoretical distribution that will be plotted (this will work also under option 1: the theoretical distribution will beevaluated, but not plotted).
• The width parameter w is simply the number of pixels horizontally. Setting the number of pixels at zero will result in its default value 500.
• In order to research the model under extreme conditions, the width parameter can be used to get a detailed plot of the most left part. For example, specifying items = 1000, mastery = 0.1, wrong = 0.00001, bonus = 0, set width = 5000.

Advanced applet 1

This version of your browser does not support Java

new menu items

• This advanced version of The Generator adds some features to the ones adstructed at the top of this webpage.
• The second set of values option makes it possible to directly compare two situations that you have made to differ in a particular way in the values of one or more of their parameters. If two positions are offered for entering values, the first one belongs to test 1, the second one to test 2.
• It is no use denying that many tests use the multiple choice item format, introducing a disturbance in the results by the guessing behaviour of the student. Anyhow, specifying the probability a not known item may be guessed correctly, will produce a binomial on the parameter mastery + ( 1 - mastery ) * guessing.

  Be aware that mastery is defined as the proportion of items known in the domain. In multiple choice testing, it is not the case that the other items will be items that the candidate will guess: surely there will be a certain proportion of items that the candidate will answer wrongly. As of august 2009 the applet does not have the option to specify this misunderstanding. I will update the applet soon, however. In the meantime the probability to correctly guess the answer should absorb the probability to wrongly know the answer. Regrettably, the situation of allowing bonus points for items left unanswered, cannot be modeled correctly by the applet as it stands now. Allowing a bonus effectively makes mc-items three-valued: correct, wrong, or bonus. Just like mastery, misunderstanding (propotion wrong in the domain) should be specified. Most of the literature on guessing does not recognize this simple fact, implicitly but falsely assuming that all wrong answers on mc-items are the result of guessing.

• The option of stratified subdomain sampling has been spoken of above in the paragraph on special points. The reference point for the test is the sum of the two reference points as declared in the menu (NB: this implies full compensation between the scores on the two subtests).
• If stratified domain sampling is chosen, a further possibility is to give the items in the second subdomain a special integer valued weight, correctly answered items might result in a score of 5 instead of the usual 1, for example.
• The scale parameters scale height and scale width, the last one is the number of pixels available for evey bar, will be chosen by the program unless some positive value is specified in the menu. Trial and error will direct you to the best fitting values.
• The horizontal scale is that of test scores obtained, the lowest score being 0. Note that the score obtained might be higher than the number of items answered correctly, using the weighting option together with stratified subdomain sampling. The vertical scale will be automatically chosen and depends on the highest value of the analytical distribution, this point is tabmarked.
• The option option will make it possible to run special analyses without cluttering the menu of the applet; as of April 2005 only option = 2 has been implemented: plot the distribution(s) and leave everything else out.
• The statistics reported are the mean and standard deviation for every distribution. Because the theoretical value of the binomial statistics assumes the number of observations to be infinite, there will be small differences between the actual statitics and the statistics obtained by evaluating their formulas. Simulation and analytical statistics will differ from each other, of course; the differences wil be smaller the larger the number of observations is chosen.

Here you will find another, older, version of the module 1 applet (also its advanced version here below), based on the class DoBinomial dedicated to the binomial / the generator only. DoBinomial is the class that takes care of (most) everything that is input or output, including plotting of distributions. DoBinomial uses the class Basics: the methods Basics.getSimulatedBinomial and Basics.Binomial. The class Basics always is the most recent version, the class is used by most or all other applets of the SPA model. The methods mentioned come in two flavors: a five resp. three parameter method is the standard, a nine (Basics.Simulate) resp. eight parameter method to take care of two tests cases and/or subdomain divisions.

menu items

• misunderstanding is the proportion of all possible items, satisfying the specific criteria for a particular test or subject matter, that the student would answer incorrectly if given the opportunity. See the spa_generator text here. This parameter takes effect only in combination with a positive value for the quessing parameter. Guessing then occurs only on the items that have not been answered either correctly or incorrectly, i.e. either on the basis of mastery, or on the basis of misunderstanding. The testee need not be aware of the difference, of course. How does it work: mastery .6 and misunderstanding .2 means that the probability for an item stay unanswered (either correctly or incorrectly) is 1 - .6 - .2 = .2. A change to guess an unknown item correct of .5 will mean in this particular case that the chance that the next item will be an unknown item that will be guessed ‘correctly’ is .5 × .2 = .1. The simulation in this particular case will use as success parameter, therefore, .6 (mastery) + .1 (lucky guesses) = .7. As will the analytical plot of the binomial distribution: the binomial parameter is .7.

• option 100 will plot a 2 * n² scheme of an item pool: every item a small rectangle, filled if it would be answered correct. The scheme is what the item pool might look like for a particular student having known mastery. In fact this scheme will be generated as one big simulated test; therefore the number corret will most of the times not exactly match the assumed mastery. This option is not available in the regular applet # 1 above.

• option 101 will plot 2 * n schemes of simulated tests of n items, mastery assumed known. This option is not available in the regular applet # 1 above.

• option 102 will plot 2 * n schemes of simulated tests of n items, mastery assumed known, items ordered. This option is not available in the regular applet # 1 above.

• option 103 Same as 102, except given mastery is changed randomly for every new item simulated either +0.05 or -0.05.

• With options 100 to 103 the width parameter determines the number of pixels width/height of every items. The minimum is 2 pixels, Java's fillpolygon will not plot rectangles on less then 2 pixels width.

• This kind of scheme has been used in projects and publications since the early eighties (first year in dentistry and in law, University of Amsterdam, to inform students on the chance aspects of the tests they would have to sit that year, so they would be able to better prepare themselves or to better interpret their own results compared to those of their fellow students, see 4.3 in Voorthuis and Wilbrink (1987 html)

Here you will find another, older, version of the module 1 applet’ advanced version, based on the class SPA_DoBinomial dedicated to the binomial / the generator only. SPA_DoBinomial is the class that takes care of (most) everything that is input or output, including plotting of distributions. SPA_DoBinomial uses the class SPA_Basics

If you come across an applet that is not functioning properly, please mail me. It is not possible always to check all applets for unintended consequences of changes in classes. As this is a project in progress, such changes are made on a routine basis.

Applets are known to work correctly under:
Internet Explorer under Windows XP
Firefox 1.0.7 under Windows XP
Safari 1.2 under MacOSX 10.3.9
FireFox 1.5.0.4 under MacOSX 10.3.9

It might be the case that the applets do not open properly in browsers under Windows, or in browsers other than Safari under MacOS X: the applet field remains gray or blank.
In module chapters original applets have been replaced with screenshots; therefore applet problems should not hinder readers of the SPA project. Readers not able to use the applets in their browser, and yet willing to do so, may contact me, if preferences of the browser pertaining to Java do not seem to be the problem.

Information about Java, and applets in particular:
http://java.sun.com
http://java.sun.com/applets/

MacOS X: There is a problem with Java versions 1.4 for browsers other than Safari. See http://javaplugin.sourceforge.net/Readme.html; http://developer.apple.com/documentation/Java/Conceptual/Java131Development/deploying/chapter_3_section_5.html; simile.mit.edu/repository/ misc/java_embedding_plugin/readme.rtf
MacOS X: Opera, version 8.5, produces 'java.lang.UnsupportedClassVersionError: Spa_BinomialApplet (Unsupported major.minor version 48.0)
MacOS X: Internet Explorer 5.2 for Mac, [preferences: enable Java on; cookies: never ask; web content: enable plug-ins on] produces 'java.lang.UnsupportedClassVersionError: Spa_BinomialApplet (Unsupported major.minor version 48.0)

Windows: Java Applet plug-in makes it possible for your computer (including Windows¨ XP, Me, NT, 2000, 98, or 95) to run applets in your browser. http://www.mcdonalds.com/search/help/plug_play/sunmicro.html

Mail your opinion, suggestions, critique, experience on/with the SPA

  http://www.benwilbrink.nl/projecten/spa_applet1.htm

• thuis
• sitemap
• publicaties
• projecten
• toetsvragen
• SPA-model
• mail