Meta-analyse en wat werkt in onderwijs

Annotaties Ben Wilbrink

“Finally, with respect to meta-analysis, our recommendation is simple: Just say no. The suggested alternative is equally simple: Read the papers, think about them, and summarize them. Try our alternative. Trust us: You will like it.”

David A. Freedman & Richard A. Berk (2005). Statistical assumptions as empirical commitments. Reprinted in David A. Freedman (2010). Statistical models and causal inference. A dialogue with the social sciences (p. 40). Cambridge University Press. online version 2001

Margaret C. Wang, Geneva D. Haertel & Herbert J. Walberg (1993). Toward a knowledge base for school learning. Review of Educational Research, 63, 249-294. abstract

Dit is een voorloper van Hattie (2009): het overzicht berust onder andere op 91 beschikbare meta-analyses.

“In projecting future results from the present findings, great caution is necessary. Poorly implemented versions of previously successful practices, especially those shown in special circumstances, are unlikely to work as well. Some practices that work well in some settings and with some students may not work as well with others, although evidence for such exceptionality is easier to hypothesize than to show consistently. The aggregated estimates nonetheless provide one reasonable basis for formulating educational policies and practices. They represent what can be distilled from an enormous body of educational research extending over the last half century; and the independent sources of evidence show reasonable agreement. [p. 279-280]”

Jerome D’Agostino & Sonya Powers (2009). Predicting teacher performance with test scores and grade point average: A meta-analysis. American Educational Research Journal, 46, 146-182. abstract

Ik kan er weinig mee. Het gaat ook om de specifiek Amerikaanse situatie. Ik vind wel dat de auteurs heel veel aandacht hebben voor deze statistische analyses, en weinig voor vakinhoudelijke en didactische kwaliteiten (waar het uiteindelijk om draait, toch?). Een artkel dat dit artikel citeert:
Jürgen Baumert, Mareike Kunter, Werner Blum, Martin Brunner, Thamar Voss, Alexander Jordan, Uta Klusmann, Stefan Krauss, Michael Neubrand & Yi-Miau Tsai (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47, 133-180. abstract

John Hattie (2011). Visible learning, tomorrow’s schools, the mindsets that make the difference in education. ppt 1,5 Mb

John Hattie (2009). Visible Learning: A Synthesis of over 800 Meta-Analayses relating to Achievement. Routledge. inhoudsopgave

E. D. Bolhuis & T. M. Willemse: bespreking. pdf

Eén van de meta-analyses is de volgende, over het gebruik van rekenmachines (in de lijst van Hattie (2011) is het # 45 als ik het goed heb). Ik maak een wat uitvoeriger annotatie bij deze publikatie, gezien de actualiteit (aangenomen motie om het gebruik van rekenmachine terug te dringen; OCW moet er nog op antwoorden, waarschijnlijk in april).

Aimee Ellington (2003). A Meta-Analysis of the Effects of Calculators on Students' Achievement and Attitude Levels in Precollege Mathematics Classes. Journal for Research in Mathematics Education, 34, 433-463. eerste pagina. Annotaties zie hier.

John Hattie (2011). Visible Learning for Teachers. Routledge. inhoudsopgave

Dit boek is niet een nieuwe meta-analyse, maar geeft de visie van Hattie op wat goed onderwijs door de leraar is. Hij baseert zich vooral op de resultaten van zijn meta-meta-analyse (Hattie, 2009). Ik zal dit boek geduldig vanaf bladzijde 1 bestuderen en van aantekeningen voorzien. Op voorhand ben ik achterdochtig, omdat zijn meta-meta-analyse maar een beperkt deel van de onderwijswerkelijkheid kan laten zien, en het zelfs dan nog onduidelijk is wat de resultaten precies kunnen betekenen voor de praktijk. Ik zal dus voortdurend buiten de smalle kaders van Hattie proberen te treden. Ik denk dat ik dat in eerste instantie zal doen door een contrast te maken met drie andere boeken. Li & Kaiser (2011) Expertise in Mathematics Instruction, omdat hier nadrukkelijk de specifieke vakinhoud van het onderwijs erbij wordt betrokken; Carpenter, Fennema & Romberg(1993) omdat de inhoud hier nog weer concreter is gekozen als het domein van de rationele getallen (breuken, proporties,enz); en Ericsson (2009) omdat het directe doel van het onderwijs toch is om te reiken naar beginnend expertschap van de leerlingen.

Yeping Li & Gabriele Kaiser (Eds.) (2011). Expertise in Mathematics Instruction, An International Perspective. Springer http://www.springerlink.com/content/x5k432/#section=831055&page=1 [Via een UB online beschikbaar].
Thomas P. Carpenter, Elizabeth Fennema & Thomas A. Romberg (Eds.) (1993). Rational Numbers. An Integration of Research. Erlbaum. zie ook hier voor inhoudsopgave enz.
K. Anders Ericsson (Ed.) (2009). Development of professional expertise: Toward measurement of expert performance and design of optimal Learning Environments. Cambridge University Press. inhoudsopgave

Preface (Hattie, 2011). Een mooie tekst. Ik weet niet wat ik ervan moet denken; het is een voorwoord, het privilege van de schrijver. De rozevingerige dageraad van een pleidooi gericht op ‘impact’. Een bericht uit een andere cultuur, waar positieve overdrijving de norm is? Het geeft mij geen aanwijzingen over wat ik mag verwachten, behalve dat ijdelheid aan de auteur niet vreemd is. Ik vind het nogal over the top en had liever een bescheiden verklaring gezien dat het er in het onderwijs om gaat de tijd van de leerling niet te verspillen (Deanna Kuhn, 2005).

Handboeken en readers

G. V. Glass, B. McGaw & M. L. Smith (1981). Meta-analysis in social research. SAGE.

John E. Hunter, Frank L. Schmidt & G. B. Jackson (1982). Advanced meta-analysis: quantitative methods for cumulating research findings across studies. SAGE.

Thomas R. Kratochwill & Joel R. Levin (Eds) (1992). Single-case research design and analysis. Erlbaum.

Meta-analysis for single-case research

John E. Hunter & Frank L. Schmidt (1990). Methods of meta-analysis. Correcting error and bias in research findings<. SAGE.

Brian Mullen (1989). Advanced BASIC meta-analysis. Erlbaum.

Dit is programmatuur, maar dat had u al begrepen.

Artikelen over meta-analyses

R. Rosenthal and M. R. DiMatteo (2001). Meta-Analysis: Recent Developments in QuantitativeMethods for Literature Reviews. Annual Review of Psychology, 52, 59-82. abstract

W. R. Shadish (1996). Meta-analysis and the exploration of causal mediating processes: a primer of examples, methods, and issues. Psychological Methods, 1, 47-65.

Theresa Ann Sipe & William L. Curlette (Guest eds.) (1997). A meta-synthesis of factors related to educational achievement: A methodological approach to summarizing and synthesizing. International Journal of Educational Research, 25 #7, 583-698.

Rana M. Tamim, Robert M. Bernard, Eugene Borokhovski, Philip C. Abrami, and Richard F. Schmid (2011). What Forty Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study. Review of Educational Research, 81, 4-28.

Giovanni W. Sosa, Dale E. Berger, Amanda T. Saw, and Justin C. Mary (2011). Effectiveness of Computer-Assisted Instruction in Statistics: A Meta-Analysis. Review of Educational Research, 81, 97-128.

R. E. Slavin (1986). Best-evidence synthesis: an alternative to meta-analytic and traditional reviews. Educational Researcher, 15 (9), 5-11.

reprinted as chapter 6 in Slavin’s Education for all

Gene V. Glass (1978). Integrating findings: The meta-analysis of research. Lee S. Shulman (Ed.) (1978). Review of research in education volume 5—1977. F.E. Peacock Publishers.

Abrami, Cohen, & d'Apollonia (1988). Implementation problems in meta-analysis. Review of Educational Research, 58.

H. G. Osburn & J. Callender (1992). A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalzation. Journal of Applied Psychology, 77, 115-122.

R. Rosenthal & D.B. Rubin (1989). Effect size estimation for one-sample multiple-choice-type data: design, analysis and meta-analysis. Psychological Bulletin, 106, 332-337

Meta-analyses uit de literatuur

H. Cooper, B. Nye, K. Charlton, J. Lindsay & S. Greathouse (1996). The effects of summer vacation on achievement test scores: a narrative and meta-analytic review. Review of Educational Research, 66, 227-268.

Ut Na Sio and Thomas C. Ormerod (2009). Does incubation enhance problem solving? A meta-analytic review. Psychological Bulletin, 135, 94-120.

Arthur E. Poropat (2009). A Meta-Analysis of the Five-Factor Model of Personality and Academic Performance. Psychological Bulletin, 135, 322-338.

Slavin, R. E. (1987). Ability grouping and student achievement in elementary schools: a best-evidence synthesis. Review of Educational Research, 57, 347-350.

reprinted as chapter 7 in Slavin’s Education for all

R. E. Slavin (1990). Achievement effects of ability grouping in secondary schools: a best evidence synthesis. Review of Educational Research, 60, 471-500.

met repliek en dupliek
reprinted as chapter 8 in Slavin’s Education for all

Smith & Glass (1980). Meta-analysis of research on class size and its relationship to attitudes and instruction. American Journal of Educational Research, 17, 419-433.

M. Ph. Born, N. Bleichrodt & H. van der Flier (1987). Cross-cultural comparison of sex-related differences on intelligence tests. A meta-analysis. Journal of Cross-Cultural Psychology, 18, 283-314.

Nathan R. Kuncel, Marcus Credé & Lisa L. Thomas (2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of Educational Research, 75, 63-82. preview

Steven B. Robbins, Kristy Lauver, Huy Le, Daniel Davis, Ronelle Langley & Aaron Carlstrom (2004). Do Psychosocial and Study Skill Factors Predict College Outcomes? A Meta-Analysis. Psychological Bulletin, 130, 261-288.

R. L. Linn & C.N. Hastings (1984). A meta-analysis of the validity of predictors of performance in law school. Journal of Educational Measurement, 21, 245-259.

White (1973). Individual differences in speed, accuracy and persistence. In H. J. Eysenck: The measurement of intelligence MTP Medical and Technical Publishing Co. Ltd.

uit het abstract: Using meta-analysis techniques, almost 200 studies that considered the relation between SES and academic achievement were examined. Results indicated that as SES is typically defined (income, education, and/or occupation of household heads) and typically used (individuals as the unit of analysis), SES is only weekly correlated (r = .22) with academic achievement. With aggregated units of analysis, tyically obtained correlations netween SES and academic achievement jump to .73. Family characteristics, such as home atmosphere, sometimes incorrectly referred to as SES, are substantially correlated with academic achievement when individuals are the unit of analysis (r = .55). Factors such as grade level at which the measurement was taken, type of academic achievement measure,type of SES measure, and the year in which the data were collected are significantly correlated statistically with the magnitude of the correlation bewteen achievement and SES. Variables considered in the meta-analysis accouted for 75% of the variance in observed correlation coefficients in the studies examined.

Avraham N. Kluger & Angelo DeNisi (1996). The Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory. Psychological Bulletin, 119, 254-284. pdf

citeert: To relate feedback directly to behavior is very confusing. Results are contradictory and seldom straight-forward. (Ilgen, Fisher, & Taylor, 1979, p. 368) The effects of manipulation of KR [knowledge of results] on motor learning . . . reveal . . . some violent contradictions to earlier beliefs about KR, and some glaring absences in our knowledge. (Salmom, Schmidt, & Walter, 1984, p. 378). Feedback does not uniformly improve performance. (Baleazar, Hopkins, & Suarez, 1985, p. 65) Few concepts in psychology have been written about more uncriti-. cally and incorrectly than that of feedback. . . . Actually, feedback is only information, that is, data, and as such has no necessary consequences at all. (Latham & Locke, 199 1, p. 224). p. 278: Implications. Before we conclude, we must reflect on the applied implication of our study. The identification of a number of moderators suggests that in certain situations, FT can yield a large and positive effect on performance. Specifically, an FT provided for a familiar task, containing cues that support learning, attracting attention to feedback-standard discrepancies at the task level (velocity FI and goal setting), and is void of cues to the metatask level (e.g., cues that direct attention to the self) is likely to yield impressive gains in performance, possibly exceeding I SD. However, even such an Fl is not always an efficient intervention. Even when FI has considerable positive effects on performance, its utility may be marginal or even negative. When an FT increases performance through an increase in task motivation, the effect may depend on a continuous FT. Removal of such an F1 may result in a reversal as some field studies have demonstrated (Komaki et al., 1980). Therefore, the cost of maintaining a continuous intervention should be considered in evaluating such an intervention. If, however, F1 affects perforthance through task-leaming processes, the effect may create only shallow learning and interfere with more elaborate learning. Lack of elaborate learning affects the ability to use the learned material in transfer tasks where the task is similar but not identical (e.g., Carroll & Kay, 1988). Moreover, the evidence for any learning effect here was minimal at best. Indeed, in the MCPL literature, several reviewers doubt whether Fls have any learning value (Balzer et al., 1989; Brehmer, 1980) and suggest alternatives to FT for increasing learning, such as providing the learner with more task information (Balzer et al., 1989). Another alternative to an FT is designing work or learning environments that encourage trial and error, thus maximizing learning from task feedback without a direct intervention (Frese & Zapf, 1994). These considerations of utility and altemative interventions suggest that even an F1 with demonstrated positive effects on performance should not be administered whenever possible. Rather, additional development of FIT is needed to establish the circumstance under which positive F1 effects on performance are also lasting and efficient and when these effects are transient and have questionable utility. This re3earch must focus on the processes induced by FTs and not on he general question of whether FIs improve performancelook at how little progress 90 years of attempts to answer the latter question have yielded.

Karen M. La Paro & Robert C. Pianta (2000). Predicting Children's Competence in the Early School Years: A Meta-Analytic Review. Review of Educational Research, 70, 443-484.

C. Thomas Holmes & Kenneth M. Matthews (1984). The effects of nonpromotion on elementary and junior high school pupils: a meta-analysis. Review of Educational Research, 54, 225-236.

"In all cases, the outcomes for promoted pupils were more positive than for retained pupils."
Conclusion "Those who continue to retain pupils at grade level do so despite cumulative evidence showing that the negative effects consistently outweighs positive consequences. Because this cumulative research evidence consistently points to negative effects of nonpromotion, the burden of proof legitimately falls on proponents of retention plans to show there is compelling logic indicating success of their plans when zo any other plans have failed."

T. Morrison & M. Morrison (1995). A meta-analytic assessment of the predictive validity of the quantitative and verbal coponents of the Graduate record Examination with graduate grade point average representing the criterion of graduate success. Educational and Psychological Measurement, 55, 309-316.

Probleem is dat de auteurs met geen woord reppen over restriction of range, dus ik moet aannemen dat de correlaties alleen op de GRE geselecteerde graduate students betreffen. Het resultaat is een verklaarde variantie van 6%, tel uit je winst. Fantastisch, maar zonder gegevens over restriction of range is het natuurlijk niet overtuigend.

A. Mitra, G. D. Jenkins Jr & N. Gupta (1992). A meta-analytic review of the relationship between absence and turnover. Journal of Applied Psychology, 77, 879-889.

P. W. Hom, F. Caranikas-Walker, G. E. Prussia & R. W. Griffeth (1992). A meta-analytical structural equations analysis of a model of employee turnover. Journal of Applied Psychology, 77, 890-909.

Ik hoef hier niet toe te lichten dat vertrek van werknemers net zoiets is als vertrek van leerlingen.

L. A. Jackson, L. A., Hunter, J. E., & Hodge, C. N. (1995). Physical attractiveness and intellectual competence: a meta-anallytic review. Social Psychology Quarterly, 58, 108-122.

Leuk en relevant artikel. geen kopie. Geen echt harde effecten aangetoond. Aantrekkelijker mensen, vooral mannen, worden hoger ingeschat wat intellectuele capaciteiten betreft, wanneer er geen andere informatie voorhanden is tenminste. Voor kinderen op school ligt dan nog weer iets anders.

N. Falchikov & D. Boud (1989). Student self-assessment in higher education: a meta-analysis. Review of Educational Research, 59, 395-430.

Nancy Falchikov & Judy Goldfinch (2000). Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research, 70, 287-322

Michael C. Rodriguez‌. (2005). Three options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement Issues and Practice 24 #2, 3.

David Gijbels, Filip Dochy en Mien Segers (2005). Effects of Problem-Based Learning : A Meta-Analysis From the Angle of Assessment. Review of Educational Research, 75, 27-62.

B. de Jager & G. J. Reezigt (1996). Onderwijseffectiviteit en metacognitieve vaardigheden. Groningen: GION. SVO 95715.

“Wat de leerlingen betreft hangen prestaties en metacognitieve vaardigheden licht negatief samen. Dat geen positieve samenhang kon worden gevonden (zoals op grond van de literatuur was te verwachten) kan liggen aan de wijze waarop de metacognitieve vaardigheden zijn geoperationaliseerd.”

Daar heb je weer zo’n wezenloze afsluiting bij een resultaat dat juist aan het denken had moeten zetten. Crombag c.s. vonden iets als dit al veel eerder (TOR, 1976, 1, 1-17) en waren zeker niet de eersten. Waarom niet eens gekeken of er mechanismen in de gebruikte didactiek en vooral in de toetsing zijn waardoor de inzet van metacognitieve vaardigheden wordt afgestraft? Maar in de klas is natuurlijk niet gekeken, want het gaat hier om meta-anayse op VOCL-data.

D. W. Johnson, G. Maruyama, R. Johnson, D. Nelson & L. Skon (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: a meta-analysis. Psychological Bulletin, 89, 47-62.

The results of the meta-analyses indicate (a) that cooperation is considerably more effective than interpersonal competition and individualistic efforts, (b) that cooperation with intergroup competition is also superior to interpersonal competition and individualistic efforts, and (c) that there is no significant difference between interpersonal competitive and individualistic efforts. Through multiple regression, a number of potentially mediating variables for these results are identified.

L. V. Hedges & W. Stock, W. (1983). The effects of class size: an examination of rival hypotheses. American Educational Research Journal, 20, 63-85.

Kulik, Kulik & Cohen (1980). Effectiveness of computer-based college teaching: a meta-analysis of findings. Review of Educational Research, 50, 525-544.

J. A. Kulik (1994). Meta-Analytic Studies of Findings on Computer-Based Instruction. In Baker, E. L., & O'Neil, H. F., J. (Eds.) (1994). Technology Assessment in Education and Training. Erlbaum.

C-L. C. Kulik & J. A. Kulik (1982). Effects of ability grouping on secondary school students: a meta-analysis of evaluation findings. American Educational Research Journal, 19, 415-428.

alleen maar positieve uitkomsten! Kan dat waar zijn?

J. A. Kulik, R. L. Bangert-Drowns & C-L. C. Kulik (1984). Effectiveness of coaching for aptitude tests. Psychological Bulletin, 95, 179-188.

M. R. Barrick & M. K. Mount (1991). The big five personality dimensions and job performance: a meta-analysis. Personnel Psychology, 44, 1-26.

A. Schaefli, J. R. Rest & S. J. Thoma (1985). Does moral education improve moral judgment? A meta-analysis of intervention studies using the Defining Issues test. Review of Educational Research, 55, 319-352.

P. A. Mabe III, & S. G. West (19820. Validity of self-evaluation of ability: a review and meta-analysis. Journal of Applied Psychology, 67, 280-296.

B. J. Becker & L. V. Hedges (Eds. ) Special issue on meta-analysis. Journal of Educational Statistics, 17 # 4.

P. A. Cohen (1980). Effectiveness of student-rating feedback for improving college instruction. A meta-analysis of findings. Research in Higher Education, 13, 321-341.

P. A. Cohen (1981). Student ratings of instruction and student achievement: a meta-analysis of multisection validity studies. Review of Educational Research, 51, 281-309.

J. B. Dusek & G. Joseph (1983). The bases of teacher expectancies: a meta-analysis. Journal of Educational Psychology, 75, 327-346.

David Watkins (2001). Correlates of Approaches to Learning: A Cross-Cultural Meta-Analysis. In Robert J. Sternberg & Li-Fang Zhang: Perspectives on Thinking, Learning, and Cognitive Styles. Erlbaum.

B. B. Gaugler et al. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493-511.

John E. Hunter & Frank L. Schmidt (1983). Quantifying the Effects of Psychological Interventions on Employee Job Performance and Work-Force Productivity. American Psychologist, 473-478.

Frank L. Schmidt & John E. Hunter (1983). Individual differences in productivity: an empirical test of estimates derived from studies of selection procedure utility. Journal of Applied Psychology, 68, 407-414.

abstract: Meta-analysis of the cumulative research on various predictors of job performance shows that for entry-level jobs there is no predictor with validity equal to that of ability; which has a mean validity of .53. For selection on the basis of current job performance, the work sample test, with mean validity of .54, is slightly better. For federal entry-level jobs, substitution of an alternative predictor would cost ftom $3.12 billion (job tryout) to $15.89 billion per year (ap). Hiring on ability has a utility of $15.61 billion per year, but affects minority groups adversely. Hiring on ability by quotas would decrease this utility by 5%. A third strategy — using a low cutoff score — would decrease utility by 83%. Using other predictors in conjunction with ability tests might improve validity and reduce adverse impact, but there is as yet no data base for studying this possibility.
Werkt met macro-aggregatie van selectieopbrengsten, vgl. Hofstee over landelijke gesommeerde effecten. Zie p. 72: A crucial element in the maintenance of high productivity in both government and private industry is the selection of people with high ability for their jobs. For most jobs, the only presently known predictive devices with high validity are cognitive and psychomotor ability tests. Recent work on validity generalization (Schmidt, Gast-Rosenberg, & Hunter, 1980; Schmidt & Hunter, 1977; Schmidt & Hunter, 1980; Schmidt, Hunter, Pearlman, & Shane, 1979) has shown that most findings of low validity are due to artifacts, mainly statistical error due to small sample size. Similar and broader conclusions follow from reanalyses of past meta-analyses by Lent, Aurbach, and Levin (1971), as noted by Hunter (1980b) and from inspection of unpublished large sample studies done in the United States Army by Heim, Gibson, and Brogden (1957), in Hunter (1980c), and in Schmidt, Hunter, and Pearlman (1981). High selection validity results into considerable financial savings for most organizations. Hunter (1979a) estimated that if the Philadelphia, Pennsylvania, Police Department were to drop its use of a cognitive ability test to select entry-level officers the cost to the city would be more than $170 million over a 10-year period. Schmidt, Hunter, McKenzie, and Muldrow (1979) show that over a 10-year period, with a 50% selection ratio, the federal government could save $376 million by using a cognitive ability test to select computer programmers rather than by selecting at random.' Hunter and Schmidt (I 982b) applied a utility model to the entire national work force. Gains from using cognitive tests rather than selecting at random are not as spectacular at the national level as might be predicted by findings from single organizations. This is because high-ability people who are selected for crucial top-level jobs will not be available for lower level jobs, where they would ???? higher productivity. However, even considering this cancellation of effect, Hunter and Schmidt estimated in 1980 that productivity differences between complete use of cognitive ability tests and no use of the tests would amount to a minimum of $80 billion per year. That is, productivity differences due to use or nonuse of present tests would be, in the current state of the economy, about as great as total corporate profits, or about 20% of the total federal budget.
Maar let op: wat voor een individueel bedrijf of instituut geldt, geldt hier niet voor de hele economie! Iedereen vist uiteindelijk in dezelfde vijver met talent. Hunter en Schmidt blunderen hier gigantisch.

J. E. Hunter & R. F. Hunter (1984). Validity and utility and alternative predictors of job performance. Psychological Bulletin, 96, 72-98.

M. A. McDaniel, Frank L. Schmidt & John L. Hunter (1988). A meta-analysis of the validity of methods for rating training and experience in personnel selection. Personnel Psychology, 41, 283-314.

Frank L. Schmidt (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.

D. Z. Ones, C. Viswesvaran & F. L. Schmidt (1993). Comprehensive meta-analysis of integrity test validityt: findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679-550.

M. A. McDaniel, D. L. Whetzel, F. L. Schmidt & S. D. Maurer (1996). The validity of employment interviews: a comprehensive meta-analysis. Journal of Applied Psychology, 79, 599-616.

John E. Hunter & Frank L. Smith (1996). Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, and Law, 2, 324-347.

Frank L. Schmidt & John E. Hunter (1998). The Validity and Utility of Selection Methods in Personnel Psychology Practical and Theoretical Implications of 85 Years of Research Findings. Psychological Bulletin, 124, 262-274.

J. Cameron & W. D. Pierce (1994). Reinformcement, reward, and intrinsic motivation: a meta-analysis. Review of Educational Research, 64, 363-424.

Edward L. Deci, Richard Koestner & Richard M. Ryan (2001) Extrinsic Rewards and Intrinsic Motivation in Education: Reconsidered Once Again Review of Educalional Research, 71, 1-27

Judy Cameron (2001). Negative Effects of Reward on Intrinsic Motivation — A Limited Phenomenon: Comment on Deci, Koestner, and Ryan (2001). Review of Educational Research, 71, 29-42.

Edward L. Deci, Richard M. Ryan, & Richard Koestner (2001). The Pervasive Negative Effects of Rewards on Intrinsic Motivation: Response to Cameron (2001). Review of Educational Research, 71, 43-51.

Luiten, Ames & Ackerson (1980). A meta-analysis of the effects of advance organizers on learning and retention. American Educational Research Journal, 17, 211-218.

M. W. Lipsey & D. B. Wilson (1993). The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

N. R. Kuncel, S. A. Hezlett & D. S. Ones (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Pychological Bulletin, 127, 162-181. Kopie.

B. D. Haig (1991). Meta-analysis, methodology and research intgration. International Journal of Educational Research , 15, 569-586.

V. Hasselblad & L. V. Hedges (1995). Meta-analysis of screening and diagnostic tests. Psychological Bulletin, 117, 167-178.

H. L. Miller, Jr. (Ed.) (1997). The New York City Public Schools Integrated learning Systems Project; Evaluation and Meta-Evaluation. International Journal of Educational Research, 27 #2, 89-184. Special issue.

A. I. Huffcutt , Ph. L. Roth & M. A. McDaniel (1996). A meta-analytic investigation of cognitive ability in employment interview evaluations: moderating characteristics and implications for incremental validity. Journal of Applied Psychology, 81, 459-473.

N. S. Raju, M. J. Burke, J. Normand & G. M. Langlois (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432-446.

Folkert Haanstra (1994). Effects of art education on visual ability and aesthetic perception: two meta-analyses. Proefschrift Universteit van Amsterdam.

S. W. Raudenbush & A. S. Bryk (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75-98.

Schmitt, Gooding, Noe & Kirsch (1984). (Meta-analysis of validity studies published between 1964 and 1982). Personnel Psychology, 1984. (nog niet gecopieerd). selectie

D. P. Tett, D. N. Jackson & M. Rothstein (1994). Personality measures as predictors of job performance: a meta-analytic review. Personnel Psychology, 44, 703-742.

Hier ontstaat een woordenwisseling over:
Tett, D. P. , Jackson, D. N. , Rothstein, M. , & Reddon, J. R. (1994). Meta-analysis of personality-job performance relations: a reply to Ones, Mount, Barrick, and Hunter (1994). Personnel Psychology, 47, 157-172.
Ones, Deniz S. , Mount, Michae K. , Barrick, Murray R. , & Hunter, John E. (1994). Personality and job performance: a critique of the Tett, Jackson, and Rothstein (1991) meta-analysis. Personnel Psychology, 47, 147-156.

E. M. Whitener (1989). A meta-analytic review of the effect on learning of the interaction between prior achievement and instructional support. , 65-86

Aptitude Treatment Interaction

W. H. Wiesner & S. F. Cronshaw (1988). A meta-analytic investigation of the impact of interview format and degree of structure on the validity of the employment interview. Journal of Occupational Psychology, 61, 275-290.

V. L. Willson & R. R. Putnam (1982). A meta-analysis of pretest sensitization effects in experimental design. American Educational Research Journal, 19, 249-258.

I. T. Robertson & A. Kinder (1993). Personality and job competences: the criterion-related validity of some personality variables. Journal of Occupational and Organizational Psychology, 66, 225-244.

Niet iedere onderzoeker vindt het nodig om in de titel van zijn artikel te vermelden dat er een meta-analyse is gedaan

P. R. Sackettt & C. L. Z. DuBois (1991). Rater-ratee race effects on performance evaluation: challenging meta-analytic conclusions. Journal of Applied Psychology, 76, 873-877.

J. J. Hox & E. D. de Leeuw (1993). Nonrespons bij schriftelijke vragenlijsten, telefonische interviews en face-to-face interviews; een vergelijkende meta-analyse. Kennis en Methode, 43, 59-78.

Adrian Simpson (2016). The misdirection of public policy: comparing and combining standardized effect sizes.

Daniël Lakens, Joe Hilgard and Janneke Staaks (2016). On the reproducibility of meta-analyses: six practical recommendations. BMC PsychologyBMC series – open, inclusive and trusted20164:24 DOI: 10.1186/s40359-016-0126-3 open access

Robert Slavin (June 21, 2018). John Hattie is Wrong blog

juni, 2018 \ contact ben at at at benwilbrink.nl

http://www.benwilbrink.nl/literature/meta-analyse.htm