Effect sizes in primary studies and meta-analysis Methods of calculation and methods of interpretation

Document Type : Original Article

Author

Professor of Curriculum and Instruction, Faculty of Education (Cairo), Al-Azhar University

Abstract

 
 
 
 

Highlights

 

 

Keywords


 

 
American Psychological Association (2005). Publication manual of the American Psychological Association (5th ed.). American Psychological Association
American Psychological Association (2010). Publication manual of the American Psychological Association (6th ed.). American Psychological Association
American Psychological Association (2020). Publication manual of the American Psychological Association (7th ed.)https://doi.org./10.1037/0000165-000
Baugh, F. (2002). Correcting effect sizes for Score reliability: A reminder that measurement and substantive issues are linked inextricably. Educational and Psychological Measurement,62(2),254-263https://doi.org/10.1177/0013164402062002004
Borenstein, M., Hedges, L.V. (2019). Effect size for meta- analysis. In H. Cooper, L. V. Hedges, J. C. Valentine (Eds.), The handbook of research synthesis and meta- analysis (3rd ed., pp. 208-243). Russell Sage Foundation.
Borenstein, M., Hedges, L.V., Higgins, J.P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley 
Card, N. A. (2012). Applied mate-analysis for social science research. The Guilford Press.
Cohen, J. (1988). Statistical power analysis for behavioral sciences. Erlbaum.
Cohen, J. (1994). The earth is round (p American Psychologist, 49, 997-1003. https://psycnet.apa.org/doi/10.1037/0003-066X.49.12.997
Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta- analysis. Routledge.
Durlak, J. A. (2009). How to select, calculate, and interpret effect sizes. Journal of Pediatric Psychology, 34(9). 917- 928.
Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and interpretation of results. Cambridge University Press
Fan, X. (2001) Statistical significance and effect size in education research: Two sides of a Coin. The Journal of Educational Research, 94(5), 275-282. https: doi.org/ 10.1080/00220670109598763
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). Sage.
Glass, G. V., & Hopkin, K. D. (1996). Statistical methods in education and psychology (3rd ed.). Allyn and Bacon.
Green, S. B., & Salkind, N. J. (2014). Using SPSS for Windows and Macintosh:  Analyzing and understanding data (5th ed.). Pearson.
Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analysis relating to achievement. Routledge.
Hill, C. G., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development perspectives, 2(3), pp. 172-177.
        https://doi.org/10.1111/j.1750-8606.2008.00061.x 
Hogarty, K., Y. & Kromrey, J., D. (2000, April). We have been reporting some effect sizes: Can you guess what they mean? [Paper presentation} American Research Association, Annual Meeting , Seattle, WA.
Huck, W. S. (2012). Reading statistics and research. Pearson.
Kirk, R. E. (1995). Experimental design: Procedures for behavioral sciences (3rd ed.). Brooks/cole.
Kirk, R. E. (1996). Practical significance. A concept whose time has come. Education and Psychological Measurement, 56,746-759.
 https://doi.org/10.1177%2F0013164496056005002
Knapp, T. R. (1998). Comments on the statistical significance testing articles. Research in the Schools, 5(2), 39-41
Kotrlik, J. W. & Williams, H. (2003). The incorporation of effect size in information technology, learning, and performance research. Information Technology, Learning, and Performance Journal, 21(1), 1-7.
Krantz, D. H. (1999). The null hypothesis testing controversy in psychology. Journal of the American Statistical Association, 94, 1372-1381.
Kraemer, H. C., & Andrews, G. (1982). A nonparametric technique for meta-analysis effect size calculation. Psychological Bulletin, 91(2), 404–412. https://doi.org/10.1037/0033-2909.91.2.404
Levin, J. R. (1998). What if there were no more bickering about statistical significance test? Research in the Schools, 5,43-53.
Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka- Fry, k., Cole, M. W., Roberto, M., Anthony, K. S., & Busick, M. d., (2012). Translating the statistical representation of effects of education interventions into more readily interpretable forms. (NCSER 2013- 3000) National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. Retrived June 3, 2021 from http://ies.ed.gov/ncser/.
Lomax, R. G., & Vaughn, D. L. (2012). Statistical concepts: A second Course (4th ed.). Routledge.
Marzano, R. S., Pickering, D. G., & Pollock, J. E. (2001). Classroom instruction that works. Association for Supervision and Curriculum Development.
Mayer, J. L., Well, A. D. (2003). Research design and statistical analysis (2ed ed.). Lawrence Erlbaum Associate.
McGraw, K. N., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin,111(2), 361- 365. https://doi.org/10.1037/0033-2909.111.2.361
McLean, J. E.; O'Neal, M. R. & Barnett, J. J. (2000, November). Are all effect sizes created equal? [Paper Presentation]. Mid-South Educational Research Association Annual Meeting, Bowling Green, K Y (ERIC Document Reproduction Service No. ED 448188).
Meline, T. & Wang, B. (2004). Effect-size reporting practices in AJSLP and other ASHA Journals, 1999-2003. American Journal of Speech-Language Pathology, 13, 202-207. https: doi. org/ 10.1044/1058-0360(2004/021)
Morgan, G.A., Leech, N. L., Gloeckner, G. W., & Barrett, K. C. (2013). IBM SPSS for introductory statistics: Use and interpretation (5th ed.). Roulege.
Nickerson, R. S. (2000). Null hypothesis significance testing. A review of an old and continuing controversy. Psychological Methods, 5, 241-301. https: doi. org/ 10.1037/1082-989x.5.2.241
Nix, T. W & Barnett, J. J. (1998). The data analysis dilemma: Ban or abandon. A review of null hypothesis significance testing. Research in the School, 5, 3-14.
Nunnally, J. & Berntein, I (1994). Psychometric theory (3rd ed.).  Mc Grew-Hill.
O'Grady, K. E. (1982). Measures of explained variance: Cautions and limitations. Psychological Bulletin, 92, 766-777.
Olejnik, S. & Alegina, J. (2000). Measurement of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286. Retrieved September. 4, 2007 from http://www.personal.psu.edu/jx614/m554/articles/olejnic&gina2000.pdf
Onwuegbuzie, A, J. & Daniel, L. G. (2000). Reliability generalization: The importance of considering sample specificity, confidence intervals, and subgroup differences [Paper presentation]. Mid-south Educational Research Association Annual Meeting, Bowling Green, KY (ERIC Document Reproduction Service No. ED 448.204).
Rosenthal, R., & Rubin, D. B. (1982). A simple general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 73(2), pp. 166-169.https://doi.org/10.1037/0022-0663.74.2.166
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reform. Educational Researcher, 25(2), 26-30. https://doi.org/10.2307/1176337
Thompson, B. (1998). Five methodological errors in educational research: The pantheon of statistical significance and other faux pas. Invited address presented at the annual meeting of the American Research Association, San Diego: (ERIC Document Reproduction Service No. ED 419-023). 
Thompson, B. (1999). If statistical significance tests are broken misused, what practices should supplement or replace them? Theory & Psychology, 9(2), 165-181. https://doi.org/10.1177%2F095935439992006
Thompson, B. (2002). "Statistical", "Practical", and "Clinical": How many kinds of significance do counselors need to consider? Journal of counseling & Development,80,64-71. https://doi.org/10.1002/j.1556678.2002.tb00167.x
Thompson, B; Diamond, K; Mc William, R; Snyder, P & Snyder, S. (2005). Evaluating the quality of evidence from correlational research for evidence –based practice. Exceptional Children, 71(2), 181-194. https://doi.org/10.1177%2F001440290507100204
Trusty, J. ; Thompson, B.; & Petrocelli, V. (2004). Practical guide for reporting effect size in quantitative research In the Journal of counseling & Development. Journal of Counseling & Development, 82, 107-110. https://psycnet.apa.org/doi/10.1002/j.1556-6678.2004.tb00291.x
Valentine, J. C., Aloe, A. M., Wilson, S. J. (2019). Interpreting effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta- analysis (3rd ed., pp. 433-452). Russell Sage Foundations