{"id":332,"date":"2014-12-05T21:58:55","date_gmt":"2014-12-05T21:58:55","guid":{"rendered":"http:\/\/labmath.org\/?p=332"},"modified":"2015-01-05T18:58:45","modified_gmt":"2015-01-05T18:58:45","slug":"effect-and-substance-not-p","status":"publish","type":"post","link":"https:\/\/labmath.org\/?p=332","title":{"rendered":"Effect and substance, not p"},"content":{"rendered":"

There is an excellent resource by Paul Ellis at:<\/p>\n

http:\/\/effectsizefaq.com<\/span><\/a><\/span><\/p>\n

where I just clicked on<\/p>\n

http:\/\/effectsizefaq.com\/2010\/05\/30\/how-do-researchers-confuse-statistical-with-substantive-significance\/<\/span><\/a><\/span><\/p>\n

This page talks about the problem with p-level being the be-all and end-all of way too many scientific studies. \u00a0You might be aware that there is discussion about this in the scientific literature. \u00a0(I think statisticians deserve prizes for still trying to get the rest of us to pay attention.) \u00a0The problem is that the ONLY thing p-level tells you is the probability that rejecting the null hypothesis is the wrong thing to do. \u00a0That’s why small is good: \u00a0you want the probability of a false positive to be as small as possible. \u00a0And that is all it does. \u00a0Nothing else. \u00a0\u00a0Take a moment, if you will, to consider all the other ways you could be wrong: false negative, wrong question, wrong control, etc. \u00a0Most importantly, it does not tell you if your result is significant in any\u00a0scientifically meaningful<\/em>\u00a0way.<\/p>\n

This is what Paul Ellis is talking about when he uses the vocabulary “substantive significance” and the link above goes right to the heart of the matter: \u00a0researchers are confusing statistical significance with substantive significance, and journals are letting them get away with it. \u00a0In my ideal world every result comes with, at least, the standard deviation, and the four things noted below: alpha, beta, sample size, and effect size, that last being accompanied by some sentences describing why that effect size was chosen.<\/p>\n

You may be lucky enough to have as big a sample size as you want, but you still must use your brain to decide what matters, to design an experiment that actually answers your question, and to do appropriate controls so that you can make interesting comparisons. \u00a0A large sample size may allow you to find very small differences, but if the differences are that small, do they matter? \u00a0They very well might, but you must think that through, no statistic can do that for you.<\/p>\n

There is, however, a numerical representation of that “minimum important difference” called the effect size. \u00a0The best way to plan an experiment is to decide in advance<\/em> on the effect size that you, with your brain, think is important enough to be worth detecting, and<\/em> decide in advance how low you want the probability of a false positive to be (alpha, or, p-level), and<\/em> decide how low you want the probability of a false negative to be (beta), and<\/em> decide on the most information-packed way to measure, which includes deciding on the appropriate statistical test to use, THEN<\/em>\u00a0calculate the sample size and stick to it.<\/p>\n

To learn more about effect size, go to effectsizefaq.com and read all about it.<\/p>\n","protected":false},"excerpt":{"rendered":"

There is an excellent resource by Paul Ellis at: http:\/\/effectsizefaq.com where I just clicked on http:\/\/effectsizefaq.com\/2010\/05\/30\/how-do-researchers-confuse-statistical-with-substantive-significance\/ This page talks about the problem with p-level being the be-all and end-all of way too many scientific studies. \u00a0You might be aware that there is discussion about this in the scientific literature. \u00a0(I think statisticians deserve prizes for […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[10,4],"tags":[],"_links":{"self":[{"href":"https:\/\/labmath.org\/index.php?rest_route=\/wp\/v2\/posts\/332"}],"collection":[{"href":"https:\/\/labmath.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/labmath.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/labmath.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/labmath.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=332"}],"version-history":[{"count":7,"href":"https:\/\/labmath.org\/index.php?rest_route=\/wp\/v2\/posts\/332\/revisions"}],"predecessor-version":[{"id":401,"href":"https:\/\/labmath.org\/index.php?rest_route=\/wp\/v2\/posts\/332\/revisions\/401"}],"wp:attachment":[{"href":"https:\/\/labmath.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/labmath.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=332"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/labmath.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}