Monday, March 28, 2016

Protecting Yourself Against Statistical Crimes

I have a dream that one day every child will take a class which will teach them to recognize statistical crimes. It would replace another high school math or science class, like calculus, trigonometry, geometry, or Newtonian physics, because these are totally useless for 90% of the population. (I was a physics major. I’m allowed to say these things.) Statistics is not like that. Send a child into the world unable to recognize statistical crimes and you are preparing them to be perpetually lied to -- by politicians pushing agendas, journalists facing tight deadlines, and scientists trying to get published.


This class would not be a math class. I don’t care if kids understand how to do a chi-2 test. I just want to make them very paranoid. It would be like that the scene in Harry Potter where the students are taught “constant vigilance” against the Dark Arts.


“[The teacher] gave a harsh laugh, and then clapped his gnarled hands together. ‘The sooner you know what you’re up against, the better. How are you supposed to defend yourself against something you’ve never seen?’ ”
And then instead of torturing a spider (seriously, who hired that guy? Don’t wizards have any teaching standards?) you could enumerate a bunch of statistical crimes. Which, to reinforce the fact that this class is necessary, I’m now going to do. I spent a month annotating every single article I read that discussed data for a popular audience (sample titles: “White Female Republicans are the Angriest Republicans”, “Study: More Useless Liberal Arts Majors Could Destroy ISIS”, “The Reproductive Rights Rollback of 2015”). In total I annotated 49 articles; you can see my annotations here and a note on my methodology here [1].


These are my overall impressions. They are not statistical; they’re a qualitative summary. Throughout I use “article” to refer to the general-interest publication and “study” to refer to the original scientific work it describes.


  1. Sites which specialized in statistical writing, like the NYT’s Upshot and FiveThirtyEight, wrote about data more reliably.
  2. Almost all the articles had something I could push back on. Most frequently, I had questions the original article didn’t answer or caveats it didn’t mention. This isn’t necessarily the journalist’s fault: most general-interest articles are shorter than the studies they describe, and so details get lost. But I also found a third of the articles were substantially misleading. (I’m not labeling those articles in the spreadsheet since I don’t want to be mean and the cutoff is somewhat arbitrary: maybe you could argue I’m an overly anal statistician and the actual fraction is a fourth or a fifth.) So if you want to know what a study says, reading a general-interest article about the study is not a reliable way to figure it out unless you really trust the journalist or outlet -- you have to at least glance through the study. General-interest articles often misdescribe studies, presenting correlational studies as causal, or presenting theoretical models as though they actually analyzed data. You don’t always have time to skim the original study, but I think you should before you repost it on Facebook or Twitter.
  3. Article titles are particularly likely to mislead. Outlets have incentives to use clickbait titles, the title is often not written by the author of the article, and it’s hard to summarize a complex topic in a dozen words. Please do not repost something after only reading the title.
  4. Be particularly suspicious of results which are politically charged or published in politically biased outlets (Jezebel, Breitbart), especially if the article substantiates the outlet’s worldview. (Also be suspicious of results which substantiate your worldview -- if you’re like me, you’re less inclined to question them.)
  5. Here are some questions to ask. If an article says, “A new study shows that X” your first question should be: how? Was it an experiment? A survey? A meta-analysis? A theoretical model? Sometimes this will be pretty obvious. If an article says, “Study shows that ⅔ of Americans prefer chocolate to vanilla”, the scientists probably ran a survey. But if an article says, “Study shows that increasing the minimum wage increases unemployment” -- it makes a huge difference whether the authors found a new natural experiment or did a meta-analysis of the past literature or are a bunch of undergrads who wrote up a theoretical model after passing Econ 101.
Once you understand how the study was conducted, push back on the study itself. If they claim to have “controlled for other factors” -- controlling for other factors is really hard. If they ran a survey -- was the population actually representative? Could non-response bias explain their results? In general: are the effects large enough to actually matter? Are their results statistically significant? Did they look at a hundred different things and only report the one which they liked? Are the numbers they are reporting the ones we care about, and are they properly contextualized? Try to think of other explanations for their data besides the one they favor. Be creative and obnoxious. You can find examples of how I think about articles in my annotations.


I close on a gentler note. The fact that you can make statistical arguments against an article does not mean that the author is incompetent or ill-intentioned or that the article is bad. All work has caveats -- certainly you can argue with all my blog posts -- and that’s fine as long as they’re clear. But some caveats are subtle and not clearly acknowledged (or deliberately hidden) which is why we need to teach our children to defend themselves. Avada Kedavra!


While I was working on this project, the New York Times and the Wall Street Journal both published op-eds arguing we should teach statistics. I dream of a world where statistical literacy is so common that statistical errors, like spelling errors, make it impossible to be taken seriously; where publications that use only anecdotes get demands for data. It would be a world where we paid attention to gun violence not because of mass shootings, but because of the far larger numbers of people who are shot and go unnoticed every day; where terrorists could no longer sow fear by killing a far smaller fraction of the population than die annually from heart disease; where we donated to charities that saved lives as opposed to making us feel good; where we conducted randomized controlled trials to test which government programs worked best. I truly believe that millions of people would lead better lives if everyone understood and applied basic statistical reasoning. That’s just not true of trigonometry. Let’s teach statistics instead.


Notes:
[1] When I first wrote this piece, it was 6:10 AM and I just couldn’t take it anymore and I ranted about three articles which I thought were bad. After I calmed down I decided that was both mean and unpersuasive, so I did a more systematic annotation. My reading material skews towards the New York Times, so to get a more representative sample I annotated not just the articles I would read naturally: I also went back and read statistical articles in other widely read publications like Gawker, Buzzfeed, and Breitbart (I Googled “new study” + publication name).  (I was doing this quickly, so if you think I’ve been unfair or misunderstood an article, my apologies -- let me know and I’ll fix the spreadsheet.)

28 comments:

  1. I love this article, but "don't be terrorized by something that kills fewer people than heart disease" if a pretty high standard!

    I might have gone with "kills fewer people than traffic accidents".

    ReplyDelete
  2. Oooh, I just found your blog. It is awesome. I have long been advocating for the same thing: replace trigonometry and calculus with statistics for most high-school students. (I majored in computer science and electrical engineering; I can say things like that.) Bad news, though. It looks like education isn't solving the problems that you and I hope it will. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992

    ReplyDelete
  3. I actually imposed this on my kids. They must take prob/stats to leave HS. Like Shakespeare, you can't consider yourself well educated without an understanding that your brain does heuristics, not probability.

    ReplyDelete
  4. Given how hard it is to revise the high school mathematics curriculum, here's a stopgap: have every student read Darrell Huff's "How to Lie with Statistics" twice -- once as part of Algebra I, once as part of studying U. S. government.

    ReplyDelete
  5. If you take a closer look at this post https://topgoodessays.net/essay-writing-rules/ you will find good collection of different essay writing rules

    ReplyDelete
  6. To setup and install your 123 HP printer go to 123.hp.com/setup for mac . For installation of drivers, seek help immediately from the website 123.hp.com/setup . We’ll help you download the right HP printer software and drivers.To setup and install your 123 HP printer go to 123.hp.com/setup for mac .

    ReplyDelete
  7. สล็อตxoFantastic postings, Cheers!
    help writing grad school essay inexpensive resume writing services
    We are a gaggle of volunteers and starting
    a brand new scheme in our community.

    ReplyDelete
  8. A printer driver is software that your computer uses to talk to a physical printer, which may be connected to your computer or another computer on your network. You can download printer drivers and software from our website 123.hp.com/setup . You can visit this site to install printer setup. you can take help from our website 123.hp.com/setup .To setup and install your 123 HP printer go to 123.hp.com/setup for mac . For installation of drivers, seek help immediately from the website 123.hp.com/setup .เว็บสล็อตแตกง่าย

    ReplyDelete
  9. สล็อตทุนน้อย
    Thanks for a very interesting blog. What else may I get that kind of info written in such a perfect approach?I’ve a undertaking that I am simply now operating on, and I have been at the look out for such info.

    ReplyDelete
  10. Harry Potter where the students are taught “constant vigilance” against the Dark Arts.
    สล็อตเว็บตรง

    ReplyDelete
  11. I guess I am not the only one having all the enjoyment here keep up the good work ทดลองเล่นสล็อต xo

    ReplyDelete
  12. When you use a genuine service, you will be able to provide instructions, share materials and choose the formatting style.สล็อตออนไลน์

    ReplyDelete
  13. This particular papers fabulous, and My spouse and i enjoy each of the perform that you have placed into this. I’m sure that you will be making a really useful place. I has been additionally pleased. Good perform!สล็อตวอเลท

    ReplyDelete
  14. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work...สล็อตแตกง่าย

    ReplyDelete
  15. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much.บา คา ร่า วอ เลท

    ReplyDelete
  16. It was wondering if I could use this write-up on my other website, I will link it back to your website though.Great Thanks.บา คา ร่า วอ เลท

    ReplyDelete
  17. Thanks For sharing this Superb article.I use this Article to show my assignment in college.it is useful For me Great Work.บาคาร่าวอเลท

    ReplyDelete
  18. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. เว็บสล็อต

    ReplyDelete
  19. You have done a terrific career on this text. It’s really specific and extremely qualitative. You've got even managed to make it readable and easy to study. You have some serious composing talent. Thank you a great deal.เว็บสล็อต

    ReplyDelete
  20. มือใหม่เริ่มต้นแทงบอลออนไลน์ผ่านหน้าเว็บไซต์ ต้องทำอย่างไรมาดูกัน
    ถ้าหากท่านนั้นเป็นมือใหม่ยังมีประสบการณ์น้อยในการ เเทงบอลออนไลน์ วันนี้เราเลยอยากจะมาแนะนำวิธีการแทง แทงบอลออนไลน์ ให้มือใหม่ ที่กำลังหัดแทงบอล สำหรับมือใหม่เองเราแนะนำให้ท่านนั้นได้เลือกเล่นบอลเดี่ยวก่อน ทีเด็ดบอลเดี่ยว เพราะเป็นการแทงบอลที่ง่ายที่สุด ง่ายต่อการวิเคราะห์รูปแบบเกมมากกว่าการเล่นบอลเต็งที่ทุกคู่ที่เลือกจะต้องชนะทั้งหมด ถ้าแพ้แม้แต่ผู้เดียวก็เสียเดิมพันเลยทันที ต่างจากบอลเดี่ยวที่ความเสี่ยงมีน้อยกว่านั่นเอง ปัจจัยในการแทงบอลออนไลน์ และสิ่งที่เราจะเสนอต่อไปนี้เป็นสิ่งที่นักเดิมพันต้องรู้จะมีอะไรบ้างนั้นไปชมเลย

    1. ให้ท่านนั้นเริ่มต้นแทงบอลด้วยเงินน้อยๆก่อนไม่ต้องรีบหากเล่นได้ค่อยเพิ่ม
    2. การแทงบอลเดี่ยว เว็บเเทงบอล บอลออนไลน์ ใจต้องถึงอาจจะต้องใช้เวลาดูเชิงสักหน่อย หากมั่นใจว่าชนะให้แทงทันที
    3. สำหรับการแทงบอลชุดให้ดูยาวๆต้องทำการวิเคราะห์ให้มากกว่าบอลเดี่ยว
    4. นักเดิมพันมือใหม่อย่าเลือกบอลต่อหรือรอง แค่ฟุตบอลสามารถพลิกไปพลิกมาได้ตามสถานการณ์ จึงต้องเลือกที่ปัจจัยต่างๆมากกว่าความชื่นชอบ

    ReplyDelete
  21. I am very thankfull to you for sharing this fantastic article, Useful to us

    ReplyDelete
  22. You’ve written nice post, I am gonna bookmark this page, thanks for info.

    ReplyDelete
  23. Thanks a lot for this great stuff here. I am very much thankful for this site.

    ReplyDelete
  24. I will bookmark this blog and check here again. Keep doing your blog post

    ReplyDelete
  25. Really very informative and creative.

    ReplyDelete