Thursday, April 21, 2016

Why Has The Number of NYT Headlines About Trump Fallen By a Factor of Two?

Over the past few weeks I’ve gotten the sense that, thank merciful heaven, the New York Times has stopped writing so many articles about Donald Trump. First I downloaded their data [1] to see whether this was all in my head. It wasn’t. Whether we look at headlines per week mentioning Trump (top graph) or articles per week mentioning Trump (bottom graph) we see that the NYT has been giving him much less coverage.
Why? Here are a couple ideas.

  1. Maybe the NYT, after repeatedly complaining about Trump’s free media coverage, decided to stop giving him so much free media coverage. Here’s their analysis of his free coverage on March 15 -- right around the time his coverage begins to decrease. (Nick Kristof, a NYT columnist, writes a similar piece on March 26.)
  2. Maybe this isn’t specific to Trump -- maybe the NYT’s getting bored of the primaries in general. So I looked at headlines about the other candidates. The Democratic candidates, if anything, have been getting more coverage in the last few weeks; top is Sanders, bottom is Clinton.
On the other hand, both Cruz and Kasich have also seen drops in coverage. Here are Cruz, Kasich, and Trump at the bottom for comparison.

So maybe the NYT is ignoring all the Republicans, not just Trump. (It’s worth noting that coverage of candidates may be correlated -- if you write about Trump fighting with Cruz, you’re covering both candidates.) Susan Athey, an economist who studies the internet, points out that a lot of the difference between liberal and conservative media sources is not that they cover the same topic differently, but that they cover different topics. (So if you read Fox News you’ll hear about Benghazi and if you read the NYT you’ll hear about climate change.) Given this, I’d be curious to know if you’d see a similar pattern in Fox News coverage of the Republicans.

3. Maybe the NYT is covering Trump less because he’s no longer as strong a candidate. Here are Trump’s odds of winning the Republican nomination as provided by PredictWise; Trump’s the red line. Note the fall in late March - early April.

(Or maybe causality goes in the other direction? Trump’s odds fall because he no longer gets as much free NYT coverage? Or maybe there’s a third variable -- Trump’s odds fall and he gets less NYT coverage because it becomes increasingly apparent that he isn’t worth talking about? Time series are fun.)

I don’t know which if any of these hypotheses is correct, so if you quote this post and say I’m making causal claims I will hunt you down. I’m just asking questions. Here’s a final one. No one reporter could account for the drop in Trump’s coverage -- it’s a difference of hundreds of articles. So something has to be coordinating the behavior of lots of reporters. How does that work? Is some editor at the NYT saying to the newsroom, “We SHALL NO LONGER write so many articles about the quasi-Nazi with little hands?” Or do the reporters look at what their peers are focusing on and choose to focus on that as well? Or are they all just responding to the same external factors?

Discriminating between these hypotheses is going to be hard using only data. But there are people who can help answer this question. So if you work at the NYT, feel free to explain what’s going on :)


[1] Data; my code.

Monday, March 28, 2016

Protecting Yourself Against Statistical Crimes

I have a dream that one day every child will take a class which will teach them to recognize statistical crimes. It would replace another high school math or science class, like calculus, trigonometry, geometry, or Newtonian physics, because these are totally useless for 90% of the population. (I was a physics major. I’m allowed to say these things.) Statistics is not like that. Send a child into the world unable to recognize statistical crimes and you are preparing them to be perpetually lied to -- by politicians pushing agendas, journalists facing tight deadlines, and scientists trying to get published.

This class would not be a math class. I don’t care if kids understand how to do a chi-2 test. I just want to make them very paranoid. It would be like that the scene in Harry Potter where the students are taught “constant vigilance” against the Dark Arts.

“[The teacher] gave a harsh laugh, and then clapped his gnarled hands together. ‘The sooner you know what you’re up against, the better. How are you supposed to defend yourself against something you’ve never seen?’ ”
And then instead of torturing a spider (seriously, who hired that guy? Don’t wizards have any teaching standards?) you could enumerate a bunch of statistical crimes. Which, to reinforce the fact that this class is necessary, I’m now going to do. I spent a month annotating every single article I read that discussed data for a popular audience (sample titles: “White Female Republicans are the Angriest Republicans”, “Study: More Useless Liberal Arts Majors Could Destroy ISIS”, “The Reproductive Rights Rollback of 2015”). In total I annotated 49 articles; you can see my annotations here and a note on my methodology here [1].

These are my overall impressions. They are not statistical; they’re a qualitative summary. Throughout I use “article” to refer to the general-interest publication and “study” to refer to the original scientific work it describes.

  1. Sites which specialized in statistical writing, like the NYT’s Upshot and FiveThirtyEight, wrote about data more reliably.
  2. Almost all the articles had something I could push back on. Most frequently, I had questions the original article didn’t answer or caveats it didn’t mention. This isn’t necessarily the journalist’s fault: most general-interest articles are shorter than the studies they describe, and so details get lost. But I also found a third of the articles were substantially misleading. (I’m not labeling those articles in the spreadsheet since I don’t want to be mean and the cutoff is somewhat arbitrary: maybe you could argue I’m an overly anal statistician and the actual fraction is a fourth or a fifth.) So if you want to know what a study says, reading a general-interest article about the study is not a reliable way to figure it out unless you really trust the journalist or outlet -- you have to at least glance through the study. General-interest articles often misdescribe studies, presenting correlational studies as causal, or presenting theoretical models as though they actually analyzed data. You don’t always have time to skim the original study, but I think you should before you repost it on Facebook or Twitter.
  3. Article titles are particularly likely to mislead. Outlets have incentives to use clickbait titles, the title is often not written by the author of the article, and it’s hard to summarize a complex topic in a dozen words. Please do not repost something after only reading the title.
  4. Be particularly suspicious of results which are politically charged or published in politically biased outlets (Jezebel, Breitbart), especially if the article substantiates the outlet’s worldview. (Also be suspicious of results which substantiate your worldview -- if you’re like me, you’re less inclined to question them.)
  5. Here are some questions to ask. If an article says, “A new study shows that X” your first question should be: how? Was it an experiment? A survey? A meta-analysis? A theoretical model? Sometimes this will be pretty obvious. If an article says, “Study shows that ⅔ of Americans prefer chocolate to vanilla”, the scientists probably ran a survey. But if an article says, “Study shows that increasing the minimum wage increases unemployment” -- it makes a huge difference whether the authors found a new natural experiment or did a meta-analysis of the past literature or are a bunch of undergrads who wrote up a theoretical model after passing Econ 101.
Once you understand how the study was conducted, push back on the study itself. If they claim to have “controlled for other factors” -- controlling for other factors is really hard. If they ran a survey -- was the population actually representative? Could non-response bias explain their results? In general: are the effects large enough to actually matter? Are their results statistically significant? Did they look at a hundred different things and only report the one which they liked? Are the numbers they are reporting the ones we care about, and are they properly contextualized? Try to think of other explanations for their data besides the one they favor. Be creative and obnoxious. You can find examples of how I think about articles in my annotations.

I close on a gentler note. The fact that you can make statistical arguments against an article does not mean that the author is incompetent or ill-intentioned or that the article is bad. All work has caveats -- certainly you can argue with all my blog posts -- and that’s fine as long as they’re clear. But some caveats are subtle and not clearly acknowledged (or deliberately hidden) which is why we need to teach our children to defend themselves. Avada Kedavra!

While I was working on this project, the New York Times and the Wall Street Journal both published op-eds arguing we should teach statistics. I dream of a world where statistical literacy is so common that statistical errors, like spelling errors, make it impossible to be taken seriously; where publications that use only anecdotes get demands for data. It would be a world where we paid attention to gun violence not because of mass shootings, but because of the far larger numbers of people who are shot and go unnoticed every day; where terrorists could no longer sow fear by killing a far smaller fraction of the population than die annually from heart disease; where we donated to charities that saved lives as opposed to making us feel good; where we conducted randomized controlled trials to test which government programs worked best. I truly believe that millions of people would lead better lives if everyone understood and applied basic statistical reasoning. That’s just not true of trigonometry. Let’s teach statistics instead.

[1] When I first wrote this piece, it was 6:10 AM and I just couldn’t take it anymore and I ranted about three articles which I thought were bad. After I calmed down I decided that was both mean and unpersuasive, so I did a more systematic annotation. My reading material skews towards the New York Times, so to get a more representative sample I annotated not just the articles I would read naturally: I also went back and read statistical articles in other widely read publications like Gawker, Buzzfeed, and Breitbart (I Googled “new study” + publication name).  (I was doing this quickly, so if you think I’ve been unfair or misunderstood an article, my apologies -- let me know and I’ll fix the spreadsheet.)

Tuesday, February 9, 2016

Monday, January 18, 2016

How Do Fraternity Men Talk About Women When There Are No Women Around?

Content note: this piece contains quotes which are graphically sexual, sexist, and racist, as well as quotes which endorse rape.

There are numerous allegations of sexism against fraternities; men who join are three times more likely to commit rape, even though they are no more likely to commit rape before joining. This makes you wonder just what these men are saying behind closed doors, a question that becomes particularly pressing when you consider that fraternity culture doesn’t magically dissolve upon graduation: it becomes incorporated into bastions of power that include Wall Street firms and billion-dollar companies as fraternity members disproportionately assume leadership roles in society.

It’s of course difficult for a woman to know how fraternity men talk when they’re alone. (My friends vetoed my plans to subpoena a fraternity’s emails or “fall asleep” at a fraternity party and listen to what happened.) But there is a public data source that claims to capture fraternity culture: the website Total Frat Move (TFM), which receives more than 1.2 million monthly visitors. The site covers all aspects of fraternity life, but its “Girls” section posts pictures of women for site users to comment on. I statistically analyzed the more than sixteen thousand comments on the Girls section. Most posts fell into one of three categories. 56% were “Instagram Babes of the Day”, containing Instagram photos of a single woman; 14% were collections of topless photos (careful where you open that); 17% were collections of photos from a particular sorority. To provide context for my analysis of the comments, I also contacted fraternity men who posted on the site and fraternity men I knew socially.

While TFM does not provide a perfectly representative sample of fraternity members (and non-fraternity members can comment on the site), it does provide a far larger sample than, for example, a story about a sexual assault at a single fraternity. It represents a substantial slice of fraternity culture. To give you a taste of what wins you approval on this site, here are some of the comments about women which got the most upvotes.

“There’s just something about those knee-high socks that screams ‘I do anal’.”
“10/10 would do… 0/10 would take home to mom.”
“When I think of all the money I could save by marrying that ass and not having to buy tits or a brazzers membership, it’s really just prudent investing.”
“I bet her butthole tastes like fruit loops.”
“She looks like her life will consist of manicures, attending her children’s equestrian events, and country club fundraiser dinner parties.”
“9/10 would follow from a safe distance.”
“Dear Santa, I can explain… My dad wasn’t around much.”
“I’ll say it, she’s getting fat. Those Hindenburg’s are great, but the tit-to-waist ratio is way off.”
“She really needs to stop fucking around and just get fully nude.”
“Someone tell Kelly that I’ve seen her butt, and I’ll pee there as soon as I can…”
“Look, it’s not like this site could piss feminists off anymore than it already does. Can you just get rid of the fucking sailboats and let us see some nipple?”
“Am I the only one who thinks an American flag somewhere on a women automatically makes her 10 times hotter?”
“Why do some of these girls think it’s ok to take pictures with zits on their boobs and asses? I can’t unsee that shit.”

The most frequently mentioned body part is “ass” (outnumbering “smile” 25 to 1, and “eyes” by 10 to 1), followed by “tits”; “face” comes in third, followed by “butt” and “boobs”. After reading these comments, I had two questions: 

1. Do fraternity men really talk this way in person? 
2. Why do women submit their pictures to this site?

As you can perhaps guess, I went into this project with a strong distrust of both TFM and fraternities in general. I came out, however, both enjoying my conversations with every fraternity man I spoke to and fascinated by the complex and contradictory world they inhabited. Some of the men I talked to were more fluent in feminist tropes than I was. When I asked one if men ever tried to “steal” girls from each other at parties, he gently chided me, “The very language of your question reveals the problematic views on women that lead to sexual assault."


Most fraternity men I spoke to agreed that while men were much cruder when they were talking only to other men, they were not as crude in person as TFM implied. One said that fraternity men were more willing to push boundaries online, on the fraternity email lists, even though it created a permanent record and some of them planned to run for political office.

I got a contrary perspective from one of the widest-read authorities on fraternity culture: Tucker Max, who achieved notoriety by writing New York Times bestsellers chronicling his drunken hookups.

Of course frat guys talk like that. I would say the overwhelming amount of men AND women talk to their friends in ways that would shock people if they were displayed to the world...” Tucker told me. “You have to remember that young guys are basically animals that can talk. I don't mean that literally of course, but the developmental psych evidence is very clear; most young men have vastly underdeveloped empathy, compassion, and other higher order thinking capabilities as compared to women their age. If you make it a rule to assume that a guy under about 25 probably thinks like a sociopath, then you're going to be more right than wrong.”

Okay then. Of course, Tucker’s perspective is also influenced by his own debaucherous experience. The truth, I suspect, lies somewhere in between: as one TFM commenter told me, “I do think the vibe and attitude is not far off from how fraternity guys actually talk, but it's much less exaggerated and embellished in real life.”

Why do they talk this way? Several fraternity members emphasized the strong role of social norms within fraternities. One explained that fraternities seemed to create feedback loops that produced more extreme behavior (both positive and negative) because people did not want to disappoint their friends. Another explained that members would deliberately say edgy things in order to win approval, and that younger members might try to win status by pushing boundaries.

One man told me that sexually explicit commentary was encouraged within fraternities because social standing was tied to sexual prowess; men who could not get women, or who hooked up with women deemed unattractive, were mocked. He described how members of his fraternity would congregate for hungover Sunday sessions where they detailed their sexual exploits from the night before. (These stories sometimes clashed with reality: a guy would claim he had had “amazing” sex with a girl and spent the night with her, when other members knew he had never gone home with her at all because they had seen her at another party entirely.) One TFM commenter described how these discussions served as bonding rituals, “one of the things guys can do that separates them from the girls. I think we feel like in a society that is currently trying to knock males, especially white males down a peg, this is a way we can all go to try to maintain our masculinity that in the 70s and 80s was so prevalent but now we're being told is wrong to have. I think we feel it's natural, biologically, to ogle girls and think things about them, and this is a place where we can let that out.”

Many men emphasized that comments on TFM were jokes or satire, although it was possible to go too far: for example, a commenter who made light of driving drunk was brutally criticized. The line often seemed arbitrary to me: sexism was obviously acceptable, as was racism (“She definitely has sex with multiple black guys at the same time” one commenter wrote about a woman; someone responded, “Once you go black, we don’t want you back”, and another, “Once you go black… You’re a single mom”). Rape comments were sometimes but not always okay: “No shame in how rapey I just felt…” one commenter wrote about a woman, to which someone responded, “Well there should be”; on the other hand, “Ramming is [sic] in her ass while she’s sleeping and bragging to your frat bros you get her to do anal,” received a dozen upvotes.

I am skeptical of the “just a joke” defense when it comes to problems as pervasive as rape, sexism, and racism. The difference between a rape joke and a dead baby joke is that while very few people kill babies, 5 - 10% of college men admit to committing rape. If a rape joke on TFM is read by 50 average college men and 50 average college women, there’s a 99% chance one of the men will have raped someone and an even higher chance one of the women will have been raped [1]. This makes the joke slightly less funny. More importantly, jokes are not necessarily harmless one-liners that leave the listener unchanged: studies provide some evidence that hearing sexist jokes does make men more tolerant of rape.


I found the women of TFM even more interesting than the men: why would they submit their photos to the site? A TFM site administrator initially told me that the women volunteered, and several commenters told me that this made them more comfortable making crude comments -- perhaps the online equivalent of “she was asking for it”.

But not all the women volunteer. One of the women featured on the site told me that she had never heard of TFM before the site posted her pictures; possibly someone else had sent them in without telling her. While she was not upset by this, I found multiple cases in which the featured woman had subsequently made her profile private. (I took a random sample of women’s pages and found that roughly one in five had been taken down.) So TFM appears, in at least some cases, to be making money off the pictures of college women without their consent by posting them for thousands of anonymous viewers to ogle and abuse. At least some of the women are taking measures to protect their privacy after their pictures are posted on the site.

This is legal: if a woman doesn’t like having her Instagram photos on TFM, she can request that they be removed (a TFM administrator said that the site honored these requests) although it’s unclear how many of the women are actually aware of this option, or that their pictures have been used at all. The pictures could also propagate further: for example, the line between appearing on TFM and appearing on pornographic sites can be blurry. Many comments on TFM encourage the women to become porn stars, and several women who appeared on TFM also had their images appear on pornographic sites.

But while some women on TFM may have had their pictures used without consent, it seems unlikely that all of them have. There are good reasons to pursue such attention: it could increase a woman’s online presence or aid careers, like acting or modeling, that privilege physical beauty; separate from such rational calculations, not all women dislike crude sexual attention. (This is not to say catcalling women is ever acceptable, because many women do hate and feel threatened by it.)

So let’s dig a little deeper. Individual women can apply to be featured on the site in two places: as “Instagram Babes of the Day” or “TFM Sweethearts”. There are important differences between the two. First, TFM users can comment on the photos of Instagram Babes, but not on the photos of Sweethearts. Second, Sweethearts have profiles, and Instagram Babes do not. I analyzed the profiles of 181 Sweethearts in an effort to understand them. Two thirds were single; they overwhelmingly appeared to be white; they were most disproportionately likely to come from Nevada, Alabama, Arizona, Kentucky, and South Carolina, and most frequently majored in Marketing, Psychology, Nursing, Communications, and Journalism. Most said they were planning to have careers: 30% mentioned further schooling in their post-graduation plans, and 40% mentioned work (only 9% mentioned marriage, children, or family). They were impressed by men with manners, which were more frequently mentioned than other popular answers like good conversation skills or a sense of humor (more than two dozen mentioned liking it when men opened doors).

So why were career-oriented women who valued manners above all volunteering as sweethearts for commenters who talked about peeing in women’s butts?

The women on the site were not eager to answer this question; though I contacted dozens, very few were willing to talk to me. One Sweetheart explained that Sweethearts were not a TFM invention; some real fraternities have had Sweethearts for over a century. A Sweetheart was “viewed like a member of the fraternity, privileged to wear their letters and learn some of their secrets...It’s a loftier position than dating the President or similar, and she's highly respected”. Perhaps it is this respect that makes TFM ban comments on the Sweethearts pages, even though it allows comments on the other women on the site: an administrator told me this kept the pages “clean”, which I guess is one way to put it.

I was struck by the many contradictions in fraternity culture: by the respectful lack of commentary on the Sweethearts, contrasting with the comments directed at the other women on the site; by the fraternity members who aspired to be gentlemen but spewed sexist vitriol; by the Sweethearts who admired chivalry, but nonetheless associated with such men; by the need to push boundaries to win social approval, but not to go too far; by the men who told me they loved their own fraternity, but thought fraternities in general were harmful; by the man who joked on TFM about girls’ daddy issues but addressed me via email as “ma’am”, and defended the site by saying:

We do not have prejudices, regardless of what the satirical comments may lead others to believe...We address social and political issues, we help people who need help, we offer prayers for those who need them and we love our girls and will defend them to our dying day.”

The fact that fraternity men are often civil and thoughtful in isolation makes articles that dehumanize them simplistic. But the individual civility of fraternity men is if anything a criticism, not a defense, of fraternities. It’s a little like saying, of the characters in Lord of the Flies, “But they’re such nice boys individually!” If that’s true, then take them off the island.

My journalist friends tell me that this piece does not adhere to journalistic conventions -- for example, it includes anonymous quotes. So please take it as creative non-fiction, not as journalism.


[1] I am aware that surveys of sexual assault often have low response rates. But even surveys with high response rates imply that the frequency of rape on college campuses is high, as Shengwu Li and I discuss here.