Response To The Comments On The Sexual Assault Model

Content warning: this post discusses sexual assault in some detail.  

A few weeks ago I created a statistical model which implied that someone who has been accused of sexual assault by multiple people is much less likely to be innocent than someone who has been accused of assault once. This ended up on the front page of Hacker News, which Wikipedia describes as a “social news website that caters to programmers and entrepreneurs”, and got as many views in two hours as this blog usually gets in two months. (Granted, that’s not really that much traffic, but we math nerds take what social interaction we can get.)

While many of the comments were critical, almost all of them were civil, thoughtful, and free from the kind of trolling that women worry about when discussing controversial gender issues. All this was tremendously welcome and we need more of it on the internet, so I thought I’d take the time to respond to the most interesting criticisms.

To briefly recap (skip to the next paragraph if you remember all this) the model computes the probability that someone who has been accused of sexual assault by k people has never actually committed assault. The model assumes that there are serial predators, who have a high probability of assaulting people, and non-serial predators, who have a low probability of assaulting people. If someone has been assaulted, they accuse the other person of assault with probability pag; if they have not, they accuse with lower probability pai. You can play with the model here (move the sliders).

As I note in the post, the model is a simplification of the real world, as all models are. That said, it is possible to build a model which is too simple. For example, after Thomas Duncan’s nurse became infected with Ebola, it would’ve been irresponsible to say, “Assuming the current rate of weekly doubling of the Ebola epidemic in the United States continues, we expect every US citizen to be infected 28 weeks from now.” So I will discuss three assumptions the model makes and you can judge their validity and usefulness for yourself. Overall my view is that, while the model simplifies out of necessity, its basic conclusions probably hold even under more complicated assumptions. But I also see why there is room for disagreement because of the uncertainty about basic facts about assault, and trying to build a model has definitely reinforced my belief that we need better data. There also are clear tradeoffs between reducing the number of assaults and not punishing innocent people; your background and life experiences differ from mine, so you may worry more about victims of assault and less about the wrongfully accused, or vice versa.

1. Objection 1: the model assumes that each accuser comes forward independently, but in fact each accuser may be influenced by previous accusers. This was the main concern people on Hacker News raised [1], and I think it is definitely important in high-profile cases like Bill Cosby’s, where accusers may well know about and be influenced by previous accusations. But I do not think cases like Cosby’s are the most representative or important to focus on, because the vast majority of accused and actual rapists are not famous. In these more representative cases, I do not think we should worry too much about accusers influencing each other for several reasons.

1. In my post I discuss a tool which allows accusers to file accusations with a third party who will keep the accusations confidential unless multiple accusations are levied at the same person. If this tool were used, it seems quite likely that accusations would be essentially independent because they would be kept secret. (I guess it is possible that accusers might collude and all submit accusations together, but this John Tucker Must Die scenario stretches the bounds of plausibility.)
2. Even if this tool is not used, accusers have to know about previous accusations to be influenced. This is by no means certain: how many people do you know who are currently being accused of assault? How often do you compare notes with someone who has had a sexual encounter with the same person?
3. Even if accusers are aware of previous accusations, either i) they were actually assaulted, in which case it’s a good thing if they’re more likely to come forward or ii) they were not actually assaulted, in which case it’s a little unclear why they’re now more likely to come forward. The thought process might go something like, “Well, I’m not sure exactly what happened...I thought it was consensual at the time, but if they attacked other people, I guess they attacked me too.” This seems a bit unlikely to me, but feel free to suggest more plausible reasoning.
4. Even if you disagree with all of the above speculations about probabilities, I built you another model that allows you to adjust them (you might have to zoom out to see all the sliders) and account for accusers who influence each other. I describe the mathematical details here [2]. Even if accusers can influence each other quite strongly, this model yields the same conclusion as before: someone accused of sexual assault multiple times is much less likely to be innocent. For example, I still see the same effect if I make an accuser ten times more likely to accuse an innocent person of assault if they learn they’ve been previously accused.

2. Objection 2: the model assumes that everyone is equally likely to be accused of assault. Several people pointed out that there are factors that influence how likely a person is to be accused of assault besides whether they actually assaulted someone. To take one example, someone who has sex with their partner of ten years may be less likely to be accused than someone who has sex with a person they met for the first time at a frat party, even if both encounters are consensual. More generally, we can imagine two groups of people, A and B, who have different sorts of sexual encounters, and thus different probabilities of being innocent:

I think this only alters our basic conclusion, though, if we think of a group who is a) reasonably large and b) a lot more likely to be accused of assault even when they are innocent. (Merely identifying group differences is not enough: you would have to identify a group for whom we can set the parameters of the original model such that the innocence curve no longer plunges sharply.) None of the people I spoke to about this advanced plausible candidates for such a group, but let me know if you have thoughts.

3. Objection 3: the model computes the probability that the accused is guilty of at least one assault, but that is not the probability we are interested in. In no court would a judge say, “Well, I don’t know which of these six alleged assaults the accused is guilty of, but there’s a very high probability he’s guilty of at least one, so I’m throwing him in jail.”

Still, when we’re not throwing someone in jail, we often do care about the probability the model computes. Maybe I’m in charge of student housing, someone’s been accused of assault by two of their dorm mates, and I’m trying to decide whether to move them to another house (or out of housing altogether). We frequently apply statistical reasoning that isn’t allowed in a courtroom.

Also, it turns out we can use a similar model to compute the probability that someone is guilty in a particular case (rather than in any case) given that they have k previous accusations of sexual assault, and it produces similar curves [3].

If you’ve made it this far, especially if you disagreed with me at the outset, thanks so much for reading, because I wrote this mainly for you. I imagine we still have disagreements and would welcome your comments below or via email.

Notes:

[1] Some made comparisons to witchcraft trials, where a “mob mentality” takes hold and many people accuse one person of witchcraft. I will say, first, that I think witch trials are so different as to make this comparison pretty useless. A large number of people are truly assaulted; no one is a true victim of witchcraft. I do not believe there are mobs of potential assault accusers who, having all had sex with the same person, are suddenly all possessed by groupthink and decide that their sexual encounters constituted assault.

[2] The model assumes that if someone’s been previously accused of assault, a potential accuser knows about it with probability pk; if they know about the previous accusations, they accuse with probability pag1  if the person assaulted them and pai1  if the person did not; if the potential accuser does not know about the previous accusations, they accuse with probability pag0  if the person assaulted them and pai0  if the person did not. As pk  goes to zero -- no accuser knows about the others -- we recover the old model, where every accuser acts independently. On the other hand, if we increase pk  and pai1  to 1 -- accusers are certain to know about the first accusation, and certain to accuse if they know about it -- every accusation after the first becomes uninformative, as we would expect. We see the same effect as in the old model even for quite high settings of pk  and pai1 -- which can’t be all that high by simple virtue of the fact that very few people are accused of assault -- someone who is accused multiple times is much less likely to be innocent.

[3] The lawyers I discussed this with are still uncomfortable with this kind of reasoning, where past accusations can influence present guilt. My general sense is that the legal system disallows a lot of reasoning that statisticians deem kosher -- which is probably, given the weight we place on not convicting the innocent, a good thing.

Ferguson FAQ

Recently I published an analysis of the Ferguson conflict that showed, using Twitter data, that there was a “red group” and a “blue group” who rarely talked to each other, thought very different things, came from very different backgrounds, and often were uncivil even when they did talk. Thanks to everyone who wrote to me about the analysis! Here are answers to the most common questions I’ve received.

What data did you use?

215,000 tweets containing the Ferguson hashtag collected between November 17th and 19th (prior to the announcement of the verdict).

What tools did you use to collect the data?

Python -- specifically, the tweepy library and a program I wrote which you can find here (described at more length here).

What tools did you use to analyze the data and make the visualization?

Python for analysis; Gephi for visualization. See Gilad Lotan’s excellent tutorial on how to use Gephi to analyze Twitter data.

How did you divide Tweeters into red and blue groups?

I used Gephi's community detection algorithm (on the adjacency matrix for the most frequent tweeters, where Mij was 1 if tweeter i had mentioned tweeter j in a tweet), sometimes known as the Louvain method. Essentially, this divides Tweeters into groups that mention each other frequently.

Regarding whether this grouping is valid: as I note in the piece, I am mindful of the fact that there are many ways to group data, and I think this is worth exploring further. One problem we always face is how many groups there are (see here and here). You can always sort of make it look like people hate each other by clustering the data into groups even if there isn’t necessarily any separation between the groups -- this is something to be wary of when looking at analyses like this one.

But I think several pieces of evidence (in addition to Gephi's striking visual) point to the validity of the red / blue division. The fact that the two groups are associated with the tweeters’ self-descriptions (like race and political affiliation) is revealing; the fact that the two groups are associated with tweeting different things is also revealing (and by no means something I expected to see -- for example, if you divide Twitter datasets by gender, you will frequently find that men and women tweet essentially similar things). This evidence is powerful because it is external -- it was not used to come up with the grouping, but it supports it.

In general, we often bring in such external evidence to argue that a grouping is valid. For example, in a biological analysis we might cluster genes into groups that show similar expression patterns (group A highly expressed in the liver and not in the lungs; group B highly expressed in the lungs and not in the liver). We would be more sure that the groups we had found were “real” if there was external evidence like a transcription factor that was known to turn on all the genes in group A, or a biological function that was common to all the genes in group A.

You said the blue cluster is much larger than the red cluster. What happens if you break down the blue cluster further?

I don’t know! Someone should figure this out.

Can I see your data or code?

Yes. I cannot make the data publicly available because of Twitter’s terms of service, but if you are a researcher with a project, shoot me an email. In addition to the two days of data used in this analysis, I also have several million tweets both from several months ago, when Ferguson initially made the news, and from after the verdict was announced.

As always, if you work at Twitter and have any objection to any of this, please email me -- I am acting in good faith and more than happy to comply with your requests.

Why I'm Not Flirting with Lesbians In Central Park

I have flown across the ocean to become a Very Serious Oxford Student who can read two books at once, tassel swinging:

I have been told that Oxford will actually expel me for wearing that hat. Today is my one-month anniversary of arriving in England and I’ve decided that I should write a piece or two about what I’ve learned here, in part to confirm to my family that I’m still alive. If you just want statistics, please skip this post and I promise the next one will have lots and lots of p-values.

I have never before gone weeks half-wondering if I’m dreaming. At first I thought it was just jetlag or social exhaustion, but I’ve come to realize that it’s something longer-term: I never fully understood that filling out those scholarship forms meant I would, in fact, fly across a real ocean and attend a real university. So when I sit at formal hall eating smoked duck and drinking white wine in a building about 40 times older than I am, part of me believes that I am, in this well-named “city of dreaming spires”, still asleep. That, of course, is a good dream.

Lesson 1: we forget how many ways there are to live a life. Keeping sane, I think, requires becoming willfully blind to possible lives. Eg, at the moment I am a long-haired computer science researcher in a committed straight relationship; but if I wanted to, by tomorrow I could be a spiky-haired harmonica-player flirting with lesbians in Central Park. In theory. But, of course, I don’t really consider that possibility, because it’s terrifying and paralyzing to constantly consider dumping your boyfriend, switching careers, and crossing an ocean; I get pretty overloaded just deciding what to eat for lunch. And because the grass is always greener I imagine that if we really did discard personas so lightly, we’d often do so prematurely.

But I worry that instead we go too far in the other direction. In Silicon Valley, at least, it’s easy to develop a tunnel vision which I will summarize in the following table. The middle column is somewhat hyperbolic [1], but the right column is (at least loosely) based on actual conversations I have had with people in Oxford.

Complete The Following Sentence	Answer in Silicon Valley	Other Possible Answers
“The fundamental problem is…”	“...our MySQL server won’t sync with the cloud.”	“...the lack of objective morality in a post-modern world.”
“You have to be careful when you sneak into…”	“...the front of the line at the Google cafeteria.”	“...Syria.”
“You can use social media to…”	“...disrupt the groups-larger-than-three-but-smaller-than-five space.”	“...represent the parents of the children who died at Newtown.”

There is such a range of ways to live! People here put on black robes for dinner and say grace in Latin and sit at “high table” so they can look down on us mortals and it all seems so absurd to me but they have been doing this for eight hundred years. And at the Oxford Union, the debating society, I see eighteen-year-olds in tuxedos giving grandiose speeches on subjects they don’t understand, playing at being members of parliament, and again it seems absurd to me -- but there’s a decent chance they really will be members of parliament. (I have also, incidentally, seen and heard of more sexism, racism, and classism in a month here than I did in a year working in tech companies, but we can talk about that another time.)

Perhaps more important, I think, than these differences in lifestyle is the diversity in worldviews. Part of this I’ve seen from the people who come to speak at Oxford: three-star generals who stand up and defend the Iraq War and Jan Brewer who says that the only thing Obama has done right is “be a good father”. Part of it is due to the other Rhodes scholars. It's nice to meet a bunch of people who don't, usually, code, and hear what it's really like to march in Ferguson and how one sneaks into Burma and what the hell is going on with Turkey and how you get water to remote Latin American towns and why you need boots on the ground to conduct an airstrike at all and why it’s so hard to prosecute war crimes and...

The part that really bakes my noodle is that, of course, even this relative diversity is only a tiny slice of human experience. In Palo Alto they drink $4 coffee, and in Oxford they drink $4 tea: this is a long way from how most people live. I realized that I could not remember the last time I’d had a long conversation with someone who hadn’t gone to college. (Can you?) Perhaps this shouldn’t surprise me, given my previous work on how birds of a feather flock together; we are astonishingly good at self-segregation, and we build complex mechanisms to facilitate it. After I did the birds-of-a-feather work, I was somewhat troubled to find that someone had used my results to support their dating app that only allows in elites. I’m now at a university where iron gates separate the black-robed students from the beggars outside, where even the way someone speaks is a clue to their class; I don’t think we need to build more walls.

Anyway, hit me up if you’re in Oxford and, assuming I don’t get hit by a car on the wrong side of the road, I’ll keep you posted on the other things I learn in England; also, if you have cool ideas for using statistics to understand the British, shoot me an email.

Notes:

[1] I should also mention that Stanford, of course, has very strong humanities departments and students (indeed, the other two Rhodes scholars from my year studied history and political science) even if Palo Alto feels extremely tech-focused.
[2] I should perhaps clarify that I do not believe one needs to have spiky hair, or play the harmonica, to flirt successfully with lesbians in Central Park. Indeed, I don't have any idea how one flirts with lesbians in Central Park, or even if there are any to flirt with. Sadly, for the reasons discussed above, I will probably remain ignorant, but feel free to enlighten me.

Love in the Time of Ebola

There’s been no shortage of frightening headlines about Ebola. (Don’t actually read all those headlines -- they’ll depress you -- just feel grateful that I read them for you.) One might expect this fear, like a more contagious version of Ebola itself, to infect the general public. Fear can cause more damage than the original disease: see Contagion, V for Vendetta, or (if you want a non-fictional example) this World Bank estimate. So I ran a survey to try to find out just how scared Americans are of Ebola. The results were disheartening, but not for the reason I expected: Americans aren’t that scared of Ebola, but only because they don’t much care about West Africa.

79% of people said they worried about getting Ebola once a week or less. They worried less about dying from Ebola than dying from the flu, a car crash, a plane crash, or a terrorist attack. The only thing they said they worried about less than Ebola was dying in a meteorite strike -- a fate which the Economist estimates has a 1 in 75 million chance of occurring. (I’m pretty skeptical of that number, but whatever.) Other surveys have also found that Americans aren’t that worried about getting Ebola: for example, the Pew Research Center found that just 11% were very worried they would be exposed to the virus.

People did display some irrationalities. For example, roughly a quarter worried more about dying from Ebola than from car crashes or the flu, even though these kill tens of thousands of Americans a year. But when I asked people to estimate how many people Ebola would kill in the United States, almost no one estimated anywhere near this number. Assuming the only reason to worry about Ebola is that it kills people, these people were giving inconsistent answers within the space of a single page.

I asked people to estimate how many people would die from Ebola both in the United States and worldwide. There was a significant correlation between both these variables and how much people said they worried about Ebola, but the correlation with worldwide deaths disappeared when I controlled for United States deaths. In other words, how many people die abroad doesn’t matter to an American’s level of worry if you account for how many people will die in America. Important addendum: Bob Aldrich points out that this may be because the wording in the survey emphasizes how much people are worried about their own safety, in which case it makes sense that we wouldn't see much correlation with faraway deaths after controlling for nearby ones. I suspect you would see similar results if you reran the survey with wording which did not emphasize personal safety, because several other findings point in the same direction. For one thing, almost everyone knew the number of US patients who had died, but less than half could identify the number of patients who had died worldwide [1]. Worse, at the end of the survey, I offered people the chance to forfeit their payment for taking the survey: in exchange, I would donate five times the payment to Doctors Without Borders, an international humanitarian aid organization which fights Ebola. People received just nine cents for taking the survey, yet two thirds of them refused to give up their payment. It’s possible they didn’t believe I would actually pay [2] or that they worried that not getting paid would damage their reputation on Mechanical Turk (I assured them it wouldn’t), but is either one of these motivations (or less than a dime in payment) enough to deny help to an Ebola patient?

Some important caveats to these results. First, the numbers on how worried people are will almost certainly change as the situation evolves and coverage continues. For example, the number of Americans who consider Ebola a “major threat” to the United States has doubled in recent weeks. (I conducted the survey prior to the announcement of a second patient in Dallas, so things may already have changed.) Also, how worried people appear to be depends a lot on how you ask the question; for example, seven in ten Americans report being at least somewhat concerned about an Ebola outbreak. It’s also possible that people are understating how worried they are because they don’t want to look silly. Finally, I conducted the survey using Mechanical Turk, a website commonly used for research in psychology, economics, and political science; Mechanical Turk has its own idiosyncrasies which I describe here [3], and in particular might lead to underestimates of how worried people are.

But my results mirror the indifference to faraway deaths that we’ve seen in the broader response to Ebola. For example, Google searches for “Ebola” peak only when the virus threatens the United States: once after the first two Americans were brought back to the United States for treatment, and once after Thomas E. Duncan flew back to Dallas with Ebola. Tweets about Ebola show a similar pattern. Or consider the plunge in airline stock prices after news of Duncan’s illness broke on September 30th:

That’s a movement of more than 2 billion dollars -- far more than has been pledged to fight the entire Ebola epidemic. When it comes to moving money, evidently, “US airline valuations” is a more potent force than “saving West African lives”.

People’s lack of fear is heartening, but our lack of compassion isn’t. The media would do well if, rather than trying to frighten us with the small mistakes made by American hospitals, it focused on the people who often cannot reach a hospital at all. More broadly, I’ve been getting increasingly worried about our ability to ignore the suffering of people who are a) unseen b) far away c) different from us, because it seems like a cognitive bias that plausibly costs millions of lives -- because it allows us to a) not get terribly upset about foreign policies that kill lots of people b) not donate as much to charity as we would if we actually felt other people’s suffering the way we feel a nice warm cup of coffee c) donate to charities that seem close and tangible rather than ones that actually do the most good. I could go on about this but truthfully I’m a bit of a hypocrite because the reason I conducted this survey in the first place is because I had a nightmare about getting Ebola; these are by no means biases that I am myself immune to. So I’ll close with just two notes:

1. Giving What We Can and GiveWell are two organizations which I think make a pretty persuasive utilitarian argument that a) we should donate a lot more to charity than is conventionally accepted and b) most of that should go to developing countries.
2. Email me if you have thoughts on how to close the empathy gap -- how to get people to value lives equally, and not write off someone's death just because they’re far away and look different (or worse, write off the deaths of a million people because, you know, a million is a really big number.)

Notes:

[1] Not the exact number; they just had to say "Between 1,000 and 5,000".

[2] I actually got so annoyed with my survey takers that I just donated regardless of what they said their preference was.

[3] Previous studies of Mechanical Turk have concluded that it yields data that agrees with other sources [1, 2, 3], offering less biased results than, for example, the “convenience samples” of college students that have historically been used in research. But most samples are biased in some way, and Mechanical Turk workers do tend to be younger, poorer, better educated, and more liberal than the general population. This may explain why my survey-takers appeared especially unconcerned about Ebola: previous polls have found that Americans who are more liberal and more educated are less likely to worry about the virus. Please keep these caveats in mind in interpreting my results (although if you happen to have the resources to run a survey on a less skewed population, we statisticians are always looking for survey daddies).