The null hypothesis

I was cued to write this mini-rant by a conversation on Twitter. I don’t really feel that I should have to explain the null hypothesis to people within this community, but in contexts like sexual assault and rape, it seems that all proportional skepticism goes right out the window.

Ami Angelwings (of Escher Girls fame) tweeted about accusations, and how the accusation that “she’s making it all up” is actually itself an accusation, and needs to be vetted out. She went on to say that the null hypothesis in this case is not that “she’s making it up”. Someone I greatly respect in the skeptical community replied to my retweeting that, saying “it is, give evidence”.

Frankly, that’s a load. That isn’t how the null hypothesis works.

In science, when you put together a scenario that you’d like to test and you hypothesize that something will happen as a result of testing it, you are in fact playing a probability game. To give you an example, if you were to fire a loaded gun, the null hypothesis — assuming everything about the gun and bullet is in working order and that I know how to fire a gun — is that a bullet would come out (and subsequently put a hole in the thing I’m pointing it at). An alternative hypothesis would be to suggest that perhaps the bullet might be a dud and nothing happens, and that no bullet would come out. That would be a rarity, and if a pilot study of a single trigger-pull was involved, you might get the wrong idea about the hypothesis based on that one dud and that therefore bullets never come out of guns. But, it is a plausible alternative outcome. If you were to suggest that instead of a bullet, a shark would come out of the gun, that would be an extraordinary claim, and that would require an extraordinary level of evidence.

Now, let’s say the scenario I give you is my claim that I was at a shooting range and fired a gun. This is not a scientific experiment, it’s not something we can duplicate; it’s a claim, and it’s in the past. This might be a perfectly plausible claim to you (because I now live in America, and guns are practically full citizens here). This doesn’t require any significant evidence, because shooting ranges exist here, I have transportation, I’m of age. You have no particular reason to doubt that I have made that suggestion earnestly. However, it’s, for me, an extraordinarily implausible scenario, because I’m pretty anti-gun. I have trouble firing Nerf guns at human targets, and I am frankly a little afraid of holding a directed explosive charge in my hands in general. Too many things could conceivably go wrong, even if those things are grossly implausible, and I’d simply rather not take the risk.

Now, knowing THAT about me, you might think that me going to a gun range is now an implausible scenario, and you have reason to doubt that I did so. The null hypothesis is no longer that you have no reason to doubt that I claimed something absolutely ordinary to other people, because you now know that it’s very out of the ordinary for me. What would normally be the null hypothesis is now suddenly less likely in my specific case than the alternative, that I did not do the things I claimed to do. It would require more explanation — did I have a change of heart? Was I lying originally when I claimed my distaste for fireamrs? Or am I lying now?

What if I had claimed to go to the store? Seems simple enough, common enough. The null hypothesis is that I have no reason to lie, and people (and I) go to the store all the time, so I probably did. Now, if I said I went to the store and picked up a case of dragon feed for dirt-cheap, you’d be hard pressed to believe me because dragons don’t exist and “dragon feed” doesn’t seem to be a euphemism for anything in particular, so you’d need more explanation before believing me in toto. The circumstances of the claim make the claim implausible, not the claim itself. That dragon feed doesn’t exist, doesn’t actually mean I DIDN’T go to the store. Skepticism would require you to investigate the claim about dragon feed, but skepticism against the claim of going to the store is actually hyper-skepticism, skepticism from which is borne “Truthers” of all stripes.

With regard to events like rape or sexual assault, given that they apparently happen all the time, and that only a very small proportion of those are reported, and that only a very small proportion of those that ARE reported are “made up” in the sense that the person who claims to have been assaulted were not actually assaulted, the null hypothesis in the case of “I was raped” is not, in fact, “they made it all up.” That is, in fact, a leap, predicated on some lack of trust of the person in question, often derived from societal biases against the class of the person making the claim.

Now, one’s word is certainly not enough proof to convict a person. That doesn’t mean that when faced with a claim — “that man raped me” — and a counterclaim — “no I didn’t” — that the null hypothesis there is actually “no I didn’t”. The null hypothesis here is actually something more like “we can’t get to the truth of these dueling claims but can take no action in absence of hard evidence”. Yes, it is possible for the “null hypothesis” in a given situation to be misframed. What people believe the null hypothesis to this proposition to be — “she’s making it up” — actually requires that the woman is always lying. It isn’t enough evidence to prove that he DID rape her, but with the burden of proof being set for criminal trials to require evidence beyond a shadow of a doubt, the man will likely not be convicted regardless. But a lack of conviction does not mean a rape did not happen.

Rape, you should know, is distressingly common. Put a guy and a girl in a room together, put them in a scenario where the guy thinks he can get away with it (e.g. the girl is unconscious), and one in three of these guys would have sex with the girl without consent. Mind you, to get those results, you have to not use the word “rape” in the questions, because “everyone” knows rape is wrong. The problem, here, is that people don’t know what rape actually IS.

So, between that, and the fact that we absolutely know that rape is drastically underreported, we can then presume that the majority of rape claims are actually true (or at least true to the extent that the woman was in fact raped), because the number of rape claims is significantly lower than the number of rapes. Because this is a common occurrence, and because there are pressures against admitting that you’re a victim of such a common occurrence, the majority of such claims are true. The null hypothesis when confronted with a woman claiming to have been raped is that yes, she very probably was raped, because it is not an extraordinary claim, and ordinary claims require only ordinary evidence. Testimony absolutely counts as ordinary evidence.

That is not to say that the person claiming to have been raped knows enough of the assailant to accurately make an accusation, or has enough evidence to get that person put in jail, and those cases where the woman was indeed raped but cannot level an accusation that sticks are all often counted against the “false accusations” statistic.

I will further note here that because most individuals are not rapists, any specific claim against a specific person requires extraordinary evidence. It’s not like you’re accusing a person of pooping — most people poop, after all. You’re accusing them of doing something to someone else without their express consent, and the vast majority of people understand that to be wrong (though that number could be buffed significantly and I support every effort toward educating people that violating consent is wrong). That is why one’s word is insufficient evidence to put someone in jail. Hell, even with twenty-plus accusations floating around about Bill Cosby, one cannot simply throw him in jail without hard evidence of one of them, but with that many accusations surfacing, and with the statistics regarding accusations, it seems far more likely that he actually took advantage of people than that there is some conspiracy to ruin him.

But under no circumstances should the null hypothesis in this case be that the woman making a claim is simply making it all up, because it’s actually vanishingly unlikely that that’s the case. That is one of the lower probability outcomes of the scenario in question. Given all the social stigma of making the accusation, given the blowback you get just for saying you were raped (much less directing the accusation at a person, coupling an actor with the passive-voice action), the counterclaim of “she’s making it all up” actually requires evidence in and of itself. In fact, the vast majority of reported rapes go unpunished specifically because the scenario was manipulated so as not to leave evidence, so the perpetrator gets away with it.

That was Ami Angelwings’ point. One cannot simply give the “she’s a lying liar” claim a pass because it plays toward what you incorrectly think the null hypothesis is in the case of the claim that the “liar” charge is countering. That does not follow. That counterclaim is also a claim, and one must treat that claim with the same healthy skepticism as you treat the original claim.

TL;DR: if you’re willing to be skeptical of a claim without evidence, be equally skeptical of a counterclaim without evidence — especially when the counterclaim offers an even less likely scenario.

Skepticsm 101, folks.

The null hypothesis

14 thoughts on “The null hypothesis

  1. 1

    Whenever I see the “false accusations” cry, the first thing I think is “ok, what’s more likely: that someone would level a false accusation of rape, or that a rapist would lie about whether they had committed rape?” Because rapists who have been accused of rape have a pretty strong, clear motivation to lie.

  2. 3

    Another good analogy is atmospheric conditions. If I tell you it rained today where I live, you’re very likely to believe me because it’s a plausible, everyday event, and I’m unlikely to lie about it (unless I have something to gain by it). If I tell you the moon appeared brown in the sky, it’s far less likely, but still within the realm of possibility and could be verified if you were willing to check weather sites for the facts. And if I tell you the clouds were all different colours of the rainbow, you’d rightfully call me deluded or accuse me of trolling.

    The probability of something happening factors into believability. Rape is unfortunately a common (or even everyday) occurrence, so every case should definitely be considered possibly or probably true and never treated with disdain, especially when there are many claims against someone (or even just one accusation). Rape accusations are not far-fetched claims, and women/people don’t make those accusations lightly.

  3. 4

    Really nice explanation of null hyposthesis, thanks. Hopefully those who run skepticism workshops and the like will take note and include this type of lesson in their programming.

    Rape, you should know, is distressingly common. Put a guy and a girl in a room together, put them in a scenario where the guy thinks he can get away with it (e.g. the girl is unconscious), and one in three of these guys would have sex with the girl without consent. Mind you, to get those results, you have to not use the word “rape” in the questions, because “everyone” knows rape is wrong. The problem, here, is that people don’t know what rape actually IS.

    These numbers are frightening. To think of so many silent predators lurking in the background just waiting for an opportunity to pounce, so many willing to do awful things if they think they can get away with it. Working in an office with about 30 employees in total, we’re now wondering specifically which 10 or so would be willing to rape. At the grocery store, which dozen or so of the other shoppers are the ones who would be willing? Which of our neighbors, and even friends? With these numbers, they must be everywhere.

    Does the study you cited here give any reason for hope or optimism?

  4. 5

    The problem, here, is that people don’t know what rape actually IS
    Also likely, they do know what rape is, but are banking on human tendency to minimize the awful and discomfort with direct confrontation in order to get away with it.

  5. 6

    It would actually be better to avoid using the term “null hypothesis” far outside the context where it has a clear meaning, which is in scientific experiments where the researcher is varying one thing (the “independent variable”) and testing whether it has an effect on another (the “dependent variable”). The null hypothesis is that it will have no effect – and that’s true even if you already have very good reason to think that it will. The null hypothesis is then rejected if results are such that the chance of a difference in the dependent variable, at least as great as that found, occurring if the null hypothesis is true, are sufficiently low. It also makes good sense to use the term where you are systematically collecting and correlating observations which the world makes available, outside an experimental context (e.g. amounts of rainfall and width of tree rings), but beyond that, it just generates confusion and invites obfuscation. The “default assumption” is a more accurate term in anything like the cases discussed in the OP – and because we know that rape is considerably more common than false rape accusations, the default assumption with regard to most such accusations is that the accusation is basically true.

  6. 7

    Nick: I largely agree, partly because it muddies the waters, and partly because I’m generally irritated with the pressganging of scientific terms-of-art into other uses (cf. “theory”). However, that’s how it’s being used here, so I’m confronting it as such.

    For what it’s worth, Ami’s original tweets were actually about a specific case, which I didn’t realize at the time —

    A woman intervened in a domestic violence case and got headbutted; the man was put in jail and many news articles were published. MRAs are presently trying to accuse this girl of “making it all up” and are presenting the “default assumption” that nothing happened, despite there being enough evidence of this to land the dude in jail.

    EVEN in cases where there was enough evidence for a conviction, EVEN in cases where the guy doesn’t deny any of the accusations, MRAs will still rally to attack anyone perceived as having “attacked” men by telling the truth about one man’s violent actions. It’s very much a bullying tactic, like “stop hitting yourself” or “who smelt it, dealt it”.

  7. 8

    We are Plethora: the study cited is about one of those college student studies, which unfortunately skews the average age by necessity. There could be some maturity components, or components related to peer pressure (e.g. “frat culture”) there. It means we need to teach consent earlier, and we need to take steps to dismantle toxic rape-culture-enabling systems within the college magisterium. But that means we know a good place where we could make inroads at least.

  8. 9

    “Now, one’s word is certainly not enough proof to convict a person.” Let me assure you, as a trial attorney of many years, people are convicted every day on the testimony of a single person. Lots.

  9. 10

    Jason:”VEN in cases where there was enough evidence for a conviction, EVEN in cases where the guy doesn’t deny any of the accusations, MRAs will still rally to attack anyone perceived as having “attacked” men ”

    Hence my earlier comment about Bayesian inference. If your personal a priori probabilities reflect your belief that no rape claims are likely to be true, then even with lots of evidence your opinion won’t be swayed. If you start with a fairly neutral view, then evidence either way sways you.

  10. 11

    Interesting topic.

    I agree with Nick Gotts that all of this has little to do with the “null hypothesis” of statistical inference as used in science, both because I see no independent variable and because a single event is being measured. I also agree with david that Bayes might be the better option, as it can be used with an IV and for single events.

    But the larger criticism that I have of the original post is that it seems to put claims and counter-claims on exactly equal footing. While I agree with Jason Thibeault that skepticism should be applied to both, as it should be applied to any claim, I do not see claims and counter-claims as having equal burdens of proof because they are critically different. While a claim can come out of nowhere, a counter-claim is always a response to a previous claim. The goal of a claim is to convince the audience of some proposition. The goal of a counter-claim is often to show that the evidence in favor of the proposition is insufficient or at least isn’t being agreed to by stipulation. Thus, a counter-claim is often an example of skepticism in action, while the previous claim is often the one that is needing more skepticism. Thus, again, while I agree that all claims should be viewed with skepticism, I don’t agree with the subtle message of this post, which I see as asking that counter-claims receive more skepticism.

    – billie

  11. 12

    Billie: I am actually stating that since the counterclaim involves making an assertion, that assertion should not be weighted such that it is “correct” should there be insufficient evidence for the original claim.

    “I went to the store” and “No you didn’t, you went to the center of the Earth” are not equal claims. One is plausible, the counter is not plausible at all. Even if I DIDN’T go to the store, that doesn’t mean the center of the earth was my actual destination.

    Likewise with “I was raped by that guy” and “she’s making it all up”. The latter actually involves NOTHING happening at all, when it’s more likely she was in fact raped, given the other facts about rape, like how unlikely it is for a rape victim to actually speak up when they were raped. So even if there’s insufficient evidence to prove rape, that doesn’t mean she DID make ANYTHING up. And yet, that’s how the mental heuristic works thereafter. See every rape claim in the past ten years that’s made headlines, and how people react to the person making the claims.

  12. 13

    Let’s put this another way. Creationism is not proven even if evolution is disproven. Is that easier to grasp, without all the apparently impenetrable rape language? I don’t know why with regard to gender issues, it’s like all the rules of debate and dialog break down and everyone reverts to the same mental traps they’d overcome in other areas. No wonder some people default to assuming WILLFUL ignorance from people who should otherwise know better.

  13. 14

    I wonder over gender bias as well (in view of tropes such as the “men are honest and don’t sugarcoat the truth”/”bitches be lyin'” that are not only popular among MRAs). Are gender-swapped but otherwise completely analogous cases where a man who made a serious accusation against a woman out of nowhere and the woman promptly rejects the accusation treated identically? (I’m thinking specifically of claims that a woman has not earned her position on the objective merits of her work alone.) It’s hard not to suspect that MRAs would immediately jump to the defence of the man, especially if the man was in a position of privilege/power or widely admired (“why would a man in his position lie, consider his achievements”), and not only them. In fact, I can think of such a case right away, the dismissal of Tarja from Nightwish.

Comments are closed.