Tags

, ,

[here enters the Great Reverend]

I said earlier that

we’re calculating the probability of every possible observation combination, excluding one: that no events occurred. However, by definition the sum of all probabilities must be one. So if we calculate the odds of that single combination and subtract it from one, we know the sum of the odds for every other combination. We can accomplish 2^N – 1 calculations for the cost of one!

This was a bit of a simplification. The evidence can support more than one theory: an object falling to the Earth is compatible with Aristotelian, Newtonian, and Einsteinian physics, for instance. By focusing on one hypothesis alone, we can convince ourselves there’s overwhelming evidence for it when in fact that evidence is even more overwhelming for another theory. Conversely, the evidence can partly support a theory, or have no real effect on it. In short, doubt and uncertainty exist. What really matters here is not the odds of at least one nest or attempt, but those odds relative to the odds of no nest or attempt.

Fortunately, I came prepared. All of part four was dedicated to determining the likelihood of a nest or attempt not occurring; now we just need to walk through the same process again, but this time think of all the ways it could occur at least once. If you read my series on Bayesian Hypothesis Testing, you know how to combine the results in a Bayesian framework.

wpid-bayes_odds_form.pngAnd while I was fairly harsh on frequentism, I did show a decent way to handle it in that framework too, via Neyman-Pearson hypothesis testing.

wpid-likelihood_ratio.pngNotice that neither method will result in a radically different conclusion to my earlier result. The “at least one” and “none” theories are both composites of multiple other theories (only Smyth is truthful, everyone’s lying or mistaken but Myerson/Puppy where honestly fooled, and so on). If we assume each of those sub-theories have roughly equal probability, then the relative odds of the composite theories is roughly proportional to the number of sub-theories each contains.

When comparing two hypotheses which are composed of sub-hypotheses, if every sub-hypothesis has about the same likelihood then the likelihood ratio of the two parent hypotheses is about the same as the count of sub-hypotheses in the first divided by the count in the second.This sub-theory approach does increase the number of ways there could be no nesting, as it treats “honest belief of a false first-hand report” as being distinct from “manufactured or misunderstood second-hand report.” So we might lose a few nines via this analysis, but the results should still be in the same ballpark.

Before we get started, though, we have some issues to attend to. Remember the Biff Jag case?

The full description of the probabilities relating to Biff Jag.We consolidated all of those possibilities into three cases: everyone was telling the truth, Jag was honestly fooled by a false claim, and Jag invented something out of thin air or misunderstood what they’d heard. Notice that I’m not equivocating “false” and “deliberate falsehood” here, as accidental misunderstandings are a possibility too. We can convert all this a truth table.

A truth table of how the Smyth and Jag claims combine. Things get weird when Jag is false (as in lying or honestly misunderstanding).That third line covers two possibilities: Smyth is either truthful or not, while Jag is not truthful. There are good reasons to think Smyth would speak up if Jag were spreading falsehoods about the nest or attempt that she truthfully saw, but she also might see a benefit in staying silent if the pros of greater attention outweighed the cons of discovery. So we need to pop in a new variable: how likely would a second-hand report remain uncorrected, if it was false and referenced a person with a truthful first-hand account? Since there are only two people who qualify in this model, Smyth and Gray, it’s easy to give them each a variable.

If neither Smyth nor Jag are truthful, they must either have independently decided to invent a claim or colluded together. The former case is already handled via combining the odds of Smyth having a false claim and that duo inventing/misunderstanding one, while I consider the latter so unlikely as to be unworthy of consideration. As the saying goes, three people can keep a secret if two of them are dead; while collusion seems like a great way to bump up the likelihood of falsehood, it requires everyone to maintain a perfect narrative even when they’re unable to communicate with their collaborators. There must also be a shared incentive, and a reason to maintain the conspiracy if that incentive isn’t achieved.

If you disagree, feel free to alter the model accordingly!

Another complication is maryann. As we don’t know their real name, there’s a chance they’re actually one of the named people and thus not a separate claim of nesting/attempts. You’d expect said person to mention this if they stepped forward, but it’s also possible said person forgot about their older claim or didn’t want to out that pseudonym. While daufnie_odie claimed their source for a nest/attempt was unique, it’s also possible they didn’t know about maryann’s claim. So we have three possibilities here, each with a different way to effect the odds.

Three ways to handle maryann being the pseudonym of someone known.Each carries its own relative likelihood, of course, and each incorporates all the changes above. This requires further tap-dancing to handle, as well as a new suite of variables:

New variables introduced in the new relative-likelihood model for the nesting claims.This expanded model is a lot more complicated: instead of taking a single pass to calculate a likelihood, we now need 4,096! Thank goodness for computers.

Based on the variables, the likelihood ratio of "at least one nest or attempt" to "none" is 134136.423206:1
The most likely "at least one" scenario is 999, with likelihood 0.0116539479
The most likely "none" scenario is 0, with likelihood 0.0000003690
(totals for "none" and "at least one" are 0.0000009796 and 0.1314027579, respectively)

It looks like there’s quite a bit of difference between the likelihood of the sub-hypotheses, contradicting my earlier assumption. How badly does this skew the results? We’ll need to convert the likelihood ratio back into a probability in order to compare it to my earlier result, which is terribly easy since both hypotheses cover all possibilities.

The Bayes Factor of 134136 translates to an H_(>=1) probability of 99.999254% or so.Whew, we only lost a bit of a nine compared to the simpler model, so long as we stick with pure Neyman-Pearson. The Bayesians in the audience still need to mix in their prior probabilities, though. Since I’m not sure what everyone’s priors are, I’ll calculate against a number of them and let you pick and choose.

A comparsion of the posterior odds ratio from a number of priors, plus an estimate of the probability of p( H_>=1 ).I should also check how sensitive these variables are, as per last time.

A table of each variable, and how sensitive it is around my initial values. Yet again, the most influential variables are those taken from large-scale studies with well-defined error bars.This revised model has a flatter response, thanks to all those new variables. The more dials you can turn, after all, the less you have to turn each to make the same overall difference. The greedy search’s results are fairly boring, so let’s skip to the multivarate search.

The top seven variables which changed the most during the multivarate search. Not much new here, though a few values related to human motivations are ranked higher.Yet again, you get the best bang for the buck by adjusting the odds of a false nesting claim or the prevalence of nestings and/or attempts; while some variables related to human motivation creep into the top 7 chart, including two new ones, they still don’t carry much impact.

I think this about wraps up the subject. Hopefully I’ve shown that you can analyze pretty much anything, and in some cases even fairly fuzzy inputs can lead to solid conclusions. Also, believe claims by default until further information casts them in doubt.

Advertisements