Before getting back into the weeds, I should get you familiar with Poisson.
That’s the statistical distribution, not fish. It describes the odds of a certain number of rare events occurring over a continuous time-span. When the time-span is short, relative to the odds of the event, Poisson looks a bit like a power distribution; when it’s long, Poisson looks more like the binomial or Gaussian distribution. The formula for it is pleasantly short.
If there was one thing that triggered this series of blog posts, it was this table.
Soooo much wrong-ness. First off, what they’re measuring is very rare, relative to their sample size; over ninety trials, it was rare for a trial to exceed three observed values and many had less. This means that randomness plays a very large role in what they observed, relative to the actual frequency of the events. If you don’t believe me, try figuring out how often you’d get snake eyes in general, via only 36 tosses of two dice. You’ll likely end up with the wrong number, even if you pool together a few trials, because the distribution is quite skewed probably leading you to underestimate how often it occurs. Compare that with calculating the odds of rolling a total of seven. The distribution is more spread out and symmetric, so repeated sampling will quickly place the mean near the true value.
Secondly, look at that last footnote. The expected number of gliomas in the control condition is 90/50, or 1.8, but it could range between 0 and 7. But if that’s the control condition, why is the treatment group hovering around that number?
Here’s the Poisson distributions for the ranges given by that footnote. The λ=0 curve is smushed against the Y axis, the λ=7 curve predicts quite a few more events than what occurred, but that λ=1.8 curve is pretty close to what we observe. Oh, and that λ=1.69 curve? That’s what gnuplot’s curve fitting function is the best fit for the observed data. The error bars are pretty wide, as it predicts we should have seen more “1”‘s in Table 1 and fewer “3”‘s, but it consistently re-converges there even if I try to shake the parameter values.
In other words, the “treatment” group’s brain lesion incidence is a good match for what we’d expect of a control group.
Speaking of which, what’s up with that control group?
For a Poisson distribution with λ=1.8, the odds of getting zero observations in two trials is 2.7%. The most likely outcome is a 1-2 punch, sitting at 15.9%, as the odds of the individual occurrences are maximal and there’s two ways to make a differing pair (we could have two observations on the first run and one on the second, or vice-versa). A crude combination of those two suggests a 5.8:1 odds ratio for the control group not being a control at all!
Now, I am assuming gilial cell hyperplasia has the same incidence rates in controls as malignant gliomas, something the footnote doesn’t explicitly endorse. But this isn’t the only table where the control group records zero events. If we stick strictly to observed conditions with an explicit baseline rate, we get:
In every single control group, zero events were recorded. The other tables aren’t as bad as the first, thanks to their lower expected values, but that’s a double-edged sword: a lower expected value for the control means the treatment has to have a large effect size merely to show up. If the cancer rate doubled, for instance, then all treatment groups in table 2 have a 28% chance of recording a single observation among them, a 23% chance of recording two, and a 15% chance of recording none.
And when you try to incorporate variability, those zeros look even more out of place. Remember, the odds of a zero drops off quickly in the Poisson distribution as λ increases.
This gets even more ridiculous when you focus in just on Table 5 and 6, combining both sexes into one.
The incidence rates between control and treatment are vast when looking at the heart, but small when looking elsewhere in the body? Even though the RF transmitter wasn’t set up to target a specific body part?
It’s all very fishy. And I haven’t even gotten to the time-to-appearance numbers yet…