1  Frameworks for statistical inference and estimation

Use statistical procedures from a range of schools and strictly adhere to their respective methods and interpretation. For example, do a Fisherian significance test properly and interpret it properly. Then set up a formal Neymann-Pearson (sic) test and interpret it formally (this means setting both Type I and II error rates beforehand, among other things). Then do an estimation procedure. Then switch hats and do a Bayesian analysis. Take the results of all four, noting their different behavior, and come to your conclusion. Good analysis and interpretation are as important as the fieldwork, so allot adequate time and resources to both.

- Francis H. J. Crome, ``Researching Tropical Forest Fragementation: Shall We Keep On Doing What We're Doing?'' ch. 31 in Tropical Forest Remants: Ecology, Management and Conservation of Fragmented Communities (ed. W. F. Laurance and R. O. Bierregaard, Jr.), University of Chicago, Chicago: 1997 (p. 501).

1.1  Frequentist

1.1.1  Fisherian

1.1.2  Neyman-Pearson

Note: modern "bad" statistical practice as criticized by Yoccoz and Johnson essentially muddles these two schools.

1.2  Likelihood

1.2.1  ``Classical'' likelihood

To establish confidence limits, you calculate likelihood profiles (how likelihood drops off as you go away from the MLE).

Example: suppose we want to test whether a coin is fair (p(head = 0.5); we flip it 10 times and get 7 heads. The likelihood that the probability of getting a head is p is proportional to p⁷ (1-p)³ (assuming flips are independent) and the log-likelihood is C + 7 ln(p) + 3 ln(1-p).

What can we tell?

• The MLE of p is 7/10 (unsurprisingly).
• p = 0.7 is 2.28 times more likely than p = 0.5 (do you think that is ``significant''?)
• If we set limits of head-probabilities that are 10 times less likely than the maximum, we get limits (0.37,0.93).
• If we set limits of head-probabilities that are 1.92 log-likelihood units (e^1.92=6.82 times) less likely than the maximum, we get limits (0.39,0.92).

Why should we use 1.92 log-likelihood units? The likelihood ratio test says that if we allow n parameters to vary, the q% confidence limits are (asymptotically) the q quantiles (upper tails) of c²_n/2: c²₁(0.95)/2 = 1.92. Essentially, this is a frequentist argument sneaking back in that says how close we can expect to get to the true MLE in repeated trials.

Example: do two populations have the same mean?

Suppose we have two populations (assume we know they are normally distributed and that they have the same [known] variance s², for simplicity). The normal distribution is ~ e^{-(x-m)2/(2 s2)}; therefore the joint log-likelihood of a set of values drawn independently from a normal distribution with mean m (the sum of the individual log-likelihoods) is µ -å_i (x_i-m)² - the sum of squares. If we assume there is really just a single distribution from which all of the values are drawn the log likelihood is µ -å_i (x_i - [`X]); if we are allowed different means for the two different populations we get µ -å<sub>i</sub> (x<sub>i</sub> - [`X]₁) - å_i(x_i-[`X]<sub>2</sub>). The likelihood for the second case can't be worse than the first (since we could always set [`X]₁ = [`X]<sub>2</sub> = [`X]). If it is more than 1.92 greater than the first, we can conclude that the two populations have ``significantly'' different means. We can also find confidence limits, etc..

Points about likelihood:

• Usually gives ``sensible'', unbiased answers
• Only relative likelihoods matter
• Gaussian, constant errors ® least-squares
• ``Everybody likes likelihood'' - it is an important component of all statistical schools, although they all use it differently

1.2.2  Information-theoretic

1.3  Bayesian

Remember, what would really like to know is P(H₀|data), the probability of our hypothesis given the data we've observed.

Use Bayes' Rule (see below) to get probabilities on hypotheses and parameter values. In frequentist statistics, the underlying hypotheses/parameters/models are true and the data come from a probability distribution; in Bayesian statistics the data are true and the underlying hypotheses etc. have probability distributions.

(Bayesian decision analysis: associate costs with different outcomes, maximize expected return.)

Bayesian analysis

• is not ``more powerful'' (unless you cheat) than other approaches
• can be used to cheat, but should not ...
• does not allow e.g. unreplicated designs, although (i) it does provide a relatively easy framework for estimating dependence among units and (ii) it gets around the philosophical problems of unreplicated designs (e.g. evolutionary biology)

	Fisher	N-P	Likelihood	Information	Bayes
depends (at least conceptually) on replicated outcomes	yes	yes	yes	no	no
outcome depends on sampling rules (violates Likelihood principle)	yes	yes	no	no	no
gives decision rules	no	yes	no	(no)	yes
requires alternative hypotheses specified	no	yes	yes	yes	yes
intuitive probability interpretation	no	no	yes	no	yes
subjective	no	no	no	no	yes
requires specified priors	no	no	no	no	yes
allows integrating previous results	no	no	no	no	yes

2  Bayesian approaches in a little more detail

How can we figure out the probability of the hypothesis, P(H₀|data) from the likelihood, P(data|H₀)? (The likelihood P(data|H₀) is a probability of data, not hypotheses: the likelihoods for all our candidate hypotheses don't even add up to 1, as they would if they were probabilities of different hypotheses ...)

How? Bayes' Rule.

P(H0|data) = P(data|H0) ·P(H0) P(data) = P(data|H0) ·P(H0) å i  P(data|Hi) ·P(Hi)

(1)

In words: multiply the likelihood P(data|H₀) by the prior probability P(H₀), and divide by the sum of (likelihood × prior) for all candidate hypotheses.

False-positive/testing example

My favorite example (which I will probably not have time to present in class): suppose the probability of having some deadly but rare disease (D) is 10^-4. There is a test for this disease which has no false negatives: if you have the disease, you will test positive (P(+|D) = 1). However, there are occasionally false positives; 1 person in 1000 who doesn't have the disease) will test positive anyway (P(+|not D) = 10^-3). We want to know the probability that someone who has a positive test is actually ill.

Using Bayes' Rule

P(D|+) = P(+|D) P(D) P(+) .

(2)

We know P(+|D) (=1) and P(D) ( = 10^-4), but we have to figure out P(+), the overall probability of testing positive. You can test positive if you are really diseased or if you are really healthy;

P(+) = (p(D and +)) + (p(not D and +)),

(3)

according to the rule that if A and B are mutually exclusive (you can't be both ill and not ill) P(A and B) = P(A)+P(B). We can then say

P(+) = P(D) P(+|D) + (1-P(D)) P(+|not D)

(4)

by the rule that P(A and B) = P(A) P(B|A). Putting it all together,

P(D|+) = P(+|D) P(D) P(D) P(+|D) + (1-P(D))P(+|not D) = 1 ×10-4 1 ×10-4 + (1-10-4) ×10-3 » 10-4 10-3 = 1 10 .

Even though false positives are rare, the chance of being ill if you test positive is still only 10%!

Priors If P(H₀) is constant reduces to likelihood rule (except that we can say something about probability) but what does a "flat" prior mean? (scale changes, subdividing hypotheses)

How do you do this?

(Continue with previous examples: coin-flipping and testing two populations.)

Coin-flipping:

• flat prior
• ``probably fair'' prior
• (paranoid prior?)

Two-population example:

• testing point null hypotheses deprecated by Bayesians ...
• can give (e.g.) probability that m₁-m₂ is greater (or less) than a certain value

Credible intervals (symmetric, contains 95% of probability).

Issues:

• although the prior (and the interpretation of the probability) are subjective, the updating rule (Bayes' rule) is the rational response to fresh evidence
• Simple models can be (practically speaking) harder: you may have to ``roll your own'' more often
• Conjugate priors
• Complex models: cf. Crome comment, MCMC

Final conclusions:

• be suspicious of big differences between the conclusions of different approaches: is someone cheating?
• usually differences are because different frameworks allow us to ask different questions: really think about what question you want to answer!