Hypothesis Testing

We need to compare two distributions of data

The two samples will always be different

Sampling error and measurement error are unavoidable

Statistics are used to distinguish between trivial and meaningful differences

A statistical test works much like a jury in criminal trial

Both situations operate without absolute proof

You may make wrong decision

A jury trial is a test for guilt, while a statistical test is a test for differences

You do not stand trial to show you are innocent

You do not design experiment unless you expect an effect

Both situations consider one of two mistakes more serious

Finding innocent man guilty in a jury trial

Concluding a difference was "real" when it was not

A statistical test

Is based on the data

Begins by assuming "no difference" (= null hypothesis)

It is difficult to reject the null hypothesis

Must be > 95% differences observed did not occur by chance

Now you accept the alternative hypothesis

Science progress by rejecting null hypotheses

To carry out most kinds of statistical tests

Use data to calculate test statistic

Look up the critical value of the test statistic

Compare to see if the test statistic is in the rejection region