Significance Levels and Confidence Intervals

Confidence intervals (only) provide a plausible range for population parameters. For instance, the population proportion would be within the range $(p_1, p_2)$ with 95% confidence level. That's if we take $N$ samples of the same sample size, 95% of the time the sample proportion would be within that range. Note that 95% is the confidence level. And the significance level, often denoted by $alpha$, is the complement of the confidence level only when the hypothesis testing is two sided. That is $text{confidence level} = 1 - text{significance level}$ in that scenario. For the one sided hypothesis testing $text{confidence level} = 1 - 2*text{significance level}$. The area of the confidence interval is always symmetry.

The $text{significance level} = P(text{Type 1 error} | H_0 text{true})$, and the type 1 error implies we reject the null hypothesis wrongly, then the bigger the significance level the more likely we would make a false rejection.

P-Values

We suppose that the null hypothesis is true and the p-value represents the probability the extreme(or more extreme) sample proportion is observed by chance: $P(text{observed or more extreme sample statistic}|H_0 text{true})$. The bigger the p-value the more likely the observed sample proportion is resulted by chance and the less we would reject the null hypothesis, but the rejection depends on the significance level we set beforehand. If the p-value is smaller than the significance level we would be confident enough that the observed sample proportion is not led by chance and we would reject the null hypothesis.