1

Statistical Inference from the Ordinary Least Squares Model

The reading assignments have derived the OLS estimator. Using assumptions A – E given in the Assumptions Reading Assignment, the mean and the variance of the OLS estimator is derived. The distribution of the estimator, however, has not been defined. The mean and variance do not generally define a statistical distribution. To conduct statistical tests, the distribution (normal, t-statistic, F distribution, Chi-Squared) must be known.

The subject of this reading assignment is inference, statistical and economic, from the OLS regression. Statistical inference involves several tests on the estimated parameters. These tests will involve tests of associated with either a single parameter or a group of parameters. Economic inference includes these statistical tests, but also includes estimated parameter’s magnitude and sign. Inference concerning the previously discussed goodness-of-fit measure, R2, will also be expanded in this reading assignment.

Additional Assumptions Necessary for Statistical Inference

Because the distribution of the OLS estimator is not given by Assumptions A – E, we need to rely on one of two assumptions to perform statistical tests. Under either of these two assumptions, the distributions for the statistical tests can be derived. In both cases, the distribution is the same. Deriving the distributions is beyond the scope of this class.

Assume Normality

The most restrictive assumption is the error terms in the model are normally distributed. We have already assumed they have a mean of zero and constant variance. This assumption is written as .

Central Limit Theorem

Instead of assuming the error terms are normally distributed, we can rely on the Central Limit Theorem. In either case, the same distributions are obtained. The Central Limit Theorem (CLT) states:

given any distribution that has a mean, , and a variance,2, the distribution of sample means drawn at random from the original distribution approaches the normal distribution with a mean  and variance as the sample size increases.

This theorem can also be stated in the form, let q1, q2, q3, . . ., qn be independent and randomly distributed from any distribution that has a mean of  and a variance 2. The mean of qi is:

then

.

In this form, the CLT states that the average value of n independent random variables from any probability distribution (as long as it has as mean and a variance) will have an approximately a standard normal distribution after subtracting its mean and dividing by its standard deviation, if the sample size, n, is large enough. The standard normal distribution is N(0, 1), that is a mean of zero and a variance of one. Why and not ? We are interested in the standard error of a series of means and not the standard error of a sample of random numbers. Second, in practice, n is large, therefore, n and n - 1 are not very different. The sample means of the random variables approximate a normal distribution.

Why is the CLT important and how is it use? To answer these questions, we need to examine the error term, u. What is u? Given the model set-up, the error term includes all factors affecting the dependent variable that are not included as an independent variable. That is, the error term includes everything affecting y except the x’s. Therefore, u is a sum of many different factors affecting y. Because u is a sum of many different factors, we can invoke the CLT to conclude u approximates the normal distribution. Why is the CLT important? If we consider the error term as a sum of many different factors, the CLT states the error term will approximate the normal distribution. The CLT use is similar to assuming the error terms are normally distributed. Thus, by invoking the CLT, statistical distributions can be derived. These distributions allow for statistical tests on the estimated OLS parameters.

Although, the assumptions of either normality or invoking the CLT are the most restrictive assumptions made to use OLS, these assumptions allow statistical tests to be performed. These statistical tests are maybe the most important component of running an OLS regression. Statistical inference is a powerful aspect of OLS. You must understand the following statistical tests.

Inference Concerning Parameters

As noted earlier inference involves statistical tests and examining the estimated coefficients’ magnitude and sign. In this section, aspects concerning individual parameters are discussed.

t-tests

Individual t-tests can be done for each estimated coefficient. Under Assumptions A - E and either assuming normality of the error terms or invoking the CLT, the following can be shown:

(1).

This equation states the result obtained by dividing the quantity obtained by subtracting any value, j, from the estimated coefficient, , by the standard error of the estimate for will be distributed as a student t-distribution (t-distribution) with n - k degrees of freedom. Note, before a particular sample is taken, the estimates for are a random number. That is, the estimator has a distribution associated with it. This is why the above equation is a distribution. After the OLS estimates are obtained for a particular sample, become as fixed number. The statistics and mathematics necessary to derive this result are beyond this class. We will take this result as a given.

t-distribution. Before applying the above result to your OLS estimates, it is informative to review, hypothesis formulation and testing, the student t-distribution, and the use of the t-distribution in statistical testing. As noted in the statistics reading assignment, the t-distribution is one of the most important statistical distributions. See the statistics reading assignment for the general t-test. You are responsible for knowing this test upside down and backwards.

The t-distribution is a symmetric bell-shaped distribution, but the shape (probabilities) depends on the degrees of freedom of the distribution. For different degrees of freedom, the t-distribution has different critical values. As the degrees of freedom increase, the t-distribution approaches the normal distribution. On the web page is a file containing a table associated with the t-distribution. At this point, you should download this file and confirm the distribution various by degrees of freedom.

Given a distribution, statistical tests associated with various hypotheses can be conducted. To conduct hypotheses testing, a null hypothesis and an alternative hypothesis are necessary. It is important that the alternative and null hypothesis cover all possible outcomes. The null hypothesis is commonly denoted as Ho, whereas, the alternative is commonly denoted as HA. Several alternative and null hypotheses are:

Null H0:j = 0j > 0j < 0

AlternativeHA:j 0j 0 j 0.

In this list, three different null hypotheses are given in the top row and the associated alternative in the second row. For each null, the alternative hypothesis covers all possible alternatives not given by the null hypothesis. For example, consider the first null hypothesis, H0: j = 0. Given this null, two different alternatives are possible, j could be less than zero or j could be greater than zero. Both alternatives are covered by the alternative hypothesis of j 0. An alternative hypothesis of j > 0 would be inappropriate for the null hypothesis of H0: j = 0. It is inappropriate because it does not cover the potential for j < 0. If your test statistics was such that it implied j < 0, you would not be able to make any inference from your two hypothesis. It is important to set-up your null and alternative hypothesis such that they cover all possible alternatives.

Given the different null and alternative hypotheses, the t-test can be either a two-tailed test or a one-tailed test. Knowing if the test is one- or two-tailed is important in conducting the test, the value associated with the critical value will depend on the tails, and in interpreting the test inference. A two-tailed test is given by

H0:j = d

HA:j d.

An example of a one-tailed test is

H0:j d

HA:j < d.

In general, these examples show that when using a t-test, any value can be tested as given by the general notation, d. It is not necessary to that d = 0, as given in the previous examples. The number d can be any value.

The “fail to reject” and “rejection” regions are different between one- and two-tailed tests. To conduct a test, a level of significance must be chosen. The level of significance is given by . The probability of a “Type I” error is given by the level of significance. A Type I error occurs when the null hypothesis is rejected, but the hypothesis is true. Associated any level of significance is a critical value. The critical value is the point of demarcation between the acceptance and region regions.

Before proceeding, a few words about Type I and II errors is appropriate. The two types of errors are defined in table 1.

Table 1. Type I and Type II Errors Defined
Decision Regarding Statistical Test / States of the World
Null hypothesis true / Null hypothesis false
Reject null / Type I error / Correct decision
Do not reject null / Correct decision / Type II error

We can fix the probability of a Type I error by picking a value for . Unfortunately, the same control over a Type II error is not possible. The probability of a Type II error can only be calculated if we know the true value for the estimated parameters. If we know the true value of the parameters, there is no reason to perform statistical tests. We can, however, state three important aspects concerning Type I and II errors.

1) The probabilities of Type I and Type II errors are inversely related. This means as you decrease the probability of a Type I error, you are increasing the probability of a Type II error and vice versa.

2) The closer the true value is to the hypothesized value, the greater the chance for a Type II error.

3) The t-test may be the best test, because for a given probability for a Type I error, the test minimizes the probability of a Type II error.

For a two-tailed test, the fail to reject and rejection regions are given by α/2. This is shown in figure 1.

Figure 1. Two-tailed test

A one-tailed test fail to reject and rejection regions are defined by α probability in one of the tails as shown in figure 2.

Figure 2. Two Cases Associated with a One-tailed Test

For either a two-tailed or a one-tailed test, you calculate a value based on equation (1). Then the null hypothesis is either failed to reject or rejected based on where the calculated t-statistic values falls.

Key Point: the test is testing hypothesis concerning the population parameters. The test is not testing hypothesis about the estimated parameters from a particular sample. Once a sample is taken, the estimated values are a fixed number. It makes no sense to test if a given number is equal to some other number. We know the value for a given number.

Application to OLS. At the beginning of this inference section, we stated we could use the estimated parameters and their estimated variance to obtain a statistic that is distributed as a t-distribution. Combining our knowledge of hypothesis testing with this result, it is clear we can conduct tests using the estimated parameters. These tests are concerning with hypothesis concerning the true parameters and are not testing hypothesis about the sample. Recall, for a given sample, your OLS estimates are a unique set of fixed numbers.

As an application to OLS, let’s assume you have estimated the following equation, with estimated standard errors beneath each estimated parameter:

(2).

It is not uncommon to see estimated equations written in this form. Here, the estimated slope is 5.2 and the estimated intercept is 1.5. These estimated values come from the OLS estimator given by the equation . The standard errors (the square root of the variance) of the estimated parameters are 0.25 for the intercept and 5.2 for the slope parameter. These standard errors are the square root of the diagonal elements from the estimator of the variance of the estimated parameters given by. Lets test the following hypothesis:

H0:1 2

HA:1 < 2.

Inserting the values from equation (2) into equation (1), the t-statistic becomes:

(3).

The next step is to use a t-table to obtain the critical value for your assumed level of significance. Assuming 28 degrees of freedom (n - k, n = 30 and k = 2) and an  = 0.05, the critical value is -1.701. At this point, you should look at the t-table given on the class web site and convince yourself you know how the critical value was determined. This is a one-tailed test and we are interested in the left-hand tail. Notice the hypotheses are concerned with the true parameter value, , and not the estimated value, . Graphically, the problem is stated in figure 3.

Figure 3. One-tailed t-test Example

In this example, the calculated value falls into the rejection region. Therefore, we would reject the null hypothesis the 1 = 0. If we chose level of significance equal to 0.025, the critical value would be -2.048. At this level of significance, we would fail to reject the null hypothesis. This examples illustrated that different statistical conclusions (inferences) can be reached depending of the level of significance chosen. It is important for you to think about what are the statistical and economic implications of choosing different  levels?

As a second example, lets test the following null hypothesis;

H0:2 = 0

HA:2 0

Inserting the values from equation (2) into equation (1), the t-statistic becomes:

(3).

As before, the next step is to use a t-table to obtain the critical value for your assumed level of significance. Assuming 28 degrees of freedom and an  = 0.05, the critical values are -2.048 and 2.048. This is a two-tailed test. At this point, you should look at the t-table given on the class web site and convince yourself you know how the critical value was determined. Another point is even through the significance level is the same between the two examples; the critical values differ. This is caused by the one- versus two-tailed aspect. Convince yourself why this occurs. Graphically, the problem is stated in figure 4.

Figure 4. Two-Tailed Example

In this test, the calculated value falls into the fail to reject region. We would state that we fail to reject the null hypothesis, 2 = 0.

Significance of a Variable. Most regression packages, including Excel, print out a specific t-test for every estimated parameter. This test is

H0:j = 0

HA:j  0.

This test is often referred to as testing if the variable is significant. If the true parameter value is equal to zero, independent variable, xj, has no affect on the dependent variable. This is what is meant by significance of the variable. You will need to know and understand this test.

p-values. In addition to printing out the specific t-test associated with the significance of a variable, most regression packages also print out the probability value. The probability value is known as the p-value. It is increasingly common to state the p-value associated with the test statistics rather than choosing a level of significance. It is, therefore, important you understand the meaning of a p-value.

The probability value is the probability that the test statistic, t-statistic, takes a value larger than the calculated value. In other words, the p-value for a given t-statistic is the smallest significance level at which the null hypothesis would be rejected. Because the p-value represents the area under a probability density function, p-values range from 0 to 1. P-values are reported as decimals.

An illustration will help in the understanding of p-values. From the hypothesis-testing example associated with 2, we obtained a calculated t-value equal to 1. The test was a two-tailed test. The p-value is given graphically in figure 5.

Figure 5. p-values Areas Defined for Two-Tailed Test

As illustrated in figure 5, the calculated t-statistics are place on the student t-distribution graph. The p-value is the areas in the two tails for a two-tailed test using the calculated t-statistic as the demarcation point between the reject and fail to reject regions. Computer programs automatically compute these areas by integration. A concept that is beyond this class. P-values can also be associated with one-tailed tests. For a one-tailed test, obviously we are interested the area given by only one of the tails. In the example illustrated, the p-value would equal 0.32.

P-values can be used several ways. The first is the p-value gives the level of significance associated with the calculated t-statistic. In the above example, if you were to choose a level of significance equal to 0.32, your two-tailed test critical values would equal -1 and 1. At this level of significance, your critical value and test statistic are equal. In other words, you can report the exact level of significance that provides the cut off value between rejecting and failing to reject the null hypothesis. Second, the p-values can be used as follows, the null hypothesis is rejected if the p-value is less than or equal to your chosen level of significance, . In the above example, a  level of 0.05 was chosen and a p-value of 0.32 was given. At this level of significance, we would fail to reject the null hypothesis; the p-value is larger than the level of significance.

Graphically, we can show the use of p-values for the two-tailed test as follows.

We are concerned with two levels of significance, = 0.05 and =0.50. At the 5% significance level, the critical values are -2.048 and 2.048, whereas at the 50% level, the critical values are -0.67 and 0.67. The calculated t-statistic fails in the range between these two levels of significance by design for this example. At the 5% critical level, we reject the null hypothesis. At the 50% level we would fail to reject the null hypothesis. The decision rule is given by