Archives for September, 2010

A linear regression is a statistical method that helps one understand the relationship between two (or more) variables.  It does this in three ways:

1. It uses one variable to predict the value of another variable
2. It tests hypotheses concerning the relationship between two variables
3. It quantifies the strength of the relationship between two variables

As we did in our discussion of linear correlation, we will denote two variables as X and Y; X is the independent variable, Y the dependent one.  A linear regression assumes that there is a linear relationship between X and Y, and is given by the following formula:

Y_i = b₀ + b₁X_i + ε_i for i = 1, …, n

where:

Y_i is the i^th value of the dependent variable

b₀ is the y-intercept

b₁ is the slope coefficient

X_i is the i^th value of the independent variable

ε_i is the i^th value of an error term

i is the index of a particular variable

n is the maximum value of i

In English, the value of the dependent variable Y_i is equal to {the value of dependent variable when the independent variable’s value is zero (b₀)} plus {the product of the slope b₁ and the independent variable b₁} plus {some error term ε_i}. The error term is that part of Y_i that is not explained by X_i . We call b₀ and b₁ the regression coefficients.

When we speak about the relationship between two variables, we think in terms of many contemporaneous observations (a cross-sectional series) or observations over a period of time (a time-series). Observations are indexed by values 1 to n.  For example, you may be interested in the effect in various countries of money supply (X_i where i refers to a particular country) on the country’s inflation rate (Y_i) – that would be a cross-sectional analysis.  Conversely, you would use a time-series analysis to test the money supply/inflation rate relationship in one country over a period of time.

A perfect linear regression would be one where all of the error terms equaled zero.  This would indicate that all changes to Y were accounted for by changes to X.  For instance, if I eat every cookie handed to me, then there would be no error values when I plot cookies offered versus cookie consumed. In this case, the regression line’s y-intercept would be zero and the slope would equal 1; all actual data values would be points directly on the regression line. Thus, if you offered me 3 cookies, I’d eat 3 cookies. Obviously this example is unrealistic when the number of cookies offered rises above some critical value, say 3-dozen in my case.

A more realistic case is one that plots a straight regression line through the data in which the errors are minimized – the best fit.  In real life, we are interested in imperfect correlations, so we need a method to achieve the best fit, which we define as the regression line that minimizes the sum of the squared vertical distances (deviations) between observations and the regression line.  This method is called the linear least-square method.  Nifty, but how do we calculate the best fit?

To achieve the best fitting regression line, we need to find the slope b₁ and y-intercept b₀ that produces the minimum sum of the squared errors. (We square the errors, which are simply the vertical deviations from the regression line, because we don’t want positive and negative values to cancel each other out).  How do we find these magic regression coefficients? We need to make estimates, which we call the fitted parameters, according the following formula:

The funny little hat (^) above b₀ and b₁ designates that the regression coefficients are estimated. We are summing, for all index values of i, the squares of the following difference: the actual value of the dependent variable minus the predicted value of the dependent variable. When this sum (the sum of the squared error terms) is minimized, we have a best-fit regression line. The actual method of calculating this minimum is complicated, and we leave it to a computer spreadsheet or math package to do the nitty-gritty work.

A note about the slope coefficient b₁: when a linear regression contains a single independent variable, the slope coefficient is equal to the following:

b₁ = Cov(Y, X) / Var(X) = Cov(Y, X) / s_xs_x where s = standard deviation

which is the covariance of Y and X divided by the variance of X.  Alert readers will recall from the previous blog that this formula is very similar to that for the correlation coefficient (r). The difference here is that the denominator, the variance of X, is the equivalent to the square of the standard deviation of X (s_x). For the correlation coefficient, the denominator is the product of the standard deviations for X and Y:

r = Cov(Y, X) / s_xs_y

Conceptually, one can see that the coefficients are very similar – they both give a scale to the covariance of the two variables.

Next time, we will address the assumptions one makes in order to calculate a proper linear regression.