OLS regression

171

models. I assume you’ve had exposure to least squares regression as typically taught in a second-semester undergraduate statistics course. But I’ve made efforts to keep the mathematical notation to a minimum and focus on practical rather than theoretical issues. A number of excellent texts are available that cover the statistical material outlined in this chapter. My favorites are John Fox’s Applied Regression Analysis and Generalized Linear Models (for theory) and An R and S-Plus Companion to Applied Regression (for application). They both served as major sources for this chapter. A good nontechnical overview is provided by Licht (1995).

8.2OLS regression

For most of this chapter, we’ll be predicting the response variable from a set of predictor variables (also called regressing the response variable on the predictor variables— hence the name) using OLS. OLS regression fits models of the form

Yi = β0 + β1X1i + ... + βkXki i = 1 ... n

where n is the number of observations and k is the number of predictor variables. (Although I’ve tried to keep equations out of these discussions, this is one of the few places where it simplifies things.) In this equation:

Yi is the predicted value of the dependent variable for observation i (specifically, it’s the estimated mean of the Y distribution, conditional on the set of predictor values).

Xji is the jth predictor value for the ith observation.

β0 is the intercept (the predicted value of Y when all the predictor variables equal zero).

βj is the regression coefficient for the jth predictor (slope representing the change in Y for a unit change in Xj).

Our goal is to select model parameters (intercept and slopes) that minimize the difference between actual response values and those predicted by the model. Specifically, model parameters are selected to minimize the sum of squared residuals:

∑ni=1(Yi − Yi)2 = ∑ni=1(Yi − β0 + β1X1i + ... + βkXki)2 =∑ni=1 ε i2

To properly interpret the coefficients of the OLS model, you must satisfy a number of statistical assumptions:

■Normality—For fixed values of the independent variables, the dependent variable is normally distributed.

■Independence—The Yi values are independent of each other.

■Linearity—The dependent variable is linearly related to the independent variables.

172	CHAPTER 8 Regression

■Homoscedasticity—The variance of the dependent variable doesn’t vary with the levels of the independent variables. (I could call this constant variance, but saying homoscedasticity makes me feel smarter.)

If you violate these assumptions, your statistical significance tests and confidence intervals may not be accurate. Note that OLS regression also assumes that the independent variables are fixed and measured without error, but this assumption is typically relaxed in practice.

8.2.1Fitting regression models with lm()

In R, the basic function for fitting a linear model is lm(). The format is

myfit <- lm(formula, data)

where formula describes the model to be fit and data is the data frame containing the data to be used in fitting the model. The resulting object (myfit, in this case) is a list that contains extensive information about the fitted model. The formula is typically written as

Y ~ X1 + X2 + ... + Xk

where the ~ separates the response variable on the left from the predictor variables on the right, and the predictor variables are separated by + signs. Other symbols can be used to modify the formula in various ways (see table 8.2).

Table 8.2 Symbols commonly used in R formulas

Symbol	Usage

~Separates response variables on the left from the explanatory variables on the right. For

example, a prediction of y from x, z, and w would be coded y ~ x + z + w.

+Separates predictor variables.

:Denotes an interaction between predictor variables. A prediction of y from x, z, and the

	interaction between x and z would be coded y ~ x + z + x:z.
*	A shortcut for denoting all possible interactions. The code y ~ x * z * w expands to
	y ~ x + z + w + x:z + x:w + z:w + x:z:w.
^	Denotes interactions up to a specified degree. The code y ~ (x + z + w)^2 expands
	to y ~ x + z + w + x:z + x:w + z:w.
.	A placeholder for all other variables in the data frame except the dependent variable. For
	example, if a data frame contained the variables x, y, z, and w, then the code y ~ .
	would expand to y ~ x + z + w.
-	A minus sign removes a variable from the equation. For example, y ~ (x + z + w)^2
	– x:w expands to y ~ x + z + w + x:z + z:w.
-1	Suppresses the intercept. For example, the formula y ~ x -1 fits a regression of y on
	x, and forces the line through the origin at x=0.

	OLS regression	173
Table 8.2 Symbols commonly used in R formulas
Symbol	Usage
I()	Elements within the parentheses are interpreted arithmetically. For example, y ~ x +
	(z + w)^2 would expand to y ~ x + z + w + z:w. In contrast, the code	y ~ x +
	I((z + w)^2) would expand to y ~ x + h, where h is a new variable created by
	squaring the sum of z and w.
function	Mathematical functions can be used in formulas. For example, log(y) ~ x + z + w
	would predict log(y) from x, z, and w.

In addition to lm(), table 8.3 lists several functions that are useful when generating a simple or multiple regression analysis. Each of these functions is applied to the object returned by lm() in order to generate additional information based on that fitted model.

Table 8.3 Other functions that are useful when fitting linear models

Function	Action
summary()	Displays detailed results for the fitted model
coefficients()	Lists the model parameters (intercept and slopes) for the fitted model
confint()	Provides confidence intervals for the model parameters (95% by default)
fitted()	Lists the predicted values in a fitted model
residuals()	Lists the residual values in a fitted model
anova()	Generates an ANOVA table for a fitted model, or an ANOVA table comparing
	two or more fitted models
vcov()	Lists the covariance matrix for model parameters
AIC()	Prints Akaike’s Information Criterion
plot()	Generates diagnostic plots for evaluating the fit of a model
predict()	Uses a fitted model to predict response values for a new dataset

When the regression model contains one dependent variable and one independent variable, the approach is called simple linear regression. When there’s one predictor variable but powers of the variable are included (for example, X, X2, X3), it’s called polynomial regression. When there’s more than one predictor variable, it’s called multiple linear regression. We’ll start with an example of simple linear regression, then progress to examples of polynomial and multiple linear regression, and end with an example of multiple regression that includes an interaction among the predictors.

8.2.2Simple linear regression

Let’s look at the functions in table 8.3 through a simple regression example. The dataset women in the base installation provides the height and weight for a set of 15 women

174

CHAPTER 8

ages 30 to 39. Suppose you want to predict weight from height. Having an equation for predicting weight from height can help you to identify overweight or underweight individuals. The analysis is provided in the following listing, and the resulting graph is shown in figure 8.1.

Figure 8.1 Scatter plot with

Regression

	160
(in pounds)	140 150
Weight	130
	120

regression line for weight predicted from height

58

60

62

64

66

68

70

72

Height (in inches)

Listing 8.1 Simple linear regression

>fit <- lm(weight ~ height, data=women)

>summary(fit)

Call:

lm(formula=weight ~ height, data=women)

Residuals:
Min	1Q	Median	3Q	Max
-1.733	-1.133	-0.383	0.742	3.117
Coefficients:

Estimate Std. Error t value				Pr(>|t|)
(Intercept) -87.5167		5.9369	-14.7	1.7e-09	***
height	3.4500	0.0911	37.9	1.1e-14	***
---
Signif. codes:	0 '***' 0.001 '**' 0.01			'*' 0.05	'.' 0.1 '' 1

Residual standard error: 1.53 on 13 degrees of freedom

Multiple R-squared: 0.991, Adjusted R-squared: 0.99

F-statistic: 1.43e+03 on 1 and 13 DF, p-value: 1.09e-14

>women$weight

[1]115 117 120 123 126 129 132 135 139 142 146 150 154 159 164

>fitted(fit)

1	2	3	4	5	6	7	8	9
112.58	116.03	119.48	122.93	126.38	129.83	133.28	136.73	140.18
10	11	12	13	14	15
143.63	147.08	150.53	153.98	157.43	160.88

> residuals(fit)