Econometrics2011

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

133

p

as n ! 1:

from which it follows that V ! V

We also need to show

that

p

p

, from which it will follow that

V

p

V

b b

! and !

!

and

p

V : Since

it is su¢ cient to show that p . Notice that

e

b

V

b !

nb

e !

1

e e

1ie^i2

e

b

X

1

n

1

n

= n i=1 xixi0 h(1 hii) 2

X

X

=

n

i=1 xixi0 h(1 hii) 2

1iei2 +

n

i=1 xixi0 h(1 hii) 2 1i

e^i2 ei2

:

Note that Theorem 6.7.1 states max1 i n hii = op(1); and thus by the CMT

max

2

= o (1):

Thus

p

1 i n (1 hii) 1

	b
e

1

n

1

n

X

X

n

xixi0

n

e^i2 ei2

n i=1

(1 hii) 2

1 ei2

+ n i=1

xixi0

(1 hii) 2

1

2

X

1

X

x

x

i0

e2

max

(1

h

) 2

1

+

x

x

i0

max

(1 hii)

2

1

max

e^2

e2

n i=1

i

i

1 i n

ii

n i=1

i

1 i n

1 i n

i

i

=Op(1)op(1) + Op(1)op(1)op(1)

=op(1)

p

p

p

Since ! it follows that ! and

! :

Proofbof Theorem 6.11.1.

By Theorem 6.9.2, p

d

n

e

!

t

( ) =

b

bs( )

n

=

p

nb

bV

q

bb

b p

N (0; V ) and V^ ! V : Thus

d	N (0; V )
!	p
		V
= N (0; 1)
The last equality is by the property that linear scales of normal distributions are normal.

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

134

Exercises

Exercise 6.1 Take the model yi = x01i 1 +x02i 2 +ei with Exiei = 0: Suppose that 1 is estimated by regressing yi on x1i only. Find the probability limit of this estimator. In general, is it consistent for 1? If not, under what conditions is this estimator consistent for 1?

Exercise 6.2 Let y be regression estimator

n 1; X be n k (rank k): y = X + e with E(xiei) = 0: De…ne the ridge

=	n	xixi0 + Ik! 1	n	xiyi!	(6.39)
b	Xi		X
	=1		i=1

where	as n	! 1	: Is consistent for ?
	> 0 is a …xed constant. Find the probability limit of b		b

Exercise 6.3 For the ridge regression estimator (6.39), set = cn where c > 0 is …xed as n ! 1:

b

Find the probability limit of as n ! 1:

Exercise 6.4 Verify some of the calculations reported in Section 6.5. Speci…cally, suppose that x1i and x2i only take the values f 1; +1g; symmetrically, with

Pr (x1i = x2i = 1)

Pr (x1i = 1; x2i = 1)

2	j
E ei2		x1i = x2i
E ei	j x1i 6= x2i

Verify the following:

1.Ex1i = 0

2.Ex21i = 1

1

3. Ex1ix2i = 2

4. E e2i = 1

5.E x21ie2i = 1

6.E x1ix2ie2i = 78:

Exercise 6.5 Show (6.18)-(6.21).

Exercise 6.6 The model is

yi

E(xiei)

=Pr (x1i = x2i = 1) = 3=8

=Pr (x1i = 1; x2i = 1) = 1=8

=54

=14:

=x0i + ei

=0

=E xix0ie2i :

b b

Find the method of moments estimators ( ; ) for ( ; ) :

bb

(a)In this model, are ( ; ) e¢ cient estimators of ( ; )?

(b)If so, in what sense are they e¢ cient?

CHAPTER 6. ASYMPTOTIC THEORY FOR LEAST SQUARES

135

Exercise 6.7 Of the variables (yi ; yi; xi) only the pair (yi; xi) are observed. In this case, we say that yi is a latent variable. Suppose

yi = xi0 + ei
E(xiei) =	0
yi =	yi + ui
where ui is a measurement error satisfying
E(xiui)	=	0
E(yi ui)	=	0
denote the OLS coe¢ cient from the regression of y			i	on x	:
Let b				i

(a) Is the coe¢ cient from the linear projection of yi on xi?

b

(b) Is consistent for as n ! 1?

p b

(c) Find the asymptotic distribution of n as n ! 1:

Exercise 6.8 Find the asymptotic distribution of pn ^2 2 as n ! 1:

Exercise 6.9 The model is

yi

E(ei j xi)

where xi 2 R: Consider the two estimators

b

=

e

=

=xi + ei

=0

Pn

Pi=1 xiyi

n x2 i=1 i

1 Xn yi :

n i=1 xi

(a)Under the stated assumptions, are both estimators consistent for ?

(b)Are there conditions under which either estimator is e¢ cient?

Chapter 7

Restricted Estimation

7.1Introduction

In the linear projection model

yi = x0i + ei

E(xiei) = 0

a common task is to impose a constraint on the coe¢ cient vector . For example, partitioning x0i = (x01i; x02i) and 0 = 01; 02 ; a typical constraint is an exclusion restriction of the form2 = 0: In this case the constrained model is

yi = x01i 1 + ei

E(xiei) = 0

At …rst glance this appears the same as the linear projection model, but there is one important di¤erence: the error ei is uncorrelated with the entire regressor vector x0i = (x01i; x02i) not just the included regressor x1i:

In general, a set of q linear constraints on takes the form

R0 = c

(7.1)

where R is k q; rank(R) = q < k and c is q 1: The assumption that R is full rank means that the constraints are linearly independent (there are no redundant or contradictory constraints).

The constraint 2 = 0 discussed above is a special case of the constraint (7.1) with

	R =	0	;	(7.2)
		I

a selector matrix, and c = 0.

Another common restriction is that a set of coe¢ cients sum to a known constant, i.e. 1+ 2 = 1: This constraint arises in a constant-return-to-scale production function. Other common restrictions include the equality of coe¢ cients 1 = 2; and equal and o¤setting coe¢ cients 1 2 = 0:

A typical reason to impose a constraint is that we believe (or have information) that the constraint is true. By imposing the constraint we hope to improve estimation e¢ ciency. The goal is to obtain consistent estimates with reduced variance relative to the unconstrained estimator.

The questions then arise: How should we estimate the coe¢ cient vector imposing the linear restriction (7.1)? If we impose such constraints, what is the sampling distribution of the resulting estimator? How should we calculate standard errors? These are the questions explored in this chapter.

136

CHAPTER 7. RESTRICTED ESTIMATION

137

7.2Constrained Least Squares

An intuitively appealing method to estimate a constrained linear projection is to minimize the least-squares criterion subject to the constraint R0 = c. This estimator is

		argmin SSE			( )	(7.3)
where	e =	R0 =c		n
n
Xi			2 = y0y 2y0X + 0X0X :
SSEn( ) =	yi xi0
=1

The estimator minimizes the sum of squared errors over all such that the restriction (7.1)

holds. We call the constrained least-squares (CLS) estimator. We follow the convention of
using a tilde “~”e rather than a hat “^” to indicate that is a restricted estimator in contrast to
the unrestrictedeleast-squares estimator ; and write it as				when we want to be clear that the
estimation method is CLS.		e	cls
One method to …nd the solution tob(7.3) uses the techniquee					of Lagrange multipliers. The
problem (7.3) is equivalent to the minimization of the Lagrangian
L( ; ) =	1	SSEn( ) + 0 R0		c	(7.4)
	2

over ( ; ); where is an s 1 vector of Lagrange multipliers. The …rst-order conditions for minimization of (7.4) are

	@		( ; ) =					X0y + X0X + R = 0			(7.5)
and	@ L e e									e e
				@		( ; ) = R0				c = 0:	(7.6)
			@ L
								e

Premultiplying (7.5) by R0 (X0X) 1 we obtain

where = (X0X)

1

R0 + R0 + R0

X0X 1 R = 0

0

X0y is the

b

e

e

c

unrestricted least-squares estimator. Imposing R

(7.6)

and solving for we …nd

e

b

e

e

b

1

= hR0

X0X

Ri 1 R0 c :

(7.7)

= 0 from

e

Substuting this expression into (7.5) and solving for we …nd the solution to the constrained minimization problem (7.3)

e b		1 R hR0		1	Ri 1 R0	b	(7.8)
= X0X			X0X			c :

This is a general formula for the CLS estimator. It also can be written as

= Qxx1R hR0Qxx1Ri 1			R0 c	:
e b	b	b	b

CHAPTER 7. RESTRICTED ESTIMATION

138

7.3Exclusion Restriction

While (4.4) is a general formula for the CLS estimator, in most cases the estimator can be found by applying least-squares to a reparameterized equation. To illustrate, let us return to the …rst example presented at the beginning of the chapter –a simple exclusion restriction. Recall the unconstrained model is

yi = x10

i 1 + x20

i 2 + ei

(7.9)

the exclusion restriction is 2 = 0; and the constrained equation is

yi = x10

i 1 + ei:

(7.10)

In this setting the CLS estimator is OLS of yi on x1i: (See Exercise 7.1.) We can write this as

1 =	n	x1ix10 i! 1	n	x1iyi!:
e	Xi		X
	=1		i=1

The CLS estimator of the entire vector 0 = 01; 02 is

e	e
=		:	(7.11)
	01

It is not immediately obvious, but (7.8) and (7.11) are algebraically (and numerically) equivalent. To see this, the …rst component of (7.8) with (7.2) is

e

b b

b

b

1

0

1

0

1

0 I #:

1 = I 0

" Qxx

I

0

I

Qxx

I

Using (4.28) this equals

1

1 = 1 Q121 Q22 1

2

e

=

+ Q

2

Q Q

Q

1

2

b1

b11

b12

22

b

22

= Q 1

Q Q Q 1Q + Q 1 Q Q 1Q Q 1

Q Q Q 1Q

b

11 2

b

1y

b

12 b22

b

21

11 b

1y

b b b

b

b

b b b

b

1

1y

12

1

2y

1

12

22

1

2y

21

11 2 b

b b

b

b

11 2

22

22 1

11

= Q

Q

Q Q Q Q Q

= Q 1

Q

Q Q 1Q

1

=

b

1

b b b

b b

11

1yb

11

b

11 2

b

b

12 b22

b

21 bQ11bQ1y

Q Q

b

b

which is (7.11) as originally claimed.

7.4Minimum Distance

The CLS estimator is a special case of a more general class of constrained estimators. To

b

see this, rewrite the least-squares criterion as follows. Let be the unconstrained least-squares

0 b

estimator, and write the uncontrained least-squares …tted equation as yi = xi + e^i: Substitute

CHAPTER 7. RESTRICTED ESTIMATION									139
this equation into SSEn( ) to obtain
	n
SSEn( ) =	Xi			2
	n
	=1
=	X	xi0 + e^i xi0 2
	i=1
	n	b	0		n		!	b
	X	b			X
=	i=1 e^i2 +				i=1 xixi0				(7.12)
=	n^2 + n 0				Qxx	:
		b			b	b

where the third equality uses the fact that Pni=1 xie^i = 0: Since the …rst term on the last line does not depend on it follows that the CLS estimator minimizes the quadratic on the right-side of

b

(7.12): This is a (squared) weighted Euclidean distance between and : It is a special case of the general weighted distance

Jn ( ; W n) = n	0	W n 1
	b		b

for W n > 0 a k k positive de…nite weight matrix. In summary, we have found that the CLS estimator can be written as

= argmin J			( ; Q 1)
e	R0 =c	n	bxx

More generally, a minimum distance estimator for is

	(W		) = argmin J		( ; W		)	(7.13)
emd		n	R0 =c	n		n

e

where W n > 0. We have written the estimator as md(W n) as it depends upon the weight matrix

W n:

An obvious question is which weight matrix W n is appropriate. We will address this question after we derive the asymptotic distribution for a general weight matrix.

7.5Computation

A general method to solve the algebraic problem (7.13) is by the method of Lagrange multipliers.

The Lagrangian is

L( ; ) = 12Jn ( ; W n) + 0 R0 c which is minimized over ( ; ): The solution is

e	b			1	R0	b	(7.14)
md(W n) = W nR R0W nR						c :

(See Exercise 7.5.)

b 1

If we set W n = Qxx then (7.14) specializes to the CLS estimator:

	(Q 1) =
emd	bxx	ecls

In this sense the minimum distance estimator generalizes constrained least-squares.

CHAPTER 7. RESTRICTED ESTIMATION

140

7.6Asymptotic Distribution

We …rst show that the class of minimum distance estimators are consistent for the population parameters when the constraints are valid.

Assumption 7.6.1 R0 = c where R is k q with rank(R) = q:

Theorem 7.6.1 Consistency

Under Assumption 1.5.1, Assumption 3.16.1, Assumption 7.6.1,

p		(W )	p	as n	:
and W n ! W > 0; emd		n	!		! 1

Theorem 7.6.1 shows that consistency holds for any weight matrix, so the result includes the CLS estimator.

Similarly, the constrained estimators are asymptotically normally distributed.

Theorem 7.6.2 Asymptotic Normality

p

Under Assumption 1.5.1, Assumption 6.4.1, Assumption 7.6.1, and W n ! W > 0;

d

pn md(W n)

(7.15)

! N (0; V (W ))

as n

! 1

; where

e

0

R0

0

0

1 R0W

V

(W ) =

V

W R R0W R

1

R0V

V

R R0W R

1

1 R W

(7.16)

+W R

W R

R V R R W R

and

V = Qxx1 Qxx1:

Theorem 7.6.2 shows that the minimum distance estimator is asymptotically normal for all positive de…nite weight matrices. The asymptotic variance depends on W . The theorem includes the CLS estimator as a special case by setting W = Qxx1:

Theorem 7.6.3 Asymptotic Distribution of CLS Estimator

Under Assumption 1.5.1, Assumption 6.4.1, and Assumption 7.6.1, as n ! 1

d

pn cls ! N (0; V cls)

where

e

V

cls

= V

Q 1R R0Q 1R 1

R0V

V

R R0Q 1R

1 R0Q 1

xx

xx

xx

xx

xx

xx

1 R

xx

1

xx

1

Q

1R

V

R R Q

1R

R Q 1

+Q

R R0

0

0

0

CHAPTER 7. RESTRICTED ESTIMATION

141

7.7E¢ cient Minimum Distance Estimator

Theorem 7.6.2 shows that the minimum distance estimators, which include CLS as a special case, are asymptotically normal with an asymptotic covariance matrix which depends on the weight matrix W . The asymptotically optimal weight matrix is the one which minimizes the asymptotic variance V (W ): This turns out to be W = V has shown in Theorem 7.7.1 below. Since V is unknown this weight matrix cannot be used for a feasible estimator, but we can replace V with

a consistent estimate V and the asymptotic distribution (and e¢ ciency) are unchanged. We call
the minimum distancebestimator setting W n = V the e¢ cient minimum distance estimator
and takes the form		b	1	R0 c	(7.17)
= V		R R0V R
e b	b	b		b

This estimator has the smallest asymptotic variance in the class of minimum distance estimators, The asymptotic distribution of (7.17) can be deduced from Theorem 7.6.2.

Theorem 7.7.1 E¢ cient Minimum Distance Estimator

Under Assumption 1.5.1, Assumption 6.4.1, and Assumption 7.6.1, for de…ned
in (7.17) ,	p			d			e
		n		! N 0; V
as n ! 1; where			e
V	= V V R R0V R 1 R0V :						(7.18)
Since
			V	V			(7.19)

the estimator (7.17) has lower asymptotic variance than the unrestricted estimator. Furthermor, for any W ;

V V (W )

(7.20)

so (7.17) is asymptotically e¢ cient in the class of minimum distance estimators.

Theorem 7.7.1 shows that the minimum distance estimator with the smallest asymptotic variance is (7.17). One implication is that the constrained least squares estimator is generally inef- …cient. The interesting exception is the case of conditional homoskedasticity, in which case the optimal weight matrix is W = V 1 = 2Qxx so in this case CLS is an e¢ cient minimum distance estimator. Otherwise when the error is conditionally heteroskedastic, there are asymptotic e¢ ciency gains by using minimum distance rather than least squares.

The fact that CLS is generally ine¢ cient is counter-intuitive and requires some re‡ection to understand. Standard intuition suggests to apply the same estimation method (least squares) to the unconstrained and constrained models, and this is the most common empirical practice. But our statistical analysis has shown that this is not the e¢ cient estimation method. Instead, the e¢ cient minimum distance estimator has a smaller asymptotic variance. Why? The reason is that the least-squares estimator does not make use of the regressor x2i: It ignores the information E(x2iei) = 0. This information is relevant when the error is heteroskedastic and the excluded regressors are correlated with the included regressors.

e

Inequality (7.19) shows that the e¢ cient minimum distance estimator has a smaller asymptotic

b

variance than the unrestricted least squares estimator : This means that estimation is more e¢ cient by imposing correct restrictions when we use the minimum distance method.

CHAPTER 7. RESTRICTED ESTIMATION

142

7.8Exclusion Restriction Revisited

We return to the example of estimation with a simple exclusion restriction. The model is

yi = x01i 1 + x02i 2 + ei

with the exclusion restriction 2 = 0: We have introduced three estimators of 1: The …rst is unconstrained least-squares applied to (7.9), which can be written as

					1	= Q 1		Q	1y 2	:
							11 2
				its asymptotic variance is
From Theorem 6.28 and equation (6.19) b							b	b			Q1112:
	avar( 1) = Q1112	11 Q12Q221 21 12Q221Q21 + Q12Q221 22Q221Q21
The	second estimator of		1	is the CLS estimator, which can be written as
	b
							= Q 1Q			:
				e1;cls			b11 b1y

Its asymptotic variance can be deduced from Theorem 7.6.3, but it is simpler to apply the CLT directly to show that

avar(	1;cls	) = Q 1		11	Q 1	:	(7.21)
e		11			11

The third estimator of 1 is the e¢ cient minimum distance estimator. Applying (7.17), it equals

	=	V	V	1			2	(7.22)
e1;md	b	1 b	12 b	22		b
where we have partitioned	= "	b	b		#:
b		V 11	V 12
		b	b
V		V 21	V 22

From Theorem 7.7.1 its asymptotic variance is

avar(	1;md	) = V	11	V	12	V 1V	21	:	(7.23)
e						22

In general, the three estimators are di¤erent, and they have di¤erent asymptotic variances.

It is quite instructive to compare the asymptotic variances of the CLS and unconstrained leastsquares estimators to assess whether or not the constrained estimator is necessarily more e¢ cient than the unconstrained estimator.

First, consider the case of conditional homoskedasticity. In this case the two covariance matrices simplify to

	avar(	) = 2Q 1
and	b1		11 2
	avar(		) = 2Q 1	:
	e1;cls		11

If Q12 = 0 (so x1i and x2i are orthogonal) then these two variance matrices equal and the two estimators have equal asymptotic e¢ ciency. Otherwise, since Q12Q221Q21 0; then Q11 Q11 Q12Q221Q21; and consequently

Q111 2 Q11 Q12Q221Q21 1 2:

e

This means that under conditional homoskedasticity, 1;cls has a lower asymptotic variance matrix

b

than 1: Therefore in this context, constrained least-squares is more e¢ cient than unconstrained least-squares. This is consistent with our intuition that imposing a correct restriction (excluding an irrelevant regressor) improves estimation e¢ ciency.