An Introduction to Statistical Signal Processing

An Introduction to Statistical Signal Processing

f−1(F )

f

F

Pr(f F ) = P ({ω : ω F }) = P (f−1(F ))

May 5, 2000

ii

An Introduction to Statistical Signal Processing

Robert M. Gray

and

Lee D. Davisson

Information Systems Laboratory

Department of Electrical Engineering

Stanford University

and

Department of Electrical Engineering and Computer Science

University of Maryland

iv

c 1999 by the authors.

v

to our Families

vi

Contents

Preface				xi
Glossary				xv
1	Introduction			1
2	Probability			11
	2.1	Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . .		11
	2.2	Spinning Pointers and Flipping Coins . . . . . . . . . . . .		15
	2.3	Probability Spaces . . . . . . . . . . . . . . . . . . . . . . .		23
		2.3.1	Sample Spaces . . . . . . . . . . . . . . . . . . . . .	28
		2.3.2	Event Spaces . . . . . . . . . . . . . . . . . . . . . .	31
		2.3.3	Probability Measures . . . . . . . . . . . . . . . . . .	42
	2.4	Discrete Probability Spaces . . . . . . . . . . . . . . . . . .		45
	2.5	Continuous Probability Spaces . . . . . . . . . . . . . . . .		56
	2.6	Independence . . . . . . . . . . . . . . . . . . . . . . . . . .		70
	2.7	Elementary Conditional Probability . . . . . . . . . . . . .		71
	2.8	Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .		75
3	Random Objects			85
	3.1	Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . .		85
		3.1.1	Random Variables . . . . . . . . . . . . . . . . . . .	85
		3.1.2	Random Vectors . . . . . . . . . . . . . . . . . . . .	89
		3.1.3	Random Processes . . . . . . . . . . . . . . . . . . .	93
	3.2	Random Variables . . . . . . . . . . . . . . . . . . . . . . .		95
	3.3	Distributions of Random Variables . . . . . . . . . . . . . .		104
		3.3.1	Distributions . . . . . . . . . . . . . . . . . . . . . .	104
		3.3.2	Mixture Distributions . . . . . . . . . . . . . . . . .	108
		3.3.3	Derived Distributions . . . . . . . . . . . . . . . . .	111
	3.4	Random Vectors and Random Processes . . . . . . . . . . .		115
	3.5	Distributions of Random Vectors . . . . . . . . . . . . . . .		117

vii

viii

CONTENTS

3.5.1 Multidimensional Events . . . . . . . . . . . . . . . 118

3.5.2 Multidimensional Probability Functions . . . . . . . 119

3.5.3Consistency of Joint and Marginal Distributions . . 120

3.6	Independent Random Variables . . . . . . . . . . . . . . . .		127
	3.6.1	IID Random Vectors . . . . . . . . . . . . . . . . . .	128
3.7	Conditional Distributions . . . . . . . . . . . . . . . . . . .		129
	3.7.1	Discrete Conditional Distributions . . . . . . . . . .	130
	3.7.2	Continuous Conditional Distributions . . . . . . . .	131
3.8	Statistical Detection and Classification . . . . . . . . . . . .		134
3.9	Additive Noise . . . . . . . . . . . . . . . . . . . . . . . . .		137
3.10	Binary Detection in Gaussian Noise . . . . . . . . . . . . .		144
3.11	Statistical Estimation . . . . . . . . . . . . . . . . . . . . .		146
3.12	Characteristic Functions . . . . . . . . . . . . . . . . . . . .		147
3.13	Gaussian Random Vectors . . . . . . . . . . . . . . . . . . .		152
3.14	Examples: Simple Random Processes . . . . . . . . . . . . .		154
3.15	Directly Given Random Processes . . . . . . . . . . . . . .		157
	3.15.1	The Kolmogorov Extension Theorem . . . . . . . . .	157
	3.15.2	IID Random Processes . . . . . . . . . . . . . . . . .	158
	3.15.3	Gaussian Random Processes . . . . . . . . . . . . . .	158
3.16	Discrete Time Markov Processes . . . . . . . . . . . . . . .		159
	3.16.1	A Binary Markov Process . . . . . . . . . . . . . . .	159
	3.16.2	The Binomial Counting Process . . . . . . . . . . . .	162
	3.16.3	Discrete Random Walk . . . . . . . . . . . . . . . .	165
	3.16.4	The Discrete Time Wiener Process . . . . . . . . . .	166
	3.16.5	Hidden Markov Models . . . . . . . . . . . . . . . .	167
3.17	Nonelementary Conditional Probability . . . . . . . . . . .		168
3.18	Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .		170
4 Expectation and Averages			187
4.1	Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . .		187
4.2	Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . .		190
	4.2.1	Examples: Expectation . . . . . . . . . . . . . . . .	192
4.3	Functions of Several Random Variables . . . . . . . . . . . .		200
4.4	Properties of Expectation . . . . . . . . . . . . . . . . . . .		200

4.5Examples: Functions of Several Random Variables . . . . . 203

	4.5.1	Correlation . . . . . . . . . . . . . . . . . . . . . . .	203
	4.5.2	Covariance . . . . . . . . . . . . . . . . . . . . . . .	205
	4.5.3	Covariance Matrices . . . . . . . . . . . . . . . . . .	206
	4.5.4	Multivariable Characteristic Functions . . . . . . . .	207
	4.5.5	Example: Di erential Entropy of a Gaussian Vector	209
4.6	Conditional Expectation . . . . . . . . . . . . . . . . . . . .		210
4.7	Jointly Gaussian Vectors . . . . . . . . . . . . . . . . . . .		213

CONTENTS		ix
4.8	Expectation as Estimation . . . . . . . . . . . . . . . . . . .	216
4.9	Implications for Linear Estimation . . . . . . . . . . . . .	222
4.10	Correlation and Linear Estimation . . . . . . . . . . . . . .	224
4.11	Correlation and Covariance Functions . . . . . . . . . . . .	231
4.12	The Central Limit Theorem . . . . . . . . . . . . . . . . .	235
4.13	Sample Averages . . . . . . . . . . . . . . . . . . . . . . . .	237
4.14	Convergence of Random Variables . . . . . . . . . . . . . .	239
4.15	Weak Law of Large Numbers . . . . . . . . . . . . . . . . .	244
4.16	Strong Law of Large Numbers . . . . . . . . . . . . . . . .	246
4.17	Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . .	251
4.18	Asymptotically Uncorrelated Processes . . . . . . . . . . . .	256
4.19	Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .	259
5 Second-Order Moments		281
5.1	Linear Filtering of Random Processes . . . . . . . . . . . .	282
5.2	Second-Order Linear Systems I/O Relations . . . . . . . . .	284
5.3	Power Spectral Densities . . . . . . . . . . . . . . . . . . . .	289
5.4	Linearly Filtered Uncorrelated Processes . . . . . . . . . . .	292
5.5	Linear Modulation . . . . . . . . . . . . . . . . . . . . . . .	298
5.6	White Noise . . . . . . . . . . . . . . . . . . . . . . . . . . .	301
5.7	Time-Averages . . . . . . . . . . . . . . . . . . . . . . . . .	305
5.8	Di erentiating Random Processes . . . . . . . . . . . . . .	309
5.9	Linear Estimation and Filtering . . . . . . . . . . . . . . .	312
5.10	Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .	326
6 A Menagerie of Processes		343
6.1	Discrete Time Linear Models . . . . . . . . . . . . . . . . .	344
6.2	Sums of IID Random Variables . . . . . . . . . . . . . . . .	348
6.3	Independent Stationary Increments . . . . . . . . . . . . . .	350
6.4	Second-Order Moments of ISI Processes . . . . . . . . . .	353
6.5	Specification of Continuous Time ISI Processes . . . . . . .	355
6.6	Moving-Average and Autoregressive Processes . . . . . . . .	358
6.7	The Discrete Time Gauss-Markov Process . . . . . . . . . .	360
6.8	Gaussian Random Processes . . . . . . . . . . . . . . . . . .	361
6.9	The Poisson Counting Process . . . . . . . . . . . . . . . .	361
6.10	Compound Processes . . . . . . . . . . . . . . . . . . . . . .	364
6.11	Exponential Modulation . . . . . . . . . . . . . . . . . . .	366
6.12	Thermal Noise . . . . . . . . . . . . . . . . . . . . . . . . .	371
6.13	Ergodicity and Strong Laws of Large Numbers . . . . . . .	373
6.14	Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .	377

x			CONTENTS
A	Preliminaries		389
	A.1	Set Theory . . . . . . . . . . . . . . . . . . . . . .	. . . . . 389
	A.2	Examples of Proofs . . . . . . . . . . . . . . . . . . .	. . . . 397
	A.3	Mappings and Functions . . . . . . . . . . . . . . . .	. . . . 401
	A.4	Linear Algebra . . . . . . . . . . . . . . . . . . . . .	. . . . 402
	A.5	Linear System Fundamentals . . . . . . . . . . . . .	. . . . 405
	A.6	Problems . . . . . . . . . . . . . . . . . . . . . . . .	. . . . 410
B	Sums and Integrals		417
	B.1	Summation . . . . . . . . . . . . . . . . . . . . . . .	. . . . 417
	B.2	Double Sums . . . . . . . . . . . . . . . . . . . . . .	. . . . 420
	B.3	Integration . . . . . . . . . . . . . . . . . . . . . . .	. . . . 421
	B.4	The Lebesgue Integral . . . . . . . . . . . . . . . .	. . . . 423
C	Common Univariate Distributions		427
D	Supplementary Reading		429
Bibliography			434
Index			438