Kluwer - Handbook of Biomedical Image Analysis Vol

Three-Dimensional Registration Methods

147

Maes, F., Suetens, P., Vandermeulen, D., vandenElsen, P. A., Napel, S., Sumanaweera, T. S., Harkness, B., Hemler, P. F., Hill, D. L. G., Hawkes, D. J., Studholme, C., Maintz, J. B. A., Viergever, M. A., Malandain, G., Pennec, X., Noz, M. E., Maguire, G. Q., Pollack, M., Pelizzari, C. A., Robb, R. A., Hanson, D., and Woods, R. P., Comparison and evaluation of retrospective intermodality brain image registration techniques, Journal of Computer Assisted Tomography, Vol. 21, No. 4, pp. 554–566, 1997.

[32]Malone, S., Crook, J. M., Kendal, W. S., and Szanto, J., Respiratoryinduced prostate motion: Quantification and characterization, International Journal of Radiation Oncology Biology Physics, Vol. 48, No. 1, pp. 105–109, 2000.

[33]Lee, Z., Sodee, D. B., Duerk, J. L., Nelson, A. D., and Berridge, M. S., Automatic registration of SPECT-MRI in the pelvis, Journal of Nuclear Medicine, Vol. 41, No. 5, pp. 232–232, 2000.

[34]Lee, Z., Sodee, D. B., Faulhaber, P. F., Lancaster, T. L., MacLennan, G. T., and Wilson, D. L., Comparison of SPECT and PET imaging for prostate cancer with histological correlation, Journal of Nuclear Medicine, Vol. 42, No. 5, pp. 1226–1226, 2001.

[35]Pluim, J. P. W., Maintz, J. B. A., and Viergever, M. A., Image registration by maximization of combined mutual information and gradient information, IEEE Transactions on Medical Imaging, Vol. 19, No. 8, pp. 809–814, 2000.

[36]Lau, Y. H., Braun, M., and Hutton, B. F., Non-rigid image registration using a median-filtered coarse-to-fine displacement field and a symmetric correlation ratio, Physics in Medicine and Biology, Vol. 46, No. 4, pp. 1297–1319, 2001.

[37]Studholme, C., Hill, D. L. G., and Hawkes, D. J., Automated threedimensional registration of magnetic resonance and positron emission tomography brain images by multiresolution optimization of voxel similarity measures, Medical Physics, Vol. 24, No. 1, pp. 25–35, 1997.

[38]Rohr, K., Stiehl, H. S., Sprengel, R., Buzug, T. M., Weese, J., and Kuhn, M. H., Landmark-based elastic registration using approximating

148	Fei, Suri, and Wilson

thin-plate splines, IEEE Transactions on Medical Imaging, Vol. 20, No.

6, pp. 526–534, 2001.

[39]Maurer, C. R., Hill, D. L. G., Martin, A. J., Liu, H. Y., McCue, M., Rueckert, D., Lloret, D., Hall, W. A., Maxwell, R. E., Hawkes, D. J., and Truwit,

C.L., Investigation of intraoperative brain deformation using a 1.5-t interventional MR system: Preliminary results, IEEE Transactions on Medical Imaging, Vol. 17, No. 5, pp. 817–825, 1998.

[40]Davis, M. H., Khotanzad, A., Flamig, D. P., and Harms, S. E., A physicsbased coordinate transformation for 3-D image matching, IEEE Transactions on Medical Imaging, Vol. 16, No. 3, pp. 317–328, 1997.

[41]Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D. L. G., Leach, M. O., and Hawkes, D. J., Nonrigid registration using free-form deformations: Application to breast MR images, IEEE Transactions on Medical Imaging, Vol. 18, No. 8, pp. 712–721, 1999.

[42]Denton, E. R. E., Sonoda, L. I., Rueckert, D., Rankin, S. C., Hayes, C., Leach, M. O., Hill, D. L. G., and Hawkes, D. J., Comparison and evaluation of rigid, affine, and nonrigid registration of breast MR images, Journal of Computer Assisted Tomography, Vol. 23, No. 5, pp. 800–805, 1999.

[43]Meyer, C. R., Boes, J. L., Kim, B., Bland, P. H., Zasadny, K. R., Kison,

P.V., Koral, K., Frey, K. A., and Wahl, R. L., Demonstration of accuracy and clinical versatility of mutual information for automatic multimodality image fusion using affine and thin-plate spline warped geometric deformations, Medical Image Analysis, Vol. 1, No. 3, pp. 195–206, 1996.

[44]Krucker, J. F., Meyer, C. R., LeCarpentier, G. L., Fowlkes, J. B., and Carson, P. L., 3D spatial compounding of ultrasound images using image-based nonrigid registration, Ultrasound in Medicine and Biology, Vol. 26, No. 9, pp. 1475–1488, 2000.

[45]Kimiaei, S., Noz, M., Jonsson, E., Crafoord, J., and Maguire, G. Q., Evaluation of polynomial image deformation using anatomical landmarks for matching of 3D-abdominal MR-images and for atlas construction, IEEE Transactions on Nuclear Science, Vol. 46, No. 4, pp. 1110–1113, 1999.

Three-Dimensional Registration Methods

149

[46]Kim, B., Boes, J. L., Frey, K. A., and Meyer, C. R., Mutual information for automated unwarping of rat brain autoradiographs, Neuroimage, Vol. 5, No. 1, pp. 31–40, 1997.

[47]Bharatha, A., Hirose, M., Hata, N., Warfield, S. K., Ferrant, M., and Zou, K. H., Three-dimensional finite element-based deformable registration of preand intraoperative prostate imaging, Radiology, Vol. 221, pp. 224–224, 2001.

[48]Goshtasby, A., Registration Of Images With Geometric Distortions, IEEE Transactions on Geoscience and Remote Sensing, Vol. 26, No. 1, pp. 60–64, 1988.

[49]Bookstein, F. L., Thin-plate splines and the atlas problem for biomedical images, Lecture Notes in Computer Science, Vol. 511, pp. 326–342, 1991.

[50]Lee, Z., Kemper, C., uzic, R. F., erridge, M. S., and ilson, D. L., Quantitative pulmonary imaging based on PET-CT co-registration with warping, Journal of Nuclear Medicine, Vol. 42, No. Suppl, pp. 10–11, 2001.

[51]Thacker, N. A., Jackson, A., Moriarty, D., and Vokurka, E., Improved quality of re-sliced MR images using re-normalized sinc interpolation, Journal of Magnetic Resonance Imaging, Vol. 10, No. 4, pp. 582–588, 1999.

Chapter 4

Stereo and Temporal Retinal Image Registration by Mutual Information Maximization

Xiao-Hong Zhu,1 Cheng-Chang Lu,2 and Yang-Ming Zhu3

4.1 Introduction

Image registration is the process of determining a one-to-one mapping between the coordinates in one space and those in another such that points in the two spaces that correspond to the same point are mapped to each other. The mappings, which are also called transformation, are two dimensional (2D) for 2D spaces and three dimensional (3D) for 3D spaces. The simplest examples are the rigid-body transformations, which are transformations in which the distances among all points are preserved. The rigid-body transformations are typically used to compensate for different imaging orientations of rigid objects. Other possible transformations include affine where parallel lines remain parallel, projective that is from 3D to 2D, and warping which is nonlinear in general.

The most prominent application of image registration is multimodality medical image fusion. Different image modalities can provide different information about the imaged organs, and most of the time this information is complementary in nature. For example, computer tomography (CT) image reveals the anatomical structure of the organ, magnetic resonance (MR) image is capable of

1 X.H. Zhu, Zircoa, Inc., 31501 Solon Rd., Solon, OH 44139

2 C.C Lu, Dept of Computer Science, Kent State Univ., Kent, OH 44242

3 Y.M. Zhu, PET Engineering, Philips Medical Systems, Cleveland, OH 44143

151

152	Zhu, Lu, and Zhu

depicting soft-tissue anatomy, and photon emission tomography (PET) and single photon emission computed tomography (SPECT) show the functional property of the organ under study. Before different types of images are fused, these images have to be registered. After registration, the skeletal structures and areas of contrast enhancement seen in CT images can be overlaid on MR images, and likewise, functional lesions detected with PET or SPECT can be viewed in the context of anatomy imaged with CT or MR. Image registration can also be applied to multiple data sets obtained with the same modality at different times for the purpose of quantitative comparison, which increases the precision of treatment monitoring with serial images.

Another major biomedical application of image registration is to retinal image matching. Retinal or fundus photographs are standard diagnostic tools in ophthalmology. In the follow-up of age-related macular degeneration, drusen deposits need to be tracked and compared (see Sbeh et al. [1], Rapantzikos et al. [2]). Screening of diabetic retinopathy can involve a follow-up over many years (see [3, 4]). To determine the progression of glaucoma, a series of optic- nerve-head topographies are assessed and compared [5]. Serial photographs of the flow of fluorescein dye are also used to determine areas of ischemia, hemorrhaging, neovascularization, and occlusions in diseases such as diabetic retinopathy (see [5]). A noise reduction technique is reported for laser scanning ophthalmoscope using image registration [6]. Multimodality registration is also performed in retinal imaging. In glaucoma diagnosis, for example, the optic-nerve-head is assessed from color stereo images and the nerve fiber layer is assessed from red-free images [7]. In retinal analysis, two types of images, fluorescein images (angiographic images taken under ultra violet light after injection of fluorescein dye) and green images (taken under natural light with a green filter), are often used for the diagnosis of the gravity of diabetic retinopathy [3]. Physicians often use more than one image to identify a lesion and assess its seriousness, or base their diagnosis on detection of various image features in different modality images. To make this comparison and assessment objective, it is necessary that all images be registered.

The research of image registration has a relatively short history. Due to diverse applications, many registration algorithms have been developed from different perspectives. Brown summarized the research work before 1992, mostly for 2D-2D registration [8]. Since the most important and fruitful application of image registration is in medical imaging, several authors reviewed registration

Stereo and Temporal Retinal Image Registration

153

algorithms from a medical imaging perspective. Gerlot-Chiron and Bizais have presented a unified description of existing registration methods [9]. Maurer and Fitzpatrick later adopted a similar scheme when they reviewed the registration algorithms within the neurosurgery context [9]. Van den Elsen et al. reviewed and classified medical image registration algorithms [10]. Their classification criteria have been augmented and detailed by Maintz and Viergever recently [11]. In addition to various survey articles and book chapters (e.g. Fitzpatrick et al. [12]), a monograph on image registration has also been published [13]. For further elaboration, the reader is advised to refer to the original surveys, book chapters, and monograph.

The increasingly complex schemes for classification reflect the sheer amount of literature on image registration methodologies. It is impossible for us to detail these algorithms here. However, we do want to point out a recent trend in image registration research and practice, i.e., the voxel property-based registration methods have become increasingly popular. Compared to other registration algorithms, the voxel property-based methods using the full image content offer several advantages: they work on the image gray-value without any prior data reduction; they can be automated and the results are objective; they require no segmentation and involve little or no user interaction.

To this category, various paradigms have been reported, including crosscorrelation in spatial or (Fourier) transformed domain, minimization of variance of intensity ratios, minimization of variance of gray values within segments, histogram clustering and minimization of histogram dispersion, minimization of the joint histogram entropy of different images, and maximization of mutual information, among many others. Studholme et al. [14] compared five similaritybased algorithms and Fitzpatrick et al. [15] compared 16 of these algorithms. The reports from various independent groups confirm that the mutual information maximization approach to image registration is one of the most robust and has superior performance.

Mutual information image registration was independently proposed by Maes et al. [16] and Wells et al. [17]. However, their development is a natural consequence of early effect on the analysis of voxel value joint histogram (see [18]). Hill et al. [19] used third-order moments of the joint histogram as well as other measures to characterize the clustering of the joint histogram at registration. Collignon et al. [20, 21] used joint entropy as a criterion for registration, but reported that it had a small capture range, i.e., only when the initial

154	Zhu, Lu, and Zhu

position of two images is sufficiently close to the optimal alignment can they be registered well. To increase the capture range, naturally the next step is to exploit mutual information. Due to its super performance, mutual information becomes the first choice for automatic image registration. The advancement on mutual information registration has been reviewed recently by Pluim et al. [22].

4.1.1 Review of Retinal Image Registration

Retinal image registration is the main focus of this chapter. This registration generally involves large x translation, due to changes between sittings and smaller y translation from changes in position of the chin cup. Rotation occurs due to tilting of the head and through ocular torsion, and scaling is caused by changes in the distance between the camera and the head, due to equipment changes or differing head positions (see [5, 23]). This section reviews some registration methods as applied to retinal images. By no means is this review complete. The interested readers may refer to [3, 5, 23] and references therein for more related work.

Peli et al. [24] reported on a correlation method that preprocesses the images using an adaptive threshold procedure to select vessel points. The normalized sum of differences is then calculated with these vessel points. Via an exhaustive search, this method produces pixel-level registration for x- and y-translation only. It is not robust toward large changes in image intensity and white noise. The absolute value of difference of pixel intensities was also used as a comparison measure for retinal image registration. The images can be processed twice, using optic discs as features for coarse alignment and the blood vessels as features for fine alignment [25].

If one image is a scaled, rotated, and translated version of another image, then the Fourier transform of that image is a scaled and rotated version of the Fourier transform of the other image. Thus, image registration can also be done in Fourier transformed space. Cideciyan et al. [26] computed the scaling and rotation differences of the Fourier transformed images by cross-correlation. These results are then used to transform one image in spatial domain. The final translation differences in spatial domain are then found via cross-correlation. When the images are taken at different times, where the translation difference and the image intensity difference may be large, this approach is problematic. This approach is not applicable to multimodality image registration.

Stereo and Temporal Retinal Image Registration

155

Hart et al. [27] exploited a blood vessel filter to detect the ends of blood vessel segments and then used them as control points to register retinal images. A specific scheme was designed to eliminate erroneous point pairs until either there are four control points left or the mean square error given by the least square fitting drops below five pixels. This method has some problems as pointed out by Ritter et al. [5].

Ritter et al. [5] recently applied the mutual information maximization to retinal image registration. To find a global optimum, simulated annealing is used in the multiresolution optimization. Although this method can successfully find the global optimum registration with translation, rotation, and scaling, it is time consuming. To assess the accuracy of registration, Ritter et al. compared the registration results against the solutions obtained by an exhaustive search. This comparison has an intrinsic drawback. What they studied is how the simulated annealing behaves which is an implementation artifact, not how the mutual information maximization behaves as a registration criterion.

Matsopoulos et al. [28] used matched filters (see [29]) to segment the vessel trees and registered the segmented trees automatically. To ensure that a global optimal registration is found, simulated annealing and genetic algorithms were employed. They also studied the suitability and efficiency of different image transformation models. The criterion used in the optimization is a correlation function defined on the segmented, binary images.

Zana et al. [3] reported on a multimodal retinal registration scheme based on vessels detection and Hough transform. The vascular tree is segmented first, and then the bifurcation points are detected. Those tree and points are features used to register the images. Although their algorithm is attractive, it involves a fair amount of user interaction in the preprocessing and in the final registration selection (the solution given by Bayesian selection is not necessarily the best). Laliberte et al. developed a similar technique that also used the blood vessel bifurcation points, but did not need the assumption of a Gaussian shape vessel intensity profile which is inappropriate for low resolution optical images (see [23]). In spite of about 10 adjustable parameters in the algorithm, it seems that the success rate for this latter method is low (36 out of 61 pairs). Can et al. [30] developed a hierarchical scheme to match the feature points in two images, using a progressively complex transformation model and a reduced set of matching points. This algorithm is attractive when one builds the retinal map since warping is generally required.

156	Zhu, Lu, and Zhu

This chapter reports on our development of an object-oriented software system for automatic retinal image registration by mutual information maximization. For maximum portability the software is written in Java which “is written once, but runs everywhere”, using MVC (model-view-control) framework. We use the simplex downhill method (see [31]) as the optimization algorithm which is easy to implement, and is quick in practice. We demonstrate that this algorithm registers temporal and stereo retinal image pairs of four patients with a very high success rate (86%), a satisfactory registration accuracy compared to point matching results, and within a clinically acceptable time (12 ± 3 sec.).

4.2Registration by Mutual Information Maximization

4.2.1 Mutual Information

For two random variables A and B, the mutual information is

		pAB(a, b)
I( A, B) =	pAB(a, b) log	pA(a) pB(b)	,
A	B

where pAB(a, b) is the joint probability density function (pdf), and pA(a) and pB(b) are marginal pdfs, Gonzalez et al. [32]. I( A, B) is related to entropy [H( A),

H(B)], conditional entropy [H( A|B), H(B| A)], and joint entropy [H( A, B)] by

I( A, B) = H( A) + H(B) − H( A, B)

=H( A) − H( A|B)

=H(B) − H(B| A).

Mutual information measures the interdependence of two random variables. If two variables are independent, then their joint pdf is the product of their marginal pdfs, i.e., pAB(a, b) = pA(a) pB(b). Substituting this into the definition of mutual information one gets zero. That is to say, the mutual information is minimal. On the other hand, if two random variables are related by a one-to- one mapping, the mutual information is maximal. In fact, in the latter case,

H( A) = H(B).

Stereo and Temporal Retinal Image Registration

157

Two images involved in registration are called reference and floating images. The floating image has undergone scaling, rotation, and translation to match the reference image. In the mutual information image registration context, we treat the voxel values a and b at corresponding points in two images that are to be registered as random variables A and B. Mutual information measures the interdependency between the reference image and floating image. We assume that the mutual information of random variables A and B has a maximum value when two images are registered (i.e., maximum interdependency). That is, the uncertainty of one image given another image is minimized, and we have more confidence in using one image to interpret another. Note that, the voxel values a and b are related by the registration transformation T . The mutual information registration states that the images are registered under transformation T for which I( A, B) is maximal.

If both marginal pdfs are independent of the registration parameters (i.e., no matter what T is, they would not change much), then mutual information maximization is reduced to minimization of joint entropy. If either pdf is independent of the registration parameters, which is the case when one image geometrically contains another image, maximization of mutual information is reduced to minimization of conditional entropy. However, if both images only partially overlap, which is very likely during the optimization as we will see later, the overlap will change as the transformation changes and both marginal entropies generally depend on the transformation. Mutual information takes overlapping explicitly into account.

Sometimes retinal images are not grayscale images. They have RGB color channels. One has three options to handle this multichannel registration. (1) Define the mutual information for each channel and maximize the sum of these mutual information values. (2) Convert the color images to luminance images and then register the converted images. (3) Pick a color channel and register this channel and other two channels are presumably registered once that selected channel has been registered. Since the green channel has the highest contrast, it can be registered first if one chooses the third option.

In Fig. 4.1 we illustrate the procedures involved in the mutual information maximization approach to image registration. In the following sections we discuss what kind of transformation we pose, how to compute the mutual information, how to update the transform and what is the criterion for optimal transform, etc.