Multiprocessor Considerations for Kernel-Mode Drivers - 8

Multiprocessor Considerations for Kernel-Mode Drivers

Preliminary Version - October 28, 2004

Abstract

Hyper-threading and future technologies mean that all new machines will eventually support more than one processor. Therefore, every Windows driver must be designed to handle the concurrency and synchronization requirements that multiprocessor systems impose and must be thoroughly tested on both single-processor and multiprocessor systems.

This paper is for developers of kernel-mode drivers for the Microsoft® Windows® family of operating systems.

This information applies for the following operating systems: Microsoft Windows 2000 Microsoft Windows XP Microsoft Windows Server™ 2003 Microsoft Windows Vista™

The current version of this paper is maintained on the Web at: http://www.microsoft.com/whdc/driver/kernel/MP_issues.mspx

References and resources discussed here are listed at the end of this paper.

Contents

1 Introduction 3

2 Multiprocessor Support in Windows 3

3 Simultaneous Thread Execution 5

4 Reentrant and Concurrent Routines 7

4.1 Reentrancy 7

4.2 Concurrency 7

5 Synchronizing Access and Enforcing Program Order 15

5.1 The volatile Keyword 15

5.2 Windows Synchronization Mechanisms 18

6 Memory Barriers and Hardware Reordering 19

6.1 Memory Barrier Semantics 19

6.2 Windows Kernel-Mode Memory Barrier Routines 21

6.3 Hardware Reordering on x86, x64, and Itanium Architectures 21

7 Performance and Scalability 24

7.1 Locking Issues 24

7.2 Caching Issues 26

8 Testing 27

8.1 Driver Verifier 28

8.2 Call Usage Verifier 28

8.3 Kernrate and KrView 28

8.4 DevCon 28

9 About NUMA Architectures 29

10 Best Practices for Drivers 30

11 Resources 30

Disclaimer

This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, email address, logo, person, place or event is intended or should be inferred.

© 2004 Microsoft Corporation. All rights reserved.

Microsoft, Windows, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

1Introduction

In the past, the typical Microsoft® Windows® computer had only a single processor. Multiprocessor configurations could be found in high-end servers and computing-intensive labs, but such systems were the exception rather than the rule. As technology improves and prices decrease, desktop applications are becoming more CPU intensive, requiring processing power that only a few years ago was found mainly on servers and in labs.

Hyper-threaded processors, which Windows treats as two CPUs, are already becoming common in machines for home and desktop environments. Soon, most new computers will be multiprocessor systems. All new drivers must be designed and tested for such systems.

Because the Windows kernel is fully preemptible, writing drivers to run on multiprocessor systems is no different from writing drivers to run on single-processor systems. However, errors in synchronization and locking are more likely to occur on multiprocessor systems because code from a single driver can run simultaneously on more than one processor. A driver that has been tested and debugged on single-processor systems may fail when run on a multiple-processor system because of previously undetected bugs.

To write drivers that operate correctly on all Windows platforms, you should be familiar with the following:

• Multiprocessor architectures that Windows supports

• Simultaneous thread execution

• Reentrancy and concurrency of standard driver routines

• Driver synchronization requirements

• Performance and scalability issues

• Tools for testing on multiprocessor hardware

In addition, Microsoft Windows Server™ 2003 includes limited support for cache-coherent non-uniform memory access (ccNUMA) architectures; expanded support is planned for the next client version of Windows, Microsoft Windows Vista™. This paper includes a brief discussion of driver issues for such architectures.

2Multiprocessor Support in Windows

Windows supports symmetric multiprocessor (SMP) architectures. In SMP architectures, all CPUs are identical and have uniform access to memory and I/O control registers. Windows treats hyper-threaded architectures as two-CPU multiprocessors.

Figure 1 shows how CPUs, memory, and devices might be configured on an SMP system.

Figure 1. Organization of Traditional SMP System

On SMP architectures, each CPU has uniform access to memory, so that operations have the same effect regardless of which CPU issues them. Each CPU has its own cache.

By default, Windows assumes that any device can interrupt on any processor, although some chip sets might favor one CPU over another. When the device interrupts, its InterruptService routine runs immediately on the same processor that the device interrupted, and its DpcForIsr routine subsequently runs on the same processor also.

Microsoft provides separate executable images of the Windows kernel and several other system files for single-processor and multiprocessor machines. During installation, the correct files are loaded onto the hardware.

The number of processors supported depends on the edition of Windows, as summarized in Table 1. These numbers represent physical processors, not logical processors. Therefore, a hyper-threaded processor counts as only one processor for licensing purposes, even though the operating system treats it as two processors.

Table 1. Number of Processors Licensed for Windows Editions

Operating System Version	Edition	Number of processors
Windows Server 2003	Web	2
	Enterprise	8
	Standard	4
	Datacenter	32 (32-bit architectures) 64 (64-bit architectures) 128 (in 2 partitions)
Windows XP	Home	1
	Professional	2
Windows 2000	Professional	2
	Server	4
	Advanced Server	8
	Datacenter Server	32