Beginning Python (2005)

Network Programming

How It Works

Getting a message by unique ID requires four IMAP commands. First and second, the client must connect to the server and select a mailbox, just as in the previous IMAP example. Third, the client needs to run a SEARCH command that returns a list of message UIDs. Finally, the client can pass in one of the UIDs to a FETCH command and get the actual message.

The last two steps both go through the IMAP4.uid method; if UIDs weren’t involved, they would use the search and fetch methods, respectively.

Using imaplib to interact with an IMAP server can be a pain, but it’s not as bad as communicating directly with the server.

POP3 servers also support UIDs, though it’s less common for multiple clients to access a single POP3 mailbox simultaneously. A POP3 object’s uidl method will retrieve the UIDs of the messages in its mailbox. You can then pass a UID into any of a POP3 object’s other methods that take message IDs: for instance, retr and top. IMAP’s UIDs are numeric; POP3’s are the “message digests”: hexadecimal signatures derived from the contents of each message.

Secure POP3 and IMAP

Both the POP3 or IMAP examples covered earlier in this section have a security problem: They send send your username and password over the network without encrypting it. That’s why both POP and IMAP are often run atop the Secure Socket Layer (SSL). This is a generic encryption layer also used to secure HTTP connections on the World Wide Web. POP and IMAP servers that support SSL run on different ports from the ones that don’t: The standard port number for POP over SSL is 995 instead of 23, and IMAP over SSL uses port 993 instead of port 143.

If your POP3 or IMAP server supports SSL, you can get an encrypted connection to it by just swapping out the POP3 or IMAP4 class for the POP3_SSL or IMAP4_SSL class. Each SSL class is in the same module and has the same interface as its insecure counterpart but encrypts all data before sending it over the network.

Webmail Applications Are Not E-mail Applications

If you use a webmail system such as Yahoo! Mail or Gmail, you’re not technically using a mail application at all: You’re using a web application that happens to have a mail application on the other side. The scripts in this section won’t help you fetch mail from or send mail through these services, because they implement HTTP, not any of the e-mail protocols (however, Yahoo! Mail offers POP3 access for a fee). Instead, you should look at Chapter 21 for information on how web applications work.

The libgmail project aims to create a Python interface to Gmail, one that can treat Gmail as an SMTP, POP3, or IMAP server. The libgmail homepage is at http://libgmail.sourceforge.net/.

Socket Programming

So far, we’ve concerned ourselves with the protocols and file formats surrounding a single Internet application: e-mail. E-mail is certainly a versatile and useful application, but e-mail-related protocols account for only a few of the hundreds implemented atop the Internet Protocol. Python makes it easier

331

TEAM LinG

Chapter 16

to use the e-mail-related protocols (and a few other protocols not covered in this chapter) by providing wrapper libraries, but Python doesn’t come with a library for every single Internet protocol. It certainly won’t have one for any new protocols you decide to create for your own Internet applications.

To write your own protocols, or to implement your own Python libraries along the lines of imaplib or poplib, you’ll need to go down a level and learn how programming interfaces to IP-based protocols actually work. Fortunately, it’s not hard to write such code: smtplib, poplib, and the others do it without becoming too complicated. The secret is the socket library, which makes reading and writing to a network interface look a lot like reading and writing to files on disk.

Introduction to Sockets

In many of the previous examples, you connected to a server on a particular port of a particular machine (for instance, port 25 of localhost for a local SMTP server). When you tell imaplib or smtplib to connect to a port on a certain host, behind the scenes Python is opening a connection to that host and port. Once the connection is made, the server opens a reciprocal connection to your computer. A single Python “socket” object hides the outgoing and incoming connections under a single interface. A socket is like a file you can read to and write from at the same time.

To implement a client for a TCP/IP-based protocol, you open a socket to an appropriate server. You write data to the socket to send it to the server, and read from the socket the data the server sends you. To implement a server, it’s just the opposite: You bind a socket to a hostname and a port and wait for a client to connect to it. Once you have a client on the line, you read from your socket to get data from the client, and write to the socket to send data back.

It takes an enormous amount of work to send a single byte over the network, but between TCP/IP and the socket library, you get to skip almost all of it. You don’t have to figure out how to get your data halfway across the world to its destination, because TCP/IP handles that for you. Nor need you worry about turning your data into TCP/IP packets, because the socket library handles that for you.

Just as e-mail and the web are the killer apps for the use of the Internet, sockets might be considered the killer app for the adoption of TCP/IP. Sockets were introduced in an early version of BSD UNIX, but since then just about every TCP/IP implementation has used sockets as its metaphor for how to write network programs. Sockets make it easy to use TCP/IP (at least, easier than any alternative), and this has been a major driver of TCP/IP’s popularity.

As a first socket example, here’s a super-simple socket server, SuperSimpleSocketServer.py:

#!/usr/bin/python import socket import sys

if len(sys.argv) < 3:

print ‘Usage: %s [hostname] [port number]’ % sys.argv[0]

sys.exit(1)

hostname = sys.argv[1] port = int(sys.argv[2])

#Set up a standard Internet socket. The setsockopt call lets this

332

TEAM LinG

Network Programming

#server use the given port even if it was recently used by another #server (for instance, an earlier incarnation of #SuperSimpleSocketServer).

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

#Bind the socket to a port, and bid it listen for connections. sock.bind((hostname, port))

sock.listen(1)

print “Waiting for a request.”

#Handle a single request.

request, clientAddress = sock.accept() print “Received request from”, clientAddress

request.send(‘-=SuperSimpleSocketServer 3000=-\n’) request.send(‘Go away!\n’)

request.shutdown(2) #Stop the client from reading or writing anything. print “Have handled request, stopping server.”

sock.close()

This server will serve only a single request. As soon as any client connects to the port to which it’s bound, it will tell the client to go away, close the connection, stop serving requests, and exit.

Try It Out

Connecting to the SuperSimpleSocketServer with Telnet

The telnet program is a very simple client for TCP/IP applications. You invoke it with a hostname and a port; it connects you to that port; and then you’re on your own. Anything you type is sent over a socket to the server, and anything the server sends over the socket is printed to your terminal. Telnet is included as a command-line program in Windows, Mac OS X, and Unix installations, so you shouldn’t have trouble getting it.

Because our example socket server doesn’t really do anything, there’s little point in writing a custom client for it. To test it out, just start up the server:

$ python SuperSimpleSocketServer.py localhost 2000

Waiting for a request.

Then, in a separate terminal, telnet into the server:

$ telnet localhost 2000 Trying 127.0.0.1...

Connected to rubberfish. Escape character is ‘^]’.

-=SuperSimpleSocketServer 3000=- Go away!

Connection closed by foreign host.

Go back to the terminal on which you ran the server and you should see output similar to this:

Received request from (‘127.0.0.1’, 32958)

Have handled request, stopping server.

333

TEAM LinG

Chapter 16

How It Works

When you started the SuperSimpleSocketServer, you bound the process to port 2000 of the “localhost” hostname. When that script called socket.accept, it stopped running and began to “block” on socket input, waiting for someone to connect to the server.

When your telnet command opens up a TCP/IP connection to the SuperSimpleSocketServer, the socket.accept method call returns from its wait. At last, someone has connected to the server! The return values of socket.accept give the server the tools it needs to communicate with this client: a socket object and a tuple describing the network address of the client. The server sends some data to the client through the socket and then shuts down. No further socket connections will be accepted.

The only obscure thing here is that client address tuple: (‘127.0.0.1’, 32958). You’ve seen 127.0.0.1 already; it is a special IP address that refers to “this computer”: it’s the IP address equivalent of “localhost”. A connection to the server from 127.0.0.1 means that the client is coming from the same computer that’s running the server. If you’d telnetted in from another machine, that machine’s IP address would have shown up instead.

32958 is a temporary or “ephemeral” port number for the client. Recall that what looks like a single, bidirectional “socket” object actually contains two unidirectional connections: one from the client to the server and one from the server to the client. Port 2000 on localhost, the port to which the server was bound when we started it up, is the destination for all client data (not that this client got a chance to send any data). The data sent by the server must also have a destination hostname and port, but not a predefined one. While a server port is usually selected by the human in charge of the server, ephemeral ports are selected by the client’s operating system. Run this exercise again and you’ll see that each individual TCP/IP connection is given a different ephemeral port number.

Binding to an External Hostname

If you tried to telnet into the SuperSimpleSocketServer from another machine, as suggested above, you might have noticed that you weren’t able to connect to the server. If so, it may be because you started the server by binding it to localhost. The special “localhost” hostname is an internal hostname, one that can’t be accessed from another machine. After all, from someone else’s perspective, “localhost” means their computer, not yours.

This is actually very useful because it enables you to test out the servers from this chapter (and Chapter 21) without running the risk of exposing your computer to connections from the Internet at large (of course, if you are running these servers on a multiuser machine, you might have to worry about the other users on the same machine, so try to run these on a system that you have to yourself). However, when it comes time to host a server for real, and external connections are what you want, you need to bind your server to an external hostname.

If you can log into your computer remotely via SSH, or you already run a web server, or you ever make a reference to your computer from another one, you already know an external hostname for your computer. On the other hand, if you have a dial-up or broadband connection, you’re probably assigned a hostname along with an IP address whenever you connect to your ISP. Find your computer’s IP address and do a DNS lookup on it to find an external hostname for your computer. If all else fails, you can bind servers directly to your external IP address (not 127.0.0.1, as that will have the same problem as binding to “localhost”).

334

TEAM LinG

Network Programming

If you bind a server to an external hostname and still can’t connect to it from the outside, there may be a firewall in the way. Fixing that is beyond what this book can cover. You should ask your local computer guru to help you with this.

The Mirror Server

Here’s a server that’s a little more complex (though not more useful) and that shows how Python enables you to treat socket connections like files. This server accepts lines of text from a socket, just as a script might on standard input. It reverses the text and writes the reversed version back through the socket, just as a script might on standard output. When it receives a blank line, it terminates the connection:

#!/usr/bin/python import socket

class MirrorServer:

“””Receives text on a line-by-line basis and sends back a reversed version of the same text.”””

def __init__(self, port):

“Binds the server to the given port.”

self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.socket.bind(port)

#Queue up to five requests before turning clients away. self.socket.listen(5)

def run(self):

“Handles incoming requests forever.” while True:

request, client_address = self.socket.accept()

#Turn the incoming and outgoing connections into files. input = request.makefile(‘rb’, 0)

output = request.makefile(‘wb’, 0) l = True

try:

while l:

l = input.readline().strip() if l:

output.write(l[::-1] + ‘\r\n’) else:

#A blank line indicates a desire to terminate the #connection.

request.shutdown(2) #Shut down both reads and writes. except socket.error:

#Most likely the client disconnected. pass

if __name__ == ‘__main__’: import sys

if len(sys.argv) < 3:

print ‘Usage: %s [hostname] [port number]’ % sys.argv[0] sys.exit(1)

hostname = sys.argv[1] port = int(sys.argv[2])

MirrorServer((hostname, port)).run()

335

TEAM LinG

Chapter 16

Try It Out

Mirroring Text with the MirrorServer

As with the SuperSimpleSocketServer, you can use this without writing a specialized client. You can just telnet into the MirrorServer and enter some text. Enter a blank line and the server will disconnect you. In one terminal, start the server:

$ python MirrorServer.py localhost 2000

In another, telnet into the server as a client:

$ telnet localhost 2000 Trying 127.0.0.1...

Connected to rubberfish. Escape character is ‘^]’. Hello.

.olleH

Mirror this text! !txet siht rorriM

Connection closed by foreign host.

$

The Mirror Client

Though you’ve just seen that the mirror server is perfectly usable through telnet, not everyone is comfortable using telnet. What we need is a flashy mirror server client with bells and whistles, so that even networking novices can feel the thrill of typing in text and seeing it printed out backward. Here’s a simple client that takes command-line arguments for the server destination and the text to reverse. It connects to the server, sends the data, and prints the reversed text:

#!/usr/bin/python import socket

class MirrorClient:

“A client for the mirror server.”

def __init__(self, server, port): “Connect to the given mirror server.”

self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.socket.connect((server, port))

def mirror(self, s):

“Sends the given string to the server, and prints the response.” if s[-1] != ‘\n’:

s += ‘\r\n’ self.socket.send(s)

#Read server response in chunks until we get a newline; that #indicates the end of the response.

buf = [] input = ‘’

336

TEAM LinG

Network Programming

while not ‘\n’ in input: try:

input = self.socket.recv(1024) buf.append(input)

except socket.error: break

return ‘’.join(buf)[:-1]

def close(self):

self.socket.send(‘\r\n’) #We don’t want to mirror anything else. self.socket.close()

if __name__ == ‘__main__’: import sys

if len(sys.argv) < 4:

print ‘Usage: %s [host] [port] [text to be mirrored]’ % sys.argv[0] sys.exit(1)

hostname = sys.argv[1] port = int(sys.argv[2]) toMirror = sys.argv[3]

m = MirrorClient(hostname, port) print m.mirror(toMirror) m.close()

The mirror server turns its socket connection into a pair of files, but this client reads from and writes to the socket directly. There’s no compelling reason for this; I just felt this chapter should include at least one example that used the lower-level socket API. Note how the server response is read in chunks, and each chunk is scanned for the newline character that indicates the end of the response. If this example had created a file for the incoming socket connection, that code would have been as simple as calling input.readline.

It’s important to know when the response has ended, because calling socket.recv (or input.readline) will block your process until the server sends some more data. If the server is waiting for more data from the client, your process will block forever. (See the sections below on select “Single-Threaded Multitasking with select” and “The Twisted Framework” for ways of avoiding this problem.)

SocketServer

Sockets are very useful, but Python isn’t satisfied with providing the same C-based socket interface you can get with most languages on most operating systems. Python goes one step further and provides SocketServer, a module full of classes that let you write sophisticated socket-based servers with

very little code.

Most of the work in building a SocketServer is defining a request handler class. This is a subclass of the SocketServer module’s BaseRequestHandler class, and the purpose of each request handler object is to handle a single client request for as long as the client is connected to the server. This is implemented in the handler’s handle method. The handler may also define per-request setup and tear-down code by overriding setup and finish.

337

TEAM LinG

Chapter 16

The methods of a BaseRequestHandler subclass have access to the following three members:

request: A socket object representing the client request: the same object obtained from socket.accept in the MirrorServer example.

client_address: A 2-tuple containing the hostname and port to which any data the server outputs will be sent. The other object obtained from socket.accept in the MirrorServer example.

server: A reference to the SocketServer that created the request handler object.

By subclassing StreamRequestHandler instead of BaseRequestHandler, you also get access to the file-like objects that let you read from and write to the socket connection. BaseRequestHandler gives you access to two other members:

rfile: The file corresponding to the data that comes in over the socket (from the client if you’re writing a server, from the server if you’re writing a client). Equivalent to what you get when you call request.makefile(‘rb’).

wfile: The file corresponding to the data that you send over the socket (to the client if you’re writing a server, to the server if you’re writing a client). Equivalent to what you get when you call request.makefile(‘wb’).

By rewriting the MirrorServer as a SocketServer server (specifically, a TCPServer), you can eliminate a lot of code to do with socket setup and teardown, and focus on the arduous task of reversing text. Here’s MirrorSocketServer.py:

#!/usr/bin/python import SocketServer

class RequestHandler(SocketServer.StreamRequestHandler): “Handles one request to mirror some text.”

def handle(self):

“””Read from StreamRequestHandler’s provided rfile member, which contains the input from the client. Mirror the text and write it to the wfile member, which contains the output to be sent to the client.”””

l = True while l:

l = self.rfile.readline().strip() if l:

self.wfile.write(l[::-1] + ‘\n’)

if __name__ == ‘__main__’: import sys

\if len(sys.argv) < 3:

print ‘Usage: %s [hostname] [port number]’ % sys.argv[0] sys.exit(1)

hostname = sys.argv[1] port = int(sys.argv[2])

SocketServer.TCPServer((hostname, port), RequestHandler).serve_forever()

338

TEAM LinG

Network Programming

Almost all of the socket-specific code is gone. Whenever anyone connects to this server, the TCPServer class will create a new RequestHandler with the appropriate members and call its handle method to handle the request.

The MirrorClient we wrote earlier will work equally well with this server, because across the network both servers take the same input and yield the same output. The same principle applies as when you change the implementation of a function in a module to get rid of redundant code but leave the interface the same.

Multithreaded Servers

One problem with both of these implementations of the mirror server is that only one client at a time can connect to a running server. If you open two telnet sessions to a running server, the second session won’t finish connecting until you close the first one. If real servers worked this way, nothing would ever get done. That’s why most real servers spawn threads or subprocesses to handle multiple connections.

The SocketServer module defines two useful classes for handling multiple connections at once:

ThreadingMixIn and ForkingMixIn. A SocketServer class that subclasses ThreadingMixIn will automatically spawn a new thread to handle each incoming request. A subclass of ForkingMixIn will automatically fork a new subprocess to handle each incoming request. I prefer ThreadingMixIn because threads are more efficient and more portable than subprocesses. It’s also much easier to write

code for a thread to communicate with its parent than for a subprocess to communicate with its parent.

See Chapter 9 for an introduction to threads and subprocesses.

Here’s MultithreadedMirrorServer.py, a multithreaded version of the MirrorSocketServer. Note that it uses the exact same RequestHandler definition as MirrorSocketServer.py. The difference here is that instead of running a TCPServer, we run a ThreadingTCPServer, a standard class that inherits both from ThreadingMixIn and TCPServer:

#!/usr/bin/python import SocketServer

class RequestHandler(SocketServer.StreamRequestHandler): “Handles one request to mirror some text.”

def handle(self):

“””Read from StreamRequestHandler’s provided rfile member, which contains the input from the client. Mirror the text and write it to the wfile member, which contains the output to be sent to the client.”””

l = True while l:

l = self.rfile.readline().strip() if l:

self.wfile.write(l[::-1] + ‘\n’)

if __name__ == ‘__main__’: import sys

if len(sys.argv) < 3:

print ‘Usage: %s [hostname] [port number]’ % sys.argv[0]

339

TEAM LinG

Chapter 16

sys.exit(1) hostname = sys.argv[1] port = int(sys.argv[2])

server = SocketServer.ThreadingTCPServer((hostname, port), RequestHandler) server.serve_forever()

With this server running, you can run a large number of telnet sessions and MirrorClient sessions in parallel. ThreadingMixIn hides the details of spawning threads, just as TCPServer hides the details of sockets. The goal of all these helper classes is to keep your focus on what you send and receive over the network.

The Python Chat Server

For the mirror server, the capability to support multiple simultaneous connections is useful but it doesn’t change what the server actually does. Each client interacts only with the server, and not even indirectly with the other clients. This model is a popular one; web servers and mail servers use it, among others.

There is another type of server, though, that exists to connect clients to each other. For many applications, it’s not the server that’s interesting: it’s who else is connected to it. The most popular applications of this sort are online chat rooms and games. In this section, you’ll design and build a simple chat server and client.

Perhaps the original chat room was the (non-networked) Unix wall command, which enables you to broadcast a message to everyone logged in on a Unix system. Internet Relay Chat, invented in 1988 and described in RFC 1459, is the most popular TCP/IP-based chat room software. The chat software you write here will have some of the same features as IRC, although it won’t be compatible with IRC.

Design of the Python Chat Server

In IRC, a client that connects to a server must provide a nickname: a short string identifying the person who wants to chat. A nickname must be unique across a server so that users can’t impersonate one another. Our server will carry on this tradition.

An IRC server provides an unlimited number of named channels, or rooms, and each user can join any number of rooms. Our server will provide only a single, unnamed room, which all connected users will inhabit.

Entering a line of text in an IRC client broadcasts it to the rest of your current room, unless it starts with the slash character. A line starting with the slash character is treated as a command to the server. Our server will act the same way.

IRC implements a wide variety of server commands: For instance, you can use a server command to change your nickname, join another room, send a private message to another user, or try to send a file to another user.

340

TEAM LinG