CASE STUDIES: RAILS RUNNING DAILY 463

By logging in to your system using the cookie, either through curl or your browser, you can also test actions that require authentication.

But ab is limited to testing a single action at the time. If you need more realistic stress testing, where multiple threads hit more than one action over a stream of interaction, you want to take a closer look at siege.16 With siege, you can define a configuration for the siege attack of your server. The resulting information can tell you how fast your application runs and also help you find the critical point where the onslaught is simply too much. Not a bad piece of information to have before you experience your first Slashdot effect.

dave> siege -u localhost:3000/store -d0 -r10 -c4

**siege 2.62

**Preparing 4 concurrent users for battle. The server is now under siege...

Transactions:	40	hits
Availability:	100.00	%
Elapsed time:	3.46	secs
Data transferred:	0.17	MB
Response time:	0.34	secs
Transaction rate:	11.56	trans/sec
Throughput:	0.05	MB/sec
Concurrency:	3.93
Successful transactions:	21
Failed transactions:	19
Longest transaction:	0.54
Shortest transaction:	0.20

While the Unix tools ab and siege are great ways to simulate realistic load and give black-box performance scores, they often offer too low a resolution to pinpoint the exact problem or measure that single bottleneck over and over again. That’s where profiling and benchmarking help, as they work down at the method level within our application. See Section 12.7, Profiling and Benchmarking, on page 171, for more information.

22.7 Case Studies: Rails Running Daily

Nothing says “But it does scale!” better than pointing to real applications that have scaled. In this section, we’ll take a quick look at some of the issues facing three real applications that have all been live for some time and are dealing with the scaling issue.

16See http://www.joedog.org/siege/.

Prepared exclusively for Rida Al Barazi

Report erratum

CASE STUDIES: RAILS RUNNING DAILY 464

Basecamp by 37signals (www.basecamphq.com)

Rails was born in Basecamp. It’s a web-based project management tool that customers pay to access on a monthly basis. The Basecamp servers run the project management application for the tens of thousands of people interacting with the system.

The hard part of Basecamp performance tuning is that it’s tough to use caching, as everyone sees a different data set depending on company affiliation and permissions. (On the positive side, you access Basecamp only if you have an account, so there’s no fear of the application being Slashdotted and flooded with large numbers of unauthenticated users.)

Basecamp does about 400,000 dynamic requests per day (everything from seeing the login page to viewing the project dashboard). That’s a sizeable number when nothing can be cached. It’s currently handled by two web/application servers that each run 15 FastCGI processes and 50–100 Apache 1.3.x processes on dual 2.4GHz Xeons with 2GB of RAM. These machines usually sit between 0.5 and 1.5 in load.

The MySQL database server is separate but shared with two other applications from 37signals (Ta-da List and Backpack). There are hundreds of thousands of rows, not millions. The biggest table is about 500,000 rows. Despite serving three applications, the database server usually runs around 0.1 to 0.3 in load—the database is not the bottleneck for Basecamp—which is great news for scaling. Just add another application/web server when the two current ones are not enough, and we’ve added a lot more capacity.

43 Things by the Robot Co-op (www.43things.com)

43 Things is a place to keep track of your goals in life. You can add goals such as “I want to learn Japanese,” keep a weblog around it, and read what others trying to reach the same goal are doing. Some parts of the application require authentication, while large parts of the site are public and accessible to unauthenticated users. These are susceptible to large changes in traffic volume. Fortunately, they’re also cacheable.

This cache is kept primarily in memcached, which is used extensively to improve site performance. Storing user session data in memcached allows either server to handle requests for any user without requiring any type of session stickiness. Results of expensive database queries are stored as marshaled Active Record objects in memcached with appropriate time-to- live stamping too.

Prepared exclusively for Rida Al Barazi

Report erratum

CASE STUDIES: RAILS RUNNING DAILY 465

43 Things processes about 200,000 dynamic requests per day (after three months online) across its two application/web servers and the single dedicated database server (all dual 3GHz Xeons with 2GB RAM). Each Apache 1.3.x web server runs 25 FastCGI processes. Load on the servers rarely exceeds 0.3 and CPU idle time is usually in excess of 80%.

Mortgage processing engine (www.rapidreporting.com)

Rapid Reporting is running their identity and income verification engine on top of a Rails system. It’s used by roughly 80% of the top 1000 mortgage underwriters in the United States and is built to handle 2 million mortgage application transactions per month.

During their initial tests to assert whether Rails was a usable platform for their application, they did 3,000 requests per second on a simple test application across their 10 machine cluster. The real application processes somewhere around 300 requests/second on a similar setup, performing interesting business logic. Processing mortgages means following GLBA compliance, so audit trails are generated and permissions are checked in many places.

The application uses PostgreSQL for the database, lighttpd on the web server, and around 10 FastCGIs per application server sitting behind a virtual server with IP tunneling.17 Using a director setup such as this enables them to add and remove additional FastCGIs without restarting the web server. That, in turn, enables the automation of the process. A daemon monitors the load and assigns more FastCGIs from their cluster to handle the load when it peaks.

This is a real commercial application. It may sound boring, but it might well help determine whether you’re going to get authorized to buy your next house.

17http://www.linuxvirtualserver.org/VS-IPTunneling.html

Prepared exclusively for Rida Al Barazi

Report erratum