Your site should be available not only to your colleagues performing tests on your website from a facility down the road, but also to real visitors who may be located on other continents with slow connections. Some large companies choose to have site performance (and availability) constantly monitored by services such as Keynote (http://keynote.com) or Gomez (http://gomez.com). These commercial services deploy worldwide networks of machines that constantly ping your web pages to record the return time. Servers then keep track of all these metrics and build you a handy-dandy dashboard to evaluate how your site performance and uptime appears from many locations around the world. Because Keynote and Gomez are deemed "objective" third parties, those statistics can be used to enforce or guide Service Level Agreements (SLAs) arranged with partner companies or sites (we'll talk more about SLAs later). Keynote and Gomez can be considered enterprise-level services. There are also plenty of low-cost alternatives, including PingDom (http://pingdom.com), SiteUptime (http://siteuptime.com), and Alertra (http://alertra.com).

It's important to understand exactly what these services measure, and how to interpret the numbers they generate. Since most of them are networks of machines rather than people, it's essential to be aware of how those web pages are being requested. Some things to consider when you're looking at service monitoring systems include:

Both for measurements purposes as well as for rapid response to changing conditions, you want your architecture to be designed so you can easily split it into parts that perform discrete tasks. In an ideal world, each component of the backend should have a single job to do, but it could still do multiple jobs well, if needed. At the same time, its effectiveness on each job should be easy to measure.

For instance, let's look at a simple, database-driven web application just starting on its path toward world domination. To get the most bang for our buck, we have our web server and our database residing on the same hardware server. This means all the moving parts share the same hardware resources, as shown in .

Let's suppose you've already read (cheating, are we?) and you have configured measurements for both system and application-level statistics for your server. You can measure the system statistics of this server via sar or rrdtool, and maybe even application-level measurements such as web resource requests or database queries-per-second.

The difficulty with the setup in is you can't easily distinguish which system statistics correspond with the different pieces of the architecture. Therefore, you can't answer basic questions that are likely to arise, such as:

As mentioned earlier, server statistics paint only a part of the capacity picture. You should also measure and record higher-level metrics specific to your application—not specific to one server, but to the whole system. CPU and server disk usage on a web server doesn't tell the whole tale of what's happening to each web request, and a stream of web requests can involve multiple pieces of hardware.

At Flickr, we have a dashboard that collects these application-level metrics. They are collected on both a daily and cumulative basis. Some of the metrics can be drawn from a database, such as the number of photos uploaded. Others can come from aggregating some of the server statistics, such as total disk space consumed across disparate machines. Data collection techniques can be as simple as running a script from a cron job and putting results into its own database for future mining.

Some of the metrics currently tracked at Flickr are:

We also track certain financial metrics, such as payments received (which lie outside the scope of this book). For your particular application, a good exercise would be to spend some time correlating business and financial data to the system and application metrics you're tracking.

For example, a Total Cost of Ownership (TCO) calculation would be incomplete without some indication of how much these system and application metrics cost the business. Imagine being able to correlate the real costs to serve a single web page with your application. Having these calculations would not only put the architecture into a different context from web operations (business metrics instead of availability, or performance metrics), but they can also provide context for the more finance-obsessed, non-technical upper management who might have access to these tools.

Obviously, the when of ordering equipment is just as important as the what and how much. Procurement timelines outlined above hint at how critical it is to keep your eye on how long it will take to get what you need into production. Sometimes external influences, such as vendor delivery times and physical installation at the data center can ruin what started out to be a perfectly timed integration of new capacity.

Startups routinely order servers purely out of the fear they'll be needed. Most newly launched companies have developers to work on the product and don't need to waste money on operations-focused engineers. The developers writing the code are most likely the same people setting up network switches, managing user accounts, installing software, and wearing whatever other hats are necessary to get their company rolling. The last thing they want to worry about is running out of servers when they launch their new, awesome website. Ordering more servers as needed can be rightly justified in these cases, because the hardware costs are more than offset by the costs of preparing a more streamlined and detailed capacity plan.

As mentioned earlier, getting to know the peaks and valleys of your various resources and application usage is paramount to predicting the future. As you gain more and more history with your metrics, you may be able to perceive more subtle trends that will inform your long-term decisions.

For example, let's take a look at , which illustrates a typical traffic pattern for a web server.

shows a pretty typical U.S. daily traffic pattern. The load rises slowly in the morning, East Coast time, as users begin browsing. These users go to lunch as West Coast users come online, keeping up the load, which finally drops off as people leave work. At this point, the load drops to only those users browsing over night.

As your usage grows, you can expect this graph to grow vertically as more users visit your site during the same peaks and valleys. But if your audience grows more internationally, the bump you see every day will widen as the number of active user time zones increases. As seen in , you may even see distinct bumps after the U.S. drop-off if your site's popularity grows in a region further away than Europe.

displays two daily traffic patterns, taken one year apart, and superimposed one on top of the other. What once was a smooth bump and decline has become a two-peak bump, due to the global effect of popularity.

Of course, your product and marketing people are probably very aware of the demographics and geographic distribution of your audience, but tying this data to your system's resources can help you predict your capacity needs.

also shows that your web servers must sustain their peak traffic for longer periods of time. This will indicate when you should schedule any maintenance windows to minimize the effect of downtime or degraded service to the users. Notice the ratio between your peak and your low period has changed as well. This will affect how many servers you can stand to lose to failure during those periods, which is effectively the ceiling of your cluster.

The time needed to acquire, install, and provision new hardware must be factored into your calculations as you determine when you're going to run out of capacity. If your capacity will be exhausted in six weeks, and it takes you three weeks to add new hardware, you only have three weeks of breathing room. Automated deployment and configuration minimizes the time spent on the phase of the process over which you have the most control—integrating machines onto your network and beginning operations.

When making changes to hosts, it's preferable to have a central location from which to push changes appropriate to the servers you're affecting. Having a central location provides a "control tower" from which to manage all aspects of your infrastructure. Unlike server architectures, in which distributed resources help with horizontal scaling, centralized configuration and management environments yield several advantages:

This is not to suggest that your configuration, installation, monitoring, and management setup should be kept on a single server. Each of these deployment components demands specific resources. Growth over time would simply overwhelm a single machine, rendering it a potential single point of failure. Separate these components from the rest of your infrastructure. Monitoring and metric collection can reside on one server; configuration management and log aggregation on another. See for an example of a typical installation, configuration, and management architecture.

Most organizations aren't happy with the operating system their vendor installs. Default OS installs are notoriously inappropriate for production environments because they are designed to run on as many different hardware platforms as possible. They usually contain packages you don't need and typically are missing those that you do. As a result, most companies create custom OS images suitable for their specific needs.

No matter how you look at it, your website needs computing and storage resources. Somewhere—whether in your direct and total control or not—a server will need to respond to client requests for data, and those requests may need some amount of computation and data retrieval.

Virtualization has been around almost as long as computing. At one time, computers were seen as equipment only managed by large financial, educational, or research institutions. Since computers were extremely expensive, IBM and other manufacturers built large-scale minicomputers and mainframes to handle processing for multiple users at once, utilizing many of the virtualization concepts still in use today. Users would be granted slices of computation time from mainframe machines, accessing them from thin, or dumb, terminals. Users submitted jobs whose computation contended for resources. The centralized system was managed via queues, virtual operating systems, and system accounting that governed resource allocation. All of the heavy lifting of computation was handled by the mainframe and its operators, and was largely invisible to the end users. The design of these systems was largely driven by security and reliability, so considerable effort was applied to containing user environments and data redundancy. illustrates the client-sever structure in a mainframe environment.

One contingency is to disable some of the site's heavier features. Building in the ability to turn certain features on or off can help capacity and operations respond significantly, even in the absence of some massive traffic event. Having a quick, one-line configuration parameter in your application with values of on or off can be of enormous value, particularly when that feature is either the cause of a problem or contributing to unacceptable performance.

For example, the webservers at Flickr perform geographic (country) lookups based on client IP addresses for logged-out users in an effort to deduce their language preferences. It's an operation that enhances the user experience, but it is yet another function the application must handle. When we launched the localized Flickr in seven different languages, we had this feature turned on with the launch. It almost immediately placed too much load on the mechanisms that carried out the country lookups, so we simply turned it off until we could learn what the issue was. The problem turned out to be an artificial throttle placed on the rate of requests the geo server could handle, which was tuned too conservatively. We isolated and fixed the issue by lifting the throttle to a more acceptable level then turned the feature (which is mostly transparent) back on. Had we not implemented the quick on/off switch for that feature—had it been hardcoded within the application—it would have taken more time to troubleshoot, disable, and fix. During this time the site would have been in a degraded state, or possibily even down.

Cacti, http://cacti.net

Zabbix, http://zabbix.com

Hyperic HQ, http://hyperic.com

Munin, http://munin.projects.linpro.no/

ZenOSS, http://www.zenoss.com/

OpenNMS, http://opennms.org

Monit, http://www.tildeslash.com/monit

KickStart, http://fedoraproject.org/wiki/Anaconda/Kickstart/

Bcfg2, http://trac.mcs.anl.gov/projects/bcfg2

Lcfg (Large-scale Unix configuration system), http://www.lcfg.org/

Dsh, http://freshmeat.net/projects/dsh/

Fabric, http://savannah.nongnu.org/projects/fab

XCat, http://xcat.sourceforge.net/

OCS Inventory NG, http://www.ocsinventory-ng.org/

Gunther, Neil. Guerilla Capacity Planning (Springer, 2006)

Menascé, Daniel A. and Virgilio A. F. Almeida. Capacity Planning For Web Services: Metrics, Models, and Methods