|
I‘m getting to know far more about servers than I ever wanted to, after
hundreds of hours of Google research trying to squeeze/beat performance out of
Apache. I do have 15 years programming experience in other areas, and I‘ve
reached the conclusion that the only experts in Apache/Linux are the programmers
who wrote (and poorly documented) all this stuff. So I‘ve gathered everything I
could learn after countless hours of frustration and I‘m writing this up in
return for the immense amount of help I‘ve received from the documentation of
others.
If you‘re reaching the limits of your Apache server because you‘re serving a
lot of dynamic content, you can either spend thousands on new equipment or
reduce bloat to increase your server capacity from 2 to 10 times. This article
concentrates on important and weakly documented ways of increasing capacity
without the need for additional hardware.
Understanding Server Load Problems
There are a few common areas of server load problems, and a thousand
uncommon. Let‘s focus on the top three I‘ve seen:
- Drive Swapping , where too many processes (or runaway processes) use
too much RAM.
- CPU , from poorly optimized DB queries, poorly optimized code, and
runaway processes.
- Network , whether hardware limits or moron attacks.
Managing Apache‘s RAM Usage
Apache processes use a ton of RAM. This issue becomes major when you realize
that after each process has done its job, the bloated process sits and
spoon-feeds data to the client, instead of moving on to bigger and better
things. This problem is compounded by a bit of essential info that should really
be more common knowledge:
If you serve 100% static files with Apache, each httpd process
will use around 2-3 megs of RAM.
If you serve 99% static files and 1% dynamic files with Apache, each
httpd process will use from 3-20 megs of RAM (depending on your
MOST complex dynamic page).
This occurs because a process grows to accommodate whatever it is serving,
and NEVER decreases until that process dies. Unless you have very few dynamic
pages and major traffic fluctuation, most of your httpd processes will soon take
up an amount of RAM equal to the largest dynamic script on your system. A very
smart web server would deal with this automatically. As it is, you have a few
options to manually improve RAM usage.
Reduce Wasted Processes by Tweaking KeepAlive
This is a tradeoff. The KeepAliveTimeout configuration setting
adjusts the amount of time a process sits around doing nothing but taking up
space. Those seconds add up in a huge way. Using KeepAlive can
increase speed for both you and the client — disable it and the serving of
static files such as images may be a lot slower. I think it‘s best to have
KeepAlive on, and KeepAliveTimeout very low, perhaps
one or two seconds.
Limit Total Processes with MaxClients
If you use Apache to serve dynamic content, your simultaneous connections are
severely limited. Exceed a certain number, and your system begins cannibalistic
swapping, getting slower and slower until it dies. Personally, I think the
system and/or web server should automatically take steps to prevent this, but by
default, they tend to allow the server to consume itself. Use trial and error to
figure out how many Apache processes your server can handle, and set this value
in MaxClients.
Note: the Apache docs on this are misleading — if this limit is reached,
clients are not "locked out," they are simply queued, and their access slows.
Based on the value of MaxClients, you can estimate the values you
need for StartServers, MinSpareServers, and
MaxSpareServers.
Force Processes to Reset with MaxRequestsPerChild
Forcing your processes to die after awhile makes them start over with lower
RAM usage. This can reduce total memory usage in many situations. The less
dynamic content you have, the more useful this will be. This is a game of
catch-up, with your dynamic files constantly increasing total RAM usage, and
restarting processes constantly reducing it. Experiment with
MaxRequestsPerChild (even values as low as 20 may work well), but
don‘t set it too low, because creating new processes does have overhead.
You can figure out the best settings under-load by examining ps axu
--sort:rss. A word of warning: the results can be impressive, but are NOT
consistent. If the only way you can keep your server running is by tweaking
this, you will eventually run into trouble. That being said, by tweaking
MaxRequestsPerChild, you may be able to increase
MaxClients as much as 50%.
Finally, think outside the box: replace or supplement Apache.
Use a Second Server
You can use a tiny, lightning fast server to handle static documents and
images, and pass more complicated requests on to Apache on the same machine.
This way Apache won‘t tie up multi-megabyte processes serving simple streams of
bytes. You can bring Apache into play, for example, only to execute PHP scripts.
Good options for this include:
Try lingerd
Lingerd takes
over the job of feeding bytes to the client after Apache has fetched the
document, but requires kernel modification. It sounds pretty good, but I haven‘t
tried it.
Use a Proxy Cache
A proxy cache can keep a duplicate copy of everything it gets from Apache,
and serve the copy instead of bothering Apache with it. This has the benefit of
also being able to cache dynamically generated pages, speeding up requests at
the expense of using more memory.
Recompile Apache
You could custom build Apache, with optimal settings for your situation,
removing parts that you don‘t require. This is not for the faint of heart.
Replace Apache Completely
If you don‘t need all the features of Apache, replace it with something more
scalable. Currently, the best options appear to be servers that use a
nonblocking I/O technology and connect to all clients with the same process. The
best include:
Apache Tuning References
Managing PHP‘s CPU and RAM Usage
Use an Accelerator
Compiling PHP scripts is usually more expensive than running them. Why not
use a simple tool that keeps them precompiled? Options include Turck MMCache
(fast and free but buggy), PHP Accelerator, APC, and Zend Accelerator. You will
likely see a speed increase of 2 to 10 times and a PHP RAM reduction of 50%-85%.
See Optimizing
PHP for more ideas.
Managing MySQL‘s CPU and RAM Usage
Optimize your Queries
This is covered in detail everywhere, so just keep in mind a few important
points. One bad query statement running often can bring your site to its knees.
Two or three bad query statements don‘t perform much differently than one. In
other words, if you optimize one query you may not see any server-wide speed
improvement. If you find and optimize ALL your bad queries you may suddenly see
a fivefold server-speed improvement. The log-slow-queries feature
of MySQL can be very helpful.
Put your Volatile Data Somewhere Else
If you don‘t need all the extra features of a relational database, storing
data with your own method can be many times faster than MySQL. For temporary
data, store it in each user session, or globally with a shared memory library
(like the one included with Turck MMCache). More important data can be stored in
files, or in your own flat-file database system. A combination of both could be
very powerful — your own flat-file database with a copy cached in memory, going
to disk only on write.
MySQL Tuning References
Other Solutions
Here are a few other things you may find useful.
Use top and ps axu to check for processes that are
using too much CPU or RAM. But beware, these programs will actually lie to you.
The total RAM used doesn‘t always match up — an application‘s threads share the
same memory pages, though it may not look that way.
Use netstat -anp | sort -u to check for network problems.
Use ApacheBench ab to benchmark your results — but keep in mind
this tool doesn‘t accurately simulate actual usage, most notably the effect of
many dialup users, who keep connections open longer than you‘d like.
References
Conclusion
If it makes you ill to think about a 20-meg process spending 15 seconds
waiting on a KeepAlive after serving a 2k JPEG file, you‘re not
alone! I come from the days of 1 MHz machines running multi-line BBSs. The idea
that gigahertz machines with gigabytes of RAM are being limited to 200
simultaneous connections suggests that we‘re far more influenced by a certain
mega-corporate mindset than we‘d like to believe. With some basic knowledge of
the software and a few of the tips here, you can decrease the bloat and get your
server running at least within a reasonable range of how it should, without
investing in more hardware.
|
