Hacking for fun and profit

The Apache HTTP server is by far the most used web server in the world. It is an excellent, feature packed and standards compliant web server. Extremely configurable, with an endless amount of modules, superb documentation and, due to its license, is being used commercially by companies like Oracle. There is only a small problem with Apache: It’s not the fastest server around. This is because historically, Apache’s priorities have been correctness and configurability, not performance. Correctness and configurability are the reasons why Apache powers almost 70% of the web today, but still, Apache has a big, fat ass (we are talking about system resources here).

First let me explain some web server models:

• Forking: In this model, every request will spawn (fork) a new process. This is because in UNIX, fork() is a very cheap thing to do. The problem is that if you are handling heavy traffic, things will slow down a lot, and if you are handling some serious heavy traffic you will run out of memory. Also, forking sucks in NT. An example of a forking webserver is Apache pre-1.3
• Preforking: This is similar to the forking webserver, but it preloads processes before they are needed by a request, so it handles traffic a lot better. It can still run out of memory with heavy traffic. Examples of preforking webservers are Apache 1.3 and Apache 2.x-prefork
• Threaded: A single process runs several threads and each thread handles a request. A thread is a lot cheaper to create than processes and less resource consuming. Examples of threaded webservers are Apache 2.x-worker
• Event driven: Also known as asynchronous programming. It uses select/poll/epoll/kqueue from a single process/thread. Examples of event driven webservers are Zeus, Litespeed and lighttpd.

Now, one of the reasons of Apache’s heavy use of resources is PHP. The default way to add PHP support in Apache is adding php as a module, that means, embedding the PHP interpreter in every instance in memory of an already fat Apache. And since the recommended way to use PHP in a production enviroment is in a non-threaded webserver, then you are stuck with Apache 1.x or 2.x-prefork. This means that every request must spawn in memory a full Apache+PHP process. In my experience, in most default installs (debian, redhat, suse), this will take like 12MB of memory for every process. If you need more PHP modules or other Apache modules, like mod_python, the memory footprint will be even higher. This is why some servers swap and run out of memory when handling heavy traffic.

When I decided to move away from a preforking Apache to a worker-mpm Apache, my first intention was to use PHP as FastCGI. FastCGI is basically a daemonized CGI. In the case of PHP, daemonized PHP instances waiting to be used by a web server. The server talks with the FastCGI daemon via UNIX sockets or TCP/IP sockets, and because of this you can separate the web server from the “application server” (the one that runs the daemonized PHP instances).

Sadly the fastcgi support in Apache is quite bad, and I got a lot of problems with this setup, from popular PHP packages. I decided to give lighttpd a try. It is a very nice event-driven webserver, developed originally as a proof of concept for the C10K problem, with all the features I need:

• mod_rewrite
• redirects
• SSL
• FastCGI
• SSI
• virtual hosts
• etc

I had no FastCGI problems with PHP and lighttpd, which leads me to think that lighty’s FastCGI support is more robust than Apache’s. What I had to do was migrating the vhost.conf files from apache to lighttpd, something quite simple since it already has mod-rewrite and mod-redirect. What was somewhat of a pain was the .htaccess files. Some tips when migrating popular PHP apps:

Wordpress (installed in /)

server.error-handler-404 = "/index.php"


Drupal (installed in /)

url.rewrite-final = (
"^/system/test/(.*)$" => "/index.php?q=system/test/$1",
"^/([^.?]*)\?(.*)$” => “/index.php?q=$1&$2″,
“^/([^.?]*)$” => “/index.php?q=$1″
)


Gallery (installed in /gallery/)

url.rewrite-once = (
"^/gallery/([^\.\?/]+)/([0-9]+)$” => “/gallery/view_photo.php?set_albumName=$1&index=$2″,
“^/gallery/([^\.\?/]+)/([A-Za-z_0-9\-]+)$” => “/gallery/view_photo.php?set_albumName=$1&id=$2″,
“^/gallery/([^\.\?/]+)/$” => “/gallery/$1″,
“^/gallery/([^\.\?/]+)$” => “/gallery/view_album.php?set_albumName=$1″
)


serendipity (installed in /)

server.error-handler-404 = "/index.php"


Docuwiki (installed in /wiki/)

$HTTP["url"] =~ “^/wiki/attic” { url.access-deny = ( “” ) }
$HTTP["url"] =~ “^/wiki/conf” { url.access-deny = ( “” ) }
$HTTP["url"] =~ “^/wiki/data” { url.access-deny = ( “” ) }
$HTTP["url"] =~ “^/wiki/inc” { url.access-deny = ( “” ) }
$HTTP["url"] =~ “^/wiki/lang” { url.access-deny = ( “” ) }


Horde

This was a real pain to install. First because, at least in FreeBSD, it won’t install if you don’t have PHP compiled as an Apache Module, and since lighttpd needs PHP compiled as FastCGI, horde will just refuse to install. This is a bug in the horde port.

Second, because of the greatness of everything PHP, PEAR wasn’t working with php-fcgi without register_argc_argv turned on in php.ini (bug).

And third, because of some bizarre reason, horde refuses to work in lighttpd. It works in the same server with apache using the same PHP fastcgi daemons, but fails in lighttpd. Here is how to fix that in yout horde install (WARNING! AWFUL HACKS INCOMING!):

• config/registry.php: assign the $webroot variable to something sane. I hardcoded mine to '/horde'.
• lib/Horde/Maintenance.php: look for the runMaintenance function and comment the header and exit sentences to avoid a weird endless redirect loop
• imp/config/prefs.php: set the 'value' item in $_prefs['compose_popup'] to 0. Reset all the compose_popup settings in your users to zero (this is easy if you store settings in SQL). For some reason the compose popup window doesn’t work so you have to deactivate it.

The web-related memory consumption in the server is now minimal, mostly php instances and due to this the server can now handle a lot more web traffic. After that I installed lighty in a couple of heavily-loaded servers to a customer with excellent results. Now I am doing benchmarks for another customer that must handle 10,000 concurrent users and so far lighty is performing a lot better than Apache. Results will be published here soon.