We recently went live with a high traffic website.  I prefer to deploy my websites using the lighttpd web server because of its awesome configuration syntax and speed.  As such, I use lighttpd's fastcgi and django's "python manage.py runfcgi" to deploy my sites.  One particular site, however, got frequent 500 Internal Server Errors when under heavy load.

In a mad scramble to get the site working more smoothly, I installed gunicorn.  I'm quite impressed with gunicorn. The load average for the system went down, and I didn't see processes running at 100% CPU in Top as much as I did with runfcgi.  However, we still got occasional 500 Internal Server Errors, and I couldn't track them down.

Finally in desperation, I tried Apache + modwsgi.  I did some benchmarking.  The website loaded faster (1.3 seconds on average) when running under gunicorn compared to apache + modwsgi (1.5 seconds on average).  However, after thousands of connections, I had hundreds of 500 errors from gunicorn and zero from modwsgi.  My vote goes to modwsgi because it seems to be more stable.

My current set up has fastcgi on the front end serving static media and taking care of SSL connections.  Lighttpd proxies all django-related requests to Apache that I have running on its own port on the server.

Note: the specific errors I was getting with gunicorn were a bad filehandler error in the notify function (seems like the temp file it tried to create didn't exist?), but couldn't trace down the exact issue.  The other error I got frequently was the nebulous "2006, MySQL has gone away" error. If anyone can shed insight into what may be the cause of these errors, I'd like to know, as I liked the architecture and design of gunicorn (and wish it didn't throw so many errors!).