admin question: howto-unresponsive server under extreme load

[Shane Hathaway]

Nicholas Leippe wrote:
> We have a webserver that occasionally gets pegged. When this happens, the 
> remote shell becomes nearly totally unresponsive--to the point that we have 
> it rebooted at the colo. We already have swap disabled. We are trying to 
> pinpoint which bit of our application is causing this. Obviously, a hard 
> reset kills any chance of it completing and writing log files with useful 
> clues.

You might try putting the log files on a separate hard drive.  Linux 
should be able to complete writes to the logging drive even when the 
main drive is swamped.  Separate partitions isn't enough; you need 
separate spindles.  Or, even better, use syslog to send the logs to 
another server.

> Is there some way to lock a root shell such that it is always responsive so 
> that we could at least kill the webserver and have a chance of the log files 
> being fully written to find the problem?
> 
> RT scheduling? Tell bash to mlock() itself into memory? Ideas?

You might consider writing a script that monitors the load level.  When 
it rises too high, automatically suspend your web server processes using 
"kill -19".  Then you can analyze the problem without actually killing 
the server.

Shane