System comes to a halt on heavy disk I/O

[Kenneth Burgener]

On 2/1/2010 9:03 AM, Charles Curley wrote:
> When I run fairly disk intensive tasks, like copying tens of gigabytes
> to this machine, it slows to a crawl. Disk I/O slows down by two
> orders of magnitude.
>    


Linux tends to use disk cache as much as possible, so until you start 
performing disk operations that fill all of the available RAM for the 
disk cache, things will appear snappier.

While you are performing your disk operations, try watching 'vmstat 2' 
under the 'wa' (IO wait) to see what percentage of the CPU time is being 
spent waiting for IO.  This number should remain as close to zero as 
possible.  If the IO queue is so backed up that things aren't being 
handled prompty, then you will quickly notice IO based apps will begin 
to crawl.  Adding more RAM usually helps with IO issues, as more of the 
disk can be cached to RAM.

Also check 'smartctl -a /dev/sda' and check to see if the error rate is 
increasing rapidly.  If the disk is spending it's time recovering from 
failures, this would decrease the throughput, and also indicates that 
the drive is probably going bad.