High Load average no CPU utilization

Steve Alligood steve at bluehost.com
Wed Mar 28 11:54:58 MDT 2007


CPU is practically idle, not out of memory, local drive is idle.

I'd check the I/O on the nfs server, and even test catting files on the 
web servers that are hosted on the nfs server during the high load.

My guess from your data would be that either the nfs server is I/O 
bound, the network between them is having issues, or you don't have 
enough nfs processes/memory/etc on the nfs server to handle the number 
of requests.

I would start looking at this as an nfs issue rather than a webserver issue.

-Steve

adam fisher wrote:
> Here is what I am seeing,sorry for the big print out.
> 
> top
> top - 04:36:52 up 7 days,  1:05,  1 user,  load average: 73.88, 62.88, 91.72
> Tasks: 207 total,   1 running, 206 sleeping,   0 stopped,   0 zombie
> Cpu(s):  8.0%us,  0.8%sy,  0.0%ni, 84.4%id,  6.3%wa,  0.2%hi,  0.3%si,  0.0%st
> Mem:   2062096k total,  1834196k used,   227900k free,   137360k buffers
> Swap:  8193140k total,       88k used,  8193052k free,  1157808k cached
> 
> 18833 apache    15   0 29240  11m 3084 S    4  0.6   0:18.77 httpd
> 18895 apache    15   0 29364  11m 3092 S    2  0.6   0:00.46 httpd
> 18831 apache    15   0 29364  11m 3216 S    2  0.6   0:02.86 httpd
> 18872 apache    15   0 29124  12m 4052 S    2  0.6   0:01.90 httpd
> 18867 apache    15   0 29316  12m 3872 D    1  0.6   0:01.51 httpd
> 18893 apache    15   0 28896  11m 3028 S    1  0.6   0:00.38 httpd
> 18884 apache    15   0 29332  11m 3088 S    1  0.6   0:00.90 httpd
> 18861 apache    15   0 29308  11m 3116 D    1  0.6   0:01.81 httpd
> 18888 apache    15   0 28904  11m 3084 S    1  0.6   0:00.59 httpd
> 
> ps -aux | grep httpd
> apache   17255  0.0  0.5  29496 12072 ?        D    04:27   0:00 /usr/sbin/httpd
> apache   17261  0.0  0.5  28840 11296 ?        D    04:27   0:00 /usr/sbin/httpd
> apache   17267  0.2  0.5  29176 11860 ?        D    04:27   0:01 /usr/sbin/httpd
> apache   17268  0.7  0.6  29552 12868 ?        D    04:27   0:04 /usr/sbin/httpd
> apache   17456  1.1  0.6  29728 13168 ?        S    04:27   0:06 /usr/sbin/httpd
> apache   17890  0.5  0.6  29928 12588 ?        D    04:28   0:02 /usr/sbin/httpd
> apache   17893  0.0  0.5  29032 11548 ?        D    04:28   0:00 /usr/sbin/httpd
> apache   17895  0.0  0.5  29184 11716 ?        D    04:28   0:00 /usr/sbin/httpd
> apache   17896  0.0  0.5  28740 11256 ?        D    04:28   0:00 /usr/sbin/httpd
> apache   17897  0.0  0.5  28912 11452 ?        D    04:28   0:00 /usr/sbin/httpd
> apache   17904  0.3  0.5  29288 11876 ?        D    04:28   0:01 /usr/sbin/httpd
> apache   17913  0.5  0.5  29316 11892 ?        D    04:29   0:02 /usr/sbin/httpd
> apache   17923  0.1  0.5  29364 12052 ?        D    04:29   0:00 /usr/sbin/httpd
> apache   17925  0.8  0.6  29336 12792 ?        D    04:29   0:04 /usr/sbin/httpd
> apache   17935  0.2  0.6  29100 12664 ?        D    04:29   0:01 /usr/sbin/httpd
> apache   17940  0.2  0.5  29328 11944 ?        D    04:29   0:01 /usr/sbin/httpd
> apache   17954  0.2  0.5  29332 11904 ?        D    04:30   0:00 /usr/sbin/httpd
> apache   17956  0.0  0.5  28400 10760 ?        D    04:30   0:00 /usr/sbin/httpd
> apache   17957  0.3  0.5  29700 12240 ?        D    04:30   0:01 /usr/sbin/httpd
> apache   17960  0.6  0.6  29416 12668 ?        D    04:30   0:02 /usr/sbin/httpd
> apache   17961  0.0  0.5  28148 10608 ?        D    04:30   0:00 /usr/sbin/httpd
> apache   17962  0.1  0.5  29120 11700 ?        D    04:30   0:00 /usr/sbin/httpd
> apache   17964  0.0  0.5  28136 10424 ?        D    04:30   0:00 /usr/sbin/httpd
> apache   17968  0.0  0.5  29184 11744 ?        D    04:30   0:00 /usr/sbin/httpd
> apache   17970  1.8  0.6  29292 12704 ?        S    04:30   0:07 /usr/sbin/httpd
> 
> iostat -x 1
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            2.00    0.00    0.00    0.00    0.00   98.00
> 
> Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00    11.00  0.00  6.00     0.00   136.00    22.67     0.00    0.50   0.17   0.10
> dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-4              0.00     0.00  0.00 17.00     0.00   136.00     8.00     0.01    0.29   0.06   0.10
> fd0               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            5.97    0.00    1.49    0.00    0.00   92.54
> 
> Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     8.00  0.00  2.00     0.00    80.00    40.00     0.00    0.50   0.50   0.10
> dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-2              0.00     0.00  0.00 10.00     0.00    80.00     8.00     0.01    0.90   0.10   0.10
> dm-3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> dm-4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> fd0               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
> 
>  vmstat
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>  0  0     88 189244 137948 1172700    0    0     0     4   30   30  1  0 99  0  0
> 
> The web root is located on an NFS share.  I restarted NFS on this box just to make sure.  When I restart httpd and the load average drops to around 10 or 11 I can browse the webpage just fine.  It is when it gets to around 150 that I can't.
> 
> Any help is appreciated
> Adam
> 
> 
> 
> ----- Ryan Simpkins <plug at ryansimpkins.com> wrote:
>> On Wed, March 28, 2007 09:30, adam fisher wrote:
>>> I just brought up a new web server on fedora core. The load aver
>> appears to grow to
>>> the size of the MaxClients which is 150. If I lower it then the load
>> average lowers.
>>> When I run ps -aux | grep httpd, I have almost every httpd process
>> in
>>> uninterruptible sleep. Are these persistent connections that just
>> aren't being
>>> closed or is it something else?
>>>
>>> Never seen this before,
>>> Adam
>> /proc/loadavg
>>               The  load  average  numbers  give  the number of jobs in
>> the run queue
>> (state R) or waiting for disk I/O (state D) averaged over 1, 5, and
>> 15
>> minutes. They  are  the same as the load average numbers given by
>> uptime(1) and other programs.
>>
>> If top doesn't show high CPU utilization, run iostat -x 1 for a bit
>> and see what
>> your disk is doing. vmstat is also helpful here.
>>
>> -Ryan
>>
>> /*
>> PLUG: http://plug.org, #utah on irc.freenode.net
>> Unsubscribe: http://plug.org/mailman/options/plug
>> Don't fear the penguin.
>> */
> 
> 
> 
> 
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3241 bytes
Desc: S/MIME Cryptographic Signature
Url : http://plug.org/pipermail/plug/attachments/20070328/c7deef74/attachment.bin 


More information about the PLUG mailing list