High Load average no CPU utilization
Steve Alligood
steve at bluehost.com
Wed Mar 28 11:54:58 MDT 2007
CPU is practically idle, not out of memory, local drive is idle.
I'd check the I/O on the nfs server, and even test catting files on the
web servers that are hosted on the nfs server during the high load.
My guess from your data would be that either the nfs server is I/O
bound, the network between them is having issues, or you don't have
enough nfs processes/memory/etc on the nfs server to handle the number
of requests.
I would start looking at this as an nfs issue rather than a webserver issue.
-Steve
adam fisher wrote:
> Here is what I am seeing,sorry for the big print out.
>
> top
> top - 04:36:52 up 7 days, 1:05, 1 user, load average: 73.88, 62.88, 91.72
> Tasks: 207 total, 1 running, 206 sleeping, 0 stopped, 0 zombie
> Cpu(s): 8.0%us, 0.8%sy, 0.0%ni, 84.4%id, 6.3%wa, 0.2%hi, 0.3%si, 0.0%st
> Mem: 2062096k total, 1834196k used, 227900k free, 137360k buffers
> Swap: 8193140k total, 88k used, 8193052k free, 1157808k cached
>
> 18833 apache 15 0 29240 11m 3084 S 4 0.6 0:18.77 httpd
> 18895 apache 15 0 29364 11m 3092 S 2 0.6 0:00.46 httpd
> 18831 apache 15 0 29364 11m 3216 S 2 0.6 0:02.86 httpd
> 18872 apache 15 0 29124 12m 4052 S 2 0.6 0:01.90 httpd
> 18867 apache 15 0 29316 12m 3872 D 1 0.6 0:01.51 httpd
> 18893 apache 15 0 28896 11m 3028 S 1 0.6 0:00.38 httpd
> 18884 apache 15 0 29332 11m 3088 S 1 0.6 0:00.90 httpd
> 18861 apache 15 0 29308 11m 3116 D 1 0.6 0:01.81 httpd
> 18888 apache 15 0 28904 11m 3084 S 1 0.6 0:00.59 httpd
>
> ps -aux | grep httpd
> apache 17255 0.0 0.5 29496 12072 ? D 04:27 0:00 /usr/sbin/httpd
> apache 17261 0.0 0.5 28840 11296 ? D 04:27 0:00 /usr/sbin/httpd
> apache 17267 0.2 0.5 29176 11860 ? D 04:27 0:01 /usr/sbin/httpd
> apache 17268 0.7 0.6 29552 12868 ? D 04:27 0:04 /usr/sbin/httpd
> apache 17456 1.1 0.6 29728 13168 ? S 04:27 0:06 /usr/sbin/httpd
> apache 17890 0.5 0.6 29928 12588 ? D 04:28 0:02 /usr/sbin/httpd
> apache 17893 0.0 0.5 29032 11548 ? D 04:28 0:00 /usr/sbin/httpd
> apache 17895 0.0 0.5 29184 11716 ? D 04:28 0:00 /usr/sbin/httpd
> apache 17896 0.0 0.5 28740 11256 ? D 04:28 0:00 /usr/sbin/httpd
> apache 17897 0.0 0.5 28912 11452 ? D 04:28 0:00 /usr/sbin/httpd
> apache 17904 0.3 0.5 29288 11876 ? D 04:28 0:01 /usr/sbin/httpd
> apache 17913 0.5 0.5 29316 11892 ? D 04:29 0:02 /usr/sbin/httpd
> apache 17923 0.1 0.5 29364 12052 ? D 04:29 0:00 /usr/sbin/httpd
> apache 17925 0.8 0.6 29336 12792 ? D 04:29 0:04 /usr/sbin/httpd
> apache 17935 0.2 0.6 29100 12664 ? D 04:29 0:01 /usr/sbin/httpd
> apache 17940 0.2 0.5 29328 11944 ? D 04:29 0:01 /usr/sbin/httpd
> apache 17954 0.2 0.5 29332 11904 ? D 04:30 0:00 /usr/sbin/httpd
> apache 17956 0.0 0.5 28400 10760 ? D 04:30 0:00 /usr/sbin/httpd
> apache 17957 0.3 0.5 29700 12240 ? D 04:30 0:01 /usr/sbin/httpd
> apache 17960 0.6 0.6 29416 12668 ? D 04:30 0:02 /usr/sbin/httpd
> apache 17961 0.0 0.5 28148 10608 ? D 04:30 0:00 /usr/sbin/httpd
> apache 17962 0.1 0.5 29120 11700 ? D 04:30 0:00 /usr/sbin/httpd
> apache 17964 0.0 0.5 28136 10424 ? D 04:30 0:00 /usr/sbin/httpd
> apache 17968 0.0 0.5 29184 11744 ? D 04:30 0:00 /usr/sbin/httpd
> apache 17970 1.8 0.6 29292 12704 ? S 04:30 0:07 /usr/sbin/httpd
>
> iostat -x 1
> avg-cpu: %user %nice %system %iowait %steal %idle
> 2.00 0.00 0.00 0.00 0.00 98.00
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 11.00 0.00 6.00 0.00 136.00 22.67 0.00 0.50 0.17 0.10
> dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-4 0.00 0.00 0.00 17.00 0.00 136.00 8.00 0.01 0.29 0.06 0.10
> fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 5.97 0.00 1.49 0.00 0.00 92.54
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 8.00 0.00 2.00 0.00 80.00 40.00 0.00 0.50 0.50 0.10
> dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-2 0.00 0.00 0.00 10.00 0.00 80.00 8.00 0.01 0.90 0.10 0.10
> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>
> vmstat
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
> r b swpd free buff cache si so bi bo in cs us sy id wa st
> 0 0 88 189244 137948 1172700 0 0 0 4 30 30 1 0 99 0 0
>
> The web root is located on an NFS share. I restarted NFS on this box just to make sure. When I restart httpd and the load average drops to around 10 or 11 I can browse the webpage just fine. It is when it gets to around 150 that I can't.
>
> Any help is appreciated
> Adam
>
>
>
> ----- Ryan Simpkins <plug at ryansimpkins.com> wrote:
>> On Wed, March 28, 2007 09:30, adam fisher wrote:
>>> I just brought up a new web server on fedora core. The load aver
>> appears to grow to
>>> the size of the MaxClients which is 150. If I lower it then the load
>> average lowers.
>>> When I run ps -aux | grep httpd, I have almost every httpd process
>> in
>>> uninterruptible sleep. Are these persistent connections that just
>> aren't being
>>> closed or is it something else?
>>>
>>> Never seen this before,
>>> Adam
>> /proc/loadavg
>> The load average numbers give the number of jobs in
>> the run queue
>> (state R) or waiting for disk I/O (state D) averaged over 1, 5, and
>> 15
>> minutes. They are the same as the load average numbers given by
>> uptime(1) and other programs.
>>
>> If top doesn't show high CPU utilization, run iostat -x 1 for a bit
>> and see what
>> your disk is doing. vmstat is also helpful here.
>>
>> -Ryan
>>
>> /*
>> PLUG: http://plug.org, #utah on irc.freenode.net
>> Unsubscribe: http://plug.org/mailman/options/plug
>> Don't fear the penguin.
>> */
>
>
>
>
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3241 bytes
Desc: S/MIME Cryptographic Signature
Url : http://plug.org/pipermail/plug/attachments/20070328/c7deef74/smime.bin
More information about the PLUG
mailing list