High Load average no CPU utilization

adam fisher afisher at circlepix.com
Wed Mar 28 11:44:19 MDT 2007


Here is what I am seeing,sorry for the big print out.

top
top - 04:36:52 up 7 days,  1:05,  1 user,  load average: 73.88, 62.88, 91.72
Tasks: 207 total,   1 running, 206 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.0%us,  0.8%sy,  0.0%ni, 84.4%id,  6.3%wa,  0.2%hi,  0.3%si,  0.0%st
Mem:   2062096k total,  1834196k used,   227900k free,   137360k buffers
Swap:  8193140k total,       88k used,  8193052k free,  1157808k cached

18833 apache    15   0 29240  11m 3084 S    4  0.6   0:18.77 httpd
18895 apache    15   0 29364  11m 3092 S    2  0.6   0:00.46 httpd
18831 apache    15   0 29364  11m 3216 S    2  0.6   0:02.86 httpd
18872 apache    15   0 29124  12m 4052 S    2  0.6   0:01.90 httpd
18867 apache    15   0 29316  12m 3872 D    1  0.6   0:01.51 httpd
18893 apache    15   0 28896  11m 3028 S    1  0.6   0:00.38 httpd
18884 apache    15   0 29332  11m 3088 S    1  0.6   0:00.90 httpd
18861 apache    15   0 29308  11m 3116 D    1  0.6   0:01.81 httpd
18888 apache    15   0 28904  11m 3084 S    1  0.6   0:00.59 httpd

ps -aux | grep httpd
apache   17255  0.0  0.5  29496 12072 ?        D    04:27   0:00 /usr/sbin/httpd
apache   17261  0.0  0.5  28840 11296 ?        D    04:27   0:00 /usr/sbin/httpd
apache   17267  0.2  0.5  29176 11860 ?        D    04:27   0:01 /usr/sbin/httpd
apache   17268  0.7  0.6  29552 12868 ?        D    04:27   0:04 /usr/sbin/httpd
apache   17456  1.1  0.6  29728 13168 ?        S    04:27   0:06 /usr/sbin/httpd
apache   17890  0.5  0.6  29928 12588 ?        D    04:28   0:02 /usr/sbin/httpd
apache   17893  0.0  0.5  29032 11548 ?        D    04:28   0:00 /usr/sbin/httpd
apache   17895  0.0  0.5  29184 11716 ?        D    04:28   0:00 /usr/sbin/httpd
apache   17896  0.0  0.5  28740 11256 ?        D    04:28   0:00 /usr/sbin/httpd
apache   17897  0.0  0.5  28912 11452 ?        D    04:28   0:00 /usr/sbin/httpd
apache   17904  0.3  0.5  29288 11876 ?        D    04:28   0:01 /usr/sbin/httpd
apache   17913  0.5  0.5  29316 11892 ?        D    04:29   0:02 /usr/sbin/httpd
apache   17923  0.1  0.5  29364 12052 ?        D    04:29   0:00 /usr/sbin/httpd
apache   17925  0.8  0.6  29336 12792 ?        D    04:29   0:04 /usr/sbin/httpd
apache   17935  0.2  0.6  29100 12664 ?        D    04:29   0:01 /usr/sbin/httpd
apache   17940  0.2  0.5  29328 11944 ?        D    04:29   0:01 /usr/sbin/httpd
apache   17954  0.2  0.5  29332 11904 ?        D    04:30   0:00 /usr/sbin/httpd
apache   17956  0.0  0.5  28400 10760 ?        D    04:30   0:00 /usr/sbin/httpd
apache   17957  0.3  0.5  29700 12240 ?        D    04:30   0:01 /usr/sbin/httpd
apache   17960  0.6  0.6  29416 12668 ?        D    04:30   0:02 /usr/sbin/httpd
apache   17961  0.0  0.5  28148 10608 ?        D    04:30   0:00 /usr/sbin/httpd
apache   17962  0.1  0.5  29120 11700 ?        D    04:30   0:00 /usr/sbin/httpd
apache   17964  0.0  0.5  28136 10424 ?        D    04:30   0:00 /usr/sbin/httpd
apache   17968  0.0  0.5  29184 11744 ?        D    04:30   0:00 /usr/sbin/httpd
apache   17970  1.8  0.6  29292 12704 ?        S    04:30   0:07 /usr/sbin/httpd

iostat -x 1
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.00    0.00    0.00    0.00    0.00   98.00

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    11.00  0.00  6.00     0.00   136.00    22.67     0.00    0.50   0.17   0.10
dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00  0.00 17.00     0.00   136.00     8.00     0.01    0.29   0.06   0.10
fd0               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.97    0.00    1.49    0.00    0.00   92.54

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     8.00  0.00  2.00     0.00    80.00    40.00     0.00    0.50   0.50   0.10
dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00  0.00 10.00     0.00    80.00     8.00     0.01    0.90   0.10   0.10
dm-3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
fd0               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

 vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0     88 189244 137948 1172700    0    0     0     4   30   30  1  0 99  0  0

The web root is located on an NFS share.  I restarted NFS on this box just to make sure.  When I restart httpd and the load average drops to around 10 or 11 I can browse the webpage just fine.  It is when it gets to around 150 that I can't.

Any help is appreciated
Adam



----- Ryan Simpkins <plug at ryansimpkins.com> wrote:
> On Wed, March 28, 2007 09:30, adam fisher wrote:
> > I just brought up a new web server on fedora core. The load aver
> appears to grow to
> > the size of the MaxClients which is 150. If I lower it then the load
> average lowers.
> >
> > When I run ps -aux | grep httpd, I have almost every httpd process
> in
> > uninterruptible sleep. Are these persistent connections that just
> aren't being
> > closed or is it something else?
> >
> > Never seen this before,
> > Adam
> 
> /proc/loadavg
>               The  load  average  numbers  give  the number of jobs in
> the run queue
> (state R) or waiting for disk I/O (state D) averaged over 1, 5, and
> 15
> minutes. They  are  the same as the load average numbers given by
> uptime(1) and other programs.
> 
> If top doesn't show high CPU utilization, run iostat -x 1 for a bit
> and see what
> your disk is doing. vmstat is also helpful here.
> 
> -Ryan
> 
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */






More information about the PLUG mailing list