JBoss: Windows vs Linux and a Solaris boot issue

Bryan Sant bryan.sant at gmail.com
Tue Mar 7 13:23:38 MST 2006


On 3/7/06, Ryan Bowman <ryanlbowman at gmail.com> wrote:
> This is not under load, but the poor performance is manifest with just one user.
> Oh, and by the way, this is now different hardware, a 32bit install,
> with only 2GB RAM total, -Xmx1408m, but the problem is exists.

OK.

> > uptime
> 19:40

I was looking for the load average portion, but that's OK.  Top has it.

> > vmstat
> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
>  0  0      0 585448  91132 907528    0    0     2     6   67    28  0  0 100  0

Was this run while you were doing something with your client?  I don't
think we're going to see anything useful unless there is a constant
load while you run these commands.  This output show that exactly
nothing is going on -- but that is correct if you aren't connecting
with enough clients to actually see something in real-time.  What I
was hoping to see is that the last column (wa -- "wait") would have
something other than 0.  A value other than 0 would indicate that the
CPU is waiting on I/O -- network or disk or something.

> > netstat -s
> Tcp:
>     508 active connections openings
>     172 passive connection openings
>     0 failed connection attempts
>     14 connection resets received
>     83 connections established
>     1476735 segments received
>     1324213 segments send out
>     60 segments retransmited
>     0 bad segments received.
>     7 resets sent
> TcpExt:
>     39 packets pruned from receive queue because of socket buffer overrun
>     ArpFilter: 0
>     411 TCP sockets finished time wait in fast timer
>     1 time wait sockets recycled by time stamp
>     70992 delayed acks sent
>     4 delayed acks further delayed because of locked socket
>     Quick ack mode was activated 210 times
>     18211 packets directly queued to recvmsg prequeue.
>     3600 packets directly received from backlog
>     42464 packets directly received from prequeue
>     1072820 packets header predicted
>     111 packets header predicted and directly queued to user
>     TCPPureAcks: 21212
>     TCPHPAcks: 914601
>     TCPRenoRecovery: 0
>     TCPSackRecovery: 0
>     TCPSACKReneging: 0
>     TCPFACKReorder: 0
>     TCPSACKReorder: 0
>     TCPRenoReorder: 0
>     TCPTSReorder: 0
>     TCPFullUndo: 0
>     TCPPartialUndo: 0
>     TCPDSACKUndo: 0
>     TCPLossUndo: 4
>     TCPLoss: 0
>     TCPLostRetransmit: 0
>     TCPRenoFailures: 0
>     TCPSackFailures: 0
>     TCPLossFailures: 0
>     TCPFastRetrans: 0
>     TCPForwardRetrans: 0
>     TCPSlowStartRetrans: 0
>     TCPTimeouts: 25
>     TCPRenoRecoveryFail: 0
>     TCPSackRecoveryFail: 0
>     TCPSchedulerFailed: 0
>     TCPRcvCollapsed: 2339
>     TCPDSACKOldSent: 210
>     TCPDSACKOfoSent: 0
>     TCPDSACKRecv: 0
>     TCPDSACKOfoRecv: 0
>     TCPAbortOnSyn: 0
>     TCPAbortOnData: 3
>     TCPAbortOnClose: 3
>     TCPAbortOnMemory: 0
>     TCPAbortOnTimeout: 6
>     TCPAbortOnLinger: 0
>     TCPAbortFailed: 0
>     TCPMemoryPressures: 0

Looks OK.  60 retransmits isn't bad considering the uptime of this box.

> > top -n 1
> top - 12:06:08 up 19:51,  5 users,  load average: 0.02, 0.02, 0.06
> Tasks: 106 total,   1 running, 105 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.2% us,  0.0% sy,  0.0% ni, 99.7% id,  0.0% wa,  0.0% hi,  0.0% si
> Mem:  2074672k total,  1490552k used,   584120k free,    91132k buffers
> Swap:  1052248k total,        0k used,  1052248k free,   907528k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 12997 root      17   0 1788m 382m  89m S  2.0 18.9   3:17.09 java

Again, we can't see anything measurable without some sustained client
load while we look.  BTW - You're brave to run jboss as root :-).

> > netstat -tn | grep -v ESTABLISHED
> Nothing.

You would probably see something with a bigger client load, but,
regardless, my theory is wrong about TCP TIME_WAITS being a problem if
you have poor performance with a single client.  Back to the drawing
board :-(.

Is it too late to rant about my feelings towards JBoss :-)?  Would you
be willing to deploy your EAR to Geronimo just to get another data
point on this?  If Geronimo is fast and JBoss is slow at least we can
isolate it to a JBoss-on-Linux problem.

I have no idea why it's working well on W2K3 and not Linux with this
data.  Can you add some logging to your app and timestamp your
internal method calls to see how long JDBC queries are taking and
other overhead.  Find out which calls are taking so long -- the
hot-spots.  Also, add the following param to jboss' command-line
-Dcom.sun.management.jmxremote -- you'll be able to connect to that
JVM with a local instance of jconsole and watch threads, memory, and
JMX instruments while the app runs.  Also, also, you can run "kill
-QUIT <jboss pid>" while the app is running to have a stacktrace of
exactly what's being executed at that moment dumped to stdout.  I'd
like to see what's on your stack while one of your client requests is
taking 5 seconds (though your timestamp logging will be more helpful
than a raw stack anyway).

-Bryan



More information about the PLUG mailing list