Top Commands to Diagnose Performance of Your Linux System
sasha at asksasha.com
Wed Jun 26 14:51:00 MDT 2013
> I know top and ps, but what are your other favorite commands to diagnose
> sluggish system performance?
gdb and strace. Here is my favorite trick with MySQL (from
gdb -ex "set pagination 0" -ex "thread apply all bt" \
--batch -p $(pidof mysqld)
and it will tell you exactly where it is stuck. This can solve a lot
of MySQL performance mysteries in a matter of seconds, and the same
idea can be applied to troubleshooting the performance of just about
anything that is either open-source or at least has decent
function-naming conventions and is not stripped so that when you see a
stack trace you will have a clue as to where it is at.
strace is the first thing I use once I know the pid of the top or
iotop dubious "winner".
It is also good to be a friend of lsof and netstat to figure out who
is on what file descriptor or port.
gdb can be used to poke around the memory of a trouble process. You
can discover some really cool things - e.g. who is holding the mutex
that is being waited upon by everybody. You can even fix up the memory
state of a running process if you have a good idea of what the result
should be. I have been able to make mysqld shut down cleanly in spite
of problems by modifying an internal variable (not a config one!) with
Fast Running Blog.
Run. Blog. Improve. Repeat.
More information about the PLUG