Top Commands to Diagnose Performance of Your Linux System

Sasha Pachev sasha at
Wed Jun 26 14:51:00 MDT 2013

> I know top and ps, but what are your other favorite commands to diagnose
> sluggish system performance?

gdb and strace. Here is my favorite trick with MySQL (from

gdb -ex "set pagination 0" -ex "thread apply all bt" \
  --batch -p $(pidof mysqld)

and it will tell you exactly where it is stuck. This can solve a lot
of MySQL performance mysteries in a matter of seconds, and the same
idea can be applied to troubleshooting the performance of just about
anything that is either open-source or at least has decent
function-naming conventions and is not stripped so that when you see a
stack trace you will have a clue as to where it is at.

strace is the first thing I use once I know the pid of the top  or
iotop dubious "winner".

It is also good to be a friend of lsof and netstat to figure out who
is on what file descriptor or port.

gdb can be used to poke around the memory of a trouble process. You
can discover some really cool things - e.g. who is holding the mutex
that is being waited upon by everybody. You can even fix up the memory
state of a running process if you have a good idea of what the result
should be. I have been able to make mysqld shut down cleanly in spite
of problems by modifying an internal variable (not a config one!) with

Sasha Pachev

Fast Running Blog.
Run. Blog. Improve. Repeat.

More information about the PLUG mailing list