How I cleaned up after a successful ssh attack
teletautala at gmail.com
Fri Dec 22 11:12:40 MST 2006
It has been suggested that I post this as it may help someone.
I was in a meeting and was notified that one of my web servers was ssh
attacking the world. I opened my laptop and shut ssh off and
continued with the meeting. Later I found out that the problem had
never been interrupted. I went to the physical box and executed top.
This showed me that there was a process pscan2 running and multiple
instances of sshd. I would pkill pscan2, but it would pop up again.
I installed and ran rkhunter (a rootkit hunter or in other words a
virus scanner), but to no avail. I remembered something I learned at
a troubleshooting class at GuruLabs about lsof. I found the pid of
pscan2. I did lsof -p <pid of pscan2> and found the location of the
files in use. I did cd into the folder and found the sshd program
that was being executed. I did chmod a-rx on the whole folder. This
stopped the traffic. I gave the files to the security officer for
diagnostics and further investigation. I have since turned iptables
on and inserted the lines:
[0:0] -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp
--dport 22 -j ACCEPT
[0:0] -I INPUT -p tcp -m state --state NEW --dport 22 -i eth0 -m recent --set
[0:0] -I INPUT -p tcp -m state --state NEW --dport 22 -i eth0 -s
22.214.171.124/255.255.0.0 -m recent --update --seconds 60 --hitcount 4
[0:0] -I INPUT -p tcp -m state --state NEW --dport 22 -i eth0 -s !
126.96.36.199/255.255.0.0 -m recent --update --seconds 300 --hitcount 2
This will prevent 4 attempts to connect within any 60 second period
from within the organization. The last line prevents 2 connection
attempts within any 300 second (5 minute) period from out side the
organization. I also created a command line program using grep and
awk that finds the ip addresses of all the failed attempts to connect
to the server from without the organization and checks to see if they
are in the /etc/hosts.deny file. If they are not they are added to
that file. I have since found a false positive and have put that
address in the /etc/hosts.allow file.
In order to make this server more secure I should change the port on
which ssh communicates and I should have reblasted the machine. It
has also been suggested that I have an off-site list of the files and
their md5 hashes (or the like) to know which file, if any, has
changed. There doesn't seem to be an immediate danger and nor any
further issues stemming from the attack. UEN (Utah Education Network)
monitors our traffic, because we get access through them. They are
the ones that saw the effects of the attack first and have not
mentioned anything about it since. I know I have not done everything
to ensure without a doubt a perfect clean up, but I have other issues
to tackle and I feel fairly confident that there will be no further
affects nor incidents.
I hope this helps someone.
More information about the PLUG