How I cleaned up after a successful ssh attack

[Clint Savage]

Daniel,

Nice Follow-up.  I enjoyed the read and always appreciate how to do
something when it's resolved.  I find it frustrating when one hundred people
have the problem and one person says "I fixed it" but doesnt give an
explanation.

Thanx for following-up it could be very helpful to so many.

Cheers,

Clint

On 12/22/06, Daniel <teletautala at gmail.com> wrote:
>
> It has been suggested that I post this as it may help someone.
> I was in a meeting and was notified that one of my web servers was ssh
> attacking the world.  I opened my laptop and shut ssh off and
> continued with the meeting.  Later I found out that the problem had
> never been interrupted.  I went to the physical box and executed top.
> This showed me that there was a process pscan2 running and multiple
> instances of sshd.  I would pkill pscan2, but it would pop up again.
> I installed and ran rkhunter (a rootkit hunter or in other words a
> virus scanner), but to no avail.  I remembered something I learned at
> a troubleshooting class at GuruLabs about lsof.  I found the pid of
> pscan2.  I did lsof -p <pid of pscan2> and found the location of the
> files in use.  I did cd into the folder and found the sshd program
> that was being executed.  I did chmod a-rx on the whole folder.  This
> stopped the traffic.  I gave the files to the security officer for
> diagnostics and further investigation.  I have since turned iptables
> on and inserted the lines:
> [0:0] -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp
> --dport 22 -j ACCEPT
> [0:0] -I INPUT -p tcp -m state --state NEW --dport 22 -i eth0 -m recent
> --set
> [0:0] -I INPUT -p tcp -m state --state NEW --dport 22 -i eth0 -s
> 205.118.0.0/255.255.0.0 -m recent --update --seconds 60 --hitcount 4
> -j DROP
> [0:0] -I INPUT -p tcp -m state --state NEW --dport 22 -i eth0 -s !
> 205.118.0.0/255.255.0.0 -m recent --update --seconds 300 --hitcount 2
> -j DROP
> This will prevent 4 attempts to connect within any 60 second period
> from within the organization.  The last line prevents 2 connection
> attempts within any 300 second (5 minute) period from out side the
> organization.  I also created a command line program using grep and
> awk that finds the ip addresses of all the failed attempts to connect
> to the server from without the organization and checks to see if they
> are in the /etc/hosts.deny file.  If they are not they are added to
> that file.  I have since found a false positive and have put that
> address in the /etc/hosts.allow file.
> In order to make this server more secure I should change the port on
> which ssh communicates and I should have reblasted the machine.  It
> has also been suggested that I have an off-site list of the files and
> their md5 hashes (or the like) to know which file, if any, has
> changed.  There doesn't seem to be an immediate danger and nor any
> further issues stemming from the attack.  UEN (Utah Education Network)
> monitors our traffic, because we get access through them.  They are
> the ones that saw the effects of the attack first and have not
> mentioned anything about it since.  I know I have not done everything
> to ensure without a doubt a perfect clean up, but I have other issues
> to tackle and I feel fairly confident that there will be no further
> affects nor incidents.
>
> I hope this helps someone.
> -Daniel
>
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */
>