Successful SSH Attack - Need help cleaning up

[Daniel]

On 10/27/06, Ryan Simpkins <plug at ryansimpkins.com> wrote:
> Secondly, and to back up a bit, how do you know that it was via SSH they gained
> access? Is SSH the only service running on your system?
>
> Did they infiltrate your system using another method, and then gain escalated access
> via SSH? If so - reinstalling and changing SSH ports won't slow them down much.
>
I plead the 5th on who's fault it is, but there was a test user that
was created with a weak password for testing purposes.  This was done
on a Thursday or Friday.  The following Tuesday morning we found that
someone was scanning ports and trying to ssh different servers.
I installed a rootkithunter and found nothing then froze so I killed
it.  I did a top and saw pscan2.  I then did lsof on pscan2.  I found
that it was in /dev/shm/.\ /hosts/
--w-------  1 1234565 123123     307 May 11 01:32 a
--w-------  1 1234565 123123     200 Oct 10 08:45 nobash.txt
--w-------  1 1234565 123123  121007 May 11 01:35 pass.txt
--w-------  1 1234565 123123    5944 May 15  2005 pscan2
--w-------  1 1234565 123123    5797 May 15  2005 pscan2.c
--w-------  1 1234565 123123     307 May 11 01:33 scan
--w-------  1 1234565 123123       0 Oct 10 11:11 scan.log
--w-------  1 1234565 123123 1384518 Jun  5  2005 sshd
--w-------  1 1234565 123123    3632 May 11 01:33 start
--w-------  1 1234565 123123      47 Oct 10 05:18 vuln.txt

I did chmod a-x on all the files in that folder.  pscan2 stopped.  I
copied these files to the security officer for analysis.  I thought
everything was fine so I opened up port 22.  I shut off outside access
through port 22 when I found out it wasn't logging to /var/log/secure.
 It was logging to /var/log/messages instead.  I have now reinstalled
ssh and it is logging to /var/log/secure.
This is probably way too much information, but this is what happened.
I need to give the patrons notice that the webserver will be down so I
will reinstall the OS on Friday.  I will try to use a different port
and implement the iptables approach to deterring attacks.

Thanks for all your help.
-Daniel