compromised Linux box

[Shane Hathaway]

Matthew Frederico wrote:
> On 4/12/07, Dallin Jones <squitoey at gmail.com> wrote:
>> I had a server of mine compromised earlier today, and it made me
>> contemplate the measures and steps every one takes to ensure that
>> their box doesn't get compromised and when it does happen, how do you
>> know that it happened? In the meantime, I'll get back to the
>> re-imaging of my server. (Thank goodness for working backups!!!)
> 
> 
> The best way to know that you've been compromised is when you start getting
> calls from your hosting provider or co-lo facility that people are
> complaining about spam, or your web site has turned into a porn site
> overnight.  That's a dead giveaway so I've been told.

Ha ha.

I've had two Linux boxes compromised before.  On the first, which was 
connected to the Internet via a modem (!), the shell started behaving 
strangely.  I don't remember what it did exactly, but the root kit that 
hit the machine replaced some executables without noticing that the 
replacements linked with the wrong libraries.  Duh.  Then I not only 
wiped the machine, I switched distributions.

On the second box, I noticed one day via netstat (which I use often as a 
simple network debugging tool) that there was an extra TCP server 
running.  The process that was apparently listening did not show up in 
"ps" or "top".  I didn't know why they wouldn't show up, so I 
investigated more and found that some executables had been replaced with 
a version that hides the root kit, but others had not.  I didn't wipe 
the machine right away, but I eventually did and switched distributions 
again.

Not wanting to be burned again, I'm now using Linux-VServer as a method 
of containing break-ins.  I have a front-end web server that runs 
software with a history of vulnerabilities.  I have second server that 
handles mail.  I have a third server that accepts SSH connections.  All 
three servers are in a single box.  Outside the virtual contexts, there 
are cron jobs that back up the virtual servers and run rkhunter every night.

I think the box is fairly safe now, but new vulnerabilities appear 
daily, so I watch the advisories on lwn.net.  I also plan to set up 
remote incremental backups, since my family is starting to store 
irreplaceable information on that server.

Shane