Scanning for Rootkits

Source: O'Reilly Network

Usually, the first sign that a server might be compromised is simple anomalies in the behavior of the server. One of the more common anomalies one might notice is a change in how one or more of the core system utilities behave. For instance, a command-line switch to 'netstat' or 'ps', which you used to use without a problem everyday, might start returning an error message. The reason for this is that intruders replace these utilities with versions designed to hide their malicious activities. The utility they replace your original one with might be a different version, or it could have been compiled with different options, and as a result, it does not have the same options you are used to.

Another anomaly that should raise a big red flag is a change in your bandwidth-usage patterns. If you or your hosting company routinely monitors your bandwidth usage, you might notice an increase in the amount of traffic your server is pushing compared with your normal traffic patterns. This is usually caused by intruders using your server to distribute copyrighted software, commonly known as 'warez'. Remember that you might be the target of a legal action as a result of such activities on your servers.

Ideally, a server administrator should not wait until all the alarms go off before a server is checked for signs of compromise, because the less time an intruder has the opportunity to spend on a server, the less damage he or she will be able to inflict. (Although a very malicious intruder can potentially wipe out the whole system seconds after gaining root-level access.) For this reason, it is important to conduct server-security audits periodically, and to know as soon as possible when a server is compromised.