Someone managed to get a rootkit on one of the boxes at work. The first clue we had was when the password was changed. Since I log in through ssh using no-password authentication this didn't affect nor stop me logging in. At first I thought maybe someone else at work had changed the password - but then it turned out that every command that made use of dates threw up a segfault - notably ls -l.

Whilst there are other reasons for that, it seemed likely that someone was trying to prevent the dates of files being shown, so as to “disguise” the new files they'd altered. Running chkrootkit confirmed that init was infected with the SucKIT rootkit (full details from phrack).

I needed to kill this - quickly. chkrootkit had identified two processes as hidden from ps. Using chkrootkit's chkproc program, I got the process IDs for both and had a look at the command line for each of them (found in the file /proc/process_id/cmdline). Bingo! One of them was running from /usr/.zk/echo.

I still couldn't kill the process though as it was hidden from kill as well as ps.

Looking through the file using strings echo identified other files. There was a hash of some sort stored in /etc/hash.zk and some other files in /usr/share/.zk. More importantly, there was also a reference to /usr/X11R6/.zk where I found the main part of the rootkit.

Thankfully, but for an inexplicable reason, they included an “uninstall” option in SucKIT. Supposedly, typing sk u would do that (see this article, down at the bottom). However, I got a different response when I tried that.

[zkmem](1.1)
Use:
./zk <option> <args>
u <password> * uninstall
k <pid> * make pid invisible
v <pid> * make pid visible
f <0/1><key> * toggle file hiding
p <0/1><key> * toggle pid hiding

Initially stymied, I was able to make the processes visible that had been hidden and kill them, cutting off the backdoor connection that was running. A futher search on Google turned up this PDF in which the password for this variant of SucKIT had been found: “kill me”. Running ./zk u kill me worked.

Unfortunately, the system had still be compromised and a number of files (most notably init) were infected. So it was decided to reformat the machine and install a newer version of Linux. An entire day wasted then.