A few days earlier, while I was helping my friend administer his VPS, I noticed a suspicious entry in last command's output. To make sure nothing shady was running on the server, I ran ps aux to list all the processes running and started inspecting the output. By pure coincidence, I noticed that there were two cron daemons running, one with the name cron and one with the name crond. The VPS was running Ubuntu Server 14.04 and I knew from experience that the one with the name cron should be the one running by default. So why was there a process named crond running as well?

Investigation

To find the source of the crond process, I started by running the which crond command, but surprisingly got no output. By this time, I was sure that this process was fishy. To find the directory from where this process was running, I ran pwdx with its pid. The output was:

<pid>: /tmp/ .

I immediately cd'ed to the /tmp/ directory.

Inside /tmp, I ran ls -a, but to my surprise the directory was empty! There was nothing there except for the usual . and .. (at least from a quick glance). Confused, I looked at pwdx's output again and realized that the directory had been ingeniously named ' . ', i.e., 'space-dot-space'.

So, finally I cd'ed into that directory and saw a few dozen files sitting around. Rather helpfully, I even found a README file among those files. After a quick look at the file, I found that the files belonged to a common IRC bouncer named psyBNC.

Even more helpfully, I found a log file under that directory tree which showed the usernames of those connecting to psyBNC (presumably the *hackers*) and the network they were connecting to - which was Undernet. There were 3 usernames in total and I found that the log entries began two months back. So the server was presumably hacked two months ago.

Cleaning Act

Before getting rid of the infection, I first ran rkhunter to make sure there weren't any known rootkits running on the system. Thankfully, there weren't any other suprises and rkhunter even showed a warning about the infection directory due to its unconventional name.

Then I checked the username with which the crond process was running and found that (thankfully) it was not running with root privileges. Inspecting the files and directories under the infection directory didn't show any use of root privileges either.

So, to finally get rid of the infection, I first killed the crond process, then removed the directory housing it and just to be safe, deleted the username which crond was running with, as well.

Finding source of the infection

Unfortunately, we weren't able to establish the source of the infection authoritatively. After much urging by me, my friend vaguely recalled that some time back he had given the credentials used by the hackers to a support executive for a renowned e-commerce software. Comparing the date mentioned by him with the begin date of the psyBNC logs, there indeed seems to be a connection.

Oh, and in case you were wondering about the suspicious entry in the last command's output, that was a totally unrelated thing. But it sure started this chain of events ;).