Diagnosing a Compromised Server: Signs of an Active Breach

I get called into more "is this server hacked" conversations than I'd like, and almost every time the answer is buried in plain sight, in a process list, a netstat output, or a log file someone glanced at and moved past. An active breach rarely announces itself with a ransom note on day one. It shows up as a load spike that doesn't match traffic, a cron job nobody remembers writing, or an SSH session from an IP that has no business touching your infrastructure. This runbook walks through how I actually diagnose these situations on a box running Ubuntu or Debian, what each symptom usually means, and how to fix it once you've confirmed it.

Symptoms

Before you go hunting for root causes, here's the pattern of things that should make you stop and look closer. CPU usage sitting at 80-100% with no corresponding application load, often visible in

top

as a process with a random-looking name. Outbound connections to IPs you don't recognize, especially on ports like 4444, 6667, or high ephemeral ranges talking to IRC or unfamiliar VPS providers. New user accounts in

/etc/passwd

that nobody on the team created. SSH logins succeeding from countries or IP ranges outside your normal access pattern, sometimes at 3 AM your time. Files in

/tmp

,

/var/tmp

, or web-writable directories with recent modification times and no legitimate reason to exist. Cron entries you don't recognize, particularly ones that curl or wget something and pipe it to bash. And sometimes it's subtler: your monitoring shows disk I/O climbing steadily, or your mail queue on

sw-infrarunbook-01

suddenly has 40,000 outbound messages queued that your application never sent.

None of these alone is conclusive. Together, in the same 24-hour window, they're a strong signal you're dealing with an active compromise rather than a flaky app.

Root Cause 1: Rogue or Masquerading Processes

This is the most common thing I check first, and it happens because attackers who land shell access almost always drop a payload that needs to run persistently, whether that's a cryptominer, a reverse shell, or a bot client. They'll often name the binary something that looks like a legitimate system process, things like

kworker/0:1

or

systemd-udevd

but running from

/tmp

or a user's home directory instead of where the real binary lives.

To identify it, I don't just run

ps aux

and eyeball it, because a decent rootkit will hide itself from the standard tools. I cross-reference process listings against what's actually running from disk:

ps auxf --sort=-%cpu | head -20
ls -la /proc/*/exe 2>/dev/null | grep deleted
for pid in $(ls /proc | grep -E '^[0-9]+$'); do
  readlink -f /proc/$pid/exe 2>/dev/null
done | sort | uniq -c | sort -rn | head -20

That second command is the one that catches people. If a process's executable has been deleted from disk after launch (a common trick to dodge file-based AV scanning), you'll see

(deleted)

next to the path. That is a massive red flag. I've found cryptominers this way on boxes where

top

looked completely normal because the process had renamed itself to

[kthreadd]

with a bracket prefix, mimicking a kernel thread.

Once confirmed, kill the process, but don't stop there, because it will almost certainly respawn from a cron job, systemd unit, or init script:

kill -9 
grep -r "curl\|wget" /etc/cron.d/ /etc/crontab /var/spool/cron/crontabs/ 2>/dev/null
systemctl list-units --type=service --state=running | grep -v "known-good-list"

Remove the persistence mechanism first, then the binary, then kill the process. Doing it in the wrong order just gives it a chance to relaunch before you've closed the loop.

Root Cause 2: Unexplained Outbound Network Connections

A compromised server is a compromised server because it's useful to someone else, and that usefulness almost always requires it to talk to infrastructure the attacker controls, a C2 server, an IRC botnet channel, or a mining pool. This is why I always check active connections before anything else.

Frequently Asked Questions
How can I tell the difference between a legitimate high-CPU process and a cryptominer?
Check the process's executable path with readlink -f /proc/<pid>/exe. Legitimate application processes resolve to known binaries in /usr/bin or your application directory. Cryptominers typically run from /tmp, /var/tmp, or hidden dot-directories in a user's home folder, and often show sustained CPU usage across all cores with no corresponding application activity in your logs.
What is the safest first action to take once I confirm active compromise?
Isolate the network connection first (block the malicious outbound IP or take the host off the network entirely) before killing processes or deleting files. This stops active exfiltration or C2 communication while preserving forensic evidence, since killing the process first can trigger a cleanup script that erases the traces you need to understand how they got in.
Should I trust rkhunter or chkrootkit results if the server might already be rootkitted?
Treat a clean scan with caution if other symptoms persist, since a sophisticated kernel-level rootkit can hide from userspace scanners running on the infected kernel itself. If you have strong evidence of compromise but scanners come back clean, boot from external trusted media and inspect the disk from outside the running kernel for a more reliable result.
How often should I rebuild a server from scratch versus trying to clean it in place?
If you find a confirmed kernel-level rootkit, or you can't establish a reliable timeline of when the compromise started, rebuilding from a known-clean image is almost always faster and safer than in-place remediation. For contained issues like a single web shell or an unauthorized SSH key, in-place cleanup with a follow-up patch is usually sufficient.
Related Articles