Penguin’s secretes: Evidence collection in Linux

So, your task is to examine a Debian-based Linux distribution (Kali, Astra, etc.) and collect information about the host, user activity, and other artifacts that can be used in the subsequent analysis of third-party activities. Needless to say that it’s highly desirable to collect all data automatically, without using a bunch of packages with dependencies. Of course, you can manually enter all commands in the terminal, compose a checklist… But what’s the good of it? It’s much more convenient to write a Bash script and copy it to a flash drive!

You can either assemble your script from commands presented below or use our script available on GitHub. Also you can use this article as a Linux cheatsheet for rookie security guys. And of course, you can safely execute all these commands in the terminal: they only collect and display information without changing anything in the system.

If you intend to deal with Windows, format your flash drive in NTFS; if not, in ext4. Create a file ifrit.sh (stands for incident forensic response in terminal) and add a preamble to it. The script will record the execution time and create a directory for files collected on the PC under investigation (i.e. artifacts) and your logs. Its name should have the following format:

At the end of the main part of code, script will create a subdirectory in the current working folder for future artifacts and allow it to be copied; then it will record the working time spent and open this folder in a file manager. All your actions will be recorded in the general log.

Such scripts usually write everything to one integral log or to many small files; alternatively, they can try to systematize their output by creating an archive with interesting data. Your script will use the latter option (see below).

In the listings, you only specify the command and write what it does in the comments (to save space); its results are recorded in one of the theme files (e.g. >> case1.log) (this part is also omitted in the listings). In a real script, each command should be accompanied by its name and support information so that you see what’s going on:

Collecting host data

First, let’s collect basic information about the host and OS and write it to the host_info file. If you investigate multiple computers at once, you should be able to distinguish between them. The script should be run as root to ensure the most detailed command output. Where it’s absolutely imperative, commands will begin with sudo.

Basic information about activity: startups, shutdowns, laptop battery level (you might need it in some situations). In one of our investigations, a PC was turned on specially for us for the first time in five years…

Case 1. Who is screwing up?

Situation: some of the users misbehaves at work: installs Steam or writes articles for online portals… Your task is to find out who it is.

Step 1. Examining file system

First, you have to check activity in standard user directories. This is the fastest and easiest way to identify users who don’t clean up after themselves. You have to run a search for all home directories at once, which requires superuser privileges. If you are checking only one current user, write ~/ instead of /home/* in the commands below.

As a result, you can find traces of downloaded files, documents, and apps (e.g. deleted episodes of TV series in the Trash).

It can be useful to search directories for files with certain content or containing a term you are looking for. For instance, you can try to search for all files that contain the word “thereminvox” (the user could write a draft in a text file prior to publishing it online) and display two lines above and below it to understand the context.

You can also search for files with a certain extension. However, this command can take a very long time to execute.

It’s also useful to build a timeline of files in home directories, including modification dates, and save it to a CSV file. A timeline makes it possible to understand at what point something went wrong, what files were created, when, and where.

Step 2. Apps

Time to examine application lists. First, let’s see what is installed in the system in addition to the standard stuff. Check browsers, instant messengers, and mail servers: perhaps, some malicious deeds were committed using them? This information will be useful in the subsequent incident analysis.

Don’t forget to examine icons on the desktop. Below are commands and folders where third-party programs can store useful information (e.g. databases, passwords, history, and other data).

If you have a list of apps prohibited within the organization, you can store it in a separate text file and grep them from there (or use one-liners). For example:

Also, you can write down all packages installed in the system (just in case). Below are commands for various package managers, not just the standard ones included in Debian.

In the course of data collection, you can gain access to browser databases where settings, passwords, bookmarks, and browsing histories are stored (as well as settings of email clients and correspondence in them). Note that copying such files usually requires certain rights in the system. Application folders listed below exist only if the respective apps were launched and used.

At this stage, you find out what the user was doing and what apps were used. In addition, you download browser artifacts to subsequently extract from them the browsing history and saved credentials.

Step 3. Virtual malefactor

The most cunning way to perform malicious actions is to use for this purpose virtual machines or Windows apps deployed in Wine. The commands below will help you to find programs installed via Wine and examine their settings. Note that they must be run on behalf of root.

Checking for the presence of VirtualBox or QEMU.

To analyze virtual machines, you have to turn them on for manual examination or download their disks for subsequent analysis involving such utilities as Autopsy.

Step 4. Console lovers

Your key priority is to check user’s most recent activity. Keep in mind that, aside from ordinary graphical users, there is such thing as root, and some malefactors install packages using the terminal and commit various outrages on its behalf. Let’s start with the command history in the console.

Apps installed and uninstalled by the user prior to your investigation.

When you search for installed apps, it makes sense to examine the following files:

• /var/log/dpkg.log*;
• /var/log/apt/history.log*;
• /var/log/apt/term.log*; and 
• /var/lib/dpkg/status.

Sometimes you start checking a computer and, to your surprise, see a pristine Linux on it. Perhaps the main working OS is hidden somewhere?

Let’s sum up the interim results. You’ve got an idea of the most recent user’s manipulations with files, actions in the terminal, and interactions with graphical apps. You checked the Trash. You built a timeline of file changes in the main home directories. You checked for the presence of virtual machines and ported apps that could be used to perform malicious actions. This is sufficient to reconstruct actions of an ordinary user. Now let’s examine more specific cases.

Case 2. Access to Tor detected

Alarm! Someone (or something) connected to Tor from a workstation. You know the IP of the host supposedly used for this, IP of the target Tor node, and the date. The results of your investigation will be recorded in the network file.

Are you on the right host?

First of all, check whether the IP of the suspicious workstation matches the IP of the connection source:

If the PC’s network adapter is configured by DHCP, it makes sense to examine the following files:

After confirming that you are on the suspicious host, proceed with further steps.

Current connections and logs

You have to find out whether active connections include connections with the Tor IP address.

Let’s assume that nothing suspicious was found among active connections… Now you have to examine logs and configs.

You need information (including data contained in system logs) about networks the OS was connected to. All commands require superuser rights. The output will be recorded in a separate file called network_list.

Additional commands that require sudo:

This command shows the number of half-open connections:

If there are many such connections, you can assume that a SYN flood attack was delivered against the host.

At this point, you have determined the current network configuration, collected some connection history, and retrieved firewall settings. By the way, some malicious programs either completely remove the firewall (no one will notice this because a 6-month or even longer uptime is normal for Linux machines) or open and change ports for their operations.

Miscellaneous techniques

Let’s check logs for a more or less valid IP addresses using a regular expression:

Alternatively, you can try this variant:

Unfortunately, the host under investigation was connected to the same network all this time. Using data collected in Case 1, you find out that two apps were active on the day of connection: Firefox and qBittorrent. Nothing of interest is present in /var/log. However, you find in the downloads folder a torrent file downloaded on the same day and pirated content. What to do in such a situation? If you don’t know where the app writes its log, it can be useful to grep a specific IP address…

There is another weak, but workable technique that can be used to search for logs somewhere else. Log file names of different applications are highly variable (e.g. -log. *.log, log1, etc.), and there will be plenty of false positives. But then it will be easier to find logs of specific apps.

The performed analysis makes it possible to find the file containing download logs of the torrent client:

The torrent client operating time recorded in the log exactly coincides with the access to Tor. So, you start downloading this torrent on a test system, and the firewall immediately detects the IP address of the Tor node. Voila! A complete picture of the incident has been restored. Someone can say it was just luck. We won’t argue. In the most extreme cases, you can search the home directory for all files changed during a certain time interval:

Case 3. Miner

A complaint was received: some workstation operates too slowly. First of all, let’s find out in detail what’s going on in the system right now. The output will be saved to a file.

Sometimes you can find a suspicious process by simply looking at the ps output (source: Tenable blog).

In this particular case, everything is simple. All you have to do is enter the command htop, determine the PID of the process that consumes the most resources (e.g. 100% of the CPU horsepower), and kill it using the command kill -9 <your PID>. Also take a closer look at processes that were started from the /tmp/ folder or other anomalous directory.

A practical demonstration

Using the command top -bcn1 -w512, you’ve got the following output.

Some firefoxier consumes 100% of the CPU horsepower. Let’s find out what is it. Knowing its PID (62147), you get the path to the executable file; after that, you can dump it, delete it, or just be proud of yourself:

Success! In the case of a one-time accidental launch, this will at least enable you to restore normal host operation. Self-replicating malware and the ways it penetrates into the system and persists in it are beyond the scope of this article.

If you are dealing with a skilled malefactor, rootkit can be installed, and processes won’t appear on the list. However, in this particular case, the miner process was started without persistence on the host, and the careless user was reprimanded and explained why unknown software must never be run on workstations.

To collect a basic triage for subsequent submission to the SOC, you can add a special command to your script. Triage, simply put, involves procedures performed when you attend a cybercrime scene and need to quickly find and collect key information about the host. For instance, if you know that the criminal used Firefox, you have to collect browser artifacts in the first instance and then logs and other stuff depending on your area of interest.

Detailed analysis of logs is beyond the scope of this article. For instance, if you are dealing with Apache logs, you should grep all User-Agent lines of clients who had connected to your host.

In unclear situations, always use the grep command. If errors are present, grep error or fail. If you know a problematic service, grep its name or its part. If you detect connections from third-party IP addresses, see Case 2 and grep. Remember, grep is your key to success!

Case 4. Removable media

Many users plug personal USB devices into their workstations. In most cases, flash drives with movies; in worst case scenarios, Wi-Fi adapters… Therefore, it makes sense to look for traces of such safety violations. Use the commands below to view the list of previously connected hardware (will be saved to the dev_info thematic log):

An example with usbrip

The most important, of course, is to find out whether any unauthorized flash drives were covertly plugged into the corporate workstation under investigation. For Linux, there is an excellent utility called usbrip. A detailed instruction for it is available on Kali.tools. Install it and run the following command:

To view output in the terminal, press Enter. All available information for each USB drive contained in logs will be displayed, including its last disconnection date and serial number (for some reason, usbrip doesn’t display all serial numbers, even though they are listed in the logs). To view this info in the table form, type:

Self-grep

If you are too lazy to install anything and review logs yourself, use the command below (grep is your best friend, remember?):

The output volume will definitely impress you. To optimize your man-hour costs, save the output to files:

For more detailed information, let’s find out serial numbers of specific flash drives and display them in separate lines:

The output of the last command indicates that a third-party device (a flash drive) was plugged into the corporate workstation. Its serial number isn’t on the white list of the security department. Accordingly, the careless user is reprimanded and receives an extraordinary security training session.

Case 5. Malware with persistence and IOCs

This is a more severe case suitable only for seasoned threat hunters. Imagine a situation: you have to check a host for indicators of compromise (IOCs). For demonstration purposes, let’s look for files whose presence indicates a potential infection with Shishiga or RotaJakiro.

But in real life, such makeshift scripts and lists of IOC markers will be difficult to monitor and edit; so, it’s easier to use ready-made tools like YARA scanners or Fenrir, which is also written in Bash and requires only a list of files with IOCs as input.

All this information is equally relevant when it comes to searches for C2 server agents. To date, more than a dozen such agents for Linux have been analyzed. Unfortunately, to detect some of them, you have to perform traffic analysis (by User-Agent strings or JA3/JARM fingerprints), which requires special tools or separate unauthorized access protection systems. For local analysis in such situations, you have to install certain utilities and dump network traffic to a file (tcpdump) for subsequent malware activity evidence collection.

Analysis of locations used for malware persistence

Instead of such time-consuming analysis, let’s look for anomalies in typical locations where malicious programs reside and from where they could autostart themselves. Novice hackers simply google: how autostart program linux. Almost all commands listed below should be run with sudo.

If you suspect that malware is still running:

C2 emulation with cron

Cron job scheduler usually is a first-to-check candidate to detect anomalous activities. To emulate anomalous activity, let’s create a simple cron job that pings your emulated C&C server (a simple one: DNS canary token). After the reboot, this task will be executed once every 100 s.

After the reboot, you receive an alert, which indicates that the mechanism is working. Let’s figure out what commands can be used to check malware locations similar to cron.

Adding commands to the script:

Special locations

Not by cron alone; you can view jobs in the background, as well as job completion messages.

User scripts in startup:

Severe privileged malware can use GRUB for persistence in a compromised system. You can check autorun for weird constructs; first of all, this applies to programs that start after the kernel boot (constructs like init=...). To examine the details, download the list of all partitions and their identifiers (UUID) from fstab.

Review GRUB settings. In some distributions, the path is /boot/grub2:

The fstab file contains information about file systems and storage devices; sometimes, you can find in it lists of encrypted volumes and credentials (or paths to them).

Graphics lovers

Let’s examine autostart of graphical applications. You can add to its config triggers for user logon and logout, as well as system reboot. For instance, you can make the browser and messenger immediately open as soon as you log in. In our ‘evil’, demo, we will start the browser and concurrently open a port for communication with the malicious C2 server.

Save changes and check whether it works. After the logon, the browser starts, and the port opens.

You can add arbitrary command execution to text configs of such startup files (the Exec parameter).

Add validation of such files to the script: you never know what could write itself to them:

When you analyze executed commands, pay attention to /usr/bin directories: sometimes malware renames itself into wget or curl.

Configuration files

Hidden files (in Linux, they begin with a dot) in users’ home directories store various configs; accordingly, they can contain useful information about most recent activities, saved sessions, and the history of commands entered in the terminal.

In addition, shell configs are often used by malware to achieve persistence in the system. For instance, spoofed shells can be created or commands automatically executed when the terminal is started.

The Ares Python C2 server agent can be stored somewhere in home directories as a hidden folder:

Malware persistence mechanisms often involve the start and end of Bash sessions. As soon as the user opens the terminal, malware starts. A funny situation occurs when the admin decides that, after a year-long uptime, it’s time to reboot: the session ends, and a malicious agent is triggered. Such tricks include both automatic execution of commands and malicious aliases (you enter sudo, and, for some reason, rm -rf /home/ is executed). Let’s briefly examine some interesting files that can be extracted from home directories this way:

• .*profile – get default shell configuration files (e.g. .bash_profile). Commands contained in these files are executed every time the interpreter is started.;
• .*history – get the history of commands executed by all users in the terminal (if it exists);
• .bashrc – Bash parameters when an interactive shell starts, and a command is executed;
• .bash_login – terminal logon parameters and command execution;
• .bash_logout – terminal logout parameters and command execution;
• .bash_history – Bash command history;
• .zshrc – ZSH startup parameters (if installed);
• .zsh_history – ZSH command history; and 
• .history – in some distributions, the command history can be stored there.

Thoroughly examine the retrieved .bashrc files (or rc files of other shells). Since they contain commands executed when a terminal session is initialized, they will be launched automatically. Here’s an implementation example for such an injection:

In addition, rootkit can override the function execution order with the LD_PRELOAD parameter and insert in the .bashrc file a string like:

Persistence via profile parameters

Checking scripts executed every time the user logs into the system:

You can create the /etc/profile.d/evil.sh script with the following contents:

Use a Canary token as a C&C server. When the user opens the terminal, an alert is generated, and the results appear in the current variables (this can be checked using the set | grep TEST command).

Therefore, let’s add the export of all available variables to your script:

Artifact triage

To complete the investigation of this case, let’s collect files that can be useful for virus analysts. It makes sense to perform this procedure only in most severe cases, and an external hard drive may be required. Your objective is to go through all user folders and collect goodies. Let’s start with the root:

Next, you collect the same data for other users in a loop; for each user, you create folders storing artifacts retrieved from the host:

At the end, don’t forget to pack all collected artifacts:

Voila! You’ve got a folder containing all the results, collected artifacts, and configs from the analyzed host. We purposively don’t provide results obtained by each team: if this topic is of interest to you, you can collect and review them by yourself.

Case 6. Remote penetration

Imagine a situation: some evidence indicates that that an unwelcome remote guest appeared in your system. Let’s find out whether this is true.

Collect information about logons, examine basic parameters of users, and find out who of them can act as an admin.

One of the persistence techniques involves the collection and storage of SSH keys. Extracted SSH artifacts and settings make it possible to identify connections to a host performed using stored public keys or fingerprints of other hosts. Sometimes you can find host private keys; this enables you to examine its outcoming connections. This can be useful when you investigate complex incidents with multiple connections.

You have to examine the authorized_keys and known_hosts files, as well as key files generated by the ssh-keygen utility or copied from an external host using ssh-copy-id. Let’s ask the root and all users what they have in this respect:

If the obtained keys don’t belong to known systems, this could be bad news. Let’s also check remote logon attempts over SSH:

Logons over SSH can be viewed using the lastlog utility, but it’s also useful to grep logs and configs:

File anomalies

Don’t forget to check the shared folders config: what if an open share exists in the system and can be used for penetration?

Next, you have to look for files that don’t have an owner or group: their presence indicates remote penetration to the host and the use of reverse shells. For demonstration purposes, let’s search only in home directories.

CERT Societe Generale specialists recommend the following commands:

Logs and network info

To be on the safe side, let’s triage logs and save them to a file:

To check the firewall status, examine network settings. For instance, the presence of commands that disable it (ufw disable or iptables -F) in the history can indicate that something bad occurred in the system. Don’t forget to check the history for the following commands that disable protection services:

If you suspect that malware is hidden somewhere in the system, you can use iptables to block requests from known malicious user agents (e.g. Ares) on your host:

Conclusions

We purposively don’t collect memory dumps: such files are useful when you are dealing with a PC infected with ransomware, but their analysis takes time.

If you reasonably suspect that the host under investigation is infected with malware (by the way, where is your antivirus?), don’t forget to scan it using YARA rules or examine sigma rules’ approaches. To find rare specific artifacts, you can check a multi-OS artifacts collection called Digital Forensics Artifacts Repository (some of these artifacts have been discussed in this article).

Of course, the above set of commands can be refined infinitely… For instance, the next step can be to implement the proper processing of the collected data and their check for compliance. But this is a completely different level, which is beyond the scope of this article.

If you think that we forgot or missed something, don’t hesitate to blast us in the comments! Let’s collect useful techniques together.

Good luck!