Conducting Forensics on Linux: Memory, Disk, and Network Dumping for Evidence Collection

This is a new installment in our forensic series for beginners, where we explain what digital forensics is, explore the most popular analysis tools, examine a few case studies on Android devices, and investigate the theft of funds from an online banking system on a Windows 10 laptop.

There are many ways to create a dump of a hard drive’s contents or system memory. You can use native utilities included in the distribution as well as third-party programs—available under both open-source licenses and commercial licenses. I will focus, where possible, on the most well-known, straightforward, and effective tools.

To begin with, for monitoring active systems, I recommend installing the utility Auditd. It provides detailed information on system changes in audit mode.

First and foremost, we will be focusing on events such as:

• System start and shutdown (reboot, halt)
• Reading/writing system files and modifying their access permissions
• Initiating network connections and changing network settings
• Modifying user or group information
• Changing date and time settings
• Installing, removing, starting, and stopping programs and daemons
• Executing system calls

It’s also recommended to configure the system’s syslog.conf and adjust the message collection rules: increase the depth of collected alerts (level 7) and expand the pool of sources (ranging from 0 to 15). This will allow you to monitor system changes in real-time.

Key Aspects of Forensics on Linux

In a previous article focusing on a case of financial theft in online banking on Windows 10, we utilized programs with graphical interfaces whenever possible. Excluding proprietary solutions like EnCase Forensic or Belkasoft Evidence Center, most tools on Linux operate in command-line mode.

The use of test commands can significantly save time when working with software; however, it might be too complex for beginners. But then again, beginners usually don’t engage in forensic analysis! 😀

In addition to individual utilities, there are entire distributions designed for digital forensics. These include DEFT, CAINE, Sumuri PALADIN, Helix, and, of course, the well-known Kali Linux.

When it comes to literature on Linux forensics, I would first recommend what is arguably the only comprehensive book on the topic: “Linux Forensics” by Philip Polstra. Second on the list is “UNIX and Linux Forensic Analysis DVD Toolkit” by Chris Pogue and others. Lastly, consider “Digital Forensics with Kali Linux.”

Comprehensive Inspection Checklist

To search for and gather forensic evidence, the first step will be to create images (dumps) of the following system objects:

• RAM (system and user processes, daemons, potentially running malicious code, etc.);
• Hard drive (sector-by-sector copy of the HDD, including deleted partitions, unpartitioned areas of the disk, shredded files, hidden files and directories, and more);
• Network stack (active connections, open ports, “unknown” services on ports, suspicious traffic).

Within the operating system itself, we will particularly focus on the following aspects:

• List of users, groups, and privileges.
• Processes running with root permissions.
• Scheduled tasks (cron jobs).
• Files with setuid and setgid bits.
• Contents of the /etc/sudoers file.
• Hidden files and directories.
• Files currently opened for reading.
• Network interfaces, connections, ports, and routing table.
• Logs from iptables and fail2ban (Reports, Alarms, Alerts).
• Configuration of /etc/ssh/sshd_config.
• Syslog daemon logs (checking for typical alerts).
• Status of SELinux.
• List of loaded kernel modules.

As an additional option, you can generate checksums for the main system files (for example, using the tool TripWire) and later compare them with the standard values from the original distribution. However, this process is quite complex and requires a considerable amount of time and patience, so we will kindly skip the details.

Since the hardware running the entire operation is located in a secure data center, it rules out the need to search for artifacts related to Network File System (NFS), locally mounted devices, and USB-connected devices.

Creating an HDD Image

You can create a sector-by-sector copy of a hard drive without resorting to additional utilities. We’ll use the classic and reliable native tool dd. This utility enables the creation of exact bit-by-bit copies of entire drives, specific partitions, or even individual files.

First, let’s retrieve a complete list of partitions from the system using the fdisk command:

The basic syntax for the dd command is as follows:

For example, to create a copy of an HDD with a cluster size of 512 bytes:

During the process of copying an HDD, there may be damaged sectors. To ensure the software doesn’t stumble on these and halt operations, you need to add the additional flag -noerror:

However, the best global practices involve using an enhanced version of the previous utility called dcfldd. This tool was developed in the computer forensic laboratory of the DCFL and includes several specialized options for creating dumps for forensic analysis. For instance, dcfldd can hash copied data and verify its integrity. It also displays the progress of the dump creation, logs the actions performed, and saves MD5 checksums in a separate file.

Example of command execution:

Instead of using the command line, you can opt for the graphical version of the FTK utility, which we are already familiar with from the previous article.

Creating a RAM Dump

As with a hard drive, there are several ways to tackle this issue. Some options include:

• Use the native kernel module Linux Memory Extractor (LiME);
• Employ the script Linux Memory Grabber, which doesn’t require installation and can be run from a USB stick, for example;
• Utilize the utility bundle lmap and pmem, which are part of the Rekall package. These are the tools I will be using.

A few words about Rekall. It is a separate branch of the well-known Volatility Framework written in Python, specifically designed for inclusion in forensic distributions that operate using Live CDs.

Navigate to the directory containing the tool and compile it from the source code:

We load the kernel driver pmem.ko into RAM:

Checking the driver initialization:

After this, the driver creates a container file for our future RAM image: /dev/pmem.

Now, using the same dd utility, we create an image of the system’s RAM:

Finally, we unload the driver after completing the task:

It’s done!

Network Traffic Analysis

There are several tools commonly in use: primarily the command-line utility tcpdump, the classic Wireshark, and the open-source framework XPLICO, although the latter is more often utilized for subsequent data analysis rather than initial collection. For those who are not familiar with these programs, there are comprehensive tutorials available, such as using tcpdump for network traffic audits and maximizing the display filter capabilities of Wireshark.

Let’s start with tcpdump. The basic command syntax looks as follows:

Here are some of the most important options:

• -i interface — specifies the interface from which to analyze traffic;
• -n — disables IP address to hostname resolution;
• -e — enables display of link-layer data (e.g., MAC addresses);
• -v — provides additional information (such as TTL, IP options);
• -w filename — sets the filename to save the captured information (dump);
• -r filename — reads (loads) a dump from the specified file;
• -q — puts tcpdump into “quiet mode,” analyzing packets at the transport layer (protocols like TCP, UDP, ICMP) rather than the network layer (IP protocol).

We are capturing all incoming traffic from the internet to our server:

Example of creating a network traffic dump for FTP or SSH protocols on the eth0 interface:

We’re dumping everything that goes through the eth0 interface.

A suitable utility for our purposes is TCPflow. Essentially, it’s a more advanced version of tcpdump that supports additional filtering options and the ability to reconstruct “broken” packets.

If TCPflow isn’t installed by default on your system, start by installing the tcpflow package.

The basic syntax of the command is as follows:

Here is a description of the options:

• -c — console output only (do not create files);
• -d — debugging level (default is 1);
• -e — display each stream in alternating colors (blue for client-server, red for server-client, green for unknown);
• -i — network interface for listening;
• -r — read packets from a tcpdump output file;
• -s — remove non-printable characters (they will be replaced with dots).

An example of data collection from an external network to our server:

Capturing all HTTP traffic on our network:

Dumping network traffic data to a local folder:

Files containing the content of network connections will now be stored in the /tcpflowdata directory. All we need to do afterwards is to transfer them to a parser for analysis.

Accessing Collected Data

The disk and memory images are ready. Now we need to mount them on the research machine to start searching for and analyzing artifacts. Mounting a HDD image is straightforward; the output file generated by dd or dcfldd can be easily mounted as a new device.

The syntax for the mount command is as follows:

For instance, to mount a previously created HDD dump, you need to execute the command

We will use Volatility Framework to analyze the RAM image. More details will be provided in the section “Artifact Hunting” below.

A .pcap file containing network connection dumps can be opened in Wireshark or XPLICO. However, I prefer using services with semi-automatic parsing, such as PacketTotal, Pcap Analyzer, or the paid CloudShark.

Additional Useful Commands

Here are a few commands that will help us gather anything we might have missed. Some of these are just in case, as they won’t take much time.

Instantly Capture System State and Key Configuration Details

Remount All File Systems as Read-Only

For each file system we will be examining, we’ll remount it in ro (read-only) mode:

Let’s create a timestamp or timeline:

Obtaining the Status of Existing Connections and Open Sockets

Current ARP Cache Dump

Capturing the Current State of iptables Network Filter Rules

Dumping ipsets (commonly used by fail2ban and firewalld)

Saving the Process Table

Another option for more readable output:

Preserve the Original Location of Executable by PID

Viewing All Open Files on the System

If a file has been deleted, the ls command will append the flag (deleted) to the destination file’s name. However, the content can still be accessed using symbolic links in /proc/${PID}/fd. This is often helpful when dealing with malware written in interpreted languages like Perl and Python. Therefore, for further analysis, we will save all open files:

Conclusion

Now we have everything needed for further analysis. In the next and final article of the series, I will demonstrate how to use the gathered data to identify artifacts. To make it more engaging, I have an interesting case study involving the detection of malware that infiltrated a hosting provider’s servers. Stay tuned!