A Beginner’s Guide to Process Inspection and Troubleshooting on Linux

Understanding processes forms the foundation for many critical security tasks on Linux systems. Whether conducting threat hunting, incident response, or malicious file analysis, you need visibility into process activity to understand what is happening across a system.

When suspicious events occur or malware infiltrates a server, processes provide the clues to uncover the source. By inspecting process relationships and changes, a security analyst can reveal the impact of threats and track down the root cause even if other artifacts like files have been deleted.

This beginner’s guide aims to equip you with two powerful commands for Linux process troubleshooting – ps and pstree. With mastery of these tools, you will gain the ability to:

  • Scan for hidden threats through comprehensive process inspection

  • Understand the context around suspicious processes

  • Pivot investigations by revealing connections between processes

  • Build foundations for more advanced Linux analysis and hunting

Let’s dive into process inspection and troubleshooting and discovery!

Using ps to View Processes

The ps command in Linux provides comprehensive visibility into the processes running locally or system-wide at any point in time.

At a basic level, ps displays details like process ID, owner, CPU usage, executable path, and more about the processes running on the current terminal session. But its real power comes from a range of options that allow tailored inspection of all process activity – not just the small subset you can observe.

$ ps
  PID TTY          TIME CMD
 9835 pts/0    00:00:00 ps
$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND 
root           1  0.0  0.0  19356
A screenshot of a terminal window on Ubuntu showing the output of the 'ps' and 'ps aux | more' commands, listing various system processes and their details such as PID, CPU usage, and memory usage.

Common options like aux-f, and -e give you different lenses to peer into process execution:

Stacked together, ps auxef can expose all processes, users, and arguments to detect anomalies.

Beyond details, ps can also visualize hierarchical process trees with the --forest argument. Trees highlight the ancestors and descendants within a process chain – invaluable context during incident investigation.

$ ps -ef --forest
A screenshot of a terminal window on Ubuntu displaying the process status 'ps' command with the '-ef --forest' options, showing a hierarchical tree of system processes.

With crafty manipulation of ps, you can illuminate all facets of process activity to pinpoint threats.

Digging into Processes with pstree

While ps acts as a flashlight into process details, pstree focuses its firepower on mapping relationships between processes for enhanced detection and hunting.

At its core, pstree visually wires together processes into a tree based on which processes spawned child processes. This immediate context delivers insight into the chain of events tied to any suspicious process:

pstree
init─┬─apache2───9*[apache]
     ├─sshd─sshd─bash─pstree
     ├─xorg───{xorg}

Augmented with parameters like -p for PIDs and -u for users, you can precisely track owners and pivot between processes. The -g flag surfaces groupings to reason about connections between process clusters.

  • -p: Shows PIDs in the process tree

  • -u: Shows usernames that own each process

  • -g: Displays process group IDs

  • -a: Displays with argument information

A screenshot of a terminal in Ubuntu running the 'pstree' command, displaying a tree of running system processes with a focus on Apache2 processes.

Just like adversaries, pstree allows you to follow threads of execution through relationship graphs. This facilitates revealing signs ofattack tactics as well as cleanup remaining after threats.

With ps + pstree, you have a toolkit to dissect process state and behavioral connections for battling threats. Now let’s see them in action together!

pstree -g -p -a
A terminal screenshot showing the output of the 'pstree' command on an Ubuntu system, illustrating the process tree with group IDs, process IDs, and arguments for system services and the Apache2 web server.

Putting It All Together

Imagine during server monitoring you come across a Python process consuming 50%+ CPU constantly over the past few days – highly abnormal. After isolation, you still find the process running with its high workload.

See also  5 Challenges of Cyber Security in Today’s Business!

Time to go process hunting! Your goal is to determine the purpose of the process, its origin, and if it poses a threat.

Visually mapping program relationships with pstree -p reveals the parent PID of the Python process belongs to an sshd foreground worker. Expected so far.

But further inspection of Python’s command line arguments with ps -fep <PID> uncovers a reference to an unusual script /tmp/export.py.

Exporting data from /tmp is suspicious and may indicate exfiltration activity. But what access does this script have?

Using pstree -ugp exposes the Python process running as root!  Privilege escalation provides the answer – a critical piece of the puzzle.

With the context provided by creatively leveraging ps + pstree, you successfully tracked down the threat vector and overall impact of the crypto mining malware disguised as an innocent system process.

Conclusion

Process analysis forms the foundation of  Linux security  operations. With ps and pstree now in your toolkit, you have the power to illuminate process state and relationships to enhance detection, incident investigation, and threat hunting.

Some key points to take away:

ps – Inspect process details like users, CPU usage, full command lines

pstree – Map hierarchical relationships between processes

Together – Leverage process context to identify, investigate, and hunt threats

I challenge you to test your skills on your own Linux systems! Experiment with combining ps and pstree to deep dive into how processes interconnect. The more hands-on practice, the faster you will unlock these tools’ immense power. Happy hunting!

Leave a Reply

Your email address will not be published. Required fields are marked *