Anti-Analysis 101: Understanding ptrace and prctl Evasion Techniques in Linux
In cybersecurity research and Red Team simulations, developing custom tools requires a deep understanding of host-based evasion. When an agent lands on a target system, modern Blue Teams and Endpoint Detection and Response (EDR) solutions will attempt to attach a disassembler or a debugger to analyze the suspicious process.
How do these processes defend themselves against analysis? In this article, we will explore the technical details of how the Linux kernel's own mechanisms—ptrace and prctl—can be utilized for process self-defense, strictly using pure x64 Assembly.