ARCHITECTURE LOG

Memory exploit mitigation techniques

How modern systems layer defenses to make exploits unreliable, expensive, and easier to detect.

2018-03-15

The best time to stop an exploit is before it runs. The second-best time is while it is still figuring out where it is.

Modern operating systems do not rely on a single wall around the application. They stack independent mitigations so that a typical buffer overflow — the kind that owned machines in the early 2000s — has to survive address randomization, non-executable data regions, exception-handler validation, return-address integrity checks, and heap hardening before it can do anything useful. Each layer is bypassable in theory. Chaining the bypasses together is what separates a proof-of-concept from a reliable exploit.

Diagram of layered exploit mitigations in a process address space — Modern exploit mitigations raise the cost of exploitation by forcing attackers to defeat multiple independent controls.

DEP: make data non-executable

Data Execution Prevention, or NX on Linux, marks stack and heap pages as non-executable. The idea is simple: if the attacker injects shellcode into a buffer, the CPU refuses to execute it from that memory region. DEP turns a stack overflow from a direct code-execution primitive into a problem of finding somewhere else to run code.

On Windows, DEP is enforced per-process via the ExecuteDisable bit in the PEB. On Linux, the ELF header can advertise the requirement:

// gcc -o vuln vuln.c -fno-stack-protector -z execstack
#include <stdio.h>
#include <string.h>

void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input);        // classic stack overflow
}

int main(int argc, char **argv) {
    vulnerable(argv[1]);
    return 0;
}

Compile this without -z execstack and the stack is non-executable. Overwrite the return address with an address on the stack and the process segfaults instead of running attacker code. DEP does not stop the overflow; it stops the injected code from running. That single constraint is what created the entire ROP arms race.

ASLR: hide the code that is already there

If the attacker cannot run injected code, the next option is to reuse code that is already mapped and executable. ASLR makes that hard by randomizing the base addresses of executables, libraries, stack, and heap every time the process starts. A hardcoded gadget address from one run is useless in the next.

ASLR is only as strong as its entropy. A 32-bit process with limited address-space randomness can be brute-forced or leaked. A 64-bit process with high entropy and position-independent code is a much harder target. The real weakness is information disclosure: one leaked pointer from a format-string bug or an out-of-bounds read can collapse ASLR by revealing where a module is loaded.

Practical note: ASLR works best when every module participates. One DLL loaded at a fixed address becomes a reliable gadget library for every exploit in the process.

SEHOP: protect the exception handler chain

On 32-bit Windows, Structured Exception Handling used a linked list of handlers stored on the stack. A stack overflow could overwrite the next handler pointer and the current record, redirecting exception dispatch to attacker-controlled code. SEHOP validates the entire chain before dispatching: the tail must still point to the OS-defined terminator, and the links must form a consistent list.

SEHOP is a good example of a mitigation that targets one specific exploitation technique rather than the underlying vulnerability. It does not stop the overflow; it stops the overflow from weaponizing the exception path. That is the pattern these controls follow: shrink the set of things an attacker can reliably do with a memory corruption bug.

Heap mitigations and null-page protection

Heap spray attacks fill memory with attacker-controlled data at predictable offsets, then trigger a vulnerability that jumps into the sprayed region. Mitigations include reserving commonly sprayed addresses, adding guard pages, randomizing heap metadata, and delaying frees. They do not eliminate heap corruption; they make the corruption harder to aim.

Null-page protection is similar in spirit. On some 32-bit systems, the zero page is mappable. An attacker who can write a null pointer dereference into a controlled write primitive may be able to place a fake structure at address 0x00000000. Pre-allocating and guarding the null page removes that address as a useful target.

DllLoad: block remote DLL loading

Some exploits load a malicious DLL from a remote share like \\malicious.com\malware.dll. The DllLoad mitigation blocks DLL loads over remote SMB paths, closing an execution vector that does not need a memory corruption bug at all. It is a policy control, not a memory control, and it shows how exploit mitigations overlap with broader hardening.

ROP mitigations: detect the workaround

Return-Oriented Programming reuses short instruction sequences — gadgets — that already end in ret. By controlling the stack, the attacker chains these gadgets to build arbitrary behavior without injecting code. Because ROP was the direct answer to DEP, operating systems added ROP-specific detections.

StackPivot detects when the stack pointer is moved to attacker-controlled memory.
RopCall ensures that critical APIs are reached through call instructions, not jmp or ret chains.
RopHeap blocks attempts to mark heap memory executable via memory-protection APIs invoked through ROP.
RopFlow simulates the return-address chain when a sensitive API is called, looking for sequences that jump between critical functions or land in non-executable memory.

A minimal ROP chain might look like this in abstract form:

; attacker-controlled stack
0x7ffe0000  dd  0x77a10000 + pop_rcx_offset   ; gadget: pop rcx
0x7ffe0008  dd  0x0000000000000040            ; PAGE_EXECUTE_READWRITE
0x7ffe0010  dd  0x77a20000 + pop_rdx_offset   ; gadget: pop rdx
0x7ffe0018  dd  0x00001000                      ; size
0x7ffe0020  dd  0x77a30000 + VirtualProtect    ; call target
0x7ffe0028  dd  0x41414141                      ; return address

Each dd is a return address the attacker writes to the stack. The CPU pops through the gadgets, sets up arguments, and calls VirtualProtect to make the heap executable. RopCall would flag the ret-into-API pattern. RopFlow would simulate the chain and notice the jump into a sensitive API from an unusual return path.

Why layers matter

No single mitigation is enough. DEP is bypassed by ROP. ASLR is bypassed by an info leak. SEHOP is irrelevant on 64-bit processes with table-based exception handling. Heap hardening does nothing for a stack overflow. The value is in the composition: an exploit that needs an info leak, a gadget chain, a stack-pivot bypass, and a heap-spray workaround all at once is fragile, noisy, and often target-specific.

That fragility is the point. Memory-safe languages are the long-term answer, but while legacy code remains in production, these layers are what keep low-skilled exploits from becoming reliable weapons. They shift the attacker from “one bad input owns the machine” to “severerelaxed bugs, a leak, and a deep understanding of this specific binary.”

Takeaway for engineers

If you are shipping native code, treat mitigations as table stakes, not guarantees. Enable DEP and ASLR. Compile position-independent code. Remove fixed-address modules. Design your logging to catch the failure modes these controls produce: segfaults at unusual addresses, repeated exception-handler validation failures, and memory-protection calls from unexpected contexts. The goal is not to make exploitation impossible. It is to make exploitation so expensive that the attacker looks for an easier target.