Linux Distro Supply Chain Attacks: XZ Utils to Dirty COW

Quick direct answer: The XZ Utils backdoor hijacked OpenSSH by hiding obfuscated build scripts inside release tarballs. Dirty COW weaponized a copy-on-write kernel race condition to grant root privileges across every major distribution. Both flaws bypassed standard firewall rules. Run this command on your servers to see if your SSH daemon loads liblzma:

ldd /usr/sbin/sshd | grep -E 'liblzma|systemd'

On the Friday before Easter in 2024, our team spent ten hours auditing every server in our lab. Andres Freund had just discovered that an upstream maintainer spent over two years quietly backdooring xz-utils.

What caught us off guard was not the payload itself. It was the fact that upstream OpenSSH never linked to liblzma. Downstream packaging glue had created an invisible side door into the SSH daemon.

Here is what we found on our own test bench, how it compares to Dirty COW and the 2008 Debian OpenSSL disaster, and the script we now run on our hosts.

1. How Three Major Linux Failures Actually Happened

Linux runs our core infrastructure, but open-source maintenance comes with real operational risks. Here is how three major compromises compare:

2. The Packaging Glue Trap: How OpenSSH Got Hooked

The attacker operated under the pseudonym Jia Tan. Over three years, they made useful pull requests and answered bugs. They slowly earned the trust of the original maintainer, Lasse Collin.

When the attacker gained release authority, they did not push suspicious code to GitHub. They slipped obfuscated M4 macros into the official 5.6.0 and 5.6.1 release tarballs.

The build script checked whether the package was being built into an RPM or Debian package on an x86-64 system. If it was, the script extracted malicious object code from corrupted test files (bad-3-corrupt_lzma2.xz).

[HOW THE DEPENDENCY CHAIN REACHED OPENSSH]

/usr/sbin/sshd (Runs as root on port 22)
  |
  +--> Patched by Debian/Ubuntu for systemd notifications (sd_notify)
  |
  v
libsystemd.so (Loads into the sshd process address space)
  |
  +--> Links compression libraries to handle journal files
  |
  v
liblzma.so (Infected versions 5.6.0 and 5.6.1)
  |
  +--> Malicious IFUNC resolver executes BEFORE authentication starts!

Upstream OpenSSH maintained by the OpenBSD team does not link to systemd. But distribution maintainers wanted systemd to know when SSH finished binding to port 22.

They added sd_notify() calls. That required linking libsystemd.so.

Because libsystemd compresses journal logs with xz, it links liblzma.so. Just like that, an unauthenticated network daemon loaded a compromised compression library directly into root memory.

3. The Memory Hook: Abusing glibc IFUNC Resolvers

The backdoor did not touch environment variables. It did not use LD_PRELOAD.

Instead, it abused GNU Indirect Functions (IFUNC). Compilers use IFUNC to pick optimized assembly routines at runtime based on the host CPU.

The attacker attached malicious payload code to the IFUNC resolver for crc64_resolve. When sshd started, the dynamic linker resolved IFUNCs before main() ever executed.

The hook checked the process name. If the executable was /usr/sbin/sshd, it reached into the Global Offset Table (GOT). It swapped out the pointer for OpenSSL’s RSA_public_decrypt.

When an attacker sent a malicious public key payload during an SSH handshake, the hook executed the payload as root. When anyone else connected, the hook stepped aside and let normal password or key authentication proceed.

4. Testing the Blast Radius on Our Lab Machines

When the security advisory dropped, we spun up VMs across our lab to check our exposure. Here is what we found on each operating system:

Debian Sid (Unstable)

Debian testing and unstable repositories had already pulled versions 5.6.0 and 5.6.1. Running ldd on our test host confirmed that sshd loaded the bad library:

ldd /usr/sbin/sshd | grep liblzma
# Output: liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9c8d200000)

Debian 12 (Bookworm) stable was never vulnerable. It ran xz-utils version 5.4.1.

Ubuntu 24.04 LTS (Noble Numbat)

Ubuntu beta repositories pulled the bad version for a brief window. Canonical stopped package syncs and yanked the beta binaries before general availability.

Production servers running Ubuntu 22.04 LTS or 20.04 LTS stayed safe on older 5.2.x and 5.4.x releases.

Arch Linux Rolling Releases

Because Arch is a rolling release, users who ran pacman -Syu between March 26 and March 29 received the backdoored binary. Arch maintainers issued an emergency downgrade package within hours.

5. Dirty COW: The Race Condition That Broke Linux for Nine Years

Eight years earlier, our team ran into a very different Linux vulnerability: Dirty COW (CVE-2016-5195).

XZ Utils was an intentional supply chain backdoor. Dirty COW was an accidental concurrency bug that sat inside the Linux kernel copy-on-write memory code from 2007 until 2016.

How the Race Condition Broke Permissions

Linux uses Copy-on-Write (COW) when an unprivileged program requests write access to private memory mappings. The kernel makes a private copy of the page so the original file stays unchanged.

Dirty COW pitted two system calls against each other:

1. Thread A repeatedly wrote to /proc/self/mem.
2. Thread B repeatedly called madvise(addr, len, MADV_DONTNEED).

The madvise call told the kernel to discard the private copy. If the timing aligned, Thread A wrote to the original read-only file instead of the copy.

[DIRTY COW RACE TIMELINE]

Thread A: Writes to /proc/self/mem (Requests private copy)
  |
Thread B: madvise(MADV_DONTNEED) drops the private copy!
  |
Kernel:   Applies the write to the ORIGINAL root-owned file on disk!

We tested this in our lab on an old CentOS 6 virtual machine. An unprivileged user account modified /etc/passwd to grant root privileges in under three seconds.

It proved that container boundaries using cgroups and namespaces fall apart when local users can alter host kernel memory.

6. The 2008 Debian OpenSSL Flaw: When a Two-Line Fix Broke Crypto

The XZ Utils incident was not the first time downstream packaging changes created a major security flaw.

In 2008, a Debian developer tried to clean up compiler warnings generated by Valgrind. Valgrind complained that OpenSSL was reading uninitialized memory inside crypto/rand/md_rand.c.

The developer commented out two lines of code to silence the warnings:

/* Removed to silence Valgrind warnings: */
/* MD_Update(&m,buf,j); */

Those two lines happened to feed system entropy into the random number generator.

With those lines gone, the only variable left in key generation was the current process ID (PID). On Linux, PIDs cap out at 32,767.

Every SSH host key, SSL certificate, and VPN secret generated on Debian and Ubuntu between 2006 and 2008 came from a pool of only 32,767 keys. Anyone with a precomputed dictionary could crack SSH private keys in milliseconds.

7. Our Team’s Production Linux Audit Script

To verify our hosts during security alerts, we wrote this Bash script. It checks dynamic library dependencies, verifies the xz version, and scans package hashes:

#!/usr/bin/env bash
# PTW Linux Dependency & Package Integrity Audit Script
# Tests: SSH linkage, xz version, debsums, rpm verification

set -euo pipefail

echo "============================================="
echo "PTW Linux Security & Dependency Audit Engine"
echo "============================================="

# 1. Check SSH Dynamic Library Dependencies
echo "[*] Checking /usr/sbin/sshd dynamic libraries..."
if command -v ldd >/dev/null && [ -f /usr/sbin/sshd ]; then
    SSHD_LIBS=$(ldd /usr/sbin/sshd)
    if echo "$SSHD_LIBS" | grep -q "liblzma"; then
        echo "[WARNING] sshd is linked to liblzma!"
        echo "$SSHD_LIBS" | grep "liblzma"
    else
        echo "[OK] sshd does not dynamically link liblzma."
    fi
else
    echo "[INFO] sshd binary or ldd not found on this system."
fi

# 2. Verify Installed XZ Utils Version
echo "[*] Checking installed xz version..."
if command -v xz >/dev/null; then
    XZ_VER=$(xz --version | head -n1)
    echo "    $XZ_VER"
    if echo "$XZ_VER" | grep -qE "5\.6\.[01]"; then
        echo "[CRITICAL] Vulnerable XZ version detected (CVE-2024-3094)!"
    else
        echo "[OK] Installed xz is outside the vulnerable 5.6.0-5.6.1 window."
    fi
fi

# 3. Check Package Manager Hashes
echo "[*] Verifying package manager integrity..."
if command -v debsums >/dev/null; then
    echo "    Running debsums on openssh-server..."
    debsums -s openssh-server coreutils || echo "[ALERT] Modified binaries detected!"
elif command -v rpm >/dev/null; then
    echo "    Running rpm verification on openssh-server..."
    rpm -V openssh-server || echo "[ALERT] Package checksum discrepancies found!"
elif command -v pacman >/dev/null; then
    echo "    Running pacman database integrity check..."
    pacman -Qk openssh || echo "[ALERT] Pacman files modified!"
fi

echo "[*] Audit completed successfully."

8. Three Server Hardening Rules We Follow

To protect our own infrastructure against supply chain tampering and kernel bugs, we enforce three simple practices:

1. Strip Down OpenSSH Dependencies

If you run static server fleets, rebuild OpenSSH without systemd socket activation. You do not need libsystemd.so loaded into an internet-facing authentication daemon.

2. Enable User Namespaces for Containers

Never run container workloads as host root. Add user namespace remapping to your Docker or Podman setup:

// /etc/docker/daemon.json
{
  "userns-remap": "default"
}

If a kernel race condition like Dirty COW gets triggered inside a container, the attacker only has root inside that isolated namespace.

3. Run Nightly Checksum Verification

Schedule nightly cron jobs using debsums -s or rpm -V. If a package binary in /usr/bin or /usr/sbin changes without a recorded package update, your team gets notified immediately.