Recovering 18.4 Million PostgreSQL Records After a CVE-2023-54391 Ransomware Attack

Abstract

A retail company running Proxmox VE 7.x was hit by a ransomware operator who exploited CVE-2023-54391 (CVSS 9.8), an unauthenticated authentication bypass in the hypervisor’s REST API, to gain root access without a password. What followed was not a generic ransomware run: the attacker deleted the PostgreSQL WAL directory first, zeroed pg_xact and pg_filenode.map, then encrypted the first 8 GB of the virtual disk. This was a deliberate sequence designed to prevent recovery, not just cause downtime. The target had no backup, no replica, and no WAL archiving. Seven years of operational data across eight tables was at risk. We describe how we recovered approximately 18.4 million canonical records by carving PostgreSQL heap pages directly from 204 GB of raw physical blocks (without a filesystem, without the commit log, without the filenode map) using custom binary decoders and MVCC hint-bit heuristics. We conclude with a full recovery field guide and hardening recommendations that defenders can apply today.

Impact TL;DR

An attacker exploited CVE-2023-54391 to obtain a root@pam ticket on a Proxmox VE 7.x hypervisor with a single unauthenticated POST request. With hypervisor access, they deployed a wiper that first deleted the PostgreSQL Write-Ahead Log (eliminating replay-based recovery), then zeroed pg_xact and pg_filenode.map, then encrypted GB 0 to GB 8 of the virtual disk. The result: a PostgreSQL 15 instance that could not start, on a filesystem that could not be mounted, with no backup to fall back on. Forensic recovery via heap page carving over the intact GB 8 to GB 212 region returned 18.4 million canonical records covering seven years of operation, but 80% of open financial positions, an entire month of sales data (February 2026), and all monetary values from 3.5 million supplier invoices were permanently destroyed. Proxmox published PSA-2026-00043-1 on September 1, 2026, the day after the wiper completed its work.

Background

This engagement started at 4:56 PM on a Sunday. The call came in framed as “the ERP is slow.” By the time we were on a call with the IT team, the framing had changed to “the VMs won’t start.” By the time we had access to the Proxmox console, the framing was accurate: a ransomware operator had been inside the hypervisor, and the database was gone.

The personal angle is worth naming. The question “who guards the guardians?” applies here with precision: the hypervisor that was supposed to isolate and protect the VMs was the entry point. The tools we trust sit at high-leverage trust boundaries, and those boundaries deserve adversarial scrutiny.

The Attack at 4:56 PM on a Sunday in August

The system stopped responding at 4:56 PM on August 30, 2026.

For whoever was at the register, it was a strange slowdown, the kind of thing that happens on a busy weekend. For the IT manager who got the first call minutes later, it was the start of one of the worst nights of their professional life. The ERP authentication server was unreachable. Not a network outage. Not a power failure. Something else.

What was at stake was the operational memory of seven years of a business. Purchase invoices going back to 2019. Credit and debit card transactions from October 2019 through that exact Sunday. Open financial positions, customer records, sales line items, the product catalog with prices updated through the day of the attack. All of it lived on a single PostgreSQL 15 server. No replica, no backup, no WAL archiving. One copy. One point of failure.

The datacenter contracted to host this infrastructure had never configured any backup mechanism. No daily snapshot. No replication to a second server. Not even PostgreSQL’s built-in WAL archiving, a feature that, had it been active, would have allowed reconstruction of the database to within seconds of the disaster. Nothing.

At 07:49:32 UTC on August 31, 2026, PostgreSQL recorded its last checkpoint. That timestamp was extracted later from the pg_control structure at offset 9,227,304,960 on the disk. The checkpoint LSN 305/2BBF84A0 is a forensic coordinate: everything that happened after that moment was destroyed before it could be consolidated to disk.

After that final checkpoint, the destruction sequence ran. The attacker deleted pg_wal/, eliminating the transaction log. They zeroed pg_xact/, destroying the commit status record. They deleted pg_filenode.map, removing the filenode-to-table mapping. Then they encrypted GB 0 to GB 8 of the virtual disk. This was not opportunistic. It was a destruction sequence optimized to eliminate recovery options before the victim could react.

The vulnerability that opened the door had existed for more than three years.

CVE-2023-54391 and the Vulnerability That Opened the Door

Proxmox VE is an open-source Debian-based hypervisor widely adopted by small and medium businesses because a single instance can host an entire organization’s IT infrastructure. That centralization is both its greatest operational advantage and its greatest risk vector: compromise the hypervisor and you compromise every VM it hosts.

CVE-2023-54391 is classified as CWE-304 (Missing Critical Step in Authentication) and carries a CVSS score of 9.8. The mechanism is precise. Proxmox exposes a REST API over HTTPS on port 8006. When Two-Factor Authentication (TFA) is enabled, the normal flow requires the client to respond to a tfa-challenge before a session ticket is issued. The defect: when the tfa-challenge field was present in the POST body (even with an arbitrary invented value), the verification code skipped password validation and jumped directly to ticket issuance.

The legitimate authentication flow:

POST /api2/json/access/ticket
Content-Type: application/x-www-form-urlencoded

username=root@pam&password=REAL_PASSWORD

The bypass:

POST /api2/json/access/ticket HTTP/1.1
Host: <server>:8006
Content-Type: application/x-www-form-urlencoded

username=root@pam&password=anything&tfa-challenge=anything

Both requests return the same output, a valid root@pam session ticket:

{
  "data": {
    "ticket": "PVE:root@pam:XXXXXXXX::...",
    "CSRFPreventionToken": "XXXXXXXX:...",
    "username": "root@pam"
  }
}

Two fields with invented values. Full administrator access as the result. With that ticket, the attacker could list all VMs and their disks, execute commands inside them via qm exec, mount volumes, modify network configurations, and upload files to any hosted virtual machine.

The Proxmox team fixed the flaw in libpve-access-control 8.0.4 in July 2023, silently, with no public advisory, no security bulletin, and no CVE registration at the time. The formal advisory PSA-2026-00043-1 was published on September 1, 2026, three years after the fix and the day after the wiper completed its work. Proxmox VE 7.x reached end-of-life in July 2024. The target server had not been updated since then.

Immediate check: dpkg -l libpve-access-control | grep ^ii - any version before 8.0.4 is vulnerable. Upgrade to Proxmox VE 8.x and apply all available updates.

The Crime Scene Inside the QCOW2 Image

When the disk image arrived in the analysis environment, nothing was obvious. The QCOW2 file sat at 92 GB on physical disk but addressed 212 GB of virtual space (cluster_size 65,536 bytes, QCOW2 v3). The gap between 92 GB and 212 GB was not waste. It was the first data point.

file format: qcow2
virtual size: 212 GiB (227,585,736,704 bytes)
disk size: 92.1 GiB
cluster_size: 65536
compat: 1.1
corrupt: false

A byte-by-byte entropy scan across the entire image revealed three unmistakable zones:

Zone 2 is the important one. The wiper did not encrypt that region; it overwrote it with zeros. That is faster and equally effective at preventing recovery. The directories pg_wal/, pg_xact/, and pg_filenode.map were in this zone. 1,742 candidate pg_xact pages were found with exactly 0.00 bits/byte entropy, zeroed to silence.

Zone 3 (the 204 GB the wiper never reached) contained readable PostgreSQL heap pages: valid 8,192-byte blocks with recognizable headers, tuples with parseable timestamps and decodable types. This is where recovery would happen.

The entropy mapping script used for zone identification:

import math, sys

BLOCK = 256 * 1024 * 1024   # 256 MB windows
SAMPLE = 65536               # entropy sampled from first 64 KB of each window

def block_entropy(data):
    freq = [0] * 256
    for b in data:
        freq[b] += 1
    n = len(data)
    return -sum((f/n) * math.log2(f/n) for f in freq if f > 0)

with open(sys.argv[1], "rb") as f:
    gb = 0.0
    while chunk := f.read(BLOCK):
        e = block_entropy(chunk[:SAMPLE])
        label = "ENCRYPTED" if e > 7.5 else ("ZEROED" if e < 0.5 else "STRUCTURED")
        print(f"GB {gb:6.1f} | entropy={e:.3f} | {label}")
        gb += len(chunk) / (1024**3)

pg_control, the Only Witness to the Database’s Final Moment

At offset 9,227,304,960 of the raw image (cluster #140,800 in QCOW2 terms, firmly inside Zone 3), the binary signature of pg_control appeared: the PostgreSQL magic PG_CONTROL_VERSION = 1300 at byte 0, followed by catalog version 202209061 at byte 4. These two 4-byte constants together uniquely identify a PostgreSQL 15 control file.

Extracting the critical fields:

pg_control version:     1300 (PostgreSQL 15)
system_identifier:      0x67A9A4F390EF1507
checkpoint_lsn:         305/2BBF84A0
checkpoint_timestamp:   2026-08-31 07:49:32 UTC
state:                  DB_IN_PRODUCTION

Each field is a piece of evidence. system_identifier is the fingerprint of this specific PostgreSQL instance, used to validate every recovered heap page against this database. state: DB_IN_PRODUCTION confirms the database was alive when the wiper acted; there was no clean shutdown. checkpoint_lsn decomposes to the WAL segment name 00000001000003050000002B and a precise byte offset within it.

A full scan of the 204 GB Zone 3 found zero WAL segments. The directory was not just emptied; it was deleted from the filesystem before the wiper ran. The attacker deleted the recovery mechanism before destroying the data. This is the defining forensic signature of a sophisticated operator: they understood that the WAL is what makes PostgreSQL recoverable, and they removed it first.

The checkpoint_timestamp 2026-08-31 07:49:32 UTC became the temporal anchor of the entire investigation. Every record recovered from the heap would be trusted only if its timestamp fell before this value. Everything written after this moment was gone.

Without pg_filenode.map (also deleted), the schema mapping had to come from another source. The shared_buffers region of the virtual disk (memory pages flushed to disk near GB 2-4) contained partially preserved pg_attribute catalog pages. From those pages, 122 tables were recovered with complete column schemas, 52 with partial schemas.

import struct, datetime

def parse_pg_control(path, offset):
    with open(path, "rb") as f:
        f.seek(offset)
        data = f.read(512)

    system_id      = struct.unpack_from("<Q", data,   8)[0]
    state          = struct.unpack_from("<I", data,  16)[0]
    lsn_hi         = struct.unpack_from("<I", data,  24)[0]
    lsn_lo         = struct.unpack_from("<I", data,  28)[0]
    ts_raw         = struct.unpack_from("<i", data, 128)[0]

    PG_EPOCH = datetime.datetime(2000, 1, 1, tzinfo=datetime.timezone.utc)
    checkpoint_dt  = PG_EPOCH + datetime.timedelta(seconds=ts_raw)

    seg_hi  = (lsn_hi >> 8) & 0xFFFFFF
    seg_lo  = ((lsn_hi & 0xFF) << 16) | (lsn_lo >> 24)
    wal_seg = f"00000001{seg_hi:08X}{seg_lo:08X}"

    return {
        "system_identifier": f"0x{system_id:016X}",
        "state":             state,
        "checkpoint_lsn":    f"{lsn_hi:X}/{lsn_lo:08X}",
        "checkpoint_time":   checkpoint_dt.isoformat(),
        "wal_segment":       wal_seg,
    }

Heap Page Carving, Extracting PostgreSQL Records from Physical Blocks

When pg_restore, WAL replay, and filesystem mounting are all off the table, the last path is heap page carving: reading PostgreSQL’s fixed-format 8,192-byte blocks directly from raw disk, extracting tuples without any metadata layer above them.

Every PostgreSQL heap page follows the same physical layout:

┌─────────────────────────────────────────────────────────────┐
│  PageHeaderData (24 bytes)                                  │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ pd_lsn [8]  pd_checksum [2]  pd_flags [2]             │ │
│  │ pd_lower [2]  pd_upper [2]  pd_special [2]            │ │
│  │ pd_pagesize_version [2] = 0x2400  pd_prune_xid [4]   │ │
│  └────────────────────────────────────────────────────────┘ │
│  Item Pointers (4 bytes each, from byte 24 to pd_lower)     │
│  Free Space (pd_lower to pd_upper)                          │
│  Tuples (grow from end toward start; first near byte 8192)  │
└─────────────────────────────────────────────────────────────┘

The magic number is pd_pagesize_version = 0x2400, located at byte 18 of the header (not byte 14, a common off-by-four error that silently breaks every decoder that follows). A valid page also satisfies 24 < pd_lower < pd_upper <= pd_special <= 8192.

Each tuple starts with a HeapTupleHeader. The fields that matter for carving:

Offset  Field          Size    Role
------  -------------  ------  ------------------------------------------
0       t_xmin         4 B     XID of the inserting transaction
4       t_xmax         4 B     XID of the deleting transaction (0 = live)
12      t_ctid         6 B     Current physical location of this tuple
18      t_infomask2    2 B     Attribute count + MVCC flags   <- byte 18!
20      t_infomask     2 B     Type flags, nullability, hint bits
22      t_hoff         1 B     Offset to user data (from tuple start)
23      [null bitmap]  var     One bit per attribute if HEAP_HASNULL set
[t_hoff] user data starts here

The most common implementation error: reading t_infomask2 at byte 14. The correct offset is byte 18. This four-byte misalignment produces a wrong attribute count and corrupts every subsequent field offset. It is silent; the decoder will not crash.

The core carver:

import struct

PAGE_SIZE     = 8192
PG_PAGE_MAGIC = 0x2400
HEAP_XMAX_INV = 0x0800   # xmax invalid = tuple is live

def is_valid_heap_page(page):
    if len(page) < PAGE_SIZE:
        return False
    pd_lower    = struct.unpack_from("<H", page, 12)[0]
    pd_upper    = struct.unpack_from("<H", page, 14)[0]
    pd_special  = struct.unpack_from("<H", page, 16)[0]
    pd_page_ver = struct.unpack_from("<H", page, 18)[0]   # MAGIC at byte 18
    return (pd_page_ver == PG_PAGE_MAGIC and
            24 < pd_lower < pd_upper <= pd_special <= PAGE_SIZE)

def get_item_pointers(page):
    pd_lower = struct.unpack_from("<H", page, 12)[0]
    n_items  = (pd_lower - 24) // 4
    result   = []
    for i in range(n_items):
        raw      = struct.unpack_from("<I", page, 24 + i * 4)[0]
        lp_off   = (raw >> 17) & 0x7FFF
        lp_len   = (raw >>  1) & 0x3FFF
        lp_flags = raw & 0x0003            # 1 = normal tuple
        if lp_flags == 1 and lp_off >= 24 and lp_len > 23:
            result.append((lp_off, lp_len))
    return result

def parse_tuple_header(page, offset):
    t_xmin       = struct.unpack_from("<I", page, offset +  0)[0]
    t_xmax       = struct.unpack_from("<I", page, offset +  4)[0]
    t_infomask2  = struct.unpack_from("<H", page, offset + 18)[0]  # byte 18!
    t_infomask   = struct.unpack_from("<H", page, offset + 20)[0]
    t_hoff       = struct.unpack_from("<B", page, offset + 22)[0]
    n_attrs      = t_infomask2 & 0x07FF
    has_null     = bool(t_infomask & 0x0001)
    is_live      = (t_xmax == 0) or bool(t_infomask & HEAP_XMAX_INV)
    if t_hoff < 23 or t_hoff > 64:
        return None
    return {"xmin": t_xmin, "xmax": t_xmax, "n_attrs": n_attrs,
            "has_null": has_null, "is_live": is_live,
            "data_off": offset + t_hoff}

def carve(raw_path, carve_start, carve_end, expected_attrs, decoder_fn, out_path):
    PAGE = 8192
    results = 0
    with open(raw_path, "rb") as f, open(out_path, "w") as out:
        f.seek(carve_start)
        abs_off = carve_start
        while abs_off < carve_end:
            page = f.read(PAGE)
            if len(page) < PAGE:
                break
            if is_valid_heap_page(page):
                for (lp_off, lp_len) in get_item_pointers(page):
                    hdr = parse_tuple_header(page, lp_off)
                    if hdr and hdr["n_attrs"] == expected_attrs and hdr["is_live"]:
                        row = decoder_fn(page, hdr["data_off"],
                                         hdr["n_attrs"], hdr["has_null"], lp_off)
                        if row:
                            out.write("\t".join(str(v) for v in row.values()) + "\n")
                            results += 1
            abs_off += PAGE
    return results

Three PostgreSQL types required custom decoders. NUMERIC is stored in an internal base-10,000 format with a 4-field header (ndigits, weight, sign, dscale) followed by digit groups. TIMESTAMP/TIMESTAMPTZ is an int64 of microseconds since 2000-01-01 00:00:00 UTC, not the Unix epoch. Using 1970 instead of 2000 introduces a 30-year systematic error. TEXT and VARCHAR use a varlena header (1 or 4 bytes) and require checking the top bits to distinguish inline content from TOAST pointers.

After decoding each attribute, the offset must be advanced to the correct alignment boundary for the next type (int4 aligns to 4, int8/timestamp to 8, text varlena to 4). Missing this step silently shifts every subsequent attribute.

MVCC Without pg_xact, Separating Live Records from Dead Ones

With pg_xact zeroed, the commit status of each transaction cannot be formally verified. The solution relies on a PostgreSQL optimization called hint bits: flags written into the tuple header itself (in t_infomask) when PostgreSQL resolves a transaction’s status, to avoid repeated lookups. The attacker zeroed the commit log. But the hint bits were already written in the heap.

The MVCC filter without pg_xact:

def classify_tuple(t_xmax, t_infomask):
    HEAP_XMIN_INVALID   = 0x0200
    HEAP_XMAX_COMMITTED = 0x0400
    HEAP_XMAX_INVALID   = 0x0800

    if t_infomask & HEAP_XMIN_INVALID:
        return "DEAD"                       # insertion was rolled back

    if t_xmax != 0 and not (t_infomask & HEAP_XMAX_INVALID):
        if t_infomask & HEAP_XMAX_COMMITTED:
            return "DEAD"                   # deletion was committed

    if t_xmax == 0 or (t_infomask & HEAP_XMAX_INVALID):
        return "LIVE"

    return "UNCERTAIN"                      # xmax set but hints ambiguous

Sampling GB 50 to GB 52 confirmed approximately 4.6% dead tuples, a normal rate for a database between vacuum cycles and not a consequence of the attack. Deduplication after carving used a DISTINCT ON (primary_key) ... ORDER BY primary_key, xmin DESC strategy to keep the highest-xmin version of each record (the most recent before the attack).

What Survived and the Final Recovery Numbers

After scanning 204 GB of intact heap blocks, the final numbers:

The audit log’s 45.9% duplicate rate is expected: high write-frequency tables accumulate dead tuple versions rapidly between vacuum cycles. The 88% canonical coverage of the estimated 12.3 million original log records is a strong result for a database with no backup and a destroyed commit log.

What Was Lost Forever

Not every gap is equal. Some losses were technical limits; others were deliberate design choices by the attacker.

The WAL does not exist. A complete scan of all 204 GB found zero WAL segments. The directory was deleted before the wiper ran. Without the WAL, replay-based recovery is impossible. Every record the forensic team extracted came from physical heap blocks with no consistency anchor.

3.5 million invoices with no monetary value. The nfcc table was recovered with 3,524,000 records, but the valor_total field (invoice monetary value) was null in 100% of tuples. The null bitmap in each tuple header (attributes 13, 16, and 17) confirmed this was a stored null, not a decoding error. Four alternative reconstruction paths (joins via access key, transaction ID, document number, and financial complement field) all failed. Those values exist in the federal tax authority’s systems; they do not exist on disk.

February 2026 is gone. The sales line-item partition for February 2026 resided in GB 0 to GB 7 (Zone 1, encrypted with maximum entropy). Uniform 7.97 bits/byte across 97,000 pages. No tuple header, no magic number, no structure of any kind survived.

80% of open financial positions. The pendfin table recovered 801,899 records out of an estimated 4 million. Four in five open receivable and payable entries are gone.

The commit log was zeroed. 1,742 pg_xact pages, all at 0.00 bits/byte. The 4.6% dead-tuple uncertainty rate is a consequence.

The attacker deleted the WAL first. That is the signature of a wiper that understood its target. Encrypting data causes downtime. Deleting WAL causes unrecoverable data loss. The operator did both, in the right order.

Recovery Field Guide

This section is a distilled guide for incident responders and DBAs who encounter a similar scenario.

Phase 0. What NOT to do (first 30 minutes)

Do not mount the image read-write. Do not run fsck. Do not start PostgreSQL from the compromised volume. Do not run pg_resetwal. Do not boot the VM. Do not delete anything.

# First: compute and record SHA-256 before any action
sha256sum /path/to/image.qcow2 | tee image.sha256

# Create a read-only primary evidence copy
cp --reflink=auto /path/to/image.qcow2 /evidence/image_ORIGINAL.qcow2
chmod 444 /evidence/image_ORIGINAL.qcow2

# Work on a copy only
cp /evidence/image_ORIGINAL.qcow2 /work/image_WORK.qcow2

Phase 1. Triage

Convert to raw, run an entropy scan, and map exact byte offsets for encrypted, zeroed, and intact zones. Search for pg_control using the magic bytes (pg_control_version = 1300 as little-endian 14 05 00 00, confirmed by catalog_version_no = 202209061 at offset +4).

qemu-img info /work/image_WORK.qcow2
qemu-img convert -f qcow2 -O raw /work/image_WORK.qcow2 /work/image.raw

Phase 2. pg_control

Extract system_identifier, checkpoint_lsn, checkpoint_timestamp, and derive the expected WAL segment name. Search for that segment. If none is found, the disk is confirmed to be in full carving mode.

Phase 3. Filenode mapping

If pg_filenode.map is missing, locate pg_attribute pages (filenode 1249) in the shared_buffers region (typically GB 2-4). Columns attrelid, attnum, atttypid, and atttypmod give you the schema for each table. Cross-reference with pg_class (filenode 1259) for table names. Document every filenode-to-table mapping with evidence.

Phase 4. Decoder development (most error-prone phase)

For each target table: get the exact schema from pg_attribute or application source. Build the decoder column by column, checking alignment after every field. Validate on 50+ hand-inspected rows before running at scale. Any issue rate above 5% on expected fields is a decoder bug, not data corruption.

Critical type decoders:

import struct, datetime

PG_EPOCH = datetime.datetime(2000, 1, 1, tzinfo=datetime.timezone.utc)

def decode_pg_timestamp(data, offset):
    raw = struct.unpack_from("<q", data, offset)[0]
    return PG_EPOCH + datetime.timedelta(microseconds=raw)

def decode_pg_numeric(data, offset):
    ndigits = struct.unpack_from("<H", data, offset)[0]
    weight  = struct.unpack_from("<h", data, offset + 2)[0]
    sign    = struct.unpack_from("<H", data, offset + 4)[0]
    dscale  = struct.unpack_from("<H", data, offset + 6)[0]
    if sign == 0xC000:
        return None
    digits = [struct.unpack_from("<H", data, offset + 8 + i*2)[0]
              for i in range(ndigits)]
    value = sum(d * (10000 ** (weight - i)) for i, d in enumerate(digits))
    return round((-value if sign == 0x4000 else value) / (10 ** dscale), dscale)

TYPE_ALIGN = {20: 8, 23: 4, 1114: 8, 1184: 8, 1700: 4, 25: 4, 1043: 4, 16: 1}

def align_to(off, atttypid):
    a = TYPE_ALIGN.get(atttypid, 4)
    return (off + a - 1) & ~(a - 1)

Phase 5. Carve at scale

Use the main loop from the section above, bounded to the intact zone offsets. Expect 50,000-80,000 pages/second in pure Python; parallelize with multiprocessing.Pool across 8 chunks for a 204 GB image to finish in under an hour.

Phase 6. Deduplicate in isolated PostgreSQL

CREATE TABLE log_canonical AS
SELECT DISTINCT ON (transaction_id)
    transaction_id, timestamp_utc, event_type
FROM log_staging
WHERE timestamp_utc <= '2026-08-31 07:49:32+00'
ORDER BY transaction_id, _xmin DESC;

Measure dead-tuple rate, check for timestamp gaps that might indicate destroyed sectors, and run semantic validation (plausible value ranges, FK consistency, encoding quality).

Phase 7. Package with GAPS

The gaps document is as important as the data. It defines the boundary of what the client can trust and what must be sourced from alternatives (tax authority portals, bank statements, supplier copies). Deliver with SHA-256 checksums and a tested import runbook.

Decision tree:

pg_wal/ present and non-empty?   YES -> WAL replay (not this path)
                                  NO  -> heap carving required

pg_xact/ present and non-empty?  YES -> standard visibility rules
                                  NO  -> hint-bit heuristic (this guide)

pg_filenode.map present?         YES -> load directly
                                  NO  -> pg_attribute reconstruction

Entropy zone of target tables?
  Encrypted (>7.5) -> irrecoverable
  Zeroed (<0.5)    -> irrecoverable
  Structured       -> carve this range

Detection and Hardening

The attack chain moved in three steps: hypervisor access via CVE-2023-54391, wiper execution inside the VM, and database destruction. Controls exist at each step.

Lock the hypervisor management plane. The Proxmox web interface should never be reachable from the internet. Place it behind a VPN or zero-trust gateway restricted to named admin IPs. Enable MFA on all accounts. Monitor for any unauthenticated or anomalous requests to /api2/json/access/ticket. Subscribe to Proxmox security mailing lists and changelogs, not just published CVEs. The three-year gap between the silent fix and the formal advisory illustrates exactly why CVE feeds alone are insufficient.

Keep the hypervisor on a supported version. Proxmox VE 7.x was EOL in July 2024. Running EOL software on infrastructure that hosts production databases is a business decision with predictable consequences.

Archive the WAL. For any PostgreSQL database with financial or compliance data, continuous WAL archiving to an external destination (S3, remote server, object store) is the minimum acceptable posture. With WAL archiving active, the attacker could have destroyed everything on the primary disk and recovery would have been measured in hours. Without it, recovery required days of forensic carving and accepted permanent losses.

Implement the 3-2-1 rule. Three copies of data, two different media types, one offsite. A database with no backup is a single point of total failure. Test restores monthly; an untested backup is an unknown backup.

Restrict VM network access. VMs should not be able to reach arbitrary external hosts. The wiper deployed in this attack required upload from the hypervisor. Network segmentation between the management plane and the VM network limits this delivery path.

What to watch for in your SIEM:

• Unauthenticated POST to /api2/json/access/ticket with a tfa-challenge field from an external IP
• Mass deletion or rapid entropy change in any directory under the PostgreSQL data directory
• pg_wal/ or pg_xact/ directories becoming empty or shrinking unexpectedly
• VM process executing tools like dd, shred, or custom wipers against block devices
• Sudden inaccessibility of the PostgreSQL port after a period of normal traffic

Conclusion

“Who guards the hypervisor?” This engagement answered that question the hard way. The hypervisor that was supposed to isolate and protect the virtual machines was the entry point. The tools we trust sit at high-leverage junctions in the infrastructure supply chain, and those junctions deserve adversarial scrutiny.

We walked a complete chain: authentication bypass at the hypervisor (CVE-2023-54391), wiper deployment inside the VM, and systematic destruction of PostgreSQL’s recovery mechanisms in the correct order (WAL first). None of those steps relied on novel exploitation. They relied on three years of an unpatched critical vulnerability, an EOL hypervisor, and a database with no backup.

The recovery result (18.4 million canonical records from 204 GB of raw blocks, without a filesystem, without the commit log, without a filenode map) demonstrates that PostgreSQL’s physical storage format is deterministic enough to carve directly from a destroyed disk. The 8,192-byte page boundary is fixed. The magic number is reliable. The tuple format does not change. That predictability, designed into PostgreSQL for performance reasons, is what made recovery possible when everything else was gone.

But the most important lesson is not technical. One weekly backup. One WAL archive destination. One patch applied three years ago. Any one of those three things would have completely changed the outcome.

Proxmox published their security advisory the day after the attack concluded. We published ours today. The controls are straightforward, the exposure is documented, and the field guide is in this post. The rest is a decision about whether to act before the next incident or after.

Who guards the hypervisor? We do. Researchers, red teams, blue teams, and every operator who runs these platforms. Today we mapped the seams. Tomorrow, we close them.