Skip to content

Windows hard freeze with system encryption: all 8 EncryptedIoQueueItem objects exhausted by blocked completion work items (1.26.29) #1873

Description

@frcdr

Expected behavior

Windows should remain responsive while the system partition is encrypted with VeraCrypt, and normal reads/writes to the system NVMe drive should continue to complete without the VeraCrypt encrypted-I/O queue becoming permanently blocked.

Observed behavior

With VeraCrypt 1.26.29 system encryption enabled, Windows occasionally freezes completely. The UI stops responding first, while the mouse cursor may continue moving briefly; after several seconds the cursor also freezes. Ctrl+Alt+Del and keyboard lock keys stop responding, and the machine requires a hard reset unless I trigger a manually initiated crash quickly enough.

The problem appears closely tied to system encryption. After fully decrypting the Windows system partition, the freezes appeared to stop even though VeraCrypt remained installed and veracrypt.sys was still loaded. After reinstalling VeraCrypt 1.26.29 and enabling system encryption again, the freezes returned.

I configured Windows manual crash generation and captured a kernel memory dump while the system was in the frozen state. The resulting bugcheck is therefore intentionally MANUALLY_INITIATED_CRASH (0xE2); the important part is the kernel state captured immediately before the crash.

Using the official VeraCrypt 1.26.29 debugging symbols, the dump shows the following runtime queue configuration:

EncryptionIoRequestCount = 16
EncryptionItemCount      = 8
EncryptionMaxWorkItems   = 1024
EncryptionFragmentSize   = 256 KiB

At the frozen instant, VeraCrypt's MainThreadProc had been waiting for about 42 seconds in GetPoolBuffer() at EncryptedIoQueue.c:185, called from MainThreadProc at line 845. That code is trying to obtain another EncryptedIoQueueItem.

nt!KeWaitForSingleObject
veracrypt!GetPoolBuffer+0x123
veracrypt!MainThreadProc+0xb7

sizeof(veracrypt!EncryptedIoQueueItem) is 0x38. I traversed the queue's FirstPoolBuffer list and found exactly eight buffers of that size, matching EncryptionItemCount = 8, and all eight were marked InUse = 1:

EncryptedIoQueueItem buffers: total=8 inuse=8

The corresponding EncryptedIoQueue object (ffffde8c6900aa70) showed a large I/O backlog while the queue was otherwise active and not suspended:

OutstandingIoCount          = 508
IoThreadPendingRequestCount = 0
ActiveWorkItems             = 8

MainThreadQueue       = non-empty
IoThreadQueue         = empty
CompletionThreadQueue = empty

StartPending          = 0
ThreadExitRequested   = 0
Suspended             = 0
SuspendPending        = 0
StopPending           = 0

The eight active VeraCrypt work items are particularly notable because representative completion workers are blocked while completing IRPs upward through Windows. One example had also been waiting for about 42 seconds:

veracrypt!CompleteIrpWorkItemRoutine
veracrypt!TCCompleteDiskIrp
nt!IofCompleteRequest
FLTMGR!FltpPassThroughCompletion
Wof!FileProvReadCompressedCompletion
Wof!FileProvReadCompressedCompletionWorker
Wof!FileProvReadCompressedOnNewStackExtendedCompletion
Wof!FileProvDecompressChunks
nt!ExAllocateFromPagedLookasideList
nt!ExAllocatePool...
nt!ExfAcquirePushLockExclusiveEx
nt!KeWaitForSingleObject

In VeraCrypt 1.26.29, CompleteIrpWorkItemRoutine() calls TCCompleteDiskIrp() before returning the associated queue item to the pool:

item->Status = workItem->Status;
OnItemCompleted(item, FALSE);
TCCompleteDiskIrp(workItem->Irp, workItem->Status, workItem->Information);

The item is only released later in the finally block: ReleasePoolBuffer(queue, item);

This appears consistent with the captured state: all eight completion work items are active, all eight EncryptedIoQueueItem objects remain occupied, and MainThreadProc is unable to obtain another item. New encrypted I/O therefore stops progressing while the backlog grows.

The dump also contains many threads waiting in Ntfs!NtfsWaitOnIo. Several real NTFS reads and writes are pending at the VeraCrypt device. For example:

IRP_MJ_READ
\Driver\veracrypt
pending

and:

IRP_MJ_WRITE
\Driver\veracrypt
pending

The IRP that MainThreadProc was processing when it became blocked is also an IRP_MJ_READ pending at \Driver\veracrypt.

I did not find a similar concentration of threads blocked in stornvme, and Windows did not log corresponding WHEA, StorPort, StorNVMe, or disk reset/error events. The NVMe device also reports healthy status.

Based on the dump, my current interpretation is that VeraCrypt 1.26.29 can enter a resource-starvation/deadlock condition in the encrypted-I/O completion path. Completion work items become blocked while completing requests through FLTMGR/WOF, retain all available EncryptedIoQueueItem objects, and prevent the main VeraCrypt I/O thread from processing further system-disk requests. I cannot determine from this dump alone whether the initial problem is entirely inside VeraCrypt or is an interaction between VeraCrypt and Windows WOF/memory management.

I also noticed that VeraCrypt has post-1.26.29 commits related to this same area. In particular, the project later reverted the CriticalWorkQueue IRP completion dispatch because Windows instability reports suggested it was a regression risk, and other follow-up changes adjusted completion-work-item handling. This looks potentially related to the state captured in this dump.

I have preserved the approximately 1.3 GB kernel memory dump and the full WinDbg transcript. I would prefer not to attach a kernel dump publicly because it may contain sensitive memory contents, but I can provide it privately to a maintainer if needed.

Steps to reproduce

  1. Install VeraCrypt 1.26.29 on Windows and enable system encryption for the Windows system partition.

  2. Use Windows normally. In my case, after some time the system eventually becomes completely unresponsive: the UI freezes, then the cursor stops moving, Ctrl+Alt+Del stops working, and a hard reset is normally required.

  3. Fully decrypt the Windows system partition and continue using the machine. In my case, the freezes appeared to stop even though VeraCrypt remained installed and veracrypt.sys remained loaded.

  4. Re-enable VeraCrypt system encryption. In my case, the hard freezes returned.

The freeze does not appear to require a specific application. I also reproduced a freeze during heavy WinDbg activity, but normal usage has triggered it as well.

Screenshots

N/A

Your Environment

VeraCrypt version: 1.26.29

Operating system and version: Windows 11 x64 (26200)

System type: 64-bit

Boot configuration: UEFI dual boot with Linux Mint/GRUB. VeraCrypt system encryption is applied to the Windows system partition rather than the whole physical disk.

System drive: BIWIN NA80Y1M10-256G NVMe SSD, firmware SN11795

NVMe controller: Microsoft Standard NVM Express Controller

Additional Windows configuration: VBS/HVCI/Hyper-V infrastructure is enabled.

No WHEA errors or corresponding disk, stornvme, or storport errors were found around the freezes. Windows reports the NVMe drive as healthy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions