Setting the Goal: Block Store vs. Pass-Through

In a pass-through mode, guest VM I/O operations map almost directly to external storage LUNs with minimal metadata intervention. In contrast, our Block Store routes every read and write through an indexed metadata B-tree map to translate virtual disk offsets into physical block locations:

The target goal for Block Store performance was to reach parity with the pass-through mode.

Red Herring #1: CPU Flamegraphs and Thread Offloading

Initial flamegraph analysis driven by AI-generated scripts flagged Map::Find as a major bottleneck in the Block Store—accounting for 6% of execution time compared to just 0.3% in the pass-through path. However, architectural expectations and code inspection suggested that key lookup overhead should have been nearly identical. It was only upon manually comparing the visual execution stacks of both Find operations that the underlying issue became clear.
The manual flamegraph analysis revealed that metadata lookups (Map::Find) and data write submissions were executing on the same frontend thread context. In contrast, the high-performance pass-through path processed data I/O submissions on a separate thread context, avoiding lock contention on the frontend worker thread.
The Fix: I refactored the execution path to offload data write submissions to a separate frontend thread. The AI LLM model was great at generating code for this patch.
The Result: The flamegraph stack looked much cleaner and closely mirrored the direct path. However, user level throughput barely moved. Fixing thread-level contention reduced CPU overhead, but it wasn't the primary bottleneck choking system throughput.

Red Herring #2 and #3: Thread Pool Queuing and I/O Pollers

Next, I evaluated thread pool queuing metrics and I/O event pollers (such as SPDK NVMe-oF worker threads):

1. IO Engine Thread Queueing Stats: I analyzed thread pool run-time versus queue wait time ratios. This stat showed a >3x queue/run ratio for the baseline block store case, compared to the pass-through mode - indicating that there was a lot of queueing on the VM disk threads. The Fix: One solution I considered was to offload some of the work from the individual disk threads to the threadpool. Here again the AI LLM model was great at generating working code for the patch. I was able to confirm through stats that the offload was working as expected.
   The Result: Offloading to control thread pools lowered queue wait times drastically (reducing queue/run ratios from ~2.5 down to ~1.02). But overall storage throughput remained capped at ~2.76 GB/s

2. I/O Poller Serialization: I suspected that because the block store used a single consolidated storage volume connection, I/O submission/completion callbacks might be serialized on a single poller thread. To isolate this, I tested a minimal benchmark: a single VM, single disk running a 1MB sequential write workload. The pass-through path reached 2.59 GB/s, while the block store plateaued at 1.12 GB/s. Poller thread imbalance was not yet the root cause either. Some bottlenecks are secondary - they have a tendency to show up when the major one is solved.

The Breakthrough: Inspecting Storage Array Telemetry

When internal CPU profiles and queue stats fail to reveal why system throughput is throttled, it is time to look downstream at storage-level telemetry. This highlighted the limits of current AI LLM models. Despite full access to the codebase and targeted questions, the model did not pinpoint the root bottleneck. It was only after I manually identified the underlying flaw and requested analysis of specific code paths that the model confirmed the implementation detail.
While issuing a 1MB sequential write workload from the host application, I observed the following metrics on the storage array:

• Latency: ~0.33 ms

• Throughput: ~3.55 GiB/s

• Array IOPS: ~220,000 IOPS

1 MB / 220,000 IOPS ≈ 4 KB per I/O operation

The storage array was receiving 256 individual 4KB I/O requests instead of single, contiguous 1MB block writes!

Root Cause: High-Level Block Fragmentation

Traced through the block engine layer, the difference between the pass-through path and the metadata-backed store became clear. With specific prompting, the model provided a detailed difference between the two implementations:

In the metadata-backed block store, the indexing layer allocated space by looking up block maps in 4KB metadata units. Even though the block IDs allocated for the 1MB write were physically contiguous, the engine issued 256 discrete I/O calls to the storage driver.
This caused severe serialization and protocol-handling overhead at the user-space storage driver (SPDK) level, saturating the storage controller with millions of tiny I/Os instead of large sequential streams.

The Solution: Intelligent I/O Coalescing

To eliminate this overhead, I updated the write path to aggregate contiguous block requests before passing them to the underlying storage hardware. Here too, the model was very useful in generating a working coalescing algorithm that I could use to test and validate the theory.
Instead of issuing an I/O call for every 4KB metadata descriptor, the submission engine now evaluates descriptors in sequence and merges adjacent blocks:

• I/O Coalescing Logic: The write path evaluates descriptors sequentially to merge contiguous block IDs into a single combined buffer, dispatching consolidated I/O requests upon detecting continuity breaks instead of issuing hundreds of fragmented sub-block calls.

After validating this coalescing logic on writes, I applied a similar contiguous block aggregation algorithm to the sequential read path.

Results

Coalescing contiguous block requests eliminated I/O driver submission overhead and dramatically increased system throughput:

• Sequential Read Performance: Increased from 4.75 GB/s to 11.87 GB/s—a ~2.5x increase, matching pass-through performance.

• Sequential Write Performance: Increased from 2.76 GB/s to 7.57 GB/s—a ~2.7x increase.

Engineering Key Takeaways

1. Trace the Complete I/O Path: Insufficient metrics within the SPDK layer left a critical blind spot across the end-to-end I/O journey. Gaining full visibility into every stage of the request path is essential for uncovering true root causes.

2. Don't Stop at CPU Flamegraphs: A clean CPU stack does not guarantee efficient I/O. CPU profiles show where threads spend time, but on a workload that is not CPU bound, I would look elsewhere.

3. AI Tools—Double-Edged but Valuable: Although over-relying on AI suggestions led to several red herrings, these same tools significantly accelerated patch iteration and sped up the overall debugging process. The key takeaway is that asking targeted, precise questions is essential to guide AI in the right direction.