NVMe over Fabrics

A GPU server with 8 H100s and 8 NVMe drives is a sensible starting point until you realize that some workloads need 100 TB of fast storage and others need 1 TB. Local NVMe is too coarse-grained for elastic provisioning. NVMe over Fabrics (NVMe-oF) decouples the drive from the server: the drive lives in a separate JBOF (just a bunch of flash) chassis, the GPU server reads it over RDMA, and the application sees the same NVMe queue interface it would for local SSD.

What it actually does

NVMe-oF is a small layer that wraps NVMe submission and completion queues in RDMA messages. The application calls pread() or uses cuFile, the kernel's NVMe driver enqueues a work request, the NIC packages it as an RDMA write to the target, the target's NIC delivers it to its NVMe controller, and the SSD performs the I/O. The completion queue entry comes back the same way. From the application's perspective, this is indistinguishable from a local NVMe read; from the fabric's perspective, it is just another RDMA message.

The wire protocol comes in three flavors. NVMe-oF over RDMA is the dominant production form: it runs on InfiniBand or RoCE and uses the same RDMA verbs as GPUDirect RDMA. NVMe-oF over Fibre Channel is for storage networks that already speak FC. NVMe-oF over TCP exists, adds 200-400 us versus RDMA, and is the fastest-growing transport in cost-sensitive shared-NVMe deployments (Lightbits, Pure FlashArray) because it removes the RoCE / IB configuration burden. Rarely the choice for latency-sensitive AI training paths, but increasingly common in mixed-workload storage tiers.

The latency story

Local datacenter NVMe inside the same PCIe domain runs roughly 20-70 microseconds end-to-end for a 4K random read on modern SSDs (e.g. Micron 7500: ~70 us typical), dominated by the SSD controller and NAND. Optane-class drives are closer to 10 us, but conventional NAND is not. NVMe-oF over RDMA adds the round-trip time of one RDMA exchange plus target-side processing, roughly 10-20 microseconds on a tuned IB or RoCE fabric. End-to-end latency for NVMe-oF over RDMA is therefore roughly 30-90 us. NVMe-oF over TCP adds another 200-400 us on top of that. Sequential bandwidth tops out at the fabric line rate divided by RDMA efficiency; on NDR (50 GB/s per port), aggregate across many connections approaches 45 GB/s per target, though a single queue pair is usually credit-bounded lower.

The latency increase is small enough that for most AI training workloads (which are bandwidth-bound, not latency-bound), NVMe-oF is indistinguishable from local. The bandwidth is what changes: if your storage fabric has more bandwidth per server than your local PCIe lanes provide, NVMe-oF can actually be faster than local for some workloads.

Why it changes how clusters are designed

The big win of NVMe-oF is provisioning flexibility. With local NVMe, the storage capacity per GPU server is fixed at build time (e.g., 8x 7.68 TB = 61 TB per server). With NVMe-oF, the same 61 TB lives in a JBOF chassis that can be carved up: one GPU server might mount 100 TB for a large dataset, another might mount 5 TB for a small one. The total fleet capacity stays the same; the per-server slice is elastic.

This is what enables shared NVMe pools backing parallel filesystems. DDN EXAScaler / Lustre on JBOF, VAST DASE, BeeGFS, and NetApp ONTAP AI all consume NVMe-oF as their drive transport: every storage node exports its NVMe drives as NVMe-oF targets, and the filesystem layer aggregates them. WekaFS takes a different path: its cluster fabric is a proprietary DPDK/SPDK transport between client and backend, not standard NVMe-oF (though it can ingest NVMe-oF as a backend transport for raw drives). Lustre vs WekaFS covers the architectural differences in detail.

When NVMe-oF replaces local

The decision tree is roughly:

• For ephemeral scratch space (training intermediate buffers, cache for object-store reads): local NVMe is simpler and just as fast.
• For shared dataset storage that multiple jobs need: NVMe-oF as the backing layer for a parallel filesystem is the standard pattern.
• For checkpoints that need to outlive the GPU server: NVMe-oF with replication, or a parallel filesystem on top of it.
• For long-term archival: object store (S3 tier for training), not NVMe.

What this means in practice

• NVMe-oF over RDMA is the production form. Use IB or RoCE, not TCP, unless you have a specific reason to avoid RDMA.
• The fabric tier matters. NVMe-oF on a rail-optimized fat-tree will share the IB fabric with collective traffic; provision the bisection bandwidth accordingly.
• Pair NVMe-oF with GPUDirect Storage for full bypass: the NIC DMAs straight into HBM, and the JBOF NVMe controller DMAs straight onto the wire. No host memory anywhere on the path.
• Failure modes look different. Local NVMe failure takes down one GPU server; NVMe-oF target failure can affect every client mounting that target. Replication and pool-level redundancy matter.
• For provisioning: think about NVMe-oF as software-defined block storage. The same JBOF can support 10 different namespaces with different sizes, IOPS guarantees, and replication levels. Most production deployments use a thin orchestration layer (NVMe Multipathing, ZNS, vendor-specific provisioner) on top of the raw protocol.

NVMe-oF is the protocol that makes "shared NVMe" a meaningful concept. Without it, NVMe is locked to whichever server it is plugged into.