Lustre vs WekaFS

For a long time, "parallel filesystem for HPC" meant Lustre. Then GPFS, BeeGFS, and DAOS each carved out share. WekaFS arrived later, designed specifically for the IOPS demands of training workloads with millions of small files. The two that show up most in production AI clusters today are Lustre and WekaFS, and they are different enough that picking the wrong one for your workload costs you 2-5x in real bandwidth.

Architectural starting points

Lustre is a layered design. There is a metadata server tier (MDS) holding the directory tree and inode metadata, and a separate object storage server tier (OSS) holding the file data. Clients consult the MDS to open a file and learn which OSS targets hold which stripes, then read directly from those OSS targets. The MDS and OSS are separate machines (or pools), separately scaled and independently failure-prone.

WekaFS has no separate metadata tier. Every backend node holds a slice of the metadata and a slice of the data. The metadata is distributed via a consistent hash; opens, lookups, and stat calls land on whichever node owns that piece of the namespace. Reads and writes from the WEKA client run over a proprietary DPDK / UDP transport (not standard NVMe-oF or stock RDMA verbs) directly to the backend processes, which use SPDK to drive their local NVMe.

The architecture difference shows up most clearly in metadata-heavy workloads. Image-classification training with one-file-per-image opens millions of files per epoch. Lustre with a single MDS bottlenecks on open/lookup RPCs; the symptom is GPU utilization dropping to 30-40% during data loading. Lustre's DNE (Distributed Namespace) feature splits the namespace across multiple MDS, which helps but does not eliminate hot-spotting. WekaFS distributes metadata by design and tolerates the small-file pattern much better.

Bandwidth and IOPS comparison

For sequential bandwidth on similarly-sized clusters:

Lustre on HDD-backed OSTs: roughly 2-5 GB/s per OST, scales linearly. 16 OSTs gives 32-80 GB/s aggregate.
Lustre on modern NVMe-backed OSS appliances (DDN A1400X3i, EXAScaler): roughly 30-90 GB/s per appliance. A handful of appliances reach the TB/s class.
WekaFS NVMe pool: roughly 20-60 GB/s per backend node on 400 GbE / NDR class NICs, scales linearly. 32 nodes gives 600 GB/s to 2 TB/s.

For random-read IOPS (4K-256K mixed):

Lustre on HDD: roughly 5K-20K IOPS per OST. Sustained random read is much weaker than sequential.
Lustre on NVMe: roughly 100K-500K IOPS per OST. A real number, but still topology-bound.
WekaFS: 1M+ IOPS per node, sustained. The whole point of the design.

For metadata operations (open + stat per second):

Lustre single MDS: 50K-200K ops/sec. Hot-spot risk.
Lustre with DNE: 200K-1M ops/sec across multiple MDS.
WekaFS: 1M+ ops/sec, distributed naturally.

When each one is right

Lustre is right when:

You have a budget constraint and existing HPC operational experience.
Your workload is bandwidth-heavy with large files (pre-sharded TFRecord, WebDataset).
You can tolerate some metadata-side hot-spotting or run DNE.
You value open-source and vendor-independent operation.

WekaFS is right when:

Your workload is metadata-heavy (many small files, long directory trees).
You need consistent low latency on opens and stats.
IOPS-bound shuffle is the bottleneck (vision training with one-image-per-file).
The cost of WekaFS licensing is offset by the cost of pre-packing your dataset into large shards.

Other PFS options worth knowing

GPFS / IBM Spectrum Scale: mature, similar architecture to Lustre with stronger client cache coherence. Common in IBM-led AI clusters.
BeeGFS: open-source, simpler than Lustre, good for small-to-medium scale.
DAOS: open-source, originally Intel-led. Uses PMDK for persistent-memory / SCM, SPDK for NVMe, and the CART / Mercury / libfabric stack for network transport; pool / container is the namespace model. Post-Optane, DAOS 2.6+ supports SSD-only "MD-on-SSD" deployments. Fast but newer; smaller production base than Lustre.

Operational considerations

Lustre has 20+ years of production scars. The tooling (lfs, lctl, lustre_rsync) is well-known. Failures and recovery are well-documented. The on-call burden is real but the playbook is mature.

WekaFS is a younger ecosystem. The tooling is good but less universally known. Vendor support is the path of least resistance for incident response. Snapshots, replication, and tiering features are first-class.

Storage failures matter for drain-and-replace at the GPU server level. A PFS that holds active dataset and active checkpoints needs replication and recovery faster than the typical training step. Both Lustre and WekaFS support data protection (Lustre via FLR / mirroring, WekaFS via N+2 or N+4 distributed erasure coding with stripe widths up to 16+2); the operational discipline to actually use it is what differs.

What this means in practice

Pre-pack datasets into large shards (100 MB-1 GB each) when running on Lustre. WebDataset, TFRecord, and MosaicML StreamingDataset all do this. The packing investment pays back as 5-10x faster training.
For metadata-heavy workloads on Lustre, enable DNE if available. If your cluster vendor does not support DNE, accept that small-file workloads will hit a wall around a few hundred GPUs.
For WekaFS, the operational discipline is around capacity. WekaFS pools fill up faster than Lustre because everything is on NVMe; budget capacity planning carefully.
Both can use GPUDirect Storage (cuFile-aware clients exist for both). Verify that your cluster's PFS client is GDS-enabled before assuming throughput numbers.
The fabric matters. Both PFS designs run on RDMA (IB or RoCE), and both need bandwidth budget on the same fabric as the compute. See topology-aware placement and rail-optimized fat-tree.

The choice is rarely "which is faster". It is "which matches my workload's pain point". Bandwidth-heavy plus large files: Lustre. Metadata-heavy plus small files: WekaFS.