Slurm Gang Scheduling

Multi-node training jobs are not normal jobs. A 64-rank PyTorch job does not start when one rank starts; it starts when every rank starts. Until rank 63 issues torch.distributed.init_process_group, ranks 0 through 62 sit in MPI_Init (or NCCL's bootstrap) waiting for the rendezvous to complete. On Kubernetes, the default kube-scheduler binds pods one at a time, so the early ranks can sit idle for minutes while later ranks wait in the queue; if some pods never bind, the job hangs and burns its wall-clock budget. Slurm's allocator avoids that specific failure (it always allocates the full node set before srun launches any tasks), but the broader rule still applies on every scheduler: do not start a multi-rank job until every rank can start, and use gang time-slicing if you want co-allocated jobs to share GPUs without deadlocking each other.

What gang scheduling actually does

In Slurm, gang scheduling is enabled by PreemptMode=GANG (typically paired with SchedulerType=sched/backfill). The gang feature lets co-scheduled jobs time-slice on the same resources: suspended jobs swap in and out cooperatively instead of one job starving the other. Atomic multi-node allocation, the property that prevents partial dispatch, is Slurm's default behavior and does not require the gang flag; srun only launches once the full node set is reserved. The scheduling decision for a single multi-node job is binary: either every rank starts together, or the job waits.

The cost: idle time on the partial reservation. If 7 of 8 nodes are free and the 8th is finishing another job, those 7 GPUs sit idle waiting. The scheduler's bookkeeping treats them as reserved (no other job can claim them) but unproductive. For short jobs this is cheap; for jobs that wait minutes for the gang to materialize, the idle cost can be a large fraction of the wall clock. The benefit: zero deadlocks, predictable startup, and the job actually does work once it starts.

Why MPI_Init is the killer without gang

Every multi-rank framework (PyTorch DDP, Horovod, JAX pjit, Megatron) bootstraps over a collective communication layer that requires all ranks to participate before any can proceed. The bootstrap protocol is approximately:

1. Each rank registers with a coordinator (NCCL bootstrap, MPI's PMIx, or torch.distributed's TCP/file backend).
2. The coordinator waits for all N ranks to register.
3. Once registered, ranks exchange addresses for the collective ring or tree topology.
4. Only then can the first all_reduce, all_gather, or any collective happen.

Step 2 is where the deadlock lives. If the scheduler dispatched only ranks 0-15 of a 64-rank job and the other 48 are still queued, ranks 0-15 wait at step 2 forever. They consume their wall-clock allocation, then time out and fail. The user sees "MPI_Init timed out" or "NCCL rendezvous failed"; the underlying cause is the scheduler. This is the canonical Kubernetes failure mode that motivates Volcano or Kueue. On Slurm it does not happen for a single job (atomic allocation), but it can still appear when two co-scheduled jobs compete for the same GPUs without PreemptMode=GANG to coordinate the time-slice.

Without gang time-slicing, the workaround for co-scheduled jobs is per-job: framework-specific timeouts, retry logic, application-level barriers. Operationally messy and per-team. The gang flag moves the fix to the scheduler so every job benefits.

How Slurm gangs interact with backfill and fair-share

Slurm's gang scheduling does not exist in isolation. It composes with two other policies that shape who actually runs:

Backfill (SchedulerType=sched/backfill) lets short jobs slip past long-waiting jobs as long as they finish before the next reservation window. Without backfill, a 1-hour job could wait an entire day behind a queued 24-hour job; with backfill, it runs immediately if a node is free for the next hour. Backfill and PreemptMode=GANG compose: backfill finds the next slot where N nodes are simultaneously free, and the gang flag lets two such allocations time-slice on the same resources if priorities require it.

Fair-share (fair-share-queues) sets per-user or per-account priorities based on historical usage. A user who has consumed the lion's share of GPU-hours this week sees their priority dampened; the gang of a fresh user gets dispatched ahead of theirs.

The interaction sometimes surprises operators: a high-priority gang can wait behind a lower-priority job whose nodes are all free now, simply because the gang requires more nodes to be simultaneously free. Adding --exclusive or smaller node-count requests usually fixes the visible scheduling stall, at the cost of fragmenting the fleet.

What goes wrong even with gang scheduling

Gang scheduling does not solve every multi-rank issue. The classic failure modes that survive:

1. Stragglers within the gang. Even if every rank starts at t=0, if one node is thermally throttling or has a slow disk, that rank reaches the first collective late. Every other rank waits. See thermal stragglers for the operational angle.

2. One rank crashes mid-job. When rank 17 crashes 4 hours into a 12-hour run, the scheduler does not have a notion of "restart this rank." The whole job typically dies. Frameworks that support elastic training (PyTorch elastic, Ray Train) recover, but most production jobs need a checkpoint and full restart. See gang failure.

3. Topology blindness. Slurm gang scheduling guarantees that N nodes start simultaneously, but does not guarantee they share a switch or a rail. For training jobs that depend on intra-rack NVLink or single-rail InfiniBand, you also need topology-aware placement.

Practical guidance

• For any multi-node training job over 8 nodes, atomic allocation is mandatory. Slurm gives it by default; on Kubernetes, install Volcano or Kueue so pods bind as a group instead of one at a time.
• On Slurm, set PreemptMode=GANG with SchedulerType=sched/backfill if you want lower-priority co-scheduled jobs to time-slice with higher-priority ones rather than block.
• Set realistic per-partition DefaultTime / MaxTime and tune the backfill scheduler (bf_window, bf_resolution) so reservation planning has good time estimates. (MaxJobCount is a controller-wide DB cap, not a backfill knob.)
• For Kubernetes-native shops, Volcano and Kueue are the K8s analogs. They implement the same gang semantics on top of kube-scheduler.

The takeaway: gang scheduling is the difference between a multi-node training fleet that runs and one that mostly waits. Slurm has it built in; Kubernetes needs Volcano. Either way, the scheduler's job is to start every rank at once or none.