[ For AI/ML Labs ]
Catch the failuresyour dashboards miss.
Training and inference clusters kept reliable. Runs in your VPC next to DCGM, Prometheus, and Grafana.
[ Where labs lose time ]
Three ways a clusterquietly loses time.
Failure modesCommon in training clusters
01 · Silent degradation
One throttled rank slows every all-reduce.
02 · Noisy faults
Xid 79, Xid 48, and NCCL timeouts, lost in the noise.
03 · Cross-layer on-call
The fault crosses five layers. The context is asleep.
[ What changes ]
Detected, diagnosed,fixed with your approval.
Where Factryze sits in your stackHuman-in-the-loop by default
event streamexample · 2026-04-12
14:23:11GPU 4 (H100)
Xid 79: fallen off bus
14:23:11GPU 4 (H100)Xid 79: fallen off bus
14:23:11rank 4
NCCL ring timeout 5000ms
14:23:11rank 4NCCL ring timeout 5000ms
14:23:12GPU 4
PCIe x16 → x8 downtrained
14:23:12GPU 4PCIe x16 → x8 downtrained
14:23:13scheduler
job 4f72 paused
14:23:13schedulerjob 4f72 paused
agent→drain node 04, swap to node 09, ticket FZ-417
01 · Continuous detection
Every GPU, link, and driver state, read continuously.
02 · Cross-layer root cause
GPU, PCIe, NCCL, and scheduler events traced to one cause.
03 · Recommend, then remediate
Approve the fix once, or autopilot known-safe runbooks.
Works with
- DCGM
- Prometheus
- Grafana
- Telemetry stays in your network
[ Compared ]
Why not justan existing tool?
Capability comparisonFactryze vs. the usual stack
| capability | DCGM Exporter | Datadog GPU | NVIDIA Mission Control | factryze |
|---|---|---|---|---|
| Detect Xid 79 / NCCL timeouts in seconds | partial | yes | yes | yes |
| Cross-layer correlation (GPU + NCCL + PCIe + scheduler) | no | partial | partial | yes |
| Recommend the actual remediation step | no | no | no | yes |
| Execute the runbook (drain, swap, ticket) | no | no | no | yes |
| Deploys in your VPC with telemetry on-network | yes | no | partial | yes |
Detect Xid 79 / NCCL timeouts in seconds
DCGM ExporterpartialDatadog GPUyesNVIDIA Mission ControlyesfactryzeyesCross-layer correlation (GPU + NCCL + PCIe + scheduler)
DCGM ExporternoDatadog GPUpartialNVIDIA Mission ControlpartialfactryzeyesRecommend the actual remediation step
DCGM ExporternoDatadog GPUnoNVIDIA Mission ControlnofactryzeyesExecute the runbook (drain, swap, ticket)
DCGM ExporternoDatadog GPUnoNVIDIA Mission ControlnofactryzeyesDeploys in your VPC with telemetry on-network
DCGM ExporteryesDatadog GPUnoNVIDIA Mission Controlpartialfactryzeyes
yeshandles nativelypartialrequires assemblynonot supported
[ Next step ]
Stop losing checkpoints.Bring your worst incident.
30 minutes. We'll diagnose it live and tell you whether we can help.
Currently piloting on NVIDIA H100 and A100 training clusters · per-GPU pricing