Troubleshoot external compute

Fix common Runpod Hybrid Cloud problems, including failed hardware verification, machines stuck in a status, and missing GPUs in the picker.

This page covers the most common problems when you connect and use external compute, and what to do about each.

Verification failed

When a machine’s status is Verification failed, the automated hardware checks found a problem with the machine or its configuration. Running verification again without changing anything gives the same result. Fix the underlying issue first, then retry verification.

To see which checks failed, open the machine on the External compute page and go to the Verifications tab.

DockerFS

The disk mounted at /var/lib/docker is too small. The failure message shows the size it found and the size it requires.

To fix this, mount a dedicated disk that meets the Docker disk size in Requirements at /var/lib/docker, then retry verification.

DockerFstabUUID

The Docker disk is mounted in /etc/fstab by its device name, such as /dev/sdb or /dev/md0, instead of by its UUID. Device names can change when the machine reboots, which can stop the Docker disk from mounting.

To fix this:

1. Find the UUID of the Docker disk:

   sudo blkid
   

2. In /etc/fstab, replace the device name on the /var/lib/docker line with the UUID. For example:

   UUID=DISK_UUID  /var/lib/docker  xfs  defaults  0  0
   

3. Check that the disk mounts correctly, then retry verification:

   sudo mount -a
   

CUDA Open Variant

The machine has the wrong variant of the NVIDIA driver installed, usually the server variant packages. Runpod requires the open kernel module variant, and the Runpod installer installs the driver for you.

To fix this, remove any NVIDIA driver packages you installed before onboarding, reboot the machine, and retry verification. To see which NVIDIA packages are installed, run:

dpkg -l | grep nvidia

Verification timed out

Verification timed out means Runpod couldn’t connect to your machine to run the hardware checks. To fix this:

1. Make sure sshd is listening on all network interfaces, not only on localhost.
2. Check that your firewall allows the inbound and outbound traffic listed in Requirements.
3. Retry verification.

Machine stuck in pending install

Pending install means Runpod hasn’t heard from the host agent yet. To fix this:

1. Make sure the install command finished without errors. If you aren’t sure, run it again.
2. Check that the machine can make outbound connections to the internet.
3. Confirm that you rebooted the machine after you configured networking.

Machine stuck in verification in progress

Verification normally finishes on its own. If a machine stays in Verification in progress much longer than expected, the problem is most likely on the Runpod side, not your hardware. Contact the Runpod team with the machine name.

Machine doesn’t appear in the GPU picker

If you can’t find your machine’s GPU when you deploy a Pod:

• Check that the machine’s status is Online on the External compute page. Machines in any other status can’t accept Pods.
• Make sure you’ve selected the organization that owns the machine in the console.
• Check whether every GPU on your machines is already in use. The External compute page shows current usage.

Get help

Runpod Hybrid Cloud is in alpha, and each customer works with the Runpod team during onboarding. If these steps don’t fix the problem, contact your Runpod onboarding contact and include the machine name and its current status.

Next steps

• Set up external compute.
• Deploy Pods on your external compute.