Setting up NVIDIA Tesla P100

the P100 was choses for balance between performance and price (second hand ebay)

Aliexpress

PCIE 4.0 3.0 16X Riser Kabel 90/180 Graden Mount Video Grafisch

Gpu 10pin Naar 1x8pin Dual 8(6 + 2)Pin Power Adapter Kabel Voor Hp Dl380 Gen8 Gen9 Server

Cutting the Riser

a special Riser for the HP Proliant gen8 is a good idea because it has a PCIE 16x in the middle which allows for the GPU to be plugged in. (hard to find and expensive)

I choose the cheap option (evidently) and cut a hole in a riser to extend with a riser cable into this riser. Some more cutting had to be done because the GPU is too long.

For the gen9 there a cheaper riser card options, so no cutting needed…

Configure the BIOS

(changed IRQ for network card to 11, since conflict)

enter BIOS (F9) and on main bios screen / “Service options” menu item. Under this, enable “PCI Express 64-bit BAR

configure proxmox host

PROXMOX PCI-E GPU passthru

When running proxmox on this hardware, there is more config needed to enable passthru of this GPU to VM. Enable IOMMU

Edit /etc/default/grub file, modify variable GRUB_CMDLINE_LINUX_DEFAULT adding intel_iommu=on parameter Enable unsafe intremap

Create file /etc/modprobe.d/iommu_unsafe_interrupts.conf with contents options vfio_iommu_type1 allow_unsafe_interrupts=1 Blacklist nvidia drivers

Create file /etc/modprobe.d/nvidia-blacklist.conf with contents

# blacklist for nvidia gpu passthru blacklist nouveau blacklist nvidia*

Apply changes into bootloader

Run proxmox-boot-tool refresh

the idea

the idea is to get nvidia to work on the proxmox host (there will be kernel modules)

for the LXC machines there is no need for kernel drivers!! since already on the host

Download NVIDIA drivers

NVIDIA-Linux-x86_64-570.133.20.run

https://developer.nvidia.com/cuda-12-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Debian&target_version=12&target_type=deb_local

*nvidia is gonna need some kernel (promox) drivers

apt install pve-headers-$(uname -r) build-essential software-properties-common make nvtop htop -y

./NVIDIA-Linux-x86_64-570.133.20.run

copy NVIDIA driver to LXC

pct push 100 NVIDIA-Linux-x86_64-570.133.20.run /root/NVIDIA-Linux-x86_64-570.133.20.run