I have been running TrusNas in a VM under esxi for about five years. With the recent upheaval at Broadcom, I’ve moved to XCP-NG. I have a VM that has a pass through of the HBA that carries four NVME drives.
It worked well under esxi but with TrueNAS-SCALE-23.10.1 under XCP-NG it stops as its booting with a bunch of error messages. I tried TrueNAS-SCALE-22.12.0 and it worked fine. For the heck of it I tried the nightlies of 24.04 and it too works just fine there.
When you do an “lspci” these are the nvme drives:. 0
04:00.0 Non-Volatile memory controller:
Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
05:00.0 Non-Volatile memory controller: Intel Corporation Device 09ad
06:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
07:00.0 Non-Volatile memory controller: Intel Corporation Device 09ad
FWIW, I did these three things to make it work:
/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:04:00.0)(0000:05:00.0)(0000:06:00.0)(0000:07:00.0)"
/opt/xensource/libexec/xen-cmdline --set-xen "extra_guest_irqs=128"
<reboot>
xe vm-param-set other-config:pci=0/0000:04:00.0,0/0000:05:00.0,0/0000:06:00.0,0/0000:07:00.0 uuid=blah-blah-blajh
Only mentioning it in case anyone out there runs into itolivierlambertVates 🪐Co-FounderCEO 15 Jan 2024, 17:50
Hi!Thanks for your feedback. Note that the commands pointed are documented into this section: https://docs.xcp-ng.org/compute/#2-tell-xcp-ng-not-to-use-this-device-id-for-dom0edit: also edited your post with the correct Markdown syntax
Aarchw @olivierlambert15 Jan 2024, 18:01
@olivierlambert
Cool! Yeah I got my comments from the place you mentioned. I only brought it up in case someone else was also pulling their hair out with TrueNas 23.10.1 and thinking they had missed something (try this…reboot…try this…reboot…cuss a bunch…reboot…)
This article was first published here