Jump to content

Proxmox VE OSX Guide discussion


fabiosun

Recommended Posts

21 minutes ago, Driftwood said:

 

@cj750 What is your vfio.conf looking like? You are passing the addresses: for video and HDMI audio: -

ie you have the following for the Frontier card in your vfio.conf

options vfio-pci ids=1002:6863 ,1002:aaf8 disable_vga=1?

 

After every modprobe vfio change your updating the initramfs, update grub, pve-efiboot-tool refresh and shutdown now? switch off, then power back up.

 

Here's a few VFIO guide tips: https://github.com/SMeatBoy/vfio/blob/master/README.md

Here is 

 

options vfio-pci ids=1002:6863,1002:aaf8 disable_vga=1

 

looks the same

Link to comment
Share on other sites

  • Supervisor
16 hours ago, Rox67er said:

From what I can see in the picture it is exactly the same connector, I think it is called an NGFF intended for WiFi. So if you can buy the same card with the metal casing you can add wifi too without offering a PCIE slot. (Some nice info here: https://www.delock.com/infothek/M.2/M.2_e.html)

 

Or use some internal antenna like this:

----

Have you a link to buy also metal case because I have not it and it seems safer to mount a Wi-Fi bt in an original metal case

  • Like 1
Link to comment
Share on other sites

2 hours ago, fabiosun said:

Have you a link to buy also metal case because I have not it and it seems safer to mount a Wi-Fi bt in an original metal case

I searched a bit for it yesterday but didn't find anything. Sorry.

  • +1 1
Link to comment
Share on other sites

5 hours ago, cj750 said:

Here is 

 

options vfio-pci ids=1002:6863,1002:aaf8 disable_vga=1

 

looks the same

Have you tried the 5700X special settings from the original guide? It uses a slightly different unbind code. Please check over there.

 

#!/bin/bash
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

  • +1 1
Link to comment
Share on other sites

@fabiosun @Driftwood

I noticed something strange regarding my USB controller issue, in my IOMMU overview the group for the two USB Controllers looks like this:

IOMMU Group 49 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU Group 49 47:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 49 47:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 49 47:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

But for the excel file shared by @Driftwoodfrom the ASRock techie it is like this:

IOMMU Group 55 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 55 48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 55 48:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 55 48:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

I noticed that my 42:08.0 only is called "Device" while the same controller in the other dump is called "Matisse PCIe GPP Bridge".

 

Can you check how this looks like on your side?

Edited by Rox67er
Link to comment
Share on other sites

  • Supervisor
11 minutes ago, Rox67er said:

And maybe more interesting which kernel driver is loaded (lspci -nnk):


42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport

Is this the same with you @fabiosun ?

 

I have this:

IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:

42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

 

  • Like 1
  • +1 1
Link to comment
Share on other sites

34 minutes ago, fabiosun said:

 

I have this:


IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:


42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

 

 

ok so that also doesn’t seem to be the cause...

Link to comment
Share on other sites

1 hour ago, Rox67er said:

noticed something strange regarding my USB controller issue, in my IOMMU overview the group for the two USB Controllers

Im asking ASRock these very questions and to fill in all the addresses being employed on the motherboard for the two SATAs (which is controlling which ports?) and the USB 2.0 device, front panel addresses etc...

56 minutes ago, Rox67er said:

I have this:


IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:


42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

 

These will change addresses depending on what is in the PCIe slots/m.2 slots occupied. etc. Where you have 46:00.1 etc I have them at 47:00.1 etc...

Edited by Driftwood
Link to comment
Share on other sites

Type in command shell:-

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*};n=${n%%/*}; printf 'IOMMU Group %s ' " $n";lspci -nns "${d##*/}"; done;

 

and copy it into a excel spreadsheet. Could you show me your complete spreadsheet?

Edited by Driftwood
Link to comment
Share on other sites

  • Supervisor
2 hours ago, Driftwood said:

Type in command shell:-

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*};n=${n%%/*}; printf 'IOMMU Group %s ' " $n";lspci -nns "${d##*/}"; done;

 

and copy it into a excel spreadsheet. Could you show me your complete spreadsheet?

this is mine

 

MSI TRX40 PRO 10G IOMMU.xls.zip

Edited by fabiosun
edited
  • Like 2
Link to comment
Share on other sites

ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...

 

Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

15 minutes ago, fabiosun said:

@DriftwoodI thought you want to see other result.. 🙂

 

Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!

 

 

@Rox67er

ASRock techie said the report for IOMMU groups was done on the following (not using Proxmox)

It was just a fresh Ubuntu desktop 20.04 LTS installation: https://ubuntu.com/download/desktop

So that should be kernel 5.4.

Edited by Driftwood
  • Like 2
Link to comment
Share on other sites

  • Supervisor
1 minute ago, Driftwood said:

ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...

 

Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

 

Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!

 


I don't know if it is simpler or not than other motherboards, but I always wondered why other users need to block things in vfio when in my case you just have to pass them in the virtual machine configuration to be automatically passed to vfio. ... I started this journey by doing things as recommended by the proxmox wiki..to cross out all the useless things for me

  • Like 1
  • Ok 1
Link to comment
Share on other sites

9 hours ago, Rox67er said:

Have you tried the 5700X special settings from the original guide? It uses a slightly different unbind code. Please check over there.

 


#!/bin/bash
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

Rox67er,

 

I was missing the line below. But nothing changed. Question when proxmox loads. And it choses the version screen. Should i see anything on the screen right after?. As soon as ProxM loads it goes to black screen. Before i started the mac os build it would log to a login root screen. Now i can only get to the GUI login from a different computer. But the only thing that has change when i did this before was my card was in slot 3 not slot 1. But still black screen. Also i change the Bios setting 

echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

3 hours later:

 

OMG......... Speechless ................

 

So checked everything over and over. read 25 pages of post twice. And guide i rather not say how many times. It load to the proxmox screen like it does 1,000 of time on my TV, but then goes black when it loads.  This time Im like what if i choose Proxmox environment 5.4.34-1-pve instead of 5.4.41-1-pve. It goes to black and then i see the proxmox loading file screen and there it was the root login window on my screen.

I was in shock. I didnt know what to do. But i rebooted the pc again to see if it was just luck. Nope it worked on older version. So when i used web to start the VM with GPU it loaded... 

 

I cant say Thank You enough for everyone help an not giving up on me.   But now i have to maybe figure out why the new version of pro does not load my card. But im so happy im pouring me  glass of scotch.  I know i have allot to do more. But for me this was a big step.

Edited by cj750
  • whahahah 1
  • Cross Finger 1
  • +1 1
Link to comment
Share on other sites

4 hours ago, Driftwood said:

ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...

 

Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

 

Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!

 

 

@Rox67er

ASRock techie said the report for IOMMU groups was done on the following (not using Proxmox)

It was just a fresh Ubuntu desktop 20.04 LTS installation: https://ubuntu.com/download/desktop

So that should be kernel 5.4.

@Driftwood @fabiosun

Well close but not completely. After reading your post I jumped in and started tweaking the bios again....

 

Using the above settings crashes Proxmox on boot but I noticed that when enabling AER Cap a new option showed up in bios which was disabled. After also enabling that one I can stop-start-reboot and get no 99 error. Tried it now for like 8 times... 😁

 

So my current bios settings for ASRock TRX40 Creator:

(Still using Bios 1.63 but expect no impact from that to be honest)

 

* Start with default UEFI settings

* Set XMP profile for your RAM

* Tweaked FAN settings

* Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

* Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

* Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

* Advanced>AMD CBS>NBIO Common option>ACS Enable>Enable

 

I did not need to set above 4G or anything else. Let's see if this is stable but looks like we nailed it !!! 😎

  • Cross Finger 1
  • +1 1
Link to comment
Share on other sites

2 minutes ago, fabiosun said:

@cj750 send us a glass of scotch 😂

and yes it is a big step for you

now you have the hope it can work also for you👍

 

I will send! 

Well i can move the mouse and keyboard and login. So im going to take a break with my drink and play some hockey for 30mins. To clear the mind. And start the next hurdle.

  • whahahah 1
Link to comment
Share on other sites

The ASRock TrX40 Creator has two SATA controllers at 48:00.0 and 49:00.0 respectively.

I got ports 5-8 working on the SATA controller at address 48:00.0 now and passedthru: 

 

VFIO 

options vfio=pci ids=1022:7901

 

VM 100.CONF

hostpci6: 48:00.0,pcie=1,rombar=1

 

This means I can now use four internal drives on those ports (indeed, Im looking at my windows boot drive on port 5 right now!), whilst keeping the Proxmox SSD boot drive on 49:00.0, SATA port 1 (1-4 is free) from outside the hypervisor. 🙂

 

Here's v2 of my chart updated. Now lets get these USB ports figured out! @Rox67er

 

176772148_TRXCreatorMotherboardLayourtDriftwoodv2.png.aed73d5a26ae1b3068e86cb960c0e4bb.png48 00.0 SATA controller looks after ports 5-8 ASRock Creator.png

Edited by Driftwood
  • Like 1
  • +1 2
Link to comment
Share on other sites

4 hours ago, Rox67er said:

* Start with default UEFI settings

* Set XMP profile for your RAM

* Tweaked FAN settings

* Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

* Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

* Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

* Advanced>AMD CBS>NBIO Common option>ACS Enable>Enable

 

I did not need to set above 4G or anything else. Let's see if this is stable but looks like we nailed it !!! 😎

 

This BIOS looks good @Rox67er... Im reporting back to ASRock. I can now run my 64 cores stable too with these BIOS settings.

I'm running Above 4G too and you get slightly better benchmarks.

Edited by Driftwood
  • Ok 1
  • +1 2
Link to comment
Share on other sites

15 hours ago, Driftwood said:

 

This BIOS looks good @Rox67er... Im reporting back to ASRock. I can now run my 64 cores stable too with these BIOS settings.

I'm running Above 4G too and you get slightly better benchmarks.

Well I did run into the 99 error again today, I didn't change anything though. Still needs more testing I guess.

 

I tried performance run with Above 4G on / off results below (Geekbench 5 CPU and GPU Metal score)

 

Above 4G OFF

1282  /  23881  /  61463 (average of 4 runs)

 

Above 4G ON

1282  /  23714  /  60924 (average of 3 runs)

 

I would say no difference in my setup.

  • +1 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • There are no registered users currently online
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.