Jump to content

Proxmox VE OSX Guide discussion


fabiosun

Recommended Posts

Wanted to give back, so I worked on a python script to simplify how we can get all the information on our Proxmox configs.

 

https://gist.github.com/trisongz/05c0681583a97d442538c8737c74b8b5

 

It's not the cleanest, but it works for me, and hopefully makes our lives a bit easier with debugging.

 

You can copy that file or create a new file in your Proxmox and call it with

 

python3 vmcheck.py --vmid [your VM ID]

 

It should produce results like the following:

Spoiler

 

root@mac:~/cmds# python3 vmcheck.py --vmid 105

**** Explicit Passthrough Devices in vfio.conf ****

PCIe ID: 03:00.0 | VFIO ID: 1002:6863 | IOMMU Group: 14 | Used By: hostpci0      | Device: VGA compatible controller - Advanced Micro Devices, Inc.
PCIe ID: 03:00.1 | VFIO ID: 1002:aaf8 | IOMMU Group: 15 | Used By: hostpci0      | Device: Audio device - Advanced Micro Devices, Inc.
PCIe ID: 44:00.0 | VFIO ID: 1d6a:07b1 | IOMMU Group: 54 | Used By: hostpci3      | Device: Ethernet controller - Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller
PCIe ID: 46:00.1 | VFIO ID: 1022:149c | IOMMU Group: 57 | Used By: None          | Device: USB controller - Advanced Micro Devices, Inc.
PCIe ID: 46:00.3 | VFIO ID: 1022:149c | IOMMU Group: 58 | Used By: hostpci6      | Device: USB controller - Advanced Micro Devices, Inc.
PCIe ID: 49:00.0 | VFIO ID: 8086:f1a8 | IOMMU Group: 61 | Used By: None          | Device: Non-Volatile memory controller - Intel Corporation Device
PCIe ID: 4a:00.0 | VFIO ID: 1987:5012 | IOMMU Group: 62 | Used By: hostpci1      | Device: Non-Volatile memory controller - Phison Electronics Corporation E12 NVMe Controller
PCIe ID: 4b:00.0 | VFIO ID: 14e4:43a0 | IOMMU Group: 63 | Used By: hostpci2      | Device: Network controller - Broadcom Limited BCM4360 802.11ac Wireless Network Adapter
********************************************************************************************************************************************

**** In-Use Passthrough Devices ****

PCIe ID: 03:00.0 | VFIO ID: 1002:6863 | IOMMU Group: 14 | Used By: hostpci0      | Device: VGA compatible controller - Advanced Micro Devices, Inc.
PCIe ID: 03:00.1 | VFIO ID: 1002:aaf8 | IOMMU Group: 15 | Used By: hostpci0      | Device: Audio device - Advanced Micro Devices, Inc.
PCIe ID: 05:00.0 | VFIO ID: 1022:1485 | IOMMU Group: 17 | Used By: None          | Device: Non-Essential Instrumentation - Advanced Micro Devices, Inc.
PCIe ID: 22:00.0 | VFIO ID: 1022:1485 | IOMMU Group: 29 | Used By: None          | Device: Non-Essential Instrumentation - Advanced Micro Devices, Inc.
PCIe ID: 43:00.0 | VFIO ID: 1b21:3242 | IOMMU Group: 53 | Used By: hostpci4      | Device: USB controller - ASMedia Technology Inc. Device
PCIe ID: 44:00.0 | VFIO ID: 1d6a:07b1 | IOMMU Group: 54 | Used By: hostpci3      | Device: Ethernet controller - Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller
PCIe ID: 46:00.0 | VFIO ID: 1022:1485 | IOMMU Group: 56 | Used By: hostpci5      | Device: Non-Essential Instrumentation - Advanced Micro Devices, Inc.
PCIe ID: 46:00.3 | VFIO ID: 1022:149c | IOMMU Group: 58 | Used By: hostpci6      | Device: USB controller - Advanced Micro Devices, Inc.
PCIe ID: 4a:00.0 | VFIO ID: 1987:5012 | IOMMU Group: 62 | Used By: hostpci1      | Device: Non-Volatile memory controller - Phison Electronics Corporation E12 NVMe Controller
PCIe ID: 4b:00.0 | VFIO ID: 14e4:43a0 | IOMMU Group: 63 | Used By: hostpci2      | Device: Network controller - Broadcom Limited BCM4360 802.11ac Wireless Network Adapter
PCIe ID: 4d:00.0 | VFIO ID: 1022:1485 | IOMMU Group: 65 | Used By: None          | Device: Non-Essential Instrumentation - Advanced Micro Devices, Inc.
PCIe ID: 62:00.0 | VFIO ID: 1022:1485 | IOMMU Group: 76 | Used By: None          | Device: Non-Essential Instrumentation - Advanced Micro Devices, Inc.
********************************************************************************************************************************************

**** All Drivers from PCIe Devices - Active indicates vfio-pci is using this Device ****

Active: False    | Blacklisted: False    | Driver: piix4_smbus
Active: True     | Blacklisted: True     | Driver: snd_hda_intel
Active: False    | Blacklisted: False    | Driver: ccp
Active: False    | Blacklisted: True     | Driver: xhci_hcd
Active: True     | Blacklisted: True     | Driver: amdgpu
Active: True     | Blacklisted: False    | Driver: atlantic
Active: True     | Blacklisted: True     | Driver: bcma
Active: False    | Blacklisted: False    | Driver: ahci
Active: False    | Blacklisted: False    | Driver: r8169
Active: False    | Blacklisted: False    | Driver: nvme
Active: False    | Blacklisted: False    | Driver: pcieport
Active: False    | Blacklisted: False    | Driver: k10temp
************************************************************

**** Drivers Currently in Use by VM ****

Blacklisted: True        | Used By: hostpci0     | PCIe ID: 03:00.0      | Driver: amdgpu
Blacklisted: True        | Used By: hostpci0     | PCIe ID: 03:00.1      | Driver: snd_hda_intel
Blacklisted: False       | Used By: hostpci3     | PCIe ID: 44:00.0      | Driver: atlantic
Blacklisted: True        | Used By: hostpci2     | PCIe ID: 4b:00.0      | Driver: bcma
************************************************************

**** Blacklisted Drivers Not in Use ****

Blacklisted: True        | Driver: snd_hda_codec_hdmi
Blacklisted: True        | Driver: snd_hda_codec
Blacklisted: True        | Driver: snd_hda_core
Blacklisted: True        | Driver: btusb
Blacklisted: True        | Driver: snd-usb-audio
Blacklisted: True        | Driver: xhci_hcd
************************************************************

**** VM Configuration ****

CPU [Host]: 1 Socket(s), 32 Cores per Socket, 2 Threads per core
        CPU Name: AMD Ryzen Threadripper 3970X 32-Core Processor
        Architecture: x86_64
        Vendor ID: AuthenticAMD
        Virtualization: AMD-V

CPU [VM]: 64/64 Cores
CPU Allocation: 100.00%
CPU Config: VM is Allocated All of Host CPU and Possible Unsupported CPU Flags in Argument

Supported Host CPU Features: pdpe1gb, svm, npt, ssse3, popcnt, avx, avx2, aes, fma, bmi1, bmi2, xsave, xsaveopt
Unsupported Host CPU Features: invtsc, hypervisor, pcid, sse4.2, fma4

Unused Host CPU Features: fpu, vme, de, pse, tsc, msr, pae, mce, cx8, apic, sep, mtrr, pge, mca, cmov, pat, pse36, clflush, mmx, fxsr, sse, sse2, ht, syscall, nx, mmxext, fxsr_opt, rdtscp, lm, constant_tsc, rep_good, nopl, nonstop_tsc, cpuid, extd_apicid, aperfmperf, pni, pclmulqdq, monitor, cx16, sse4_1, sse4_2, movbe, f16c, rdrand, lahf_lm, cmp_legacy, extapic, cr8_legacy, abm, sse4a, misalignsse, 3dnowprefetch, osvw, ibs, skinit, wdt, tce, topoext, perfctr_core, perfctr_nb, bpext, perfctr_llc, mwaitx, cpb, cat_l3, cdp_l3, hw_pstate, sme, ssbd, mba, sev, ibpb, stibp, vmmcall, fsgsbase, smep, cqm, rdt_a, rdseed, adx, smap, clflushopt, clwb, sha_ni, xsavec, xgetbv1, xsaves, cqm_llc, cqm_occup_llc, cqm_mbm_total, cqm_mbm_local, clzero, irperf, xsaveerptr, wbnoinvd, arat, lbrv, svm_lock, nrip_save, tsc_scale, vmcb_clean, flushbyasid, decodeassists, pausefilter, pfthreshold, avic, v_vmsave_vmload, vgif, umip, rdpid, overflow_recov, succor, smca

 

 

 

  • Like 2
Link to comment
Share on other sites

  • Moderators
9 hours ago, fabiosun said:

@iGPUcould you post a screenshot of about my Mac memory section?

i would like to see how 32gb modules are seen in your rig

i have bought four modules but there I see 8 slot full populated with 16 gb modules each

also other users with 32gb modules type if possible

thank you

 

Sure. I saw your earlier msg but thought it was directed to someone else.

 

64792615_128GBDDR4.png.f94ac1556aac243a003b9dac1630691a.png

Link to comment
Share on other sites

Fabian notified me he made the Linux 5.8 pre-release official on his webpage. He also incorporated the AMD reset patch in it:

 

https://github.com/fabianishere/pve-edge-kernel/releases/tag/v5.8.0-navi-reset

 

So if you'd like to experiment with 'bleeding edge' kernels on Proxmox, now this is up to date so use his .deb releases as it's easier than building it yourself.

  • Like 1
  • +1 1
Link to comment
Share on other sites

Finally got around to running benchmarks after working through several Vega specific threads based on @meina222's device property patch, and combing through IORE

Spoiler


 

GeekBench 5

image.png.1bf4973205d802875bbddc1c16c402dc.png

 

LuxMark

1613420168_ScreenShot2020-08-11at1_46_23AM.png.a7f84f211a43a69f38c7bc1c4ce04bcf.png

 

Cinebench

1324669659_ScreenShot2020-08-11at2_03_46AM.png.3a194664ca534b9d10e5eddb70dfc437.png

 

Also tested out Cl!ng to test device/memory bandwidth

 

OpenCL

 

image.png.fb72b4ff20ff0288ecd76f6c04d4600a.png

Metal

image.png.209e770fab4609de719024bd5ae26efa.png

 

The GPU Metrics during 2 of the tests

 

For Geekbench, was around this

 

"Fan Speed(%)"=21
"Fan Speed(RPM)"=1047
"Temperature(C)"=47

 

LuxMark was when it got REALLY toasty

 

"Fan Speed(%)"=62
"Fan Speed(RPM)"=3053
"Temperature(C)"=67

 

The config I ended up using for my Vega Frontier. There's a few other PP_PhmSoftPowerPlayTable I haven't tested yet, but this was reported to be stable, so I went with this config, and so far so good.

 

image.png.51142361c8c30e88df4ac9509e18f37f.png

 


<key>DeviceProperties</key>
    <dict>
        <key>Add</key>
        <dict>
            <key>PciRoot(0x1)/Pci(0x1C,0x0)/Pci(0x0,0x0)</key>
            <dict>
                <key>AAPL,slot-name</key>
                <string>Slot-1</string>
                <key>ATY,Copyright</key>
                <string>Copyright AMD Inc. All Rights Reserved. 2005-2020</string>
                <key>ATY,DeviceName</key>
                <string>Vega Frontier Edition</string>
                <key>ATY,EFIVersion</key>
                <string>017.001.000.042.012566</string>
                <key>ATY,FamilyName</key>
                <string>Radeon</string>
                <key>ATY,Rom#</key>
                <string>113-D1820501-102</string>
                <key>CAIL_EnableLBPWSupport</key>
                <integer>0</integer>
                <key>CAIL_EnableMaxPlayloadSizeSync</key>
                <integer>1</integer>
                <key>CFG_APER_MODE</key>
                <integer>1</integer>
                <key>CFG_CAA</key>
                <integer>0</integer>
                <key>CFG_FB_LIMIT</key>
                <integer>0</integer>
                <key>CFG_FORCEMAXDPM</key>
                <false/>
                <key>CFG_FORCE_MAX_DPS</key>
                <false/>
                <key>CFG_GEN_FLAGS</key>
                <integer>0</integer>
                <key>CFG_INT_SSPC</key>
                <integer>25</integer>
                <key>CFG_NODM</key>
                <true/>
                <key>CFG_NON_ZERO_MC_LOC</key>
                <true/>
                <key>CFG_NO_HDCP</key>
                <false/>
                <key>CFG_NO_MSI</key>
                <false/>
                <key>CFG_NO_MST</key>
                <false/>
                <key>CFG_NO_PP</key>
                <false/>
                <key>CFG_NO_SLS</key>
                <false/>
                <key>CFG_PAA</key>
                <integer>0</integer>
                <key>CFG_PULSE_INT</key>
                <true/>
                <key>CFG_TRANS_WSRV</key>
                <true/>
                <key>CFG_USE_AGDC</key>
                <true/>
                <key>CFG_USE_CP2</key>
                <true/>
                <key>CFG_USE_DPT</key>
                <true/>
                <key>CFG_USE_FBC</key>
                <false/>
                <key>CFG_USE_FBWRKLP</key>
                <true/>
                <key>CFG_USE_FEDS</key>
                <true/>
                <key>CFG_USE_HDMI20</key>
                <true/>
                <key>CFG_USE_LPT</key>
                <false/>
                <key>CFG_USE_SRRB</key>
                <false/>
                <key>CFG_USE_STUTTER</key>
                <true/>
                <key>CFG_USE_SWIP</key>
                <true/>
                <key>DALReadDelayStutterOff</key>
                <integer>4</integer>
                <key>DALUseUrgencyWaterMarkOffset</key>
                <integer>0</integer>
                <key>Force_Load_FalconSMUFW</key>
                <true/>
                <key>PP_DisableClockStretcher</key>
                <integer>1</integer>
                <key>PP_DisableDIDT</key>
                <integer>1</integer>
                <key>PP_DisablePowerContainment</key>
                <integer>1</integer>
                <key>PP_DisableULPS</key>
                <integer>1</integer>
                <key>PP_DisableULV</key>
                <integer>1</integer>
                <key>PP_DisableVoltageIsland</key>
                <integer>0</integer>
                <key>PP_EnableUploadFirmware</key>
                <integer>0</integer>
                <key>PP_Falcon_QuickTransition_Enable</key>
                <integer>1</integer>
                <key>PP_FuzzyFanControl</key>
                <integer>0</integer>
                <key>PP_GfxOffControl</key>
                <integer>0</integer>
                <key>PP_PhmSoftPowerPlayTable</key>
                <data>
                tgIIAQBcAOEGAAAILAAAGwBIAAAAgKkDAPBJAgAyAAgA
                AAAAAAAAAAAAAAAAAAIBXABPAkYClACeAb4AKAF6AIwA
                vAEAAAAAcgIAAJAAqAJtAUMBlwHwSQIAcQICAgAAAAAA
                AAgAAAAAAAAABQAHAAMABQAAAAAAAAABCCADhAO2A+gD
                GgRMBGUEfgQBAUYFAQGEAwAIYOoAAABAGQEAAYA4AQAC
                3EoBAAOQXwEABAB3AQAFkJEBAAbA1AEABwEI0EwBAAAA
                gAAAAAAAAByDAQABAAAAAAAAAABwpwEAAgAAAAAAAAAA
                iLwBAAMAAAAAAAAAAMDUAQAEAAAAAAAAAABEIwIABQAA
                AAABAAAA8EkCAAYAAAAAAQAAAKhVAgAHAAAAAAEAAAAA
                BWDqAAAAQBkBAACAOAEAANxKAQAAkF8BAAAACChuAAAA
                LMkAAAH4CwEAAoA4AQADkF8BAAT0kQEABdCwAQAGwNQB
                AAcACGw5AAAAJF4AAAH8hQAAAqy8AAADNNAAAARobgEA
                BQiXAQAG7KMBAAcAAWg8AQAAAQQ8QQAAAAAAUMMAAAAA
                AIA4AQACAADEgAEABQAAAQgAmIUAAEC1AABg6gAAUMMA
                AAGAuwAAYOoAAJQLAQBQwwAAAgDhAACUCwEAQBkBAFDD
                AAADeP8AAEAZAQCIJgEAUMMAAARAGQEAgDgBAIA4AQBQ
                wwAABYA4AQDcSgEA3EoBAFDDAAAGAHcBAAB3AQCQXwEA
                UMMAAAeQkQEAkJEBAAB3AQBQwwAAARgAAAAAAAAAC+QS
                vAIkEy0ACgBUA5ABkAGQAZABkAGQAZABAAAAAAACCDEH
                9wD3APcAUQEAAFkAaQBKAEoAXwBzAHMAZABAAAAAl2CW
                AJBVAAAAAAAAAAAAAAAAAAAAAAACAtQwAAACEGDqAAAC
                EA==
                </data>
                <key>PP_PhmUseDummyBackEnd</key>
                <integer>0</integer>
                <key>PP_ThermalAutoThrottlingEnable</key>
                <integer>1</integer>
                <key>PP_ToolsLogSpaceSize</key>
                <integer>524288</integer>
                <key>PP_WorkLoadPolicyMask</key>
                <integer>32</integer>
                <key>built-in</key>
                <data>
                AA==
                </data>
                <key>device_type</key>
                <string>gpu-controller</string>
                <key>hda-gfx</key>
                <string>onboard-1</string>
                <key>model</key>
                <string>Radeon Vega Frontier Edition</string>
                <key>name</key>
                <string>ATY_GPU</string>
            </dict>
        </dict>
        <key>Delete</key>
        <dict/>
    </dict>


 

 

  • Like 2
Link to comment
Share on other sites

@tsongz - you're running a single GPU for host and VM right?

 

Do you use patched ROM as described here:

 

https://github.com/joeknock90/Single-GPU-Passthrough

 

Is this a good tutorial to set up single GPU? Asking since I have some PCIE resource issues as there are too many devices on my motherboard and want to try 1 GPU setup until I figure how to resolve it.

Link to comment
Share on other sites

On 7/23/2020 at 9:00 PM, Driftwood said:

@paulthesparky Ive been thru a lot of variables. Interesting to see the MADI card sits on a named slot! (Slot 3)

 

The firewire LSI card is almost perfect especially when the USB passthru is minimised - ie 48.00 group - 48:00.0, 48.00.1, 48:00.3 are disabled. And you run at only 32 cores.

 

With the PCIe HDSPe AIO there is definitely something like IRQ Interupt problems, possibly shared which is not quite allowing me to use it.

With PCIe cards, IOMMU handles the memory addresses, the DMA calls, and the timed interupts. And the AIO card is currently all over the place. It needs to be settled.

Investigating a few more options...

 

It would be nice I guess to get hold of a MADI card as they are newer and perhaps more vm friendly. Have you tried reading it with lspci yet ? Be interested to know the reading?

I will try and loan one from Synthax UK.

 

I booted up Ubuntu on the Mac Pro 5.1, and read the RME MADI PCIE card using lspci in terminal. Here are the results

 

 

ubuntu@ubuntu:~$ lspci 

00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:0d.0 Host bridge: Intel Corporation Device 343a (rev 22)
00:0d.1 Host bridge: Intel Corporation Device 343b (rev 22)
00:0d.2 Host bridge: Intel Corporation Device 343c (rev 22)
00:0d.3 Host bridge: Intel Corporation Device 343d (rev 22)
00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 22)
00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 22)
00:0d.6 Host bridge: Intel Corporation Device 341a (rev 22)
00:0d.7 Host bridge: Intel Corporation Device 341b (rev 22)
00:0e.0 Host bridge: Intel Corporation Device 341c (rev 22)
00:0e.1 Host bridge: Intel Corporation Device 341d (rev 22)
00:0e.2 Host bridge: Intel Corporation Device 341e (rev 22)
00:0e.3 Host bridge: Intel Corporation Device 341f (rev 22)
00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 22)
00:0f.0 Performance counters: Intel Corporation Device 3424 (rev 22)
00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 22)
00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 22)
00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 22)
00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 22)
00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2
00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3
00:1c.3 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 4
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 PCI bridge: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch (rev 01)
02:02.0 PCI bridge: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch (rev 01)
02:04.0 PCI bridge: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch (rev 01)
03:00.0 Multimedia audio controller: Xilinx Corporation RME Hammerfall DSP MADI (rev d2)
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cypress XT [Radeon HD 5870]
05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cypress HDMI Audio [Radeon HD 5830/5850/5870 / 6850/6870 Rebrand]
06:00.0 SATA controller: Kingston Technology Company, Inc. Device 0010 (rev 10)
09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01)
0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)
0d:00.0 Network controller: Broadcom Inc. and subsidiaries BCM43602 802.11ac Wireless LAN SoC (rev 01)

 

Edited by paulthesparky
Not sure how to make it hidden contents?
Link to comment
Share on other sites

On 8/11/2020 at 1:10 PM, meina222 said:

Do you use patched ROM as described here:

 

https://github.com/joeknock90/Single-GPU-Passthrough

 

No, I only use the rom from the download as you mentioned. I tested a few different PPM configurations in Device Properties and although I did notice a bit of a performance jump, (+5-7%), it made it really unstable, but at least was able to validate that modifying Power Tables does affect GPU performance without kexts/ACPIs. 

 

I've been following that thread - I'm a bit torn about going bare metal, mainly because of the docker compatibility with AMD CPUs, specifically on MacOS, and VirtualBox isn't yet supported on Big Sur.

Link to comment
Share on other sites

I have been testing bare metal. The only real downside of Proxmox is the AMD reset. Everything else is going to be so much harder on bare metal. My guess is that I will go back to Proxmox when an update bricks my bare metal. But it's too much fun to pass up. I made single GPU work.

Link to comment
Share on other sites

On 7/23/2020 at 9:00 PM, Driftwood said:

@paulthesparky Ive been thru a lot of variables. Interesting to see the MADI card sits on a named slot! (Slot 3)

 

The firewire LSI card is almost perfect especially when the USB passthru is minimised - ie 48.00 group - 48:00.0, 48.00.1, 48:00.3 are disabled. And you run at only 32 cores.

 

With the PCIe HDSPe AIO there is definitely something like IRQ Interupt problems, possibly shared which is not quite allowing me to use it.

With PCIe cards, IOMMU handles the memory addresses, the DMA calls, and the timed interupts. And the AIO card is currently all over the place. It needs to be settled.

Investigating a few more options...

 

It would be nice I guess to get hold of a MADI card as they are newer and perhaps more vm friendly. Have you tried reading it with lspci yet ? Be interested to know the reading?

I will try and loan one from Synthax UK.

 

@Driftwood I used lspci with gave only basic previous results on my last comment using ubuntu. Not being Linux proficient at this stage, I now realise I should have gone into more in-depth analysis to read other items, like the Madi output using lspci -nn etc. It takes a bit to set up Linux again on my Mac Pro 5.I, as I need to swap GPU's to boot Linux.

 

Instead, I found an interesting link/ thread dating back to 2008-2012, on the RME Forum to run different RME PCIe Audio cards using Ubuntu, which could be potentially useful. 

There is information there  which might be helpful in order to get your PCIe HDSPe AIO or Madi card, up and running on Proxmox/ OSX. 

 

Instead of using Totalmix, for your AIO card, you could possibly try using hdspmixer software?

 

If you managed to get hold of a Madi card, you could instead try to use text mode hdspmmixer (note the double M) to configure the card's internal mixer.

 

Another Madi Linux user,  with a hdspm driver, used alsamixer software, instead of Totalmix. He used alsamixer to set the sample rate, and the computer as the master clock, with word sync going to the converters, and jack at 44.1/48/88.2/ 96K sample rate. He had issues using the Madi card as a slave, with the converter as the master word clock/ sync.

 

Another Madi user got this result for using lspci -nn

 

The output from: $ lspci -nn | grep MADI:
04:00.0 Multimedia audio controller [0401]: Xilinx Corporation RME Hammerfall DSP MADI [10ee:3fc6] (rev d2)

 

Yet another Madi user lists info on IRQ, register addresses, etc.  Linux type info regarding Madi is quite sparse, and hard to find online.

 

https://forum.rme-audio.de/viewtopic.php?id=4164

 

 

 

 

 

Edited by paulthesparky
Link to comment
Share on other sites

  • Supervisor
root@proxmox:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.8.1-1)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-1
pve-kernel-helper: 6.2-1
pve-kernel-libc-dev: 5.8.1-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.3
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-5
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
root@proxmox:~# 

from https://github.com/fabianishere/pve-edge-kernel/releases
thank you @meina222for infos about this new release

 

  • Like 1
Link to comment
Share on other sites

  • Supervisor

I have had to revert to old fabian 5.78.1 kernel

for me this one is latest which works perfectly

 

With 5.8.x I have an instable system. it reboots my VM in a random way

 

I have associated this problem to my 2 10TB exsata disks or to my two 6tb data disks (not seen in osx because they are formatted in windows software raid)
these disks some time appear after few minutes in desktop..and often with 5.8.x kernel produce a vm reboot

with 5.78.1 this fact does not happen and system is perfect

 

  • Like 1
Link to comment
Share on other sites

On 8/10/2020 at 7:04 AM, fabiosun said:

@iGPUcould you post a screenshot of about my Mac memory section?

i would like to see how 32gb modules are seen in your rig

i have bought four modules but there I see 8 slot full populated with 16 gb modules each

also other users with 32gb modules type if possible

thank you

Same here.

 

 

Screen Shot 2020-08-20 at 18.27.59.png

On 8/13/2020 at 3:50 PM, paulthesparky said:

Another Madi Linux user,  with a hdspm driver, used alsamixer software, instead of Totalmix.

But it has to work under Mac OS!

Link to comment
Share on other sites

Pass thru'd and running the Sabrent Rocket 8TB for a week now. Amazing size and very good speed! Not on par with MP600s but at 8TB its plenty quick enough for pulling in raw video and sound library samples.

Edited by Driftwood
  • Thanks 1
  • +1 1
Link to comment
Share on other sites

Back on the Proxmox Train now with Big Sur.

 

- Can't use my existing bare metal Catalina install because it's installed on the Intel 660p which has issues with PCIe passthrough. 

- Added 1070ti to Slot 1, moved Vega Frontier to Slot 3

- Compiled PVE Kernel 5.4.55 with Vega 10 Patch (modified device ID in the patch with Vega FE's Device ID) and installed all the compiled debs

- Modified hook script for PVE MacOS (this now fixes the need to use RTC, and passes through cleanly. Although after VM shutdown, I still need to reboot the host)

- Have macOS running with Vega FE, and have ubuntu/Windows running with the 1070ti at the same time (has no issues with multiple resets)

- Biggest headache is the constant fixing of the Self-Assigned IP that macOS keeps popping up with. Since Big Sur doesn't place nice with the BCM adapter, any time eth breaks, keeps setting me back. I haven't found a reliable way of fixing this problem. 

 

Other than that, everything works perfectly for my work flow. Have CUDA 11.0 on the 1070ti, and can ssh into the second VM from the macOS VM. Will likely try to fix the BCM adapter issue when I get the chance and stay in Big Sur rather than Catalina, since it seems like applications that are getting ported into Big Sur have to play nicer with arm, with the trade off of less existing capabilities right now. 

 

 

Link to comment
Share on other sites

Hey, im having issues with a 3900x and a vega vii with passthrough. The machine works but freezes a lot for like 1-2 seconds and i cant play videos off of youtube for example neither, it stutters a lot. Any Ideas?

Link to comment
Share on other sites

@asynic Do you have any custom configuration for your Vega VII? And which SMBIOS model are you currently using?

 

I personally use additional device properties to tweak power settings for my Vega FE, and while the fans ramp up a bit faster than with bare metal (with same config, minus the addition of the 1070ti, so that could also be a cause), there's been no issues so far for me. I think one of the previous posts I had shared my gpu configuration.

 

You could download VGTab, and use the stock settings for it, and replace the power tables to see if that makes a difference.

 

Also @meina222 - with the patched PVE kernels installed, I no longer experience any reset issues with the Vega FE. I can start up and shut down the VM with PCIe passthrough multiple times without any problems, although sometimes a non-clean shutdown of the VM will require a full host reset. 

Link to comment
Share on other sites

@tsongz - good to hear about the reset. From what I recall you use single GPU and unload the framebuffer from the host at startup of VM. Do you then give it back to the host? Right now I've configured my hook script to shutdown the host when I shutdown the VM but if the reset patch allows you to reboot more cleanly, then I want to try single GPU and reset + giving back the framebuffer to the host. Do you mind sharing your hook?

Link to comment
Share on other sites

Sure, here's my hook. Just note that this is in conjunction with the Vega 10 patch, with specifying my Vega FE device ID in the patch (I'm not sure why they don't explicitly say that part, and just let people patch it with two of the many Vega cards), and also having a different GPU in PCIe Slot 1, and passing the Vega FE through in PCIe Slot 3. 

 

That said - I connect my single monitor to both GPUs and switch the inputs, and it does still work, despite that first part of the text from PVE on the Vega FE card. I've passed both GPUs through to 2 separate VMs with no issues.

 

It doesn't make any audible "click" sound from the RTC reset as it used to. 

 

Spoiler



#!/usr/bin/env bash

export VGA_DEVICE=0000:23:00.0 < - Change this

if [ "$2" == "release" ] ; then
if [ "$3" == "end" ] ; then
echo "unbinding, resetting and rebinding $VGA_DEVICE" > /dev/kmsg 2>&1
echo $VGA_DEVICE > /sys/bus/pci/devices/$VGA_DEVICE/driver/unbind
# Change the path to reset-test below #
/root/cmds/reset-test $VGA_DEVICE
echo $VGA_DEVICE > /sys/bus/pci/drivers/vfio-pci/bind
echo "sucessfully unbound, reset and rebound $VGA_DEVICE" > /dev/kmsg 2>&1
fi
fi


 

 

 

 
Link to comment
Share on other sites

Mine is incidentally the same pcie id but I have 1 GPU in slot 1 and I have to dump the vbios and specify the path in the VM config. I decided to give up on the 2nd GPU because I don't have the need to run 2 VMs (or at least not 2 VMs with acceleration).

 

So what kind of reset patch is that? The only one I tried involved the power clicking. And what does root/cmds/reset do?

Edited by meina222
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • There are no registered users currently online
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.