Не работает Xen PCI passthrough после переезда на Ubuntu 22.04
Форум — General
Всем доброго времени суток!
На одной и той же машине, где все отлично работает на Ubuntu 14.04, лютый же треш на Ubuntu 22.04.
Первый звоночек был - что гипервизор Xen не работает на голой операционке «из коробки». Ядра что идут с ubuntu в комплекте - 6.8.0-40-generic и 6.5.0-18-generic, вообще в паре с Xen'oм не работают без вылетов.
Перебрав с дюжину вариантов, остановился на ядре «6.1.0-1036-oem» с которым получилось добиться стабильной работы Dom0. Но пока тщетны попытки пробросить в гостевую систему любой hardware. Ошибки одинаковые, будь то usb hub, soundcard или gpu.
Конфиг DomU:
#cat /etc/xen/router
builder='hvm'
memory = 1000
name = "router"
vcpus=1
#pae=1
acpi=1
apic=0
disk = [
'file:/media/md0/vm/router_test_ubu24.img,hda,w'
]
xen_platform_pci=1
viridian=1
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'destroy'
boot='c'
sdl=0
vnc=1
vncdisplay=5
serial='pty'
pci = ['08:00.0']
Запускаю:
#xl create /etc/xen/router
libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 10:Offset 0x000e:0xffff0080 expands past register size (1)
libxl: error: libxl_pci.c:1775:device_pci_add_done: Domain 10:libxl__device_pci_add failed for PCI device 0:8:0.0 (rc -28)
libxl: error: libxl_create.c:1892:domcreate_attach_devices: Domain 10:unable to add pci devices
Если же в конфиге изменить acpi с 1 на 0, те же грабли только в профиль.
libxl: error: libxl_qmp.c:1838:qmp_ev_parse_error_messages: Domain 11:Bus 'pci.0' does not support hotplugging
libxl: error: libxl_pci.c:1775:device_pci_add_done: Domain 11:libxl__device_pci_add failed for PCI device 0:8:0.0 (rc -28)
libxl: error: libxl_create.c:1892:domcreate_attach_devices: Domain 11:unable to add pci devices
К pciback'у подключены:
# xl pci-assignable-list
0000:08:00.0
IOMMU вроде работает:
# dmesg | grep DMAR
[ 2.382298] ACPI: RMAD 0x000000007DF176B0 000110 (v01 A M I OEMDMAR 00000001 INTL 00000001)
[ 4.299817] DMAR: IOMMU enabled
Если вдруг окажется полезным, выхлоп xl info и xl dmesg:
# xl info
host : geregtigheig-pc
release : 6.1.0-1036-oem
version : #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Mar 11 17:32:20 UTC 2024
machine : x86_64
nr_cpus : 24
max_cpu_id : 23
nr_nodes : 3
cores_per_socket : 12
threads_per_core : 1
cpu_mhz : 2700.004
hw_caps : bfebfbff:77bee3ff:2c100800:00000001:00000001:00000281:00000000:00000100
virt_caps : pv hvm hvm_directio pv_directio hap shadow iommu_hap_pt_share gnttab-v1 gnttab-v2
total_memory : 65501
free_memory : 1012
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 16
xen_extra : .0
xen_version : 4.16.0
xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit2
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder
cc_compiler : x86_64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
cc_compile_by : ubuntu-devel-di
cc_compile_domain : lists.ubuntu.com
cc_compile_date : Sat Nov 26 22:06:42 UTC 2022
build_id : 4805454f1b2808877ffdb263ecb101cd100b5263
xend_config_format : 4
# xl dmesg
(XEN) Xen version 4.16.0 (Ubuntu 4.16.0-1~ubuntu2.1) (ubuntu-devel-discuss@lists.ubuntu.com) (x86_64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0) debug=n Sat Nov 26 22:06:42 UTC 2022
(XEN) Bootloader: GRUB 2.02~beta2-9ubuntu1.17
(XEN) Command line: placeholder
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN) EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN) Found 7 MBR signatures
(XEN) Found 6 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) [0000000000000000, 000000000008ffff] (usable)
(XEN) [0000000000090000, 0000000000096fff] (reserved)
(XEN) [0000000000097000, 000000000009e7ff] (usable)
(XEN) [000000000009e800, 000000000009ffff] (reserved)
(XEN) [00000000000e0000, 00000000000fffff] (reserved)
(XEN) [0000000000100000, 000000007ddb1fff] (usable)
(XEN) [000000007ddb2000, 000000007ddedfff] (reserved)
(XEN) [000000007ddee000, 000000007df17fff] (ACPI data)
(XEN) [000000007df18000, 000000007e145fff] (ACPI NVS)
(XEN) [000000007e146000, 000000007f32dfff] (reserved)
(XEN) [000000007f32e000, 000000007f7fffff] (ACPI NVS)
(XEN) [0000000080000000, 000000008fffffff] (reserved)
(XEN) [00000000fed1c000, 00000000fed3ffff] (reserved)
(XEN) [00000000ff000000, 00000000ffffffff] (reserved)
(XEN) [0000000100000000, 000000107fffffff] (usable)
(XEN) New Xen image base address: 0x7d600000
(XEN) ACPI: RSDP 000F04A0, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT 7DE39088, 009C (r1 HPQOEM SLIC-MPC 1072009 AMI 10013)
(XEN) ACPI: FACP 7DE45D40, 010C (r5 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: DSDT 7DE391B0, CB8B (r2 ALASKA A M I 4 INTL 20051117)
(XEN) ACPI: FACS 7E13DF80, 0040
(XEN) ACPI: APIC 7DE45E50, 01B4 (r3 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: FPDT 7DE46008, 0044 (r1 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: FIDT 7DE46050, 009C (r1 ALASKA A M I 1072009 AMI 10013)
(XEN) ACPI: MCFG 7DE460F0, 003C (r1 ALASKA OEMMCFG. 1072009 MSFT 97)
(XEN) ACPI: SRAT 7DE46130, 0430 (r1 A M I AMI SRAT 1 AMI. 0)
(XEN) ACPI: SLIT 7DE46560, 0030 (r1 A M I AMI SLIT 0 AMI. 0)
(XEN) ACPI: HPET 7DE46590, 0038 (r1 ALASKA A M I 1072009 AMI. 5)
(XEN) ACPI: SSDT 7DE465C8, D0CB0 (r2 INTEL CpuPm 4000 INTL 20051117)
(XEN) ACPI: EINJ 7DF17278, 0130 (r1 AMI AMI EINJ 0 0)
(XEN) ACPI: ERST 7DF173A8, 0230 (r1 AMIER AMI ERST 0 0)
(XEN) ACPI: HEST 7DF175D8, 00A8 (r1 AMI AMI HEST 0 0)
(XEN) ACPI: BERT 7DF17680, 0030 (r1 AMI AMI BERT 0 0)
(XEN) ACPI: DMAR 7DF176B0, 0110 (r1 A M I OEMDMAR 1 INTL 1)
(XEN) ACPI: SLIC 7DF177C0, 0176 (r1 HPQOEM SLIC-MPC 1 HPQ 1)
(XEN) System RAM: 65501MB (67073316kB)
(XEN) Domain heap initialised DMA width 29 bits
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 7e13df80/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1200 ... 2700 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN) Hardware hints:
(XEN) Hardware features:
(XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: No, Other: BRANCH_HARDEN
(XEN) L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 46, Safe address 300000000000
(XEN) Support for HVM VMs: RSB EAGER_FPU
(XEN) Support for PV VMs: EAGER_FPU
(XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN) PV L1TF shadowing: Dom0 disabled, DomU enabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 2700.004 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) Allocated console ring of 64 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) - APIC Register Virtualization
(XEN) - Virtual Interrupt Delivery
(XEN) - Posted Interrupt Processing
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 24 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) mtrr: your CPUs had inconsistent fixed MTRR settings
(XEN) Dom0 has maximum 1112 PIRQs
(XEN) Xen kernel: 64-bit, lsb
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4e00000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000070000000->0000000078000000 (16472653 pages to be allocated)
(XEN) Init. ramdisk: 0000001078a87000->000000107ffff9bb
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff84e00000
(XEN) Phys-Mach map: 0000008000000000->0000008007e27e30
(XEN) Start info: ffffffff84e00000->ffffffff84e004b8
(XEN) Page tables: ffffffff84e01000->ffffffff84e2c000
(XEN) Boot stack: ffffffff84e2c000->ffffffff84e2d000
(XEN) TOTAL: ffffffff80000000->ffffffff85000000
(XEN) ENTRY ADDRESS: ffffffff834681c0
(XEN) Dom0 has maximum 24 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 592kB init memory
Само собой без строки «pci = ['08:00.0']» в конфиге - зараза работает.. У кого есть мысли куда копать?
p.s. Кидается в глаза строчка: (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
но в ubuntu 14.04 (4.4.0-124-generic & xen 4.4.2), эта строка идентичная, и passthrough работает..