PCI passthrough in nested virtualization (rhel7)-白红宇

PCI passthrough in nested virtualization (rhel7)

阅读量：6314 次

发布时间：2019-06-22

本文共 5281 字，大约阅读时间需要 17 分钟。

Frist, you need an Intel host support Vt-d, and enabled nested virtualization.

By default, nested virtualization is disabled:

[root@kvm-hypervisor ~]# cat /sys/module/kvm_intel/parameters/nestedN

To enable nested virtualization:

[root@kvm-hypervisor ~]# vi /etc/modprobe.d/kvm-nested.confoptions kvm-intel nested=1options kvm-intel enable_shadow_vmcs=1options kvm-intel enable_apicv=1options kvm-intel ept=1

Save & exit the file

[root@kvm-hypervisor ~]# modprobe -r kvm_intel[root@kvm-hypervisor ~]# modprobe -a kvm_intel

Now verify whether nested virtualization feature enabled or not.

[root@kvm-hypervisor ~]# cat /sys/module/kvm_intel/parameters/nestedY

host should enable Vt-d and "iommu=pt intel_iommu=on" in kernel cmdline.

To enabled L2 guest to use the PCI passthrough, need to configure the L1 guest as below:

vmx is need for nested virtualization(L1 guest should use 'host-model' or 'host-passthrough' cpu or have the vmx as required);

The guest vIOMMU is a general device in QEMU. Currently only Q35 platform supports guest vIOMMU;

intremap=[on|off] shows whether the guest vIOMMU will support interrupt remapping. To fully enable vIOMMU functionality, we need to provide intremap=on here. Currently, interrupt remapping does not support full kernel irqchip, only "split" and "off" are supported, It depends on <ioapic driver='qemu'/>;

Most of the full emulated devices (like e1000 mentioned above) should be able to work seamlessly now with Intel vIOMMU. However there are some special devices that need extra cares. These devices are:
Assigned devices (like, vfio-pci)
Virtio devices (like, virtio-net-pci)

caching-mode=on is required when we have assigned devices with the intel-iommu device. The above example assigned the host PCI device 02:00.0 to the guest;

They will make qemu cmdline like this:
......kernel_irqchip=split .... -device intel-iommu,intremap=on,caching-mode=on

virtio devices need "iommu_platform=on,ats=on" defined in device like memballoon device as above. And "device-iotlb=on" in the iommu device;

And on L1 guest, enable "iommu=pt intel_iommu=on" in kernel cmdline.

# vim /etc/default/grub  (apend "intel_iommu=on" to GRUB_CMDLINE_LINUX)

if you use seabios:

# grub2-mkconfig -o /boot/grub2//grub.cfg

if you use OVMF:

# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Reboot the L1 guest, then on L1 guest check if the env is ok:

1). the kvm device is there, otherwise, check the 'enable nested virtualization' step(step 1)

# ls -al /dev/kvmcrw-rw-rw-. 1 root kvm 10, 232 Jul  3 14:30 /dev/kvm

# lscpuArchitecture:          x86_64CPU op-mode(s):        32-bit, 64-bitByte Order:            Little EndianCPU(s):                10On-line CPU(s) list:   0-9Thread(s) per core:    1Core(s) per socket:    1Socket(s):             10NUMA node(s):          1Vendor ID:             GenuineIntelCPU family:            6Model:                 63Model name:            Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHzStepping:              2CPU MHz:               2397.222BogoMIPS:              4794.44Virtualization:        VT-xHypervisor vendor:     KVMVirtualization type:   full......

2). Checkpoint for vIOMMU enable

# dmesg  | grep -i DMAR[    0.000000] ACPI: DMAR 0x000000007FFE2541 000048 (v01 BOCHS  BXPCDMAR 00000001 BXPC 00000001)[    0.000000] DMAR: IOMMU enabled[    0.203737] DMAR: Host address width 39[    0.203739] DMAR: DRHD base: 0x000000fed90000 flags: 0x1[    0.203776] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 12008c22260206 ecap f02[    2.910862] DMAR: No RMRR found[    2.910863] DMAR: No ATSR found[    2.914870] DMAR: dmar0: Using Queued invalidation[    2.914924] DMAR: Setting RMRR:[    2.914926] DMAR: Prepare 0-16MiB unity mapping for LPC[    2.915039] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff][    2.915140] DMAR: Intel(R) Virtualization Technology for Directed I/O

Make sure the "DMAR: Intel(R) Virtualization Technology for Directed I/O" is there – if that’s missing something went wrong – don’t be mislead by the earlier “DMAR: IOMMU enabled” line which merely says the kernel saw the “intel_iommu=on” command line option.

3). The IOMMU should also have registered the PCI devices into various groups

# dmesg  | grep -i iommu  |grep device[    2.915212] iommu: Adding device 0000:00:00.0 to group 0[    2.915226] iommu: Adding device 0000:00:01.0 to group 1...snip...[    5.588723] iommu: Adding device 0000:b5:00.0 to group 14[    5.588737] iommu: Adding device 0000:b6:00.0 to group 15[    5.588751] iommu: Adding device 0000:b7:00.0 to group 16

Now you can assgin the 3 interfaces to L2 guest.

Above steps expected to works well, but in fact, some devices share the same iommu group. How to make the devices into separated iommu group?

Reference:

转载于:https://blog.51cto.com/11527071/2135675

你可能感兴趣的文章