Difference between revisions of "KVM"

From Michael's Information Zone
Jump to navigation Jump to search
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
==Fedora 28==
+
==Installation==
 +
==Debian==
 +
<ref>https://wiki.debian.org/KVM</ref>
 +
<pre>
 +
apt install qemu-kvm libvirt-clients libvirt-daemon-system
 +
</pre>
 +
===Fedora 28===
 
I had removed zfs-fuse, which also removed a number of kvm modules that I needed. To re-install I was able to follow the following link <ref>https://unix.stackexchange.com/questions/195948/kvm-virtual-manager-connection-failed</ref>
 
I had removed zfs-fuse, which also removed a number of kvm modules that I needed. To re-install I was able to follow the following link <ref>https://unix.stackexchange.com/questions/195948/kvm-virtual-manager-connection-failed</ref>
 
<pre>
 
<pre>
Line 6: Line 12:
 
sudo systemctl start libvirtd
 
sudo systemctl start libvirtd
 
sudo dnf install qemu-kvm qemu-img libvirt-python python-virtinst libvirt-client virt-install virt-viewer device-mapper-libs libvirt-daemon-driver-qemu libvirt-daemon-config-network libvirt-daemon-kvm
 
sudo dnf install qemu-kvm qemu-img libvirt-python python-virtinst libvirt-client virt-install virt-viewer device-mapper-libs libvirt-daemon-driver-qemu libvirt-daemon-config-network libvirt-daemon-kvm
 +
</pre>
 +
==Create Storage Domain==
 +
I have a ZFS pool already established, with several volumes in use. I will be creating a new volume dedicated to this purpose and will set it as one of two pools. The other being the ISO directory where I keep ISOs.
 +
<pre>
 +
virsh # pool-list
 +
Name                State      Autostart
 +
-------------------------------------------
 +
default              active    yes             
 +
michael              active    yes     
 +
qemu                active    yes     
 +
 +
virsh # pool-autostart --pool default --disable
 +
Pool default unmarked as autostarted
 +
 +
virsh # pool-undefine --pool default
 +
Pool default undefined
 +
 +
virsh # pool-autostart --pool michael --disable
 +
Pool michael unmarked as autostarted
 +
 +
 +
virsh # pool-list
 +
Name                State      Autostart
 +
-------------------------------------------
 +
default              active    no       
 +
localstorage        active    no       
 +
michael              active    no       
 +
qemu                active    no       
 +
 +
</pre>
 +
Now I can create the new pool. I am going to use the most basic config options.
 +
<pre>
 +
virsh # pool-create-as default --type dir --target /raid5/libvirt
 +
Pool default created
 +
 +
virsh # pool-autostart --pool default
 +
Pool default marked as autostarted
 +
 +
virsh # pool-list
 +
Name                State      Autostart
 +
-------------------------------------------
 +
default              active    yes             
 +
michael              active    no       
 +
qemu                active    no       
 +
 +
virsh #
 +
</pre>
 +
 +
==Networking==
 +
==Remove Default Network==
 +
To remove the default network from the host. This can conflict with other services running if this is a shared environment. In my case I run a DNS server off the host, and the default network uses DNS forwarding with dnsmasq. When dnsmasq is running it takes port 53 which prevents my DNS service from running.
 +
 +
<ref>https://libvirt.org/sources/virshcmdref/html/sect-net-destroy.html</ref><pre>
 +
[root@nas ~]# virsh
 +
Welcome to virsh, the virtualization interactive terminal.
 +
 +
Type:  'help' for help with commands
 +
      'quit' to quit
 +
 +
virsh # net-destroy default
 +
Network default destroyed
 +
 +
virsh # net-list
 +
Name                State      Autostart    Persistent
 +
----------------------------------------------------------
 +
 +
virsh # exit
 +
</pre>
 +
===Setup Bridged Network===
 +
I already had a bridge created for LXC containers, it only made sense to use it for KVM guests as well.
 +
*Create the XML file<ref>https://libvirt.org/formatnetwork.html#examplesBridge</ref>. My existing bridge name is virb0.
 +
<pre>
 +
<network>
 +
  <name>bridge1</name>
 +
  <bridge name="virbr0" />
 +
  <forward mode="bridge" />
 +
</network>
 +
</pre>
 +
*Create the network using virsh
 +
<pre>
 +
virsh # net-list
 +
Name                State      Autostart    Persistent
 +
----------------------------------------------------------
 +
 +
vvirsh # net-define --file br.xml
 +
Network bridge1 defined from br.xml
 +
 +
virsh # net-list
 +
Name                State      Autostart    Persistent
 +
----------------------------------------------------------
 +
 +
virsh # net-start --network bridge1
 +
Network bridge1 started
 +
 +
virsh # net-list
 +
Name                State      Autostart    Persistent
 +
----------------------------------------------------------
 +
bridge1              active    no            yes
 +
 +
</pre>
 +
*Set to auto start
 +
<pre>
 +
virsh # net-autostart --network bridge1
 +
Network bridge1 marked as autostarted
 +
</pre>
 +
==Nested Virtualization==
 +
<ref>https://www.server-world.info/en/note?os=Debian_9&p=kvm&f=8</ref>
 +
 +
==PCI Passthrough==
 +
<ref>https://www.ovirt.org/documentation/install-guide/appe-Configuring_a_Host_for_PCI_Passthrough.html</ref>
 +
*Add intel_iommu=on to /etc/default/grub on the GRUB_CMDLINE_LINUX line.
 +
*Rebuild Grub with grub2-mkconfig -o /boot/grub2/grub.cfg
 +
*Reboot
 +
*Blacklist device for passthrough. First grab the IDs<ref>https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Isolating_the_GPU</ref>
 +
<pre>
 +
lspci -vmmnn
 +
...
 +
Slot: 01:00.0
 +
Class: VGA compatible controller [0300]
 +
Vendor: Advanced Micro Devices, Inc. [AMD/ATI] [1002]
 +
Device: Caicos XT [Radeon HD 7470/8470 / R5 235/310 OEM] [6778]
 +
SVendor: Dell [1028]
 +
SDevice: Radeon HD 7470 [2120]
 +
 +
Slot: 01:00.1
 +
Class: Audio device [0403]
 +
Vendor: Advanced Micro Devices, Inc. [AMD/ATI] [1002]
 +
Device: Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] [aa98]
 +
SVendor: Dell [1028]
 +
SDevice: Device [aa98]
 +
...
 +
</pre>
 +
*In this case we want 1002:6778 and 1002:aa98. Add these to '/etc/modprobe.d/vfio.conf' as follows.<ref>https://www.server-world.info/en/note?os=CentOS_8&p=kvm&f=12</ref>
 +
<pre>
 +
options vfio-pci ids=1002:6778,1002:aa98
 +
</pre>
 +
*Enable the module
 +
<pre>
 +
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
 +
</pre>
 +
 +
==SELinux==
 +
===Image Files===
 +
====New process====
 +
My previous notes are not great, working on cleaner notes.<br>
 +
*To get things going for an existing image that started to not work anymore<ref>https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/ch07s02</ref>
 +
<pre>
 +
[root@natasha ~]# ls -alZ /data/libvirt/images/unifi2.qcow2
 +
-rw-------+ 1 qemu qemu system_u:object_r:virt_image_t:s0 53695545344 Feb 21 08:41 unifi2.qcow2
 +
 +
[root@natasha ~]# chcon -t svirt_image_t /data/libvirt/images/unifi2.qcow2
 +
 +
[root@natasha ~]# ls -alZ /data/libvirt/images/unifi2.qcow2
 +
-rw-------+ 1 qemu qemu system_u:object_r:svirt_image_t:s0 53695545344 Feb 21 08:50 /data/libvirt/images/unifi2.qcow2
 +
 +
[root@natasha ~]# semanage fcontext -d "/data/libvirt/images(/.*)?"
 +
 +
[root@natasha ~]# semanage fcontext -a -t svirt_image_t "/data/libvirt/images(/.*)?"
 +
 +
</pre>
 +
 +
====Old process====
 +
After a server rebuild I wanted the disk images to be placed in another directory. SELinux would NOT work for me, even after setting the context based on Red Hat's documentation<ref>https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-single/virtualization_security_guide/index</ref>. Found an obscure posting<ref>https://unix.stackexchange.com/questions/60799/selinux-interfering-with-host-guest-file-sharing-using-kvm</ref> after an hour of searching that got me running. The only difference is svirt_image_t vs virt_image_t.<br>
 +
UPDATE : This appears to not make a difference. I am still unable to load the images. The svirt is actually just the dynamic label that gets applied after the image is started.<ref>http://selinuxproject.org/page/NB_VM</ref>
 +
<br>
 +
<pre>
 +
semanage fcontext -a -t virt_image_t "/data/libvirt/images(/.*)?"
 +
restorecon -vR /data/libvirt/images
 +
</pre>
 +
<br>
 +
At one point I needed to remove bad context.<ref>https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-selinux_contexts_labeling_files-persistent_changes_semanage_fcontext</ref> The policy is kept in /etc/selinux/targeted/contexts/files/file_contexts.local but you can't edit this directly.
 +
<pre>
 +
semanage fcontext -d "/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso"
 +
</pre>
 +
What is really annoying is that audit logs were not reporting any violations when using the troubleshooter, but I was getting the following
 +
<pre>
 +
type=VIRT_CONTROL msg=audit(1576848063.439:6601): pid=1265 uid=0 auid=4294967295 ses=4294967295 \
 +
subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="Unifi" uuid=37eed7bf-a37f-4d49-86c2-b9a6bb8682c3 \
 +
vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="unset"
 
</pre>
 
</pre>

Latest revision as of 08:55, 21 February 2020

Installation

Debian

[1] 

apt install qemu-kvm libvirt-clients libvirt-daemon-system

Fedora 28

I had removed zfs-fuse, which also removed a number of kvm modules that I needed. To re-install I was able to follow the following link [2] 

sudo dnf install libvirt-daemon
sudo systemctl enable libvirtd
sudo systemctl start libvirtd
sudo dnf install qemu-kvm qemu-img libvirt-python python-virtinst libvirt-client virt-install virt-viewer device-mapper-libs libvirt-daemon-driver-qemu libvirt-daemon-config-network libvirt-daemon-kvm

Create Storage Domain

I have a ZFS pool already established, with several volumes in use. I will be creating a new volume dedicated to this purpose and will set it as one of two pools. The other being the ISO directory where I keep ISOs. 

virsh # pool-list
 Name                 State      Autostart 
-------------------------------------------
 default              active     yes              
 michael              active     yes       
 qemu                 active     yes       

virsh # pool-autostart --pool default --disable 
Pool default unmarked as autostarted

virsh # pool-undefine --pool default 
Pool default undefined

virsh # pool-autostart --pool michael --disable 
Pool michael unmarked as autostarted


virsh # pool-list 
 Name                 State      Autostart 
-------------------------------------------
 default              active     no        
 localstorage         active     no        
 michael              active     no        
 qemu                 active     no

Now I can create the new pool. I am going to use the most basic config options. 

virsh # pool-create-as default --type dir --target /raid5/libvirt
Pool default created

virsh # pool-autostart --pool default
Pool default marked as autostarted

virsh # pool-list
 Name                 State      Autostart 
-------------------------------------------
 default              active     yes               
 michael              active     no        
 qemu                 active     no        

virsh #

Networking

Remove Default Network

To remove the default network from the host. This can conflict with other services running if this is a shared environment. In my case I run a DNS server off the host, and the default network uses DNS forwarding with dnsmasq. When dnsmasq is running it takes port 53 which prevents my DNS service from running.

[3]

[root@nas ~]# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # net-destroy default
Network default destroyed

virsh # net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------

virsh # exit

Setup Bridged Network

I already had a bridge created for LXC containers, it only made sense to use it for KVM guests as well.

  • Create the XML file[4]. My existing bridge name is virb0.
<network>
  <name>bridge1</name>
  <bridge name="virbr0" />
  <forward mode="bridge" />
</network>
  • Create the network using virsh
virsh # net-list 
 Name                 State      Autostart     Persistent
----------------------------------------------------------

vvirsh # net-define --file br.xml 
Network bridge1 defined from br.xml

virsh # net-list 
 Name                 State      Autostart     Persistent
----------------------------------------------------------

virsh # net-start --network bridge1
Network bridge1 started

virsh # net-list 
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 bridge1              active     no            yes
  • Set to auto start
virsh # net-autostart --network bridge1
Network bridge1 marked as autostarted

Nested Virtualization

[5]

PCI Passthrough

[6]

  • Add intel_iommu=on to /etc/default/grub on the GRUB_CMDLINE_LINUX line.
  • Rebuild Grub with grub2-mkconfig -o /boot/grub2/grub.cfg
  • Reboot
  • Blacklist device for passthrough. First grab the IDs[7]
lspci -vmmnn
...
Slot:	01:00.0
Class:	VGA compatible controller [0300]
Vendor:	Advanced Micro Devices, Inc. [AMD/ATI] [1002]
Device:	Caicos XT [Radeon HD 7470/8470 / R5 235/310 OEM] [6778]
SVendor:	Dell [1028]
SDevice:	Radeon HD 7470 [2120]

Slot:	01:00.1
Class:	Audio device [0403]
Vendor:	Advanced Micro Devices, Inc. [AMD/ATI] [1002]
Device:	Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] [aa98]
SVendor:	Dell [1028]
SDevice:	Device [aa98]
...
  • In this case we want 1002:6778 and 1002:aa98. Add these to '/etc/modprobe.d/vfio.conf' as follows.[8]
options vfio-pci ids=1002:6778,1002:aa98
  • Enable the module
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

SELinux

Image Files

New process

My previous notes are not great, working on cleaner notes.

  • To get things going for an existing image that started to not work anymore[9]
[root@natasha ~]# ls -alZ /data/libvirt/images/unifi2.qcow2
-rw-------+ 1 qemu qemu system_u:object_r:virt_image_t:s0 53695545344 Feb 21 08:41 unifi2.qcow2

[root@natasha ~]# chcon -t svirt_image_t /data/libvirt/images/unifi2.qcow2

[root@natasha ~]# ls -alZ /data/libvirt/images/unifi2.qcow2 
-rw-------+ 1 qemu qemu system_u:object_r:svirt_image_t:s0 53695545344 Feb 21 08:50 /data/libvirt/images/unifi2.qcow2

[root@natasha ~]# semanage fcontext -d "/data/libvirt/images(/.*)?"

[root@natasha ~]# semanage fcontext -a -t svirt_image_t "/data/libvirt/images(/.*)?"

Old process

After a server rebuild I wanted the disk images to be placed in another directory. SELinux would NOT work for me, even after setting the context based on Red Hat's documentation[10]. Found an obscure posting[11] after an hour of searching that got me running. The only difference is svirt_image_t vs virt_image_t.
UPDATE : This appears to not make a difference. I am still unable to load the images. The svirt is actually just the dynamic label that gets applied after the image is started.[12] 

semanage fcontext -a -t virt_image_t "/data/libvirt/images(/.*)?"
restorecon -vR /data/libvirt/images


At one point I needed to remove bad context.[13] The policy is kept in /etc/selinux/targeted/contexts/files/file_contexts.local but you can't edit this directly. 

semanage fcontext -d "/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso"

What is really annoying is that audit logs were not reporting any violations when using the troubleshooter, but I was getting the following 

type=VIRT_CONTROL msg=audit(1576848063.439:6601): pid=1265 uid=0 auid=4294967295 ses=4294967295 \
subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="Unifi" uuid=37eed7bf-a37f-4d49-86c2-b9a6bb8682c3 \
vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="unset"
  1. ↑ https://wiki.debian.org/KVM
  2. ↑ https://unix.stackexchange.com/questions/195948/kvm-virtual-manager-connection-failed
  3. ↑ https://libvirt.org/sources/virshcmdref/html/sect-net-destroy.html
  4. ↑ https://libvirt.org/formatnetwork.html#examplesBridge
  5. ↑ https://www.server-world.info/en/note?os=Debian_9&p=kvm&f=8
  6. ↑ https://www.ovirt.org/documentation/install-guide/appe-Configuring_a_Host_for_PCI_Passthrough.html
  7. ↑ https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Isolating_the_GPU
  8. ↑ https://www.server-world.info/en/note?os=CentOS_8&p=kvm&f=12
  9. ↑ https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/ch07s02
  10. ↑ https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-single/virtualization_security_guide/index
  11. ↑ https://unix.stackexchange.com/questions/60799/selinux-interfering-with-host-guest-file-sharing-using-kvm
  12. ↑ http://selinuxproject.org/page/NB_VM
  13. ↑ https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-selinux_contexts_labeling_files-persistent_changes_semanage_fcontext
Retrieved from "https://wiki.1701technology.com/index.php?title=KVM&oldid=1400"