Nested esxi in kvm
Purchased an refurbished Dell server with 24 drives, 20 physical cores, and ~386GB RAM.The purpose for this server has changed many times during it's time in the lab, but now it will be a storage backend for both VMware and KVM hypervisors. ZFS will be used as the filsesystem with dedup and compression enabled. Obviously the server is overkill for such a use case. So I would like to run the following
- LXD Containers for my many CentOS 7 based servers. Makes sense to use fast kernel based "virtualization" since they run the same kernel.
- KVM machines that are essentially dedicated to the site the server is hosted in.
- VMWare esxi as a nested hypervisor. This is to maintain compatibility with the existing data center.
Ultimately this will act as a replica and DR site.
If you happen to have used the Dell PERC storage controller and wanted something ZFS friendly, please wipe the meta data from the drives. Otherwise CentOS 7 will not install with an error during Anaconda initialization.
- Boot from rescue media and perform the following on all drives.
dd if=/dev/zero of=$YOUR_DEV bs=512 seek=$(( $(blockdev --getsz $YOUR_DEV) - 1024 )) count=1024
I do not have the orignal link to reference. But this would have saved me a lot of time as I was using shred on all the drives instead.
At this point I had installed the "IT Mode LSI SAS 9207-8i SATA/SAS 6Gb/s PCI-E 3.0 Host Bus Adapter LSI00301" I bought off ebay, which gave me direct disk access to all drives. ZFS is happy.
Start Installing Software
Enable epel-release and update
yum -y install epel-release yum -y update
Install Virtualization Packages
yum install -y qemu-kvm qemu-img libvirt-python python-virtinst libvirt-client virt-install virt-viewer device-mapper-libs libvirt-daemon-driver-qemu libvirt-daemon-config-network libvirt-daemon-kvm OVMF
echo -e "options kvm_intel nested=1 \ enable_shadow_vmcs=1 \ enable_apicv=1 \ ept=1\n \ options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm-nested.conf modprobe -r kvm_intel modprobe kvm_intel
You should now see the nested module enabled. At this point we can install the updated qemu packages as mentioned here 
yum -y install centos-release-qemu-ev sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev systemctl restart libvirtd
At this point we install esxi as a VM.
virt-install \ --name esxi \ --ram 160000 \ --disk path=/tank/images/esxi.qcow2,bus=sata \ --cpu host-passthrough \ --vcpus=10 \ --os-type linux \ --os-variant=virtio26 \ --network bridge=br3,model=e1000 \ --graphics spice \ --video qxl \ --cdrom /tank/iso/VMware-VMvisor-Installer-6.7.0.update01-10302608.x86_64.iso \ --features kvm_hidden=on \ --machine q35
Then edit the config
virsh edit esxi <features> . . <kvm> <hidden state='on'/> </kvm> </features> . . . <cpu mode='custom' match='exact'> . . <feature policy='require' name='vmx'/> </cpu>