Nested esxi in kvm

Purpose

Purchased an refurbished Dell server with 24 drives, 20 physical cores, and ~386GB RAM.The purpose for this server has changed many times during it's time in the lab, but now it will be a storage backend for both VMware and KVM hypervisors. ZFS will be used as the filsesystem with dedup and compression enabled. Obviously the server is overkill for such a use case. So I would like to run the following

• LXD Containers for my many CentOS 7 based servers. Makes sense to use fast kernel based "virtualization" since they run the same kernel.
• KVM machines that are essentially dedicated to the site the server is hosted in.
• VMWare esxi as a nested hypervisor. This is to maintain compatibility with the existing data center.

Ultimately this will act as a replica and DR site.

Prepare Hardware

If you happen to have used the Dell PERC storage controller and wanted something ZFS friendly, please wipe the meta data from the drives. Otherwise CentOS 7 will not install with an error during Anaconda initialization.

• Boot from rescue media and perform the following on all drives.

dd if=/dev/zero of=$YOUR_DEV bs=512 seek=$(( $(blockdev --getsz $YOUR_DEV) - 1024 )) count=1024

I do not have the orignal link to reference. But this would have saved me a lot of time as I was using shred on all the drives instead.
At this point I had installed the "IT Mode LSI SAS 9207-8i SATA/SAS 6Gb/s PCI-E 3.0 Host Bus Adapter LSI00301" I bought off ebay, which gave me direct disk access to all drives. ZFS is happy.

Start Installing Software

Enable epel-release and update

yum -y install epel-release
yum -y update

Install ZFS

yum install http://download.zfsonlinux.org/epel/zfs-release.el7_5.noarch.rpm
vi /etc/yum.repos.d/zfs.repo
...
Disable ZFS, enable ZFS-KMOD.
...

install zfs
modprobe zfs

Install related encryption packages

yum -y install cryptsetup

Encrypt the drives using luks, then create ZFS pool.

Install Virtualization Packages

yum install -y qemu-kvm qemu-img libvirt-python python-virtinst libvirt-client virt-install virt-viewer device-mapper-libs libvirt-daemon-driver-qemu libvirt-daemon-config-network libvirt-daemon-kvm OVMF

Now we can enable nested virtualization

echo "options kvm_intel nested=1" >> /etc/modprobe.d/kvm-nested.conf
echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm-nested.conf
modprobe -r kvm_intel
modprobe kvm_intel

You should now see the nested module enabled. At this point we can install the updated qemu packages as mentioned here

yum -y install centos-release-qemu-ev
sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
systemctl restart libvirtd

esxi

At this point we install esxi as a VM.

virt-install \
--name esxi \
--ram 160000 \
--disk path=/tank/images/esxi.qcow2,bus=sata \
--cpu host-passthrough \
--vcpus=10 \
--os-type linux \
--os-variant=virtio26 \
--network bridge=br3,model=e1000 \
--graphics spice \
--video qxl \
--cdrom /tank/iso/VMware-VMvisor-Installer-6.7.0.update01-10302608.x86_64.iso \
--features kvm_hidden=on \
--machine q35