How does QEMU use KVM? Explain the ioctl sequence and the run loop.
Short answer: QEMU opens /dev/kvm, creates a VM with KVM_CREATE_VM (which returns a VM file descriptor), registers guest RAM with KVM_SET_USER_MEMORY_REGION, creates each vCPU with KVM_CREATE_VCPU, mmaps the vCPU's shared kvm_run structure, initialises the vCPU (on ARM with KVM_ARM_VCPU_INIT) and sets its registers. Each vCPU thread then loops on ioctl(vcpu_fd, KVM_RUN): the guest runs until something needs QEMU, KVM returns with an exit reason in kvm_run, QEMU handles it and calls KVM_RUN again.
int kvm = open("/dev/kvm", O_RDWR);
int vmfd = ioctl(kvm, KVM_CREATE_VM, 0); /* on ARM the arg can set the IPA size */
void *ram = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
struct kvm_userspace_memory_region mr = {
.slot = 0, .guest_phys_addr = 0x40000000, .memory_size = size,
.userspace_addr = (uint64_t)ram,
};
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &mr);
int vcpu = ioctl(vmfd, KVM_CREATE_VCPU, 0);
int run_sz = ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, 0);
struct kvm_run *run = mmap(NULL, run_sz, PROT_READ | PROT_WRITE, MAP_SHARED, vcpu, 0);
struct kvm_vcpu_init init;
ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, &init);
ioctl(vcpu, KVM_ARM_VCPU_INIT, &init);
/* set PC and x0 with KVM_SET_ONE_REG ... */
for (;;) {
ioctl(vcpu, KVM_RUN, 0);
switch (run->exit_reason) {
case KVM_EXIT_MMIO: /* run->mmio.phys_addr, .data, .len, .is_write */
handle_mmio(run); break;
case KVM_EXIT_SYSTEM_EVENT: /* PSCI SYSTEM_OFF / RESET */
return;
case KVM_EXIT_INTR: /* a signal interrupted KVM_RUN */
break;
default:
abort();
}
}
In QEMU: kvm_init() (accel/kvm/kvm-all.c) creates the VM; kvm_init_vcpu() creates vCPUs; kvm_vcpu_thread_fn() calls kvm_cpu_exec(), which contains the KVM_RUN loop and dispatches exits to address_space_rw() for MMIO (or to port I/O on x86).
Inside the kernel, one KVM_RUN: load the guest's registers and stage-2 page table base, enable traps, ERET into the guest at EL1. On a trap, save guest state, restore host state, and either handle it in the kernel and re-enter (most exits) or return to user space.
Corrections to common notes
- There are no
KVM_DESTROY_VMorKVM_DESTROY_VCPUioctls. A VM and its vCPUs are destroyed when their file descriptors are closed (when QEMU exits). - On arm64 KVM is built into the kernel image (
CONFIG_KVM); there is no separatekvm-arm.ko. On x86 it iskvm.kopluskvm-intel.koorkvm-amd.ko. - Guest system calls do not cause VM exits:
SVCgoes to the guest kernel at EL1. Exits come from MMIO, trapped instructions and registers,WFI,HVC/SMC(PSCI), and interrupts.