NetBSD/src branch: netbsd-9 commit: ab5d3c2cb5f7a1301959e764a7c2be5006cc5805 (HEAD -> netbsd-9, origin/netbsd-9) author: martin date: Fri Apr 30 14:08:16 2021 +0000 $ git log -- sys/dev/nvmm lib/libnvmm commit cf95f272cacaa51f77f4c64b3f0afc895cb61b04 Author: martin Date: Sun Sep 13 11:56:44 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1078): sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.73 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.73 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.74 nvmm-x86-vmx: improve the handling of CR4 - Filter out certain features we don't want the guest to enable. This is for general correctness, and future-proofness. - Flush the guest TLB when certain flags change. nvmm-x86: improve the handling of RFLAGS.RF - When injecting certain exceptions, set RF. For us to have an up-to-date view of RFLAGS, we commit the state before the event. - When advancing RIP, clear RF. commit 90ddc7940387194a3fdf5e32960cc5682d1df5a7 Author: martin Date: Sun Sep 13 11:54:10 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1077): sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.68 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.74 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.16 Improve emulation of MSR_IA32_ARCH_CAPABILITIES: publish only the *_NO bits. Initially they were the only ones there, but Intel then added other bits we aren't interested in, and they must be filtered out. nvmm-x86-svm: improve the handling of MSR_EFER Intercept reads of it as well, just to mask EFER_SVME, which the guest doesn't need to see. nvmm-x86: improve the CPUID emulation - Mask DTES64, DS_CPL, CID, SDBG, xTPR, PN. - B10, B20 and IA64 do not exist, so just remove them. commit 7bb3c59f629ba32d954177961a91065e8f06f370 Author: martin Date: Fri Sep 4 18:53:43 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1076): sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.75 sys/arch/x86/include/specialreg.h: revision 1.172 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.72 nvmm-x86-vmx: fix detection of the BIOS lock If it's locked, ensure it's locked with VMX enabled. If it's not locked, then lock it ourselves with VMX enabled. Should fix NetBSD PR/55596. - Add a few more CPUID flags. - nvmm-x86-svm: check the SVM revision Only revision 1 exists, but check it, for future-proofness. commit 632b08c8596a1c7c470ddbb6ae27b824275a7152 Author: martin Date: Sat Aug 29 17:00:28 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1068): sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.71 sys/dev/nvmm/nvmm.c: revision 1.34 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.72 sys/dev/nvmm/nvmm.c: revision 1.35 sys/dev/nvmm/nvmm.c: revision 1.36 sys/dev/nvmm/x86/nvmm_x86_svmfunc.S: revision 1.5 sys/dev/nvmm/nvmm.c: revision 1.37 sys/dev/nvmm/x86/nvmm_x86_vmxfunc.S: revision 1.5 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.70 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.68 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.15 sys/dev/nvmm/nvmm_ioctl.h: revision 1.10 Micro-optimize: use pushq instead of pushw. To avoid LCP stalls and unaligned stack accesses. nvmm-x86: also flush the guest TLB when CR4.{PCIDE,SMEP} changes nvmm: localify a variable that doesn't need to be global nvmm: use relaxed atomics to read nmachines nvmm-x86-svm: dedup code nvmm-x86: hide more CPUID flags, mostly related to perf monitors nvmm: misc improvements - use mach->ncpus to get the number of vcpus, now that we have it - don't forget to decrement mach->ncpus when a machine gets killed - add more __predict_false() nvmm-x86-svm: don't forget to intercept INVD INVD executed in the guest can be dangerous for the host, due to CPU caches being flushed without write-back. nvmm: slightly clarify nvmm: explicitly include atomic.h commit 0ef848ac2ffc686519c5d5ca9c8c9af194010796 Author: martin Date: Wed Aug 26 17:55:48 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1058): sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.70 sys/dev/nvmm/x86/nvmm_x86.h: revision 1.19 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.69 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.71 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.69 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.11 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.12 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.13 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.14 Improve the CPUID emulation: - Hide SGX*, PKU, WAITPKG, and SKINIT, because they are not supported. - Hide HLE and RTM, part of TSX. Because TSX is just too buggy and we cannot guarantee that it remains enabled in the guest (if for example the host disables TSX while the guest is running). Nobody wants this crap anyway, so bye-bye. - Advertise FSREP_MOV, because no reason to hide it. Hide OSPKE. NFC since the host never uses PKU, but still. Improve the CPUID emulation on nvmm-intel: - Limit the highest extended leaf. - Limit 0x00000007 to ECX=0, for future-proofness. nvmm-x86-svm: improve the CPUID emulation Limit the hypervisor range, and properly handle each basic leaf until 0xD. nvmm-x86: advertise the SERIALIZE instruction, available on future CPUs nvmm-x86: improve the CPUID emulation - x86-svm: explicitly handle 0x80000007 and 0x80000008. The latter contains extended features we must filter out. Apply the same in x86-vmx for symmetry. - x86-svm: explicitly handle extended leaves until 0x8000001F, and truncate to it. commit 3274d19ae23dbb53c5684ccf9f8b88e4143decf4 Author: martin Date: Tue Aug 18 09:29:52 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1055): sys/dev/nvmm/nvmm.h: revision 1.13 sys/dev/nvmm/nvmm.h: revision 1.14 sys/dev/nvmm/nvmm.c: revision 1.33 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.67 sys/dev/nvmm/nvmm_internal.h: revision 1.17 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.67 sys/dev/nvmm/x86/nvmm_x86.c: revision 1.10 Put the few x86-specific structures under #ifdef __x86_64__, for clarity. Make it easier to understand what's going on, no functional change. Add new field definitions. Add new field definitions, and intercept everything, for future-proofness. Add CTASSERT. commit f225e41cf82800ca39ff1c455b3d5fb2e276216c Author: martin Date: Wed Aug 5 15:18:24 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1041): sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.66 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.50 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.66 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.46 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.49 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.55 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.56 pg->phys_addr > VM_PAGE_TO_PHYS(pg) Explicitly cast pointers to uintptr_t before casting to enums. They are not necessarily the same size. Don't cast pointers to bool, check for NULL instead. vmx_vmptrst(): only used when DIAGNOSTIC Simplify, remove unnecessary #ifdef DIAGNOSTIC around KASSERTs. Use ULL, to make it clear we are unsigned. commit f32847cbde359971e17a95a19789c9dc06f41c50 Author: martin Date: Sun Aug 2 11:19:09 2020 +0000 Open code preempt_needed() which is not available on this branch. Fixes build fallout from ticket #1032. commit 3c83b743258e5ce317ad29b0c1b4c5b3acc2cb5d Author: martin Date: Sun Aug 2 08:49:08 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #1032): sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.60 (patch) sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.61 (patch) sys/dev/nvmm/nvmm.c: revision 1.30 sys/dev/nvmm/nvmm.c: revision 1.31 sys/dev/nvmm/nvmm.c: revision 1.32 sys/dev/nvmm/nvmm_internal.h: revision 1.15 sys/dev/nvmm/nvmm_internal.h: revision 1.16 sys/dev/nvmm/files.nvmm: revision 1.3 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.62 (patch) sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.63 (patch) sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.59 (patch) sys/modules/nvmm/nvmm.ioconf: revision 1.2 Gather the conditions to return from the VCPU loops in nvmm_return_needed(), and use it in nvmm_do_vcpu_run() as well. This fixes two undesired behaviors: - When a VM initializes, the many nested page faults that need processing could cause the calling thread to occupy the CPU too much if we're unlucky and are only getting repeated nested page faults thousands of times in a row. - When the emulator calls nvmm_vcpu_run() and immediately sends a signal to stop the VCPU, it's better to check signals earlier and leave right away, rather than doing a round of VCPU run that could increase the time spent by the emulator waiting for the return. style Register NVMM as an actual pseudo-device. Without PMF handler, to explicitly disallow ACPI suspend if NVMM is running. Should fix PR/55406. Print the backend name when attaching. commit 361ca4783320d504e33f437bcfb5cadee814bd4e Author: martin Date: Thu May 21 10:52:58 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #919): sys/dev/nvmm/x86/nvmm_x86.c: revision 1.9 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.60 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.61 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.56 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.57 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.58 sys/dev/nvmm/nvmm.c: revision 1.29 Improve the CPUID emulation of basic leaves: - Hide DCA and PQM, they cannot be used in guests. - On Intel, explicitly handle each basic leaf until 0x16. - On AMD, explicitly handle each basic leaf until 0x0D. Respect the convention for the hypervisor information: return the highest hypervisor leaf in 0x40000000.EAX. Improve the CPUID emulation on nvmm-intel: limit the highest basic and hypervisor leaves. Complete rev1.26: reset nvmm_impl to NULL in nvmm_fini(). commit 6d68414119a6936b0401e343203b5d1402397be3 Author: martin Date: Wed May 13 12:21:56 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #898): sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.59 sys/dev/nvmm/nvmm_internal.h: revision 1.14 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.53 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.54 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.55 sys/dev/nvmm/nvmm.c: revision 1.27 sys/dev/nvmm/nvmm.c: revision 1.28 When the identification fails, print the reason. If we were processing a software int/excp, and got a VMEXIT in the middle, we must also reflect the instruction length, otherwise the next VMENTER fails and Qemu shuts the guest down. On Intel CPUs, CPUID leaf 0xB, too, provides topology information, so filter it correctly, to avoid inconsistencies if the host has SMT. This fixes HaikuOS which fetches SMT information from there and would panic because of the inconsistencies. commit c9394b4073140c64d3e5d8ac6e4675d88b848cd3 Author: martin Date: Mon Apr 27 14:56:22 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #857): sys/dev/nvmm/nvmm.c: revision 1.26 In nvmm_open(), make sure an implementation was found. This fixes an initialization bug triggerable in certain conditions. If you build nvmm inside the kernel, AND have a cpu that is not supported, AND run nvmmctl (or qemu-nvmm, both being the only binaries in the "nvmm" group), you get a page fault. This is because when nvmm is built inside the kernel, the kernel registers nvmm_cdevsw behind nvmm's back. The ioctl is therefore always accessible, and will hit NULL pointers if nvmm_init() failed. Problem reported by Andrei M. on netbsd-users@, thanks. commit fd35fb68fc89492bd415b9b5f3a0a9f56dc4cc53 Author: martin Date: Mon Feb 10 19:05:05 2020 +0000 Pull up following revision(s) (requested by maxv in ticket #688): share/man/man4/nvmm.4: revision 1.5 lib/libnvmm/libnvmm.3: revision 1.26 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.47 Mmh, as noted in PR/54847, this should be uint64_t, not uint16_t. Harmless because we use only the two lowest bits anyway. I believe this could be caught by KUBSAN; time to do another round of NVMM+K_SAN testing. Reference nvmmctl(8). commit 4d0edef5f2d14903ab71f3bc019364f33d4979f8 Author: martin Date: Mon Nov 25 16:39:29 2019 +0000 Pull up following revision(s) (requested by maxv in ticket #475): tests/lib/libnvmm/h_mem_assist.c: revision 1.18 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.45 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.54 Hide XSAVES-specific stuff and the masked extended states. Several improvements. In particular, reduce CS.limit, because Intel CPUs perform strict sanity checks, and the previous (too high) limit caused the VM entry to fail. commit d3a89c929b49222e58fe18e1f61e79942b44cc15 Author: martin Date: Sat Nov 16 20:08:45 2019 +0000 Pull up following revision(s) (requested by jmcneill in ticket #434): sys/dev/nvmm/x86/nvmm_x86.c: revision 1.8 Don't report MWAITX by default. commit f64859cc71f19f74dcd907e24e9bcc7854625228 Author: martin Date: Sun Nov 10 12:58:29 2019 +0000 Pull up following revision(s) (requested by maxv in ticket #405): usr.sbin/nvmmctl/nvmmctl.8: revision 1.2 lib/libnvmm/libnvmm.3: revision 1.24 sys/dev/nvmm/nvmm.h: revision 1.11 lib/libnvmm/libnvmm.3: revision 1.25 sys/dev/nvmm/x86/nvmm_x86.h: revision 1.16 sys/dev/nvmm/nvmm.h: revision 1.12 sys/dev/nvmm/x86/nvmm_x86.h: revision 1.17 tests/lib/libnvmm/h_mem_assist.c: revision 1.12 sys/dev/nvmm/x86/nvmm_x86.h: revision 1.18 share/mk/bsd.hostprog.mk: revision 1.82 lib/libnvmm/libnvmm.c: revision 1.15 distrib/sets/lists/base/md.amd64: revision 1.281 tests/lib/libnvmm/h_mem_assist.c: revision 1.13 lib/libnvmm/libnvmm.c: revision 1.16 tests/lib/libnvmm/h_mem_assist.c: revision 1.14 lib/libnvmm/libnvmm_x86.c: revision 1.32 lib/libnvmm/libnvmm.c: revision 1.17 tests/lib/libnvmm/h_mem_assist.c: revision 1.15 lib/libnvmm/libnvmm_x86.c: revision 1.33 lib/libnvmm/libnvmm.c: revision 1.18 usr.sbin/nvmmctl/Makefile: revision 1.1 tests/lib/libnvmm/h_mem_assist_asm.S: revision 1.7 tests/lib/libnvmm/h_mem_assist.c: revision 1.16 lib/libnvmm/libnvmm_x86.c: revision 1.34 usr.sbin/nvmmctl/Makefile: revision 1.2 tests/lib/libnvmm/h_mem_assist_asm.S: revision 1.8 tests/lib/libnvmm/h_mem_assist.c: revision 1.17 sys/dev/nvmm/nvmm_internal.h: revision 1.13 lib/libnvmm/libnvmm_x86.c: revision 1.35 lib/libnvmm/libnvmm_x86.c: revision 1.36 usr.sbin/postinstall/postinstall.in: revision 1.8 lib/libnvmm/libnvmm_x86.c: revision 1.37 lib/libnvmm/libnvmm_x86.c: revision 1.38 lib/libnvmm/libnvmm_x86.c: revision 1.39 usr.sbin/Makefile: revision 1.282 lib/libnvmm/nvmm.h: revision 1.13 lib/libnvmm/nvmm.h: revision 1.14 lib/libnvmm/nvmm.h: revision 1.15 sys/dev/nvmm/nvmm.c: revision 1.23 lib/libnvmm/nvmm.h: revision 1.16 sys/dev/nvmm/nvmm.c: revision 1.24 lib/libnvmm/nvmm.h: revision 1.17 sys/dev/nvmm/nvmm.c: revision 1.25 tests/lib/libnvmm/h_io_assist.c: revision 1.9 etc/MAKEDEV.tmpl: revision 1.209 tests/lib/libnvmm/h_io_assist.c: revision 1.10 tests/lib/libnvmm/h_io_assist.c: revision 1.11 etc/group: revision 1.35 distrib/sets/lists/man/mi: revision 1.1660 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.40 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.41 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.42 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.43 sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.44 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.51 sys/dev/nvmm/nvmm_ioctl.h: revision 1.8 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.52 sys/dev/nvmm/nvmm_ioctl.h: revision 1.9 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.53 usr.sbin/nvmmctl/nvmmctl.c: revision 1.1 lib/libnvmm/libnvmm.3: revision 1.20 distrib/sets/lists/debug/md.amd64: revision 1.106 lib/libnvmm/libnvmm.3: revision 1.21 lib/libnvmm/libnvmm.3: revision 1.22 usr.sbin/nvmmctl/nvmmctl.8: revision 1.1 lib/libnvmm/libnvmm.3: revision 1.23 Fix incorrect parsing: the R/M field uses a special GPR map when the address size is 16 bits, regardless of the actual operating mode. With this special map there can be two registers referenced at once, and also disp16-only. Implement this special behavior, and add associated tests. While here simplify a few things. With this in place, the Windows 95 installer initializes correctly. Part of PR/54611. add missing initializer Implement XCHG, add associated tests, and add comments to explain. With this in place the Windows 95 installer completes successfuly. Part of PR/54611. Improve nvmm_vcpu_dump(). Put back 'default', because llvm apparently doesn't realize that all cases are covered in the switch. Miscellaneous changes in NVMM, to address several inconsistencies and issues in the libnvmm API. - Rename NVMM_CAPABILITY_VERSION to NVMM_KERN_VERSION, and check it in libnvmm. Introduce NVMM_USER_VERSION, for future use. - In libnvmm, open "/dev/nvmm" as read-only and with O_CLOEXEC. This is to avoid sharing the VMs with the children if the process forks. In the NVMM driver, force O_CLOEXEC on open(). - Rename the following things for consistency: nvmm_exit* -> nvmm_vcpu_exit* nvmm_event* -> nvmm_vcpu_event* NVMM_EXIT_* -> NVMM_VCPU_EXIT_* NVMM_EVENT_INTERRUPT_HW -> NVMM_VCPU_EVENT_INTR NVMM_EVENT_EXCEPTION -> NVMM_VCPU_EVENT_EXCP Delete NVMM_EVENT_INTERRUPT_SW, unused already. - Slightly reorganize the MI/MD definitions, for internal clarity. - Split NVMM_VCPU_EXIT_MSR in two: NVMM_VCPU_EXIT_{RD,WR}MSR. Also provide separate u.rdmsr and u.wrmsr fields. This is more consistent with the other exit reasons. - Change the types of several variables: event.type enum -> u_int event.vector uint64_t -> uint8_t exit.u.*msr.msr: uint64_t -> uint32_t exit.u.io.type: enum -> bool exit.u.io.seg: int -> int8_t cap.arch.mxcsr_mask: uint64_t -> uint32_t cap.arch.conf_cpuid_maxops: uint64_t -> uint32_t - Delete NVMM_VCPU_EXIT_MWAIT_COND, it is AMD-only and confusing, and we already intercept 'monitor' so it is never armed. - Introduce vmx_exit_insn() for NVMM-Intel, similar to svm_exit_insn(). The 'npc' field wasn't getting filled properly during certain VMEXITs. - Introduce nvmm_vcpu_configure(). Similar to nvmm_machine_configure(), but as its name indicates, the configuration is per-VCPU and not per-VM. Migrate and rename NVMM_MACH_CONF_X86_CPUID to NVMM_VCPU_CONF_CPUID. This becomes per-VCPU, which makes more sense than per-VM. - Extend the NVMM_VCPU_CONF_CPUID conf to allow triggering VMEXITs on specific leaves. Until now we could only mask the leaves. An uint32_t is added in the structure: uint32_t mask:1; uint32_t exit:1; uint32_t rsvd:30; The two first bits select the desired behavior on the leaf. Specifying zero on both resets the leaf to the default behavior. The new NVMM_VCPU_EXIT_CPUID exit reason is added. Three changes in libnvmm: - Add 'mach' and 'vcpu' backpointers in the nvmm_io and nvmm_mem structures. - Rename 'nvmm_callbacks' to 'nvmm_assist_callbacks'. - Rename and migrate NVMM_MACH_CONF_CALLBACKS to NVMM_VCPU_CONF_CALLBACKS, it now becomes per-VCPU. Update the libnvmm man page: - Sync the naming with reality. - Replace "relevant" by "desired" and "virtualizer" by "emulator", closer to what I meant. - Add a "VCPU Configuration" section. - Add a "Machine Ownership" section. Add the "nvmm" group, and make nvmm_init() public. Sent to tech-kern@ a few days ago. Use the new PTE naming, and define CR3_FRAME_* separately. No functional change. Add a new VCPU conf option, that allows userland to request VMEXITs after a TPR change. This is supported on all Intel CPUs, and not-too-old AMD CPUs. The reason for wanting this option is that certain OSes (like Win10 64bit) manage interrupt priority in hardware via CR8 directly, and for these OSes, the emulator may want to sync its internal TPR state on each change. Add two new fields in cap.arch, to report the conf capabilities. Report TPR only on Intel for now, not AMD, because I don't have a recent AMD CPU on which to test. Mask CPUID leaf 0x0A on Intel, because we don't want the guest to try (and fail) to probe the PMC MSRs. This avoids "Unexpected WRMSR" warnings in qemu-nvmm. Add PCID support in the guests. This speeds up most 64bit guests, because since Meltdown, everybody uses PCID (including NetBSD). Change the way root_owner works: consider the calling process as root_owner not if it has root privileges, but if the /dev/nvmm device was opened with write permissions. Introduce the undocumented nvmm_root_init() function to achieve that. The goal is to simplify the logic and have more granularity, eg if we want a monitoring agent to access VMs but don't want to give this agent real root access on the system. A few changes: - Use smaller types in struct nvmm_capability. - Use smaller type for nvmm_io.port. - Switch exitstate to a compacted structure. Add nram in struct nvmm_ctl_mach_info. Add nvmmctl, with two commands for now. Macro tidyness. Sort SEE ALSO. should be fork(2), noticed by wiz Add debug entry for newly introduced nvmmctl utility. Annotate a covering switch as such to avoid warnings about missing returns. Forgot to put nvmmctl in the "nvmm" group. Add nvmm group. commit 3cccbb9c8b33acd753072c3c66503a75c9c00851 Author: martin Date: Sun Oct 6 11:04:55 2019 +0000 Pull up following revision(s) (requested by maxv in ticket #287): sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.38 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.47 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.48 sys/dev/nvmm/x86/nvmm_x86_svm.c: revision 1.49 Add definitions for RDPRU, MCOMMIT, GMET and VTE. Fix definition for MWAIT. It should be bit 11, not 12; 12 is the armed version. Switch to the new PTE naming. commit 2867c5a559ab084b85ae44a7099110cf9bd08d74 Author: martin Date: Tue Sep 24 18:14:59 2019 +0000 Pull up following revision(s) (requested by maxv in ticket #239): sys/dev/nvmm/x86/nvmm_x86_vmx.c: revision 1.37 Always set hwcode on error. Useful for debugging. commit 8c134110d67792ba7deb0a095c19b8d04dac6c34 Author: maxv Date: Sat Jul 6 05:13:10 2019 +0000 Localify two functions that are no longer used outside. Also return the error from the *_vcpu_run() functions, now that we commit the states in them (which can fail). commit 4febafc81e1fa9d1ed55d506625dd3b40e893987 Author: maxv Date: Sun Jun 16 18:30:31 2019 +0000 Make sure VMX-outside-SMX is allowed. It may not be if the BIOS decided to disable VMX. Seen on an HP laptop, where NVMM would panic because of that. commit 2d39ebf166acd9d66525b7758ae0391f7a5e5ea4 Author: maxv Date: Sat Jun 8 07:27:44 2019 +0000 Change the NVMM API to reduce data movements. Sent to tech-kern@. commit 54ebea61e129501586cb6328d90159ccc3b04162 Author: maxv Date: Sat May 18 08:55:59 2019 +0000 Now that SVS cannot be disabled at run time, MSR_LSTAR is static, so no need to save it on each VM enter. commit e66fce09b8b94eb5624b1f46bd6a75f17c24c076 Author: maxv Date: Wed May 15 04:39:52 2019 +0000 NVMM: Expose MD_CLEAR to the guests. commit 8d06476a4af78e72e2840470daac171f98e30abc Author: maxv Date: Sat May 11 07:44:00 2019 +0000 Replace "VMM" by "emulator", clearer. commit b0c3d26d9a3ac415670cce95549ae9a3d868a46d Author: maxv Date: Sat May 11 07:40:38 2019 +0000 Sync with reality. commit 08b6253b720d3b8f53c9ab0c3ed78e48dfb117d8 Author: maxv Date: Sat May 11 07:31:56 2019 +0000 Rework the machine configuration interface. Provide three ranges in the conf space: , and . Remove nvmm_callbacks_register(), and replace it by the conf op NVMM_MACH_CONF_CALLBACKS, handled by libnvmm. The callbacks are now per-machine, and the emulators should now do: - nvmm_callbacks_register(&cbs); + nvmm_machine_configure(&mach, NVMM_MACH_CONF_CALLBACKS, &cbs); This provides more granularity, for example if the process runs two VMs and wants different callbacks for each. commit 6087bfac2af9811302540232697de9be85ac714c Author: maxv Date: Wed May 1 09:20:21 2019 +0000 Use the comm page to inject events, rather than ioctls, and commit them in vcpu_run. This saves a few syscalls and copyins. For example on Windows 10, moving the mouse from the left to right sides of the screen generates ~500 events, which now don't result in syscalls. The error handling is done in vcpu_run and it is less precise, but this doesn't matter a lot, and will be solved with future NVMM error codes. commit 9b9b8400e517955de0a7298d2d980b618e657f42 Author: maxv Date: Mon Apr 29 19:03:17 2019 +0000 sync with reality commit ed9fe4d15fc1177ea499bd2708f7dc6965686646 Author: maxv Date: Mon Apr 29 18:54:25 2019 +0000 Stop taking care of the INT/NMI windows in the kernel, the emulator is supposed to do that itself. commit 604a261057e604c2616f4404e5b9f06f3e18a618 Author: maxv Date: Mon Apr 29 17:27:57 2019 +0000 Remove useless calls to nvmm_init(). commit 45e2a27ec0c25d5be79280294f4d27e2ab9e02d3 Author: maxv Date: Sun Apr 28 14:22:13 2019 +0000 Modify the communication layer between the kernel NVMM driver and libnvmm: introduce a bidirectionnal "comm page", a page of memory shared between the kernel and userland, and used to transfer data in and out in a more performant manner than ioctls. The comm page contains the VCPU state, plus three flags: - "wanted": the states the kernel must get/set when requested via ioctls - "cached": the states that are in the comm page - "commit": the states the kernel must set in vcpu_run The idea is to avoid performing expensive syscalls, by using the VCPU state cached, either explicitly or speculatively, in the comm page. For example, if the state is cached we do a direct 1->5 with no syscall: +---------------------------------------------+ | Qemu | +---------------------------------------------+ | ^ | (0) nvmm_vcpu_getstate | (6) Done | | V | +---------------------------------------+ | libnvmm | +---------------------------------------+ | ^ | ^ (1) State | | (2) No | (3) Ioctl: | (5) Ok, state cached? | | | "please cache | fetched | | | the state" | V | | | +-----------+ | | | Comm Page |------+---------------+ +-----------+ | ^ | (4) "Alright | V babe" | +--------+ +-----| Kernel | +--------+ The main changes in behavior are: - nvmm_vcpu_getstate(): won't emit a syscall if the state is already cached in the comm page, will just fetch from the comm page directly - nvmm_vcpu_setstate(): won't emit a syscall at all, will just cache the wanted state in the comm page - nvmm_vcpu_run(): will commit the to-be-set state in the comm page, as previously requested by nvmm_vcpu_setstate() In addition to this, the kernel NVMM driver is changed to speculatively cache certain states known to be of interest, so that the future nvmm_vcpu_getstate() calls libnvmm or the emulator will perform will use the comm page rather than expensive syscalls. For example, if an I/O VMEXIT occurs, the I/O Assist in libnvmm will want GPRS+SEGS+CRS+MSRS, and now the kernel caches all of that in the comm page before returning to userland. Overall, in a normal run of Windows 10, this saves several millions of syscalls. Eg on a 4CPU Intel with 4VCPUs, booting the Win10 install ISO goes from taking 1min35 to taking 1min16. The libnvmm API is not changed, but the ABI is. If we changed the API it would be possible to save expensive memcpys on libnvmm's side. This will be avoided in a future version. The comm page can also be extended to implement future services. commit 7fd08024e558cd8d5f30167e7232aefb8588350c Author: maxv Date: Sat Apr 27 17:30:38 2019 +0000 Mmh, fix nvmm_vcpu_create(), the cpuid is given, and must not be chosen from the free map. Looks like I forgot this after all my design rounds. While here reorder the initialization. commit 476039c9961b8b37a4d97b94c56da14e713693f9 Author: maxv Date: Sat Apr 27 15:45:21 2019 +0000 Reorder the NVMM headers, to make a clear(er) distinction between MI and MD. Also use #defines for the exit reasons rather than an union. No ABI change, and no API change except 'cap->u.{}' renamed to 'cap->arch'. commit 5e829d46acdcee0321e4e572bb68f62c664bb94b Author: maxv Date: Sat Apr 27 09:06:18 2019 +0000 If guest events were being processed when a #VMEXIT occurred, reschedule the events rather than dismissing them. This can happen for instance when a guest wants to process an exception and an #NPF occurs on the guest IDT. In practice it occurs only when the host swapped out specific guest pages. commit 005f3c8b3a55c052dd3915faab8b403f46f4858f Author: maxv Date: Sat Apr 27 08:16:19 2019 +0000 Optimize nvmm-intel, use inlined GCC assembly rather than function calls. commit 6e083df1253a316ead4b31c755b9be92dc1b8ad9 Author: maxv Date: Wed Apr 24 18:45:15 2019 +0000 Match the structure order, for better cache utilization. commit 5752ed64e4a0572edcb1573eff2de53bcd5e7413 Author: maxv Date: Wed Apr 24 18:19:28 2019 +0000 Provide the hardware error code for NVMM_EXIT_INVALID, useful when debugging. commit 2b64ba44dd6880f19807eee36a4dbd534a8d49b9 Author: maxv Date: Sat Apr 20 08:45:30 2019 +0000 Ah, take XSAVE into account in ECX too, not just in EBX. Otherwise if the guest relies only on ECX to initialize/copy the FPU state (like NetBSD does), spurious #GPs can be encountered because the bitmap is clobbered. commit 71d0393c414b5bd8f6682493422c2eb34d0a4d57 Author: maxv Date: Wed Apr 10 18:49:04 2019 +0000 Add the NVMM_CTL ioctl, always privileged regardless of the permissions of /dev/nvmm. We'll use it to provide a way for an admin to control the registered VMs in the kernel. Add an associated wrapper in libnvmm. commit 80bc33b376c379a8434db4527fdb5d1bd7b803e0 Author: maxv Date: Mon Apr 8 18:30:54 2019 +0000 Switch to MODULE_CLASS_MISC, from pgoyette@. commit 03a3a4384fc5891c12b53dce75f0493e6a093b84 Author: maxv Date: Mon Apr 8 18:23:46 2019 +0000 Don't forget to call (*machine_destroy) when killing VMs. commit 4077ef59599375eeed48ac452d63d440583af5a9 Author: maxv Date: Mon Apr 8 18:21:42 2019 +0000 Use the fd_clone approach, to avoid losing references to the registered VMs during fork(). We attach an nvmm_owner struct to the fd, reference it in each VM, and identify the process' VMs by just comparing the pointer. commit c5ddaabe68e9be361363faf215be0ef87108ed81 Author: maxv Date: Sun Apr 7 14:28:50 2019 +0000 Invert the filtering priority: now the kernel-managed cpuid leaves are overwritable by the virtualizer. This is useful to virtualizers that want to 100% control every leaf. commit 8f258a3edcb2d2d522f3090bba8c022fb41eb37a Author: maxv Date: Sun Apr 7 14:13:03 2019 +0000 Sync, and fix grammar. commit dbae0ceee4ed07f63929343fe58b428d04b94563 Author: maxv Date: Sun Apr 7 14:05:15 2019 +0000 Don't allow unloading when there are still VMs registered, and don't allow auto-unloading at all. Not a big problem actually, because since I changed the module class it's not auto-loadable anymore. commit 2d7d00f32f8289cc7ae72e1790add5e61c84b17a Author: maxv Date: Sat Apr 6 11:49:53 2019 +0000 Replace the misc[] state by a new compressed nvmm_x64_state_intr structure, which describes the interruptibility state of the guest. Add evt_pending, read-only, that allows the virtualizer to know if an event is pending. commit cee0a1fca1498ab9c3de45bd069b06989d6be71b Author: maxv Date: Thu Apr 4 17:33:47 2019 +0000 Check the GPA permissions too in the Assists, because it is possible that the guest traps on a page the virtualizer marked as read-only (even if it appears as read-write in the HVA). commit d4ea4f24d0f171e44ac50f48f055daade6a2d6e0 Author: maxv Date: Wed Apr 3 19:10:58 2019 +0000 VMX: if PAT is not valid, #GP on WRMSR, rather than crashing the guest. commit 142dd7189d5502fae113cded5d2b5952b714fbc3 Author: maxv Date: Wed Apr 3 18:05:55 2019 +0000 Add new VMCS bits. commit fe5d0e57b74a2dc1dccbd4262cbf1b3ef26b14c3 Author: maxv Date: Wed Apr 3 17:32:58 2019 +0000 Add MSR_TSC. commit 0895cc3a398fca7be974f1ea435e45ad6b992145 Author: maxv Date: Thu Mar 28 19:00:40 2019 +0000 Move NVMM in the "any" class, so that it can be enabled in GENERIC. Add missing files in files.nvmm, and add NVMM (commented out) in the amd64 GENERIC. Remove the "caveats" section in the man page. commit df7220ac04759dff216e0cbdb1d74429f7ae392e Author: maxv Date: Thu Mar 21 20:21:40 2019 +0000 Make it possible for an emulator to set the protection of the guest pages. For some reason I had initially concluded that it wasn't doable; verily it is, so let's do it. The reserved 'flags' argument of nvmm_gpa_map() becomes 'prot' and takes mmap-like protection codes. commit b93efd52e70760b40fadce6293a47d26b3a560c0 Author: maxv Date: Thu Mar 14 20:29:53 2019 +0000 Optimize NVMM-Intel: keep the VMCS active on the host CPU, and lazy-switch it on demand only when needed. This allows the CPU to use the cached version of the guest state, rather than the in-memory copy of it. This is much more performant. A VMCS must be active on only one CPU, but one CPU can have several active VMCSs at the same time. We keep track of which CPU each VMCS is active on. When we want to execute a VCPU, we determine whether its VMCS is loaded on another CPU, and if so send an IPI to ask it to unbusy that VMCS. In most cases the VMCS is already active on the current CPU, so we don't have to do anything and can proceed with a fast VMRESUME. We send IPIs with kpreemption enabled but with a bound LWP, because we don't want to get context-switched to the CPU we just sent an IPI to. Overall, with this in place, I see a ~15% performance increase in the guests on NVMM-Intel. commit 7c63f593d73c2584a06f794e05bff5c594be25a4 Author: maxv Date: Thu Mar 14 19:26:44 2019 +0000 Move a KASSERT, applies to all branches. commit 78aa41a956f0454eb0b8003fab03bfd139f9f84a Author: maxv Date: Thu Mar 14 19:15:26 2019 +0000 Reduce the mask of the VTPR, only the first four bits matter. commit 3c093f6fb3e30cb7eaf4a43797824a8d4702895f Author: maxv Date: Thu Mar 14 19:10:27 2019 +0000 Fail early if we're beyond the guest max ram. commit 824d33230d413d9a96d0392a69131c56377a8954 Author: maxv Date: Thu Mar 7 15:47:34 2019 +0000 Micro optimizations: - Compress x86_rexpref, x86_regmodrm, x86_opcode and x86_instr. - Cache-align the register, opcode and group tables. - Modify the opcode tables to have 256 entries, and avoid a lookup. commit 29f0099ec30e94009fcc2cbf75cfb803e8b04cef Author: maxv Date: Thu Mar 7 15:22:21 2019 +0000 Rename the internal NVMM HVA table entries from "segment" to "hmapping", less confusing. Also fix the error handling in nvmm_hva_unmap(). commit 4b725333a19d4295e7116a66e64600bf8c430d0d Author: maxv Date: Thu Mar 7 15:06:37 2019 +0000 Parse EXC_NMI on nvmm-intel, and don't return NVMM_EXIT_INVALID if we received a host NMI, otherwise the guest could get killed if an NMI comes in, typically when the host runs tprof at the same time. Already handled on nvmm-amd. commit ad584c20185c433b9a7d3abb49a27a6a145f2f40 Author: maxv Date: Sun Mar 3 07:01:09 2019 +0000 Choose which CPUID bits to allow, rather than which bits to disallow. This is clearer, and also forward compatible with future CPUs. While here be more consistent when allowing the bits, and sync between nvmm-amd and nvmm-intel. Also make sure to disallow AVX, because the guest state we provide is only x86+SSE. Fixes a CentOS panic when booting on NVMM, reported by Jared McNeill, thanks. commit a7cdcccb8494989cb2e0a70cb06e5905b8a0ce28 Author: maxv Date: Tue Feb 26 12:23:12 2019 +0000 Change the layout of the SEG state: - Reorder it, to match the CPU encoding. This is the universal order, also used by Qemu. Drop the seg_to_nvmm[] tables. - Compress it. This divides its size by two. - Rename some of its fields, to better match the x86 spec. Also, take S out of Type, this was a NetBSD-ism that was likely confusing to other people. commit f35533a8094ee9d3fc61401beaf9028ab5d6f1f6 Author: maxv Date: Tue Feb 26 10:18:39 2019 +0000 Set hardseg to -1 rather than 0, because 0 can be a valid segment. commit 3f7ba5d607892caff67ede7857c13f6b4659d0fc Author: maxv Date: Sat Feb 23 12:27:00 2019 +0000 Install the x86 RESET state at VCPU creation time, for convenience, so that the libnvmm users can expect a functional VCPU right away. commit bce11997382ba886f8576eb6df01582b9747bd89 Author: maxv Date: Sat Feb 23 10:43:36 2019 +0000 Add support for CPUs that don't have the EPT_{A,D} bits. On such CPUs, these bits are ignored by the hardware. We don't care about setting them, however, we must always assume they are set. Modify the pmap code to do that. While here, in pmap_ept_remove_pte, don't flush the TLB when it's not needed. Tested on an old Intel Celeron. commit 1712eeb2645b4a221705e694b0a61ec3f9fc8830 Author: maxv Date: Sat Feb 23 08:19:16 2019 +0000 Reorder the functions, and constify setstate. No functional change. commit d3528af0854366fe84a0e7a20921a87bbd0e725d Author: maxv Date: Fri Feb 22 12:24:34 2019 +0000 Fix omission: if we receive a guest trap on CR0, and if the original instruction would have resulted in Long Mode being enabled, we need to manually enable Long Mode ourselves. We were already doing that correctly in setstate, but not in the CR0 trap handler. Problem initially reported by Aymeric Vincent; ArchLinux wouldn't boot, now it does and works correctly. While here, add CR0_ET in the CR0 mask, for the associated shadow to be taken into account. Normally this shadow bit shouldn't be necessary, but for now I keep it regardless. commit 875518e2abe1e083fc8caacc74a0731d7bdd944d Author: maxv Date: Thu Feb 21 13:25:44 2019 +0000 Reorder the detection in vmx_ident(), to fix panic on old CPUs. We must read MSR_IA32_VMX_EPT_VPID_CAP _after_ ensuring EPT is there, because if it's not, the rdmsr faults. commit d6f8093fcd631fbae0ce8ae04447ae1492d92992 Author: maxv Date: Thu Feb 21 12:17:52 2019 +0000 Another locking issue in NVMM: the {svm,vmx}_tlb_flush functions take VCPU mutexes which can sleep, but their context does not allow it. Rewrite the TLB handling code to fix that. It becomes a bit complex. In short, we use a per-VM generation number, which we increase on each TLB flush, before sending a broadcast IPI to everybody. The IPIs cause a #VMEXIT of each VCPU, and each VCPU Loop will synchronize the per-VM gen with a per-VCPU copy, and apply the flushes as neededi lazily. The behavior differs between AMD and Intel; in short, on Intel we don't flush the hTLB (EPT cache) if a context switch of a VCPU occurs, so now, we need to maintain a kcpuset to know which VCPU's hTLBs are active on which hCPU. This creates some redundancy on Intel, ie there are cases where we flush the hTLB several times unnecessarily; but hTLB flushes are very rare, so there is no real performance regression. The thing is lock-less and non-blocking, so it solves our problem. commit 409f23287dab9681e2948875bcbb1061a2009467 Author: maxv Date: Thu Feb 21 11:58:04 2019 +0000 Clarify the gTLB code a little. commit 042f8e32f3f132fe4cd182325724da3b7ca055dc Author: maxv Date: Mon Feb 18 12:17:45 2019 +0000 Ah, finally found you. Fix scheduling bug in NVMM. When processing guest page faults, we were calling uvm_fault with preemption disabled. The thing is, uvm_fault may block, and if it does, we land in sleepq_block which calls mi_switch; so we get switched away while we explicitly asked not to be. From then on things could go really wrong. Fix that by processing such faults in MI, where we have preemption enabled and are allowed to block. A KASSERT in sleepq_block (or before) would have helped. commit b360319c237c85e717ab9e13d6b74c1ea9074404 Author: maxv Date: Sun Feb 17 20:25:46 2019 +0000 Fix handling of SIB instructions. We were jumping to the SIB node _before_ fetching the displacement, so the node would always think there was no displacement. This didn't alter the final GPA we would be touching - because it is fetched from the kernel directly and not from the computation -, but it altered the instruction length, and on some guests (like Fedora 64bit), the VCPU would resume execution at the wrong RIP and crash. Now these guests work. commit bdf95c00c81f94ebe2b7638cfbd3378cadcb0062 Author: maxv Date: Sat Feb 16 12:58:13 2019 +0000 Ah no, adapt previous, on AMD RAX is in the VMCB. commit 613287c8d80e0ae608c972974949f8297a15a1c5 Author: maxv Date: Sat Feb 16 12:40:31 2019 +0000 Improve the FPU detection: hide XSAVES because we're not allowing it, and don't set CPUID2_OSXSAVE if the guest didn't first set CR4_OSXSAVE. With these changes in place, I can boot Windows 10 on NVMM. commit 60a2d4b8fb52e73be5f5c69b46c2d323ab33a930 Author: maxv Date: Sat Feb 16 12:05:30 2019 +0000 Handle MSR_MISC_ENABLE on NVMM-Intel (Intel-specific). commit 40c6faeb0023754bcb8de68dfad4e9aaf7e9786a Author: maxv Date: Fri Feb 15 16:42:27 2019 +0000 Remove the PSE check in the 32bit-PAE MMU. Setting CR4.PAE automatically enables PSE regardless of whether CR4.PSE is set or not, so we should just ignore it. With this in place I can boot Windows 8.1 on NVMM. commit 5f77213fd2f4c5b42649f18aa9f0c4c0a99870e6 Author: maxv Date: Fri Feb 15 13:17:05 2019 +0000 Initialize the guest TSC to zero at VCPU creation time, and handle guest writes to MSR_TSC at run time. This is imprecise, because the hardware does not provide a way to preserve the TSC during #VMEXITs, but that's fine enough. commit 83a61044252c6b126e7d3210ea15dc56dae6b2e8 Author: maxv Date: Thu Feb 14 14:30:20 2019 +0000 Harmonize the handling of the CPL between AMD and Intel. AMD has a separate guest CPL field, because on AMD, the SYSCALL/SYSRET instructions do not force SS.DPL to predefined values. On Intel they do, so the CPL on Intel is just the guest's SS.DPL value. Even though technically possible on AMD, there is no sane reason for a guest kernel to set a non-three SS.DPL, doing that would mess up several common segmentation practices and wouldn't be compatible with Intel. So, force the Intel behavior on AMD, by always setting SS.DPL<=>CPL. Remove the now unused CPL field from nvmm_x64_state::misc[]. This actually increases performance on AMD: to detect interrupt windows the virtualizer has to modify some fields of misc[], and because CPL was there, we had to flush the SEG set of the VMCB cache. Now there is no flush necessary. While here remove the CPL check for XSETBV on Intel, contrary to AMD Intel checks the CPL before the intercept, so if we receive an XSETBV VMEXIT, we are certain that it was executed at CPL=0 in the guest. By the way my check was wrong in the first place, it was reading SS.RPL instead of SS.DPL. commit 9eb6ddd38e3562dec16e1271f08147ff965482c4 Author: maxv Date: Thu Feb 14 09:37:31 2019 +0000 On AMD, the segments have a simple "present" bit. On Intel however there is an extra "unusable" bit, which has a twisted meaning. We can't just ignore this bit, because when unset, the CPU performs extra checks on the other attributes, which may cause VMENTRY to fail and the guest to be killed. Typically, on Qemu, some guests like Windows XP trigger two consecutive getstate+setstate calls, and while processing them, we end up wrongfully removing the "unusable" bits that were previously set. Fix that by forcing "unusable = !present". Each hypervisor I could check does something different, but this seems to be the least problematic solution for now. While here, the fields of vmx_guest_segs are VMX indexes, so they should be uint64_t (no functional change). commit 2829a4737b63754a5375b25d9d3f4dd56448f388 Author: maxv Date: Wed Feb 13 16:03:16 2019 +0000 Add Intel-VMX support in NVMM. This allows us to run hardware-accelerated VMs on Intel CPUs. Overall this implementation is fast and reliable, I am able to run NetBSD VMs with many VCPUs on a quad-core Intel i5. NVMM-Intel applies several optimizations already present in NVMM-AMD, and has a code structure similar to it. No change was needed in the NVMM MI frontend, or in libnvmm. Some differences exist against AMD: - On Intel the ASID space is big, so we don't fall back to a shared ASID when there are more VCPUs executing than available ASIDs in the host, contrary to AMD. There are enough ASIDs for the maximum number of VCPUs supported by NVMM. - On Intel there are two TLBs we need to take care of, one for the host (EPT) and one for the guest (VPID). Changes in EPT paging flush the host TLB, changes to the guest mode flush the guest TLB. - On Intel there is no easy way to set/fetch the VTPR, so we intercept reads/writes to CR8 and maintain a software TPR, that we give to the virtualizer as if it was the effective TPR in the guest. - On Intel, because of SVS, the host CR4 and LSTAR are not static, so we're forced to save them on each VMENTRY. - There is extra Intel weirdness we need to take care of, for example the reserved bits in CR0 and CR4 when accesses trap. While this implementation is functional and can already run many OSes, we likely have a problem on 32bit-PAE guests, because they require special care on Intel CPUs, and currently we don't handle that correctly; such guests may misbehave for now (without altering the host stability). I expect to fix that soon. commit 098836b92dccbd757fe736cc3c8e5bfc67c8d3a1 Author: maxv Date: Wed Feb 13 10:55:13 2019 +0000 Drop support for software interrupts. I had initially added that to cover the three event types available on AMD, but Intel has seven of them, all with weird and twisted meanings, and they require extra parameters. Software interrupts should not be used anyway. commit e7b70c162b837efb5f76fd989aa58ffafbc36868 Author: maxv Date: Wed Feb 13 07:04:12 2019 +0000 Micro optimization: the STAR/LSTAR/CSTAR/SFMASK MSRs are static, so rather than saving them on each VMENTRY, save them only once, at VCPU creation time. commit 008a833faf902dcb502f8ae4c6bf368fa3c54654 Author: maxv Date: Wed Feb 13 06:32:45 2019 +0000 Reorder the GPRs to match the CPU encoding, simplifies things on Intel. commit 0196af49ab249be76c52d1ae4e6dd63fd820c186 Author: maxv Date: Tue Feb 12 14:54:59 2019 +0000 Optimize: the hardware does not clear the TLB flush command after a VMENTRY, so clear it ourselves, to avoid uselessly flushing the guest TLB. While here also fix the processing of EFER-induced flushes, they shouldn't be delayed. commit 43dbe2c4ed79a2aaa0b47688b2b078404598b463 Author: maxv Date: Tue Feb 12 14:50:21 2019 +0000 Optimize: fetch only 5 bytes instead of 15, the instruction can have only up to five prefixes. commit 0c66f932a362a9c62828a8dab4ea30e3f0cefc98 Author: maxv Date: Mon Feb 11 07:07:37 2019 +0000 Increase the max guest ram from 4GB to 128GB. commit 00dbf654738e05140e158c4145e06cc50cf47ac9 Author: christos Date: Sun Feb 10 19:30:28 2019 +0000 #### is not legal. commit 1f9a1ec47be5f51d6cb858092d628f7d7071a438 Author: maxv Date: Thu Feb 7 10:58:45 2019 +0000 Improvements: - Emulate the instructions by executing them directly on the host CPU. This is easier and probably faster than doing it in software manually. - Decode SUB from Primary, CMP from Group1, TEST from Group3, and add associated tests. - Handle correctly the cases where an instruction that always implicitly reads the register operand is executed with the mem operand as source (eg: "orq (%rbx),%rax"). - Fix the MMU handling of 32bit-PAE. Under PAE CR3 is not page-aligned, so there are extra bits that are valid. With these changes in place I can boot Windows XP on Qemu+NVMM. commit 1a85186570917f36ecef73c076a66a49f6745c1e Author: wiz Date: Tue Feb 5 15:03:35 2019 +0000 Mark up NULL with Dv. Remove empty line. commit a1a558cbc8b8801712536119e62333865a1325cb Author: maxv Date: Tue Feb 5 13:56:32 2019 +0000 Sync with reality, and improve. commit ef67fd308549ea0e74a403dfbf03c5c34647c610 Author: maxv Date: Mon Feb 4 12:11:18 2019 +0000 Improvements: - Guest reads/writes to PAT land in gPAT, so no need to emulate them. - When emulating EFER, don't advance the RIP if a fault occurs, and don't forget to flush the VMCB cache accordingly. commit cdb31846f88aa75d95305ec0042b31178809f208 Author: maxv Date: Fri Feb 1 06:49:58 2019 +0000 Fix two issues: * Uh I put the wrong masks in some GPRs, fuck. * When the opsize of MOVZX is 4, we need to combine the zero-extend from the instruction with the natural zero-extend of long mode. Add two associated tests. commit 6b34ef66c5bc0e04cdfce2b6f36cedc083cb803b Author: pgoyette Date: Sun Jan 27 02:08:33 2019 +0000 Merge the [pgoyette-compat] branch commit 7d511c12ce179812b31745508a6bc6ef085ad22b Author: maxv Date: Sat Jan 26 15:25:51 2019 +0000 Optimize: keep a per-VCPU buffer for the state, and copy in and out directly on it. The VCPUs are protected by mutexes, so nothing to worry about. This saves two kmem_allocs in {get,set}state. commit c034519d8c971b80e248b55eb183b1b47b539dcb Author: maxv Date: Sat Jan 26 15:12:20 2019 +0000 Remove nvmm_exit_memory.npc, useless. commit 90b0ce8a8beb73f6c4aec9738ef1f69d2a2195a6 Author: maxv Date: Sat Jan 26 14:44:54 2019 +0000 Ah, fix bug: when the opcode has an immediate, we fill the src with a register storage, but then we overwrite it without zeroing out the highest bits of the resulting immediate (which may contain garbage from the union). commit 4edf9e08c2d5da7cf40982a7d6a4b75c2bfc4dbb Author: maxv Date: Thu Jan 24 13:05:59 2019 +0000 Optimize: change the behavior of the HLT vmexit, make it a "change in vcpu state" which occurs after the instruction executed, rather than an instruction intercept which occurs before. Disable the shadow and the intr window in kernel mode, and advance the RIP, so that the virtualizer doesn't have to do it itself. This saves two syscalls and one VMCB cache flush. Provide npc for other instruction intercepts, in case someone is interested. commit e0ff52afb5d1fcb09eaa6f0fb6ecea4b7d8b81ee Author: maxv Date: Sun Jan 20 16:55:21 2019 +0000 Improvements in NVMM * Handle the FPU differently, limit the states via the given mask rather than via XCR0. Align to 64 bytes. Provide an initial gXCR0, to be sure that XCR0_X87 is set. Reset XSTATE_BV when the state is modified by the virtualizer, to force a reload from memory. * Hide RDTSCP. * Zero-extend RBX/RCX/RDX when handling the NVMM CPUID signature. * Take ECX and not RCX on MSR instructions. commit 8cb399318926d8aee03f910c1af3c094621de84e Author: maxv Date: Sun Jan 13 10:43:22 2019 +0000 Handle more corner cases, clean up a little, and add a set of instructions in Group1. commit cf712e1021547cd92b0e6ce990bf11a6abc57259 Author: maxv Date: Sun Jan 13 10:07:50 2019 +0000 Reset DR7 before loading DR0-3, to prevent a fault if the host process has dbregs enabled. commit fe65b1082f76612238471131fc9c6c66a0a04c8c Author: maxv Date: Thu Jan 10 06:58:36 2019 +0000 Optimize: * Don't save/restore the host CR2, we don't care because we're not in a #PF context (and preemption switches already handle CR2 safely). * Don't save/restore the host FS and GS, just reset them to zero after VMRUN. Note: DS and ES must be reset _before_ VMRUN, but that doesn't apply to FS and GS. * Handle FSBASE and KGSBASE outside of the VCPU loop, to avoid the cost of saving/restoring them when there's no reason to leave the loop. commit c73cb7aefbfc51da7ee63133ea164a8c908bc21a Author: maxv Date: Tue Jan 8 14:43:18 2019 +0000 Optimize: don't keep a full copy of the guest state, rather take only what is needed. This avoids expensive memcpy's. Also flush the V_TPR as part of the CR-state, because there is CR8 in it. commit c31f2b7a7f314e565fdf7bdf6e143bf974fc47af Author: maxv Date: Tue Jan 8 07:34:22 2019 +0000 Handle REPN. FreeBSD has a "repn movs", which is a bit unusual, but doesn't seem illegal as far as I can tell from the AMD SDM. With that, I can boot FreeBSD on Qemu+NVMM. commit 22f74dfea3707feb5529f4127650ed27a9c52aa3 Author: maxv Date: Tue Jan 8 07:29:46 2019 +0000 _IOWR -> _IOW commit 54bdfea1ce230e6c0aa2e785c456d27bd08c22fc Author: wiz Date: Mon Jan 7 22:17:02 2019 +0000 Remove leading zero from date. commit f7bb016339701f15208edb198c280d8baef8e050 Author: maxv Date: Mon Jan 7 18:13:34 2019 +0000 Optimize the legpref node: omit BRN (we don't care and it's the same as OVR_CS), inline the loops, sort the checks from most to least likely prefix, and use a compact structure. commit 9edc324e9b99e233a44399f04ff26eb5caf4bbc1 Author: maxv Date: Mon Jan 7 16:30:25 2019 +0000 Optimize: on single memory operand instructions, take the GPA directly from the exit structure provided by the kernel. This saves an MMU translation, and sometimes complex address computation (eg SIB). Drop the GVA field, it is not useful to virtualizers. commit def9083f44681cdf683fb584498e6256ff19a17f Author: maxv Date: Mon Jan 7 14:08:02 2019 +0000 Optimize: cache the guest state entirely in the VMCB-cache, flush it on a state-by-state basis when needed. commit 15992ed9f793d45abc86f105b6f2051936ac850a Author: maxv Date: Mon Jan 7 13:47:33 2019 +0000 Improvements and fixes: * Decode AND/OR/XOR from Group1. * Sign-extend the immediates and displacements in 64bit mode. * Fix the storage of {read,write}_guest_memory, now that we batch certain IO operations we can copy more than 8 bytes, and shit hits the fan. * Remove the CR4_PSE check in the 64bit MMU. This bit is actually ignored in long mode, and some systems (like FreeBSD) don't set it. commit f66528d8cdad2beb8654e43fbb5e26dad3d27dc8 Author: maxv Date: Sun Jan 6 18:32:54 2019 +0000 Add more VMCB fields. Also remove debugging code I mistakenly committed in the previous revision. No functional change. commit 02a074bc61ed811a75d5377ceb3325cc09f3a9cf Author: maxv Date: Sun Jan 6 16:10:51 2019 +0000 Improvements and fixes in NVMM. Kernel driver: * Don't take an extra (unneeded) reference to the UAO. * Provide npc for HLT. I'm not really happy with it right now, will likely be revisited. * Add the INT_SHADOW, INT_WINDOW_EXIT and NMI_WINDOW_EXIT states. Provide them in the exitstate too. * Don't take the TPR into account when processing INTs. The virtualizer can do that itself (Qemu already does). * Provide a hypervisor signature in CPUID, and hide SVM. * Ignore certain MSRs. One special case is MSR_NB_CFG in which we set NB_CFG_INITAPICCPUIDLO. Allow reads of MSR_TSC. * If the LWP has pending signals or softints, leave, rather than waiting for a rescheduling to happen later. This reduces interrupt processing time in the guest (Qemu sends a signal to the thread, and now we leave right away). This could be improved even more by sending an actual IPI to the CPU, but I'll see later. Libnvmm: * Fix the MMU translation of large pages, we need to add the lower bits too. * Change the IO and Mem structures to take a pointer rather than a static array. This provides more flexibility. * Batch together the str+rep IO transactions. We do one big memory read/write, and then send the IO commands to the hypervisor all at once. This considerably increases performance. * Decode MOVZX. With these changes in place, Qemu+NVMM works. I can install NetBSD 8.0 in a VM with multiple VCPUs, connect to the network, etc. commit b6c2e14c6bfa17ce0f7cecb911e286412a73a262 Author: maxv Date: Fri Jan 4 10:25:39 2019 +0000 In !64bit mode RIP-relative is null+disp32, handle that correctly. commit fac5e55a24367828649f075d3fd77fc9f88afd01 Author: maxv Date: Thu Jan 3 08:02:49 2019 +0000 Fix another gross copy-pasto. commit 1cb762c12044df197ad606bb031b92ee27edb4c8 Author: maxv Date: Wed Jan 2 12:18:08 2019 +0000 When there's no DecodeAssist in hardware, decode manually in software. This is needed on certain AMD CPUs (like mine): the segment base of OUTS can be overridden, and it is wrong to just assume DS. We fetch the instruction and look at the prefixes if any to determine the correct segment. commit 6b8576f31ede26be7f6e2e9b69db5cafaffe8129 Author: maxv Date: Sat Dec 29 17:54:54 2018 +0000 Fix the segmentation check, the limit is relative, not absolute. commit 6d0107130349e7ccc613d85a3866e5fc6356acd2 Author: maxv Date: Thu Dec 27 07:22:31 2018 +0000 Several improvements and fixes: * Change the Assist API. Rather than passing callbacks in each call, the callbacks are now registered beforehand. Then change the I/O Assist to fetch MMIO data via the Mem callback. This allows a guest to perform an I/O string operation on a memory that is itself an MMIO. * Introduce two new functions internal to libnvmm, read_guest_memory and write_guest_memory. They can handle mapped memory, MMIO memory and cross-page transactions. * Allow nvmm_gva_to_gpa and nvmm_gpa_to_hva to take non-page-aligned addresses. This simplifies a lot of things. * Support the MOVS instruction, and add a test for it. This instruction is special, in that it takes two implicit memory operands. In particular, it means that the two buffers can both be in MMIO memory, and we handle this case. * Fix gross copy-pasto in nvmm_hva_unmap. Also fix a few things here and there. commit a7d1548b50e74991bfacdee2a9f8bf4d6c154100 Author: maxv Date: Sat Dec 15 13:39:43 2018 +0000 Invert the mapping logic. Until now, the "owner" of the memory was the guest, and by calling nvmm_gpa_map(), the virtualizer was creating a view towards the guest memory. Qemu expects the contrary: it wants the owner to be the virtualizer, and nvmm_gpa_map should just create a view from the guest towards the virtualizer's address space. Under this scheme, it is legal to have two GPAs that point to the same HVA. Introduce nvmm_hva_map() and nvmm_hva_unmap(), that map/unamp the HVA into a dedicated UOBJ. Change nvmm_gpa_map() and nvmm_gpa_unmap() to just perform an enter into the desired UOBJ. With this change in place, all the mapping-related problems in Qemu+NVMM are fixed. commit e88dc7e8bf7b918df1c85a6c72dacd5e18dc7468 Author: maxv Date: Sat Dec 15 13:09:02 2018 +0000 Two changes: - Fix the I/O Assist, for INS* it is RDI and not RSI, and the register gets updated regardless of the REP prefix. - Fill in the Mem Assist. We decode and emulate certain instructions, and pass a mem descriptor to the callback to handle the transaction. The disassembler could use some polishing, and there are still a few instructions missing; but basically it works. commit 65f495e8aa2116bef51801809f26a7ad61d3472b Author: maxv Date: Thu Dec 13 16:28:10 2018 +0000 Don't forget to advance the RIP after an XSETBV emulation. commit 4cf43b42f438b161db7d8d117b77b123724c6119 Author: wiz Date: Wed Dec 12 11:40:08 2018 +0000 Remove superfluous dot. commit e3002faf2d915ce3ca295b1fb2b7ad042d25fa55 Author: maxv Date: Wed Dec 12 10:42:34 2018 +0000 Change the map/unmap functions, again. commit e785e4bc51e75f6bd156006b6aee07ecf3769f99 Author: maxv Date: Wed Dec 12 09:09:08 2018 +0000 Change the "FILES" section, in the end I don't want to commit toyvirt and smallkern, there is little interest installing them by default, rather they can be downloaded from www. It's better this way. While here add NVMM(4) in "SEE ALSO". commit 6882fb8eca9b71cad2c720fb94d273333dee0eca Author: maxv Date: Thu Nov 29 19:55:20 2018 +0000 Rewrite the gpa map/unmap functions. Dig holes in the mapped areas when there is an overlap. Close to what Qemu expects. commit 2affef406444a1316ae1662d2702c910145a788a Author: maxv Date: Sun Nov 25 14:11:24 2018 +0000 Appease the check: allow NVMM_MAX_RAM bytes of memory, and not just NVMM_MAX_RAM-1. commit aa9fffd9559d03f72c332e6e70a86dbe6c583e2a Author: maxv Date: Sun Nov 25 14:09:57 2018 +0000 Add RFLAGS in the exitstate. commit 5aea1ef46fc591f95cb4a13e4df7ac90c503813c Author: maxv Date: Thu Nov 22 07:37:12 2018 +0000 Add missing pmap_update after pmap_kenter_pa, noted by Kamil. commit ed114ff4dc2467302e81153e3b3a0fd3090ee411 Author: maxv Date: Mon Nov 19 21:45:37 2018 +0000 Fix error handling of realloc, and use memmove because the areas overlap; noted by agc@. These _nvmm_area_add/delete functions don't make a lot of sense right now and will likely be rewritten to match the behavior expected by Qemu; but still fix for the time being. Also fix a collision check while here. commit 9d4e3adbeef2598e7eeb4e0141d9ad587e3266ce Author: maxv Date: Mon Nov 19 17:35:12 2018 +0000 Rename one constant, for clarity. commit e0463a397f0f81bd1549fdb2002e8de92e90f0df Author: maxv Date: Sun Nov 18 07:42:24 2018 +0000 Ah, should be UVM_ADV_RANDOM. commit 327344f16ac2ab65ec73fff66073bfd69412bbe8 Author: maxv Date: Sat Nov 17 16:11:33 2018 +0000 Don't forget to set 'prot' when the guest has paging disabled. commit 86a74f4fcdb0da86ef214b0af2bb28e833dba5e7 Author: maxv Date: Wed Nov 14 19:14:40 2018 +0000 Take RAX from the VMCB and not the VCPU state, the latter is not synchronized and contains old values. commit 9db01b7e2c13b3ed9227fba4df27be9d2a7647a8 Author: martin Date: Tue Nov 13 09:24:37 2018 +0000 Too much magic involved - revert previous. commit d2cb6db9f3a97f343d3cb2d8d66d4fcc3d6af5ef Author: martin Date: Tue Nov 13 09:14:14 2018 +0000 Need some minimalistic support for additional things that ../Makefile requires, even if we do nothing here commit 1ccf0491dec2869fca0feaa815aaaf715f1c27bd Author: martin Date: Tue Nov 13 09:00:08 2018 +0000 Move conditionals for libnvmm to subdir makefile, requested boy mrg. commit 791fac88452f8910ed09c323800b1e23619d4f48 Author: maya Date: Tue Nov 13 06:57:14 2018 +0000 Revert my own rev 1.2, the missing include was only when building the 32-bit compat library, we no longer do this. commit 07fe401d4f470069f52053322129188bf6629ce5 Author: nakayama Date: Mon Nov 12 17:46:53 2018 +0000 No need to install shared libraries to /lib. commit 7cf5622063423def53a97e70e8f2cc4aa22ce8b9 Author: maya Date: Sun Nov 11 00:06:48 2018 +0000 Add missing include for struct nvmm_x64_state (Pointed out by the clang build) commit 38a06a522d6f5adfb9ba8037988859f476b512a7 Author: maxv Date: Sat Nov 10 10:57:06 2018 +0000 Add copyright and RCSID, from wiz@. commit 33c33657c05873631724acbc68f17cb79eb93b2b Author: maxv Date: Sat Nov 10 09:42:42 2018 +0000 Remove unused cpu_msr.h includes. commit e00c4d7eb507db478a9a082f0db453d10159e0b9 Author: maxv Date: Sat Nov 10 09:28:56 2018 +0000 Add libnvmm, NetBSD's new virtualization API. It provides a way for VMM software to effortlessly create and manage virtual machines via NVMM. It is mostly complete, only nvmm_assist_mem needs to be filled -- I have a draft for that, but it needs some more care. This Mem Assist should not be needed when emulating a system in x2apic mode, so theoretically the current form of libnvmm is sufficient to emulate a whole class of systems. Generally speaking, there are so many modes in x86 that it is difficult to handle each corner case without introducing a ton of checks that just slow down the common-case execution. Currently we check a limited number of things; we may add more checks in the future if they turn out to be needed, but that's rather low priority. Libnvmm is compiled and installed only on amd64. A man page (reviewed by wiz@) is provided. commit 5c184ba9b5764e690485213d54f1c92aade4a23a Author: maxv Date: Wed Nov 7 07:43:07 2018 +0000 Add NVMM - for NetBSD Virtual Machine Monitor -, a kernel driver that provides support for hardware-accelerated virtualization on NetBSD. It is made of an MI frontend, to which MD backends can be plugged. One MD backend is implemented, x86-SVM, for x86 AMD CPUs. We install /usr/include/dev/nvmm/nvmm.h /usr/include/dev/nvmm/nvmm_ioctl.h /usr/include/dev/nvmm/{arch}/nvmm_{arch}.h And the kernel module. For now, the only architecture where we do that is amd64 (arch=x86). NVMM is not enabled by default in amd64-GENERIC, but is instead easily modloadable. Sent to tech-kern@ a month ago. Validated with kASan, and optimized with tprof.