From memory:

Qemu sometimes needs to stop a VCPU while it is running.

The code in my Git doesn't have an nvmm_vcpu_stop() API. Instead, to stop a
VCPU you must send a signal to the thread that is running said VCPU. See
nvmm_ipi_signal() in Qemu. The principle is: signal deliveries interrupt
the target CPU, this causes a VMEXIT and there os_return_needed() returns
'false' so we break out of the loop and return to userland. Therefore by
sending signals we are able to stop VCPUs.

But it's not super efficient, because a signal gets delivered and this
costs several syscalls.

To improve performance here, there needs to be an nvmm_vcpu_stop() API that
*only* stops a VCPU, and doesn't do the extra signal delivery. The correct
way to do it would be to use a combination of two bools in the comm page,
and a kernel ioctl that sends IPIs to kick remote CPUs (and force a VMEXIT)
. I had made a design document for that, see attached. Matt if you're
interested in performance you may want to look at it too.

The change that the NetBSD people did implements only half of that logic
and therefore doesn't really stop a VCPU. Their change is not even
programmatically correct, because they access the VCPU comm page after
releasing its mutex.

In any case, I'd suggest rather focusing on getting my current NVMM working
correctly, before going into this.

Maxime
2021-07-03

================================================================================

Introduce two fields in the comm page:

 - 'bool stop': set to 'true' by userland when we want the VCPU to stop. Cleared
   to 'false' by the kernel when stopping the VCPU.

 - 'bool running': set to 'true' by the kernel when the VCPU starts executing.
   Cleared to 'false' by the kernel when the VCPU is not executing anymore.

================================================================================

user::nvmm_vcpu_stop()

	comm->stop = true;

	/*
	 * If the vCPU is running, then there is no guarantee that the hCPU saw
	 * that "stop = true". In that case, invoke the ioctl, and the kernel
	 * will deal with the hCPU.
	 */
	if (comm->running) {
		ioctl(kern::nvmm_vcpu_stop);
	}

kern::nvmm_vcpu_stop()

	hcpu = atomic_load_relaxed(&vcpu->hcpu);

	/*
	 * It is possible that the hCPU pointer is gone, if the hCPU stopped
	 * executing the vCPU while we were coming here. In that case, there
	 * is nothing to do: the kernel already has, or will soon, evaluate
	 * vcpu->stop and return to userland.
	 */
	if (__predict_false(hcpu == NULL))
		return 0;

	/*
	 * Kick the hCPU. This sends an IPI to it, which will cause a VMEXIT.
	 * Upon VMEXIT the kernel will see that vcpu->stop is true, and will
	 * return to userland.
	 */
	cpu_kick(hcpu);

	return 0;

kern::nvmm_vcpu_run()

	while (1) {
		if (vcpu->comm->stop)
			break;
		vcpu->hcpu = curcpu();
		vcpu->comm->running = true;
		do_run();
		vcpu->comm->running = false;
		vcpu->hcpu = NULL;

	}
	vcpu->comm->stop = false;