# DragonFly BSD FPU Subsystem: Specification, Code Review, and NetBSD Porting Guide

## Part 1 — How FPU State Works, In General

### 1.1 How the FPU interacts with the CPU

On x86, "the FPU" is really three historically-layered register files sharing
one save/restore mechanism:

- **x87** — 8×80-bit stack registers, status/control/tag words. Legacy,
  rarely used directly by modern compilers except for `long double`.
- **SSE/SSE2** — 16×128-bit `XMM0-15` registers plus `MXCSR` (a control/status
  register for SIMD FP exceptions and rounding mode). The compiler ABI on
  x86-64 uses XMM registers for ordinary `double`/`float` math, so *every*
  userland thread that does floating point touches this state.
- **AVX/AVX2** — widens each XMM register to a 256-bit `YMM0-15` (the XMM
  register is just the low 128 bits of the corresponding YMM register).
- **AVX-512** — widens further to 512-bit `ZMM0-31` (32 registers instead of
  16), plus 8 new 64-bit mask registers `k0-k7`.

All of this shares a single hardware ownership model:

- `CR0.TS` (Task Switched) and `CR0.EM` (Emulation) gate access. If `TS=1`
  and a thread executes any FP/SSE/AVX instruction, the CPU raises `#NM`
  (Device Not Available, vector 7) instead of executing it. `EM=1` forces
  every x87 instruction to trap (used only for pure emulation, obsolete
  today).
- The classic BSD/UNIX trick is to run with `TS` set for a thread whose FPU
  register file is *not* currently loaded into the actual hardware
  registers. The first FP instruction that thread executes takes an `#NM`
  trap; the kernel's "device-not-available" handler loads that thread's
  saved FPU state into the registers, clears `TS`, and returns to retry the
  faulting instruction. This is **lazy restore** — the reload cost is paid
  only by threads that actually use the FPU, and only once per scheduling
  quantum in the classic model.
- The complementary operation is **saving**: at some point (traditionally,
  when the kernel is about to run a different thread's FPU state into the
  registers, or when it must inspect/modify the state, e.g. `ptrace`), the
  current contents of the FPU registers must be written out to memory
  (`FSAVE`/`FNSAVE`, `FXSAVE`, or `XSAVE`-family instructions) so they are
  not lost and so the registers are free to be reused.

Two different lifecycle strategies exist for tying this to context switches:

1. **Eager-save-at-switch, lazy-restore-on-trap** (the traditional 4.4BSD /
   FreeBSD / DragonFly model): on every context switch away from a thread
   that owned the FPU, the kernel immediately saves that thread's state and
   sets `TS`. The *next* owner only gets its state loaded when it actually
   touches the FPU (`#NM` → DNA handler).
2. **Fully lazy / deferred-restore** (the model NetBSD moved to in the 2019
   rewrite, and what Linux and modern FreeBSD also effectively do): a
   context switch does **nothing** to the FPU hardware at all. It merely
   marks in the outgoing thread's software state whether its registers are
   still resident in hardware; the actual save is deferred until the kernel
   *needs* the registers (because a different thread is about to use the
   FPU, or the state must be read/written by the kernel). Restoration to
   the incoming thread is likewise deferred to the return-to-userspace path
   rather than being bound to the context-switch instruction stream at all.

The second model is strictly more general and is the crux of why the NetBSD
rewrite matters — see §3.

### 1.2 Kernel management of FPU state between kernel and userland

Each thread (LWP) needs a **save area** — a fixed or variable-size memory
block holding the architected FPU/SSE/AVX register image — that lives in the
thread's PCB (process control block) or an equivalent per-thread structure.
The kernel's bookkeeping problem is: *whose register image is currently
loaded in this CPU's hardware, if any, and is it stale or authoritative?*

Two classic pieces of state track this:

- A **per-CPU pointer** to the LWP/thread whose FPU state is currently
  resident in hardware on that CPU (e.g. NetBSD's old `ci_fpcurlwp`,
  DragonFly's `mdcpu->gd_npxthread`).
- A **per-thread flag** recording whether that thread's software copy in
  the PCB is currently authoritative, or whether the live copy is in some
  CPU's registers (e.g. NetBSD's old `pcb_fpcpu`, DragonFly's
  `TDF_USINGFP`).

Because a thread can be preempted, migrate CPUs, be examined via `ptrace`
while stopped on a *different* CPU, or be reaped while its FPU state is
still "checked out" to a CPU register file, the classic model needs a way to
forcibly reclaim FPU state that is sitting on a remote CPU — traditionally
an IPI (`fpusave_lwp()`/cross-CPU FPU IPI in old NetBSD; a similar mechanism
existed historically in FreeBSD too). This turns out to be one of the
biggest sources of complexity, and it is precisely what the NetBSD rewrite
eliminated (§3).

The kernel itself normally **does not use the FPU** for its own C code (the
compiler is told not to generate SSE/x87 code for kernel translation units).
When the kernel legitimately needs FPU/SIMD instructions — cryptography,
RAID6/XOR engines, and, per the DragonFly commit under review, GPU driver
code ported from Linux (`amdgpu`) — it must explicitly bracket that usage:
save whatever user thread's state might be resident, do the kernel-only FPU
work, then restore things so the interrupted user thread notices nothing.
This bracketing is exactly what `kernel_fpu_begin()`/`kernel_fpu_end()` (and
NetBSD's `fpu_kern_enter()`/`fpu_kern_leave()`) provide.

At the ABI boundary, userland floating-point state is exposed/injected via:
- `sigreturn`/signal delivery (`ucontext_t`/`mcontext_t` carries an FP
  save-area image, `fpu_sigreset`/`npxpush`/`npxpop` type functions).
- `ptrace(PT_GETFPREGS/PT_SETFPREGS)` and `/proc/.../regs` style interfaces
  used by debuggers.
- Core dumps (the FP save area is embedded in the NT_FPREGSET/NT_X86_XSTATE
  note).
- `fork()`/`clone()`, which must copy the parent's FPU image into the
  child's PCB.

### 1.3 XSAVE, XSAVEOPT, XSAVEC, XSAVES — mechanics and performance

`FXSAVE`/`FXRSTOR` (SSE2 era) save/restore a fixed 512-byte legacy area
(x87 + XMM0-15 + MXCSR) unconditionally, with no notion of "which parts
changed."

The `XSAVE` instruction family (introduced with AVX, extended since)
generalizes this to an open-ended, self-describing set of **state
components**, each independently enable-able:

- **`XCR0`** (accessed via `XSETBV`/`XGETBV`, requires `CR4.OSXSAVE=1`) is
  the *user*-visible enable mask: bit 0 = x87, bit 1 = SSE, bit 2 = AVX
  (YMM upper halves), bits 3-4 = MPX bounds regs (deprecated/removed on
  newer silicon), bit 5 = AVX-512 opmask (`k0-k7`), bit 6 = AVX-512
  `ZMM_Hi256` (upper 256 bits of `ZMM0-15`), bit 7 = AVX-512 `Hi16_ZMM`
  (registers `ZMM16-31`), bit 9 = PKRU. AVX-512's three bits (5,6,7) must be
  enabled as a group — enabling any one requires enabling all three.
- **`IA32_XSS`** is the parallel *supervisor*-only enable mask (processor
  trace state, CET shadow-stack state, HDC, etc.) usable only with
  `XSAVES`/`XRSTORS` from CPL0. This lets the kernel keep a smaller,
  privileged extended-state footprint that userland can't even address.
- **CPUID leaf `0x0D`** enumerates all of this: subleaf 0 reports which
  `XCR0` bits are supported and the required (non-compacted) save-area size
  in `ECX`; subleaf 1 reports which of `XSAVEOPT`/`XSAVEC`/`XGETBV1`/
  `XSAVES` are supported, plus the *compacted* save area size; subleaves
  ≥2 report the byte offset and size of each individual state component.

The instructions themselves, in increasing sophistication:

| Instruction | Layout | Key optimization |
|---|---|---|
| `XSAVE` | Fixed, standard offsets per CPUID (portable across implementations) | None — always writes every enabled component |
| `XSAVEOPT` | Same fixed layout as `XSAVE` | **Modified-optimization**: the CPU internally tracks, per state component, whether it is still equal to the *init* state established by the last `XRSTOR`. If untouched since restore, `XSAVEOPT` skips writing that component to memory entirely. |
| `XSAVEC` | **Compacted** — components are packed with no gaps for disabled/unused components | Modified-optimization *and* compaction — smaller footprint, ideal when only a couple of the many possible components (e.g. just x87+SSE) are actually in use by a given thread |
| `XSAVES`/`XRSTORS` | Compacted, privileged (CPL0-only) | Everything `XSAVEC` has, *plus* access to supervisor-only components gated by `IA32_XSS` |

Why this matters for performance: the AVX-512 XSAVE area can be roughly
2.5–2.7 KB per thread. Most threads never touch `ZMM16-31` or the opmask
registers at all. Plain `XSAVE`/`FXSAVE` still burns memory bandwidth and
cache footprint copying all of it (or, for `FXSAVE`, simply can't represent
it). `XSAVEOPT` turns "save state we never modified" into a near-no-op;
`XSAVEC`/`XSAVES` additionally shrink the *resident* memory footprint of
mostly-idle threads. A kernel scheduling thousands of context switches per
second benefits enormously from these — this is the direct performance
motivation behind Matt Dillon's remark that DragonFly's FP subsystem, which
predates broad `XSAVEOPT` usage in the tree, is not well optimized.

### 1.4 Extending to AVX-256 / AVX-512 and beyond

Practically, adding wider vector support to a kernel FPU subsystem means:

1. **Feature probing**: `CPUID.1:ECX.XSAVE[bit26]`/`.AVX[bit28]` for AVX;
   `CPUID.7,0:EBX.AVX2[bit5]`; `CPUID.7,0:EBX.AVX512F[bit16]` and its many
   sibling bits (`AVX512DQ`, `AVX512BW`, `AVX512VL`, …) for AVX-512
   sub-features. `CPUID.0xD` for XSAVE area sizing as above.
2. **Enabling**: set `CR4.OSXSAVE`, then `XSETBV(0, mask)` with the desired
   `XCR0` bits, on **every** logical CPU (this is per-core state, done once
   at boot per AP).
3. **Sizing the save area dynamically** rather than using a fixed `struct`
   — the kernel typically keeps a global `xsave_area_size` /
   `x86_fpu_save_size` computed once from `CPUID.0xD` (compacted size if
   `XSAVEC/XSAVES` are used) and allocates/`memcpy`s using that size, not
   `sizeof(some_struct)`.
4. **Choosing the best save/restore primitive available**, typically ranked
   `XSAVES > XSAVEC > XSAVEOPT > XSAVE > FXSAVE > FSAVE`, recorded once as
   an enum (this is exactly what DragonFly's `x86_fpu_save`/NetBSD's
   `x86_fpu_save` selects between).
5. **Extending the userland ABI**: `ucontext_t`/`mcontext_t` is a
   fixed-layout, backward-compatible structure that cannot simply grow to
   fit a 2.5 KB area without breaking existing binaries/signal trampolines.
   The standard solution (used by Linux, FreeBSD, and NetBSD alike) is a
   variable-length **extended state** trailer appended after the legacy
   FXSAVE-sized area, self-identified by a magic number and size field
   (NetBSD's `struct xstate`, with an `xsh_xstate_bv` bitmap mirroring the
   `XSAVE` header). `ptrace`/core-dump paths need a parallel
   `PT_GETXSTATE`/`NT_X86_XSTATE`-style interface.
6. **Validating untrusted input**: when a debugger or `sigreturn` supplies
   attacker-controlled FPU/xstate content, the kernel must sanity-check
   reserved bits and the `XSTATE_BV` header field before `XRSTOR`ing it —
   `XRSTOR` itself will `#GP` on some malformed inputs, but software
   validation (see NetBSD's `process_verify_xstate()`) is still needed to
   avoid propagating garbage or crashing in unexpected ways.
7. **Heterogeneous-core caveat**: while XCR0 is normally CPU-model-wide,
   real-world asymmetric designs (e.g. Intel hybrid P/E-core parts that
   disable AVX-512 on some cores) mean a portable kernel cannot assume every
   logical CPU in the system supports identical `XCR0` state — worth a
   defensive check even if DragonFly doesn't target such hardware today.

---

## Part 2 — Reviewing DragonFly's Current FPU Code

Only the diff you supplied (`dflybsd-kernel-fpu.diff`, Matt Dillon's Oct 2021
commit `1be00ff1`) is available to inspect directly; the surrounding
`npx.c`/`npxdna()`/`npxpush()`/`npxpop()` bodies are referenced but not
included in full. The analysis below is grounded in what the diff shows and
in the well-known lineage of this code (DragonFly's `npx.c` was forked from
FreeBSD's, which in turn is a descendant of the classic 4.4BSD/386BSD `npx`
driver — i.e., it is architecturally the **eager-save/lazy-restore,
per-CPU-ownership** model described in §1.1, the same family NetBSD's *old*
pre-2019 code belonged to).

### 2.1 Why `kernel_fpu_begin()`/`kernel_fpu_end()` are "really not well optimized"

Matt Dillon's own `XXX` comment sits directly above `kernel_fpu_begin()`:
*"really not well optimized, goes through a lot unnecessarily."* Reading the
implementation:

```c
void
kernel_fpu_begin(void)
{
	thread_t td = curthread;

	KASSERT((td->td_flags & TDF_KERNELFP) == 0, ...);
	atomic_set_int(&td->td_flags, TDF_KERNELFP);
	if (td->td_kfpuctx == NULL) {
		td->td_kfpuctx = kmalloc(sizeof(*td->td_kfpuctx), M_FPUCTX,
					 M_INTWAIT | M_ZERO | M_POWEROF2);
	}
	npxpush(td->td_kfpuctx);
	npxdna();
}
```

Several concrete inefficiencies are visible or implied:

- **It is built entirely out of the user-fault machinery, not a
  purpose-built kernel path.** `npxpush()`/`npxpop()` and `npxdna()` are
  designed to service `sigreturn`-adjacent state swaps and the `#NM` trap
  handler for *userland* threads. Calling `npxdna()` directly from
  `kernel_fpu_begin()` means every kernel FPU acquisition pays for whatever
  bookkeeping, `crit_enter()`/`crit_exit()` sections, and state-machine
  checks that trap handler was written to do for a much more general case
  (arbitrary user-mode DNA faults), not the narrow, single-purpose "give me
  the FPU right now, this thread, no faults involved" operation
  `kernel_fpu_begin()` actually needs.
- **It is unconditionally eager**, exactly the model criticized in §1.1: it
  *always* does a full save of whatever is live (`npxpush`) and then *always*
  does a full re-init/reload (`npxdna()` → `npxinit()`), regardless of
  whether the calling thread had touched the FPU at all since the last
  kernel_fpu bracket, and regardless of whether `XSAVEOPT`'s
  modified-optimization could have skipped most of the work. There is no
  fast path for "nothing needs saving because this thread's FPU state
  wasn't resident in hardware to begin with," which a lazily-tracked
  ownership model gets essentially for free.
- **Dynamic allocation on (potentially) every first use, with a blocking
  allocation flag.** `td_kfpuctx` is `kmalloc(..., M_INTWAIT | ...)` —
  `M_INTWAIT` permits the allocator to sleep/wait for memory, which is a
  latency and correctness hazard in any context that isn't guaranteed to be
  a fully preemptible kernel thread (it is *not* interrupt-safe). Combined
  with the fact that `td_kfpuctx` is sized as `sizeof(*td->td_kfpuctx)` —
  i.e., a fixed `mcontext_t`-based struct rather than the dynamically-sized,
  ideal-instruction-appropriate XSAVE area described in §1.4 — this
  allocation is neither minimal nor future-proofed for AVX-512-sized state.
- **Non-reentrant, coarse-grained locking via a single thread flag.** The
  `TDF_KERNELFP` flag plus `KASSERT` means `kernel_fpu_begin()` cannot
  nest, and its "protection" against concurrent modification is a
  same-thread flag check, not an actual light-weight critical section
  designed for the fast path — real preemption/interrupt safety appears to
  be delegated entirely to whatever `npxpush()`'s internal `crit_enter()`
  provides, which (again) is machinery designed for a broader, slower use
  case.
- **No distinction between "FPU state is trivially still ours" and
  "FPU state genuinely needs saving/restoring."** A well-optimized version
  would, like NetBSD's `fpu_kern_enter()`, simply check a lightweight flag
  (whether the FPU is currently resident for *any* owner) and skip the
  save/restore round-trip whenever it's already safe to just use the FPU
  (e.g., because nothing owns it, or because the calling thread already
  owns it) — instead the DragonFly version always executes the entire
  push+DNA-trap-emulation dance.

In short: functionally correct, but built by composing two heavyweight,
general-purpose user-fault primitives instead of writing a minimal
kernel-only save/restore that takes advantage of what is *already known*
(no untrusted user state involved, no signal delivery concerns, thread
identity known in advance) to skip work.

### 2.2 Why Matt Dillon says the entire FP subsystem needs a rewrite

Based on the API shape visible in the diff (`TDF_USINGFP`, `TDF_KERNELFP`,
per-CPU `mdcpu->gd_npxthread`, a `#NM`-trap-driven `npxdna()`, explicit
`npxpush()`/`npxpop()` for context save/restore around signal delivery),
DragonFly's `npx.c` is architecturally the same generation of design NetBSD
itself abandoned in 2019 — i.e., it predates the "deferred restore, no
cross-CPU stealing" model entirely. The likely (and historically confirmed,
based on the FreeBSD lineage) issues are:

- **Legacy eager/CPU-ownership design.** The save-at-switch,
  restore-on-trap model requires per-CPU "who owns the FPU right now"
  bookkeeping and, historically in this code family, a mechanism to forcibly
  reclaim a thread's FPU state if it's needed on a different CPU than the
  one it's resident on (e.g. a debugger reading `ptrace` FP registers of a
  stopped thread currently "checked out" elsewhere, or reaping an exiting
  thread). That reclaim path is exactly the kind of cross-CPU
  synchronization NetBSD found unnecessary complexity and removed outright
  (its old `fpusave_lwp()` IPI dance, §3). It is a correctness *and*
  performance liability: extra IPIs, extra spin-loops, extra invariants to
  maintain under SMP.
  Note: DragonFly's `npxdna()` snippet shown in the diff still references
  `td->td_flags & (TDF_USINGFP | TDF_KERNELFP)`-style gating typical of this
  older design, reinforcing that DragonFly has not yet adopted a deferred
  model.
- **No `XSAVEOPT`/`XSAVEC`/`XSAVES` awareness baked into the hot paths.**
  Even if `npxsave`/`npxrstor` internally dispatch on a saved-format enum
  the way FreeBSD's does, a design built around always-save/always-restore
  semantics forfeits most of the benefit `XSAVEOPT`'s modified-optimization
  offers, because that optimization pays off precisely when you *skip*
  saves for untouched threads — something an eager model structurally does
  less of, and something bolt-on kernel APIs like `kernel_fpu_begin()`
  (§2.1) defeat entirely by forcing a save/restore on every bracket.
- **No clean, minimal kernel-internal FPU API prior to this patch.** The
  very fact that `kernel_fpu_begin()`/`kernel_fpu_end()` had to be *added*
  in 2021, and had to be built by repurposing user-fault plumbing rather
  than calling into a small independent primitive, is itself evidence that
  the subsystem was never designed with in-kernel FPU consumers (crypto,
  RAID XOR engines, and now GPU drivers ported from Linux) in mind. A
  proper rewrite would design the low-level save/restore primitives first,
  and build *both* the user fault handler and the kernel API as thin
  wrappers over them — not derive the kernel API from the fault handler.
- **AVX-512 / wide-vector readiness.** As the person commissioning this
  spec already suspects, there's no indication in the visible DragonFly
  code of a CPUID-0xD-driven, dynamically-sized save area, or of
  `XSAVEC`/`XSAVES` support. A fixed-size `union savefpu`/`mcontext_t`
  approach is a natural outgrowth of the FXSAVE era and doesn't extend
  cleanly to variable-size, compactible AVX-512 state.
- **General maintenance burden.** Matt Dillon's phrasing ("has a lot of
  legacy burden... not well maintained... not well optimized") matches a
  codebase that has been patched incrementally to add features (like this
  `kernel_fpu_begin`) rather than redesigned around modern XSAVE
  capabilities and a deferred-restore model — technical debt accreting
  around 1990s-2000s assumptions about how expensive `CR0` reloads and FPU
  traps were relative to today's hardware and workloads (GPU drivers doing
  FP math in interrupt-adjacent contexts, for instance).

---

## Part 3 — Is Porting NetBSD's FPU Rewrite the Right Path?

### 3.1 What the NetBSD rewrite actually changed (mechanically)

Reading `netbsd-rewrite-x86-fpu.patch` end to end, the core conceptual shift
is:

- **Remove per-CPU FPU ownership tracking entirely** (`ci_fpcurlwp`,
  `pcb_fpcpu` are deleted). There is no longer a notion of "which CPU holds
  lwp X's FPU state" because state is never left resident across a context
  switch in the first place, on the outgoing side.
- **Context switch always saves-and-clears the outgoing thread's FPU state
  if it was resident** (`fpu_switch()`), and — critically — **never eagerly
  restores the incoming thread's state**. Instead it just asserts the
  incoming thread's `MDL_FPU_IN_CPU` flag is clear.
- **Restoration is deferred to the return-to-userspace path**, via a new
  `HANDLE_DEFERRED_FPU` assembly macro inlined into every trap/syscall/
  interrupt return path (`amd64_trap.S`, `locore.S`, `spl.S`, `i386_trap.S`,
  `i386/locore.S`). It's a single branch: if `MDL_FPU_IN_CPU` is already
  set, do nothing; otherwise call `fpu_handle_deferred()` (an `XRSTOR`) once
  and set the flag. This means the (historically expensive) `CR0` reload
  and FPU register reload that used to happen unconditionally on *every*
  context switch back into `cpu_switchto`'s assembly is now done **at most
  once per return to userspace**, and skipped entirely for kernel-only
  excursions (interrupts, most syscalls) that never re-enter userland FP
  code in between.
- **`fpudna()` (the actual `#NM` trap handler) becomes a `panic()`.** This
  is the strongest evidence of how thorough the redesign is: because
  restoration is now unconditionally handled by `HANDLE_DEFERRED_FPU` before
  userland code can execute *any* instruction, the CPU should structurally
  never be able to generate a genuine `#NM` fault for userland FP use again
  — if it does, that indicates the flag bookkeeping itself is broken, which
  is a bug worth panicking over rather than silently "handling."
- **Cross-CPU FPU stealing removed outright.** `fpusave_lwp()`'s IPI-based
  "steal this LWP's FPU state from whatever CPU it's resident on" is gone
  completely; the IPI handlers (`x86_ipi_synch_fpu`, `xen_ipi_synch_fpu`)
  are turned into `panic("impossible")`. This is possible only *because*
  FPU state is never left resident across a context switch anymore — it's
  either in the PCB (always true for a non-running thread) or, for the
  currently-running thread on the current CPU, directly inspectable by that
  CPU without needing an IPI.
- **A genuinely reentrant/lightweight kernel API falls out almost for free**
  (`netbsd-add-in-kernel-fpu-api.patch`): `fpu_kern_enter()`/
  `fpu_kern_leave()` just raise IPL to block preemption/interrupts briefly,
  call the *existing* `fpu_save_lwp()` primitive (the same one the rest of
  the subsystem already uses — no bespoke, redundant fault-handler
  path), and rely on `HANDLE_DEFERRED_FPU` to lazily restore userland state
  on the way back out, whenever that happens. There's no dedicated
  "kernel FPU context" allocation at all — the kernel simply
  borrows the registers and lets the existing deferred-restore machinery
  put things back later at zero extra engineering cost.

This directly validates Matt Dillon's assessment: the reason DragonFly's
`kernel_fpu_begin()`/`end()` "goes through a lot unnecessarily" is that it's
retrofitted onto an eager-restore architecture that has no cheap notion of
"FPU currently unowned, just take it"; NetBSD's rewrite produces exactly
that cheap primitive as a side effect of the larger architectural change,
not as a special case bolted on top.

### 3.2 Is it the right path for DragonFly? Yes, as a *design* — not as a literal diff port

**The lazy-deferred-restore model itself is sound and worth adopting.**
It's the same fundamental approach used by Linux (`TIF_NEED_FPU_LOAD` +
restore-on-return-to-userspace since ~4.20/5.x "lazy FPU restore" work) and
modern FreeBSD (`PCB_FPUNOSAVE`/`PCB_FPUINITDONE` deferred-restore changes
post-dating DragonFly's fork point). It eliminates an entire category of
SMP synchronization code (cross-CPU FPU IPIs), reduces context-switch-path
`CR0` traffic to near zero, and produces a natural, cheap in-kernel FPU
API — precisely the three things this analysis identified as DragonFly's
pain points.

**However, this is not a drop-in patch — it's a design to re-derive inside
DragonFly's own machine-dependent layer.** Several structural mismatches
mean the NetBSD diff cannot be mechanically applied:

1. **Different threading model.** NetBSD's rewrite is expressed in terms of
   `struct lwp`/`l_md.md_flags`/`splhigh()`/IPL levels. DragonFly's
   scheduler is LWKT-based: threads (`struct thread`, `td_flags`), critical
   sections (`crit_enter()`/`crit_exit()`) instead of `spl*()`, and its own
   IPI queue (`lwkt_send_ipiq`) instead of x86 IPI vectors wired directly
   into `ipi.c`. Every place NetBSD used `splhigh()`/`IPL_HIGH` to protect
   FPU flag transitions needs a DragonFly-native equivalent
   (`crit_enter()` plus, likely, `atomic_*` ops on `td_flags` as the
   existing `kernel_fpu_begin()` already does).
2. **Different assembly entry/exit paths.** `HANDLE_DEFERRED_FPU` has to be
   inlined into *every* return-to-userspace path. In NetBSD that's
   `amd64_trap.S`, `locore.S`'s `cpu_switchto`/`handle_syscall`, and
   `spl.S`'s `doreti_checkast`. DragonFly's equivalents live in
   `sys/platform/pc64/x86_64/exception.S`, `cpu_switchto/cpu_heavy_restore`
   in `swtch.s`, and its own `doreti`-equivalent — these are different
   files with different calling conventions, register usage, and DragonFly-
   specific concepts (kernel/user trampolines, `lwkt_switch()` semantics)
   that must each be located, understood, and modified independently. This
   is the highest-risk part of the port: bugs here manifest as silent user
   FP-state corruption or SMP-only crashes, and it must be gotten right on
   every entry point, not just the common ones.
3. **`mcontext_t`/PCB layout differences.** DragonFly's `td_kfpuctx` is
   typed `mcontext_t *` and its FPU save area shape (`union savefpu`,
   `npxpush`/`npxpop` operating on `mcontext_t`) is not identical to
   NetBSD's `struct pcb.pcb_savefpu`/`union savefpu`. Field-for-field
   mapping is required, and DragonFly's variable-size AVX-512 story (if any
   exists yet) needs to be established rather than assumed identical.
4. **No i386 (32-bit) target.** DragonFly is 64-bit-only, so roughly a
   third of the NetBSD patch (`i386_trap.S`, `i386/locore.S`,
   `i386/machdep.c`, `i386/pcb.h`, `i386/proc.h`, `i386/frameasm.h`) is
   simply inapplicable — a simplification in DragonFly's favor.
5. **No Xen PV target.** NetBSD's patch carries Xen-specific hooks
   (`HYPERVISOR_fpu_taskswitch`, `xen_ipi_synch_fpu`,
   `sys/arch/xen/x86/*`). DragonFly does not support Xen PV guests the way
   NetBSD does, so this portion should simply be dropped rather than
   ported — but it's a reminder to grep DragonFly's tree for any
   hypervisor-specific FPU hooks (e.g. VMware/HyperV paravirt CPUID quirks)
   before assuming a clean removal.
6. **No NVMM equivalent, but DragonFly *does* have virtualization/vkernel
   code.** NetBSD's patch also updates `nvmm_x86_svm.c`/`nvmm_x86_vmx.c` to
   use the new `fpu_save()` primitive for guest FPU state swapping.
   DragonFly doesn't have NVMM, but it does have its own vkernel
   (`sys/platform/vkernel64`) and, if present, any hardware-virtualization
   driver — these are exactly the kind of consumer that needs to be
   re-audited for correct interaction with the new deferred-restore model,
   the same way NetBSD had to touch its VMM backends.
7. **ACPI suspend/resume and other `fpusave_cpu()`-style call sites.**
   NetBSD's `acpi_wakeup.c` simply switched from `fpusave_cpu(true)` to
   `fpu_save()`. DragonFly will have its own ACPI suspend/resume FPU
   save call(s) that need the equivalent substitution — these are easy to
   miss because they're outside `npx.c` proper.
8. **ptrace/procfs/core-dump FP register accessors.** NetBSD's
   `process_read_fpregs_xmm()`/`process_write_xstate()`/etc. all changed
   from the old "IPI-steal-then-read" pattern (`fpusave_lwp(l, true)`) to
   the simple `fpu_lwp_area(l)` (which internally calls `fpu_save()` only
   if `l == curlwp`, and otherwise just trusts the PCB because a
   *non-running* stopped thread's FPU state is, by construction under the
   new model, always already resident in its PCB, never "checked out" on
   some other CPU). DragonFly's analogous `procfs`/`ptrace` FP-register
   code needs the same simplification, and this is a good correctness
   canary during testing: it's the code path most likely to reveal a bug in
   the deferred-restore flag transitions.
9. **Testing surface is unusually large and unusually dangerous.** This
   code sits on *every* context switch and *every* return to userspace on
   *every* CPU. A subtle bug doesn't crash reliably — it silently corrupts
   floating-point results in some unrelated, unlucky userland process,
   which is exactly the failure mode hardest to bisect. This argues for an
   intentionally staged, heavily-instrumented rollout (see §4, step 9)
   rather than a single big-bang commit — even though NetBSD itself landed
   it as one commit.

### 3.3 Verdict

**Porting the *design* (deferred/lazy restore-on-return-to-userspace,
elimination of cross-CPU FPU ownership stealing, a minimal kernel API
derived from the same primitives as the rest of the subsystem, and a
dynamically-sized `XSAVEC`/`XSAVES`-aware save area) is the right move**,
and directly addresses both of Matt Dillon's stated complaints
(`kernel_fpu_begin`'s inefficiency and the subsystem's general legacy
burden) in one coherent redesign. **Porting the NetBSD *patch* verbatim is
not feasible** — the assembly integration points, threading primitives, and
PCB layout are all DragonFly-specific and must be re-derived, not
transplanted. Treat the NetBSD commit as an excellent reference
implementation and design document, not as a diff to `patch -p1` against
DragonFly's tree.

---

## Part 4 — A Concrete Porting Guide

### Step 0 — Reconnaissance (before writing any code)

- Read the *entirety* of `sys/platform/pc64/x86_64/npx.c`, not just the
  hunks visible in the supplied diff — in particular `npxinit()`,
  `npxsave()`/`npxrstor()` (or equivalents), `npxdna()`, `npxpush()`,
  `npxpop()`, and whatever selects between `FXSAVE`/`XSAVE`/`XSAVEOPT`
  today (`x86_fpu_save`-equivalent, `npxprobemask()` referenced in the
  diff).
- Grep the whole tree for every call site of `npxpush`, `npxpop`,
  `npxdna`, `npxinit`, `gd_npxthread`, `TDF_USINGFP`, `TDF_KERNELFP`, and
  `pcb_save`/`mcontext_t` FPU fields. Build an inventory: `cpu_fork()`
  equivalent (thread/PCB fork), `sendsig()`/`sigreturn()`, `ptrace`
  `PT_GETFPREGS`/`PT_SETFPREGS`, `procfs` regs, core-dump note writer,
  ACPI suspend/resume, and any `vkernel64` platform code.
- Identify DragonFly's return-to-userspace assembly: the syscall return
  path, trap return path, and `lwkt_switch()`/`cpu_heavy_restore` context
  switch path (in `sys/platform/pc64/x86_64/exception.S` and
  `sys/platform/pc64/x86_64/swtch.s` or equivalents). These are the files
  where `HANDLE_DEFERRED_FPU`-equivalent logic must be inlined.
- Confirm what `kernel_fpu_begin()`'s current callers actually need (check
  the `amdgpu`-derived Linux compat code) so the new API's contract
  (reentrant? interrupt-safe? softint-safe?) is scoped correctly rather
  than guessed.

### Step 1 — Define the new state model

- Add a per-thread flag analogous to `MDL_FPU_IN_CPU`, e.g.
  `TDF_FPU_IN_CPU`, alongside the existing `TDF_USINGFP` (or replacing it,
  once the semantics are unified — decide whether "using FP" and "FP
  resident in hardware" should remain distinct as they were before, or
  collapse into one flag as NetBSD did).
- Remove (or plan the removal of) `mdcpu->gd_npxthread`-style per-CPU
  ownership tracking; it should become unnecessary once restore is
  deferred and cross-CPU stealing is gone.

### Step 2 — Rewrite the low-level save/restore primitives

- Implement `fpu_area_save()`/`fpu_area_restore()`-equivalents that
  dispatch on the already-existing (or newly added) `x86_fpu_save`-style
  enum, and, importantly, size their buffer from a dynamically computed
  `x86_fpu_save_size` (from `CPUID.0xD`) rather than a fixed struct —
  this is the point at which AVX-512 readiness is actually earned, not the
  assembly-integration parts.
- Prefer `XSAVEOPT` at minimum; evaluate `XSAVEC`/`XSAVES` support and
  gate on CPUID feature bits, matching NetBSD's fallback chain.

### Step 3 — Rewrite `fpu_switch()`/context-switch integration

- On switch-out: if the outgoing thread's state is `TDF_FPU_IN_CPU`, save
  it and clear the flag (this is the *only* FPU-related work a context
  switch should do).
- On switch-in: do **nothing** to the FPU. Do **not** touch `CR0`/`TS`
  speculatively for the incoming thread inside the switch path — this is
  the change most likely to be "obviously correct in theory, subtly wrong
  in DragonFly's specific assembly," so budget real review time here.

### Step 4 — Add the deferred-restore check to every return path

- Implement `fpu_handle_deferred()` (an `XRSTOR`/equivalent of the target
  thread's saved state) and a macro/inline routine equivalent to
  `HANDLE_DEFERRED_FPU`, then thread it into DragonFly's syscall-return,
  trap-return, and (if applicable) `doreti`-style interrupt-return paths.
- Turn the old `#NM`/DNA trap handler into a `panic()` once this is
  verified working, exactly as NetBSD did — it's a strong, cheap
  correctness assertion for the rest of the subsystem's lifetime.

### Step 5 — Rebuild `kernel_fpu_begin()`/`kernel_fpu_end()` on the new primitives

- With deferred restore in place, `kernel_fpu_begin()` degenerates to:
  briefly raise a critical section, save current-thread FPU state if
  resident (reusing the *same* `fpu_area_save()` the rest of the subsystem
  uses — no bespoke DNA-trap-emulation call), then let the caller use the
  FPU; `kernel_fpu_end()` just lowers the critical section and lets
  `HANDLE_DEFERRED_FPU` lazily restore userland state whenever control
  actually returns to userspace. No `kmalloc()` in the hot path should be
  necessary if `td_kfpuctx`-equivalent storage is embedded in the thread
  structure or allocated once at thread creation instead of lazily with
  `M_INTWAIT`.
- Decide whether kernel-FPU usage should be nestable/reentrant; NetBSD's
  version is not (single `ci_kfpu_spl`), matching DragonFly's existing
  non-reentrant `KASSERT`. If GPU driver code needs nesting, that's a
  deliberate scope decision to make explicitly, not an accident.

### Step 6 — Update all downstream consumers found in Step 0

- `ptrace`/procfs FP-register get/set, core dump writer, `sendsig()`/
  `sigreturn()`, `fork()`/thread-create FPU-state copy, ACPI
  suspend/resume, and vkernel platform code. Each should simplify (remove
  IPI-based stealing, if any exists) the same way NetBSD's did.

### Step 7 — Extend the userland xstate ABI for AVX-256/512

- Introduce a variable-length extended-state trailer for
  `ucontext_t`/`mcontext_t` and `ptrace`/core-dump interfaces, gated by a
  magic/size header, following the pattern in §1.4. Add
  `process_verify_xstate()`-equivalent validation before any `XRSTOR` of
  user-supplied data.
- Confirm `XCR0` enabling (`CR4.OSXSAVE`, `XSETBV`) happens correctly on
  every AP at boot, and that the AVX-512 opmask/`ZMM_Hi256`/`Hi16_ZMM`
  triad is enabled atomically as a group when present.

### Step 8 — Testing plan (do this before merging, not after)

- **Uniprocessor sanity**: boot, run an FP-heavy userland workload
  (`libm` test suite, or simply anything linked against a math-heavy
  library), confirm correctness.
- **SMP stress**: many threads doing FP work concurrently with frequent
  preemption (e.g. a synthetic benchmark pinning threads to alternating
  CPUs) to shake out any remaining cross-CPU assumption.
- **Debugger interaction**: `ptrace`-attach to a stopped, FP-active
  process from another thread/CPU and read/write FP registers; this
  exercises exactly the code path NetBSD simplified and is the highest-
  value regression test for the new model.
- **Core dumps**: verify FP/xstate notes are correct post-crash for a
  process that was actively using AVX registers.
- **Signal delivery**: FP state across `sigreturn` for a handler that
  itself uses FP.
- **`kernel_fpu_begin()`/`end()` consumer**: exercise the `amdgpu`-derived
  code path (or a synthetic stand-in) under load, confirming no user FP
  corruption and measuring the expected latency improvement versus the
  old implementation.
- **Suspend/resume**: ACPI S3 cycle with FP-active processes.
- **fork/exec**: verify FP state inheritance semantics match the old
  behavior (typically: child inherits parent's saved FP image; `exec`
  resets to the default control word).

### Step 9 — Rollout strategy

- Land the new save/restore primitives and the dynamically-sized xstate
  area first, behind the *existing* eager model (i.e., keep today's
  `npxdna()`/`gd_npxthread` scheme working, just re-plumbed onto the new
  primitives) — this validates the AVX-512-readiness work independently
  of the riskier deferred-restore/assembly-integration change.
- Land the deferred-restore/assembly-integration change as a second,
  clearly separated commit (or behind a boot-time tunable if the platform
  supports easily toggling it) so it can be bisected in isolation if a
  regression surfaces.
- Given how central this code is, coordinate directly with Matt Dillon
  (who flagged the need for this rewrite himself) before landing the
  assembly-integration step, and expect several rounds of SMP-hardware
  testing beyond a single developer's machine before merging to `master`.

---

## Summary

- DragonFly's FPU subsystem is architecturally a member of the classic
  "eager save-at-switch / lazy restore-on-trap, per-CPU ownership" family —
  the same generation of design NetBSD itself replaced in 2019.
- `kernel_fpu_begin()`/`kernel_fpu_end()` are inefficient specifically
  because they are built by composing two heavyweight, user-fault-oriented
  primitives (`npxpush()` + `npxdna()`) rather than a minimal, purpose-built
  kernel-only save/restore, and because they always do full eager
  save+restore work with no fast path.
- The deeper "needs a rewrite" complaint points at the same root cause:
  legacy eager/ownership-tracking design, no `XSAVEOPT`/`XSAVEC`/`XSAVES`
  awareness, no AVX-512-ready dynamically-sized save area, and no
  minimal in-kernel API designed in from the start.
- NetBSD's rewrite is a strong reference design — deferred restore-on-
  return-to-userspace, no cross-CPU FPU stealing, a kernel API derived from
  the same primitives as everything else — and adopting that *design* is
  the right direction. The *patch itself* cannot be ported mechanically
  because of DragonFly's different threading primitives (LWKT vs. NetBSD's
  spl/IPL model), different assembly entry/exit points, and different
  PCB/mcontext layout; it should be used as a specification to re-implement
  against DragonFly's own machine-dependent layer, landed in carefully
  separated, heavily tested stages given how central and easy-to-silently-
  corrupt this code is.