DragonFly kernel List (threaded) for 2004-01
An interesting thread (search for '[CHECKER]' in freebsd-hackers led
me to review our use of M_NOWAIT in malloc().
In a nutshell, most of the M_NOWAIT uses are incorrect. In particular,
much of the CAM code runs from an interrupt or a software interrupt
and almost universally uses M_NOWAIT for its allocations, with dire
consequences if/when allocations fail.
I think I've come up with a solution! The reason M_NOWAIT is used is
primarily because an interrupt thread may be preempting a normal thread
and cannot safely manipulate 'cache' pages in the VM page queues. This
is because reusing a cache page requires messing around with the
VM Object the cache page resides in. So M_NOWAIT causes kmem_malloc()
to only pull pages out of the VM 'free' page queue. At the same time
it is allowed to actually exhaust the free page queue whereas normal
allocations are not allowed to completely exhaust the free page queue.
But in DragonFly it is utterly trivial to (A) test to see if we are
preempting someone and (B) to switch away and when we get scheduled again
we will no longer be preempting anyone and can use a more exhaustive
This means that in DragonFly we do not have to key interrupt-time
allocations off of M_NOWAIT. Instead we can simply test to see if we
are preempting someone and if we are only allocate from the VM page
'free' queue... and if that allocation fails all we need to do is
lwkt_switch() away, and when we get cpu back we will no longer be
preempting anyone and can safely access the other queues.
This in turn (whew) means that with very few changes to the low level
kmem_malloc() code we can allow interrupt code to use M_WAITOK for
something like 90% of the cases where it currently uses M_NOWAIT.
I am going to add a couple of new M_ flags and start 'fixing' all the
improper M_NOWAIT useages to something a whole lot safer.