Userland 1:1 threading roadmap

["Simon 'corecode' Schubert" <corecode@xxxxxxxxxxxx>]

What this is about, why and how we do it.

At the moment we still use FreeBSD-4's libc_r for threading which is
POSIX compliant (I think) but implemented as 1:N, i.e. all threads of
one process run in the same process *with the same kernel thread*.
Userland does all scheduling of threads itself.  This also means that we
can't use MP for threaded programs:  All threads run on the same CPU.

The plan for DragonFly is to switch to a 1:1 threading model.
Libthread_xu already can do this, but it lacks kernel support mainly for
signals.  I've taken this oportunity to restructure the kernel to make
it directly aware of threading.  This will work in several steps as
outlined below.

The idea is the following.  We want one kernel thread (scheduling
entity) per userland thread (1:1).  While this can be done with pure
procs alone, it quickly gets messy, because all userland threads
belonging to one process share lots of state.  We keep struct proc as
the entity which is viewed as the "process" by the user:  it has a pid,
a virtual memory space, filedescriptors, a text-vnode (the binary) and
so on.  The new thing is that we move out all stuff which is in fact
per-(userland)- thread:  execution and scheduling control (lwkt),
signals pending/blocked, statistics.  Now we start (userland-)
scheduling lwps instead of procs and thus we can run multiple threads of
one process on different CPUs.

Furthermore this will help us doing MP-locking the data for processes:
All thread-local data is in struct lwp and owned by the executing cpu.
All (shared) process data is in struct proc and needs some sort of
interlocking.  But that's future, at the moment we still run under BGL.

I hope I described the concept well enough.  Now on to the roadmap of
implementation (mostly layed out by Matt Dillon):


1.
Split thread-local members of struct proc out into struct lwp.  Embed
one struct lwp in struct proc and set up preprocessor linkage.

This doesn't change semantics for the rest of the kernel:  There is one
lwp per proc and all fields of that one lwp can be accessed via macros
that indirect to the lwp:  p_sigmask is p_lwp.lwp_sigmask.

This has already been done by me.


2.
Change functions which take a struct proc as parameter but instead
should take a struct lwp, because they actually deal with the thread
state and not the process (this will be the vast majority) to actually
take a struct lwp.  This will have to be done in chains, i.e. multiple
functions could need to be changed at once.  Patches still need to be
small so that they can be easily reviewed and bugs can be spotted.
Overall this is mostly mechanical.  Also uses of td_proc which really
mean to access thread-local data need to be changed to use td_lwp.

We still will have just one lwp embedded in struct proc, but we will try
and avoid accessing it via curproc->p_lwp, but rather via
curthread->td_lwp (maybe we want to introduce a curlwp macro).

This will change the function signatures but not the location of the
data they are actually accessing!

This will be split in many small sub-steps.  Every helping hand is
appreciated!  I will coordinate efforts.


3.
Removal of the p_* compatibility macros and use of the embedded p_lwp.
At this point all kernel accesses thread-local data via a passed struct
lwp / thread->td_lwp and process data via a passed struct proc /
thread->td_proc.  Thus we have made thread-local data access explicit.

We still have just one lwp per proc, but the kernel isn't aware of this
fact anymore.


4.
At this stage we will allocate lwps directly and will remove the
embedded p_lwp.  We will implement neccessary kernel logic for multiple
lwps per process.  This means getting forking, exiting and signals
working correctly with processes which have multiple threads.  Once this
is done we have a working 1:1 threading.  This work will require some
thinking but actually not much code.


(5.)
Getting locking right before we can remove BGL (farer away).


If you have any questions, feel free to ask.  And as I said before,
feedback and helping hands are appreciated (and expected!).

cheers simon

--
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low $$$ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \

Attachment: PGP.sig
Description: This is a digitally signed message part