DragonFly kernel List (threaded) for 2005-11
Re: Question about userland scheduler
:Matt, could you please explain yet another time how usched interacts with
:LWKT scheduler. I've read your posts in kernel@ from Sat, 15 Nov 2003 and
:Tue, 7 Dec 2004 and still don't get it :(
:> The userland scheduler is responsible ONLY for scheduling threads that
:> are running in userland. The userland scheduler schedules only one such
:> thread at a time, and always at e.g. TDPRI_USER_NORM. It does not
:> schedule multiple threads (running in userland) with LWKT at the same
:What means 'only one such thread at a time' and how about other userland
:In the situation when we have to schedule TDPRI_USER_NORM thread
:(useland thread) it will be choosen by LWKT scheduler according to round-
:robin discipline. So, why we need the userland scheduler in this situation?
:I know that I'm wrong, so please show me the situation where the
:userland scheduler is used and explain how it is used on top of LWKT
:And the final question for now: what is gd->gd_uschedcp? Is this a pointer
:to the current running process?
The LWKT scheduler is a fixed priority scheduler. This is suitable for
well prioritized kernel threads but not really suitable for userland
threads. The userland scheduler is responsible for scheduling processes
running in or about to return to user mode. It does this by scheduling
the process's thread via the LWKT scheduler at near the lowest LWKT priority
(so all kernel threads will have priority over any user process thread
running in userland). But since the userland scheduler only has a handful
(3) of LWKT priorities to play with it can really only LWKT-schedule one
runnable user process at a time (per cpu) to get the desired behavior.
The biggest issue with the userland scheduler right now is the fact that
the userland process run queue and related structures are global rather
then per-cpu. I need to rewrite the whole thing, actually, and I intend to
after the release to finally make that path MP safe. I would prefer to
make the userland scheduler per-cpu, with process threads clearly assigned
to a particular cpu, and then have the per-cpu schedulers proactively 'push'
processes to other cpus based on a heuristic. This will not only give us
excellent locality of reference, it will also allow me to implement a
secondary 'desired' cpu for userland processes so we don't bounce them across
all the cpus on a SMP system (blowing up all the caches in the process).
Theoretically we could make the userland scheduler LWKT-schedule all runnable
threads and then bump the LWKT priority of the one it really wants to run
the most. In fact, I like this idea better then the current implementation.
It wasn't done this way initially because the current implementation uses a
global mutex (the Big Giant Lock) and 'pulls' runnable processes out of
the user scheduler's run queue onto an available cpu and a process's thread
can't be pulled to another cpu if it's already scheduled. When I rewrite
the scheduler into a push system it will be able to just leave all the
user process threads LWKT scheduled and simply fiddle with the LWKT priority.