Re: scheduler

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:Hi !
:
:Since I'm very fond of the responsiveness effects
:of ULE in fbsd-current, I'd like to know, how this
:very responsiveness is supposed to be achieved
:in dragonfly via userland schedulers.
:As I understand it, the kernel-level thread-scheduling
:is not to be touched at all including time-slices.
:Also I understand that preemptive cpu-moves are against
:essential dragonfly design because of locking issues.
:(Which makes perfect sense for me, though I'm just an
:  interested bystander/sysadmin.)
:I don't expect any complete explanations, just some pointers
:to relevant email-threads/documentation/ideas/whatever.
:
:I hope I don't sound too demanding...
:Just asking.
:
:Cheers
:Peter

    Well, there are two things going on here... there's the
    userland scheduler and there is the LWKT scheduler.  The
    LWKT scheduler is responsible for actually running a thread.
    It uses a fixed priority scheme but you have to keep in mind
    that the fixed priorities are differentiating major subsystems,
    not user processes.

    For example, hardware interrupt threads have the highest priority,
    followed by software interrupts, kernel-only threads, then 
    finally user threads.  A user thread either runs at user-kernel priority
    (when it is actually running in the kernel, e.g. running a syscall on
    behalf of userland), or a user thread runs at user priority.

    The user scheduler uses the dynamic priority mechanism from FreeBSD-4.x.
    It schedules *exactly* *one* user-runnable thread on each cpu in the
    system at a time, and it is responsible for dealing with the timeslice
    and forcing a reschedule to reschedule the next user-runnable thread.

    What this means is that from the point of view of a user process,
    preemption and dynamic priorities works as it always has.  What has
    changed is that preempt and dynamic priorities *ONLY* apply to threads
    related to user processes now and are not used to schedule kernel-specific
    activities such as interrupts.  In FreeBSD-5 the same ULE mechanism is
    use to schedule both user and kernel activities which imposes a great
    deal of unnecessary overhead IMHO.

    This also means that we do not have priority inversion based deadlocks,
    because any thread that is in the kernel (whether working on behalf of
    a user process, e.g. running a system call, or not), runs at a fixed
    kernel priority which is higher then the priority we use when running a
    process's thread that is currently in userland.

    Be sure not to confuse user priority based time slicing with preemption.
    A program running in user mode can be 'preempted' just fine.  It's when
    a thread is running in the kernel that preemption is either not allowed
    or follows careful rules.

    DragonFly does preempt... it just does it very carefully and only
    under particular circumstances.  An LWKT interrupt thread can preempt 
    most other threads, for example.  This mimics what FreeBSD-4.x already
    did with its spl/run-interrupt-in-context-of-current-process mechanism.
    What DragonFly does *NOT* do is allow a non-interrupt thread to preempt
    another non-interrupt thread.  FreeBSD-5 uses priority borrowing to
    allow virtually any thread to preempt virtually any other thread, either
    directly or indirectly.  This is a huge mistake IMHO which allows for
    very sloppy interrupt thread programming.  FreeBSD-5 will also preemptively
    switch a thread to another cpu.  DragonFly never preemptively switches a
    thread to another cpu if the thread is running in the kernel.

						-Matt