Re: Testing needed - HAMMER PFS exports via NFS

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:...
:> - `indefinite wait!' message is displayed when spin_lock_wr() is called
:>   from ata_start(), to lock state_mtx
:> - ata_interrupt takes too long while it's holding state_mtx when it calls
:>   ch->hw.end_transaction. digging further, it's callout_stop() right
:>   below the `end_finished' label in ata_end_transaction() that is taking
:>   long time (> sys_cputimer->freq)
:> 
:> I'll try diving into callout_stop() later.
:
:It turned out that the while loop in lwkt_wait_ipiq() performs so many
:iterations (18000000~) in such a case.  It's called from callout_stop()
:when the caller's CPU isn't the same as that of the caller of callout_reset(),
:in which case callout_stop() must issue a synchronous IPI.
:
:lwkt_wait_ipiq():
:				       :
:            cpu_enable_intr();
:            while ((int)(ip->ip_xindex - seq) < 0) {
:                crit_enter();
:                lwkt_process_ipiq();
:                crit_exit();
:				       :
:	    }
:
:Since the only caller of lwkt_wait_ipiq() is callout_stop(), I don't know
:if the number is extreme or not, but usually the number of iterations in
:this loop stays very low, and it doesn't seem to reach 5000 during
:`make -j2 buildworld' on my Athlon64x2 box.  The number depends on the
:hardware, of course, but even on VMWare, the upper bound of the iterations
:in the normal case is hundred times lower than the abnormal case.
:Assuming that the values returned by sys_cputimer->count() are not skewed,
:it takes about one second (that is, slightly above sys_cputimer->freq)
:to return from lwkt_wait_ipiq(), when `indefinite wait!' message is shown.

    FreeBSD has been struggling with callout races for the last year.
    I did callout_stop() that way because I wanted to ensure that
    upon the return from callout_stop(), the callout is guaranteed to
    have been removed and its dispatch function is guaranteed to have
    completed if it was running.

    We can probably use spinlocks there to some degree.  Most callouts
    occur on the same cpu (because the protocol threads are already
    localized), so we should not have to worry too much about the performance
    of the occassional cross-cpu callout.

    If you would like to change it, I suggest using a spinlock to
    protect the structure and then something else... perhaps even
    allow a blocking condition, if the target callout is in the middle
    of running its callout function and we have to wait for it to return
    before we can return ourselves.

    (Currently callout_stop() is considered to be non-blocking so some
    code auditing may be needed.  The synchronous IPI is just as dangerous,
    though).

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>