Serializing tokens problem

[Richard Nyberg <rnyberg@xxxxxxxx>]

Hello gang!

I'm trying to use tokens to serialize access and updates to
a hash containing nnpfs nodes. As a start I put the  hash lookup
routine between token acquisition and release. All code using
the tokens are below.

Actually I have one token per list in the hash, probably overkill :)

void
nnpfs_init_head(struct nnpfs_nodelist_head *head)
{
    int i;

    for (i = 0; i < XN_HASHSIZE; i++) {
	NNPQUEUE_INIT(&head->nh_nodelist[i]);
	lwkt_token_init(&head->nh_tokens[i]);
    }
}

struct nnpfs_node *
nnpfs_node_find(struct nnpfs_nodelist_head *head, nnpfs_handle *handlep)
{
    struct nh_node_list *h;
    struct nnpfs_node *nn;

    lwkt_tokref ilock;

    lwkt_gettoken(&ilock, &head->nh_tokens[nnpfs_hash(handlep)]);
    h = &head->nh_nodelist[nnpfs_hash(handlep)];

    NNPQUEUE_FOREACH(nn, h, nn_hash) {
	if (nnpfs_handle_eq(handlep, &nn->handle))
	    break;
    }
    lwkt_reltoken(&ilock);
    return nn;
}

I tried to do the same thing as ufs_ihashlookup but obviously
there's something I'm missing because this soon gives the following
panic (backtrace below):
panic: assertion: tok->t_cpu == gd in lwkt_reltoken

What am I doing wrong?

Summary of backtrace: arlad is writing a messages to /dev/nnpfs0
telling nnpfs that it wants to install a new node. nnpfs tries
to make sure it doesn't already contain that node, which is where
the panic occurs.

(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:508
#1  0xc0188aa7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:341
#2  0xc0188ed8 in panic (fmt=0xc02da7e0 "assertion: tok->t_cpu == gd in %s")
    at /usr/src/sys/kern/kern_shutdown.c:620
#3  0xc018de84 in lwkt_reltoken (_ref=0xcc4dc998)
    at /usr/src/sys/kern/lwkt_token.c:410
#4  0xcc56708c in nnpfs_node_find (head=0xcc571b74, handlep=0xcc5a0310)
    at ../../../nnpfs/dragonfly/nnpfs_node.c:94
#5  0xcc569141 in nnpfs_new_node (nnpfsp=0xcc571b60, node=0xcc5a0310, 
    xpp=0xcc4dca08, p=0xc121a5e0)
    at ../../../nnpfs/dragonfly/nnpfs_node-df.c:125
#6  0xcc566615 in nnpfs_message_installnode (fd=0, message=0xcc5a01f0, 
    size=416, p=0xc121a5e0) at ../../../nnpfs/dragonfly/nnpfs_message.c:98
#7  0xcc56806e in nnpfs_message_receive (fd=0, message=0xcc5a01f0, size=416, 
    p=0xc121a5e0) at ../../../nnpfs/dragonfly/nnpfs_dev-common.c:639
#8  0xcc567b7c in nnpfs_devwrite (dev=0xc1368ad8, uiop=0xcc4dcbe0, 
    ioflag=8323073) at ../../../nnpfs/dragonfly/nnpfs_dev-common.c:352
#9  0xc017961a in cdevsw_putport (port=<incomplete type>, lmsg=0xcc4dcaf8)
    at /usr/src/sys/kern/kern_device.c:125
#10 0xc018dfe1 in lwkt_domsg (port=0xc035a5e0, msg=0xcc4dcaf8)
    at /usr/src/sys/sys/msgport2.h:86
#11 0xc017996a in dev_dwrite (dev=0xc1368ad8, uio=0xcc4dcbe0, ioflag=8323073)
    at /usr/src/sys/kern/kern_device.c:285
#12 0xc01c8fd1 in svn_write (fp=0xc1345a40, uio=0xcc4dcbe0, cred=0xc13a7438, 
    flags=0, td=<incomplete type>) at /usr/src/sys/kern/vfs_vnops.c:699
#13 0xc019e14b in kern_writev (fd=8, auio=0xcc4dcbe0, flags=0, res=0xcc4dcc60)
    at /usr/src/sys/sys/file2.h:76
#14 0xc019df07 in write (uap=0xcc4dcc34) at /usr/src/sys/kern/sys_generic.c:260
#15 0xc02b2e1a in syscall2 (frame=
      {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 674654628, tf_esi = 674655388, tf_ebp = 674654040, tf_isp = -867316364, tf_ebx = 134602956, tf_edx = 8, tf_ecx = 496, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = 673538668, tf_cs = 31, tf_eflags = 515, tf_esp = 674654012, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1350
#16 0xc02a4aba in Xint0x80_syscall ()

Happy hacking!
        -Richard