Re: panic on nightly build / test box

[Jan Lentfer <Jan.Lentfer@xxxxxx>]

It's actually apache serving static pages that causes the panic, right 
away on the first client connection. Below another panic. It's a SMP system.

I am usung apache-2.2.21nb1 binary package from 
ftp://ftp.mathematik.uni-marburg.de/pub/mirror/dragonfly/packages/i386/DragonFly-2.13/pkgsrc-2011Q3/All/

Jan


dfbench# kgdb kern.1 vmcore.1
GNU gdb (GDB) 7.3
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-dragonfly".
For bug reporting instructions, please see:
<http://bugs.dragonflybsd.org/>...
Reading symbols from /var/crash/kern.1...done.

Unread portion of the kernel message buffer:
panic: vm_page_wakeup: page not busy!!!
cpuid = 0
Trace beginning at frame 0xca9bba08
panic(ffffffff,0,c07111b0,ca9bba3c,c0b61d80) at panic+0x19e 0xc0373423
panic(c07111b0,c1e55ae8,0,ca9bba5c,c03bc000) at panic+0x19e 0xc0373423
vm_page_wakeup(c1e55ae8,0,c06b3f01) at vm_page_wakeup+0x1d 0xc05855b5
sf_buf_mfree(ca941898,cb6f7398,0,0,c28b578c) at sf_buf_mfree+0x6f 0xc03bc000
m_free(caa3f800,ca9bbaac,c03b624e,caa3f800,caa3f800) at m_free+0x279 
0xc03b3eb4
m_freem(caa3f800,caa3f800,ca9bba9c,caa3f800) at m_freem+0x15 0xc03b3fd6
sbdrop(c28b578c,4f7,8b12,b04c5e0a,5000) at sbdrop+0xa6 0xc03b624e
tcp_input(ca9bbbf4,ca9bbbf0,6,14,0) at tcp_input+0x21dd 0xc0463929
transport_processing_oncpu(0,5,ca9bbc18,e,caa3c600) at 
transport_processing_oncpu+0x52 0xc045b065
ip_input(caa3c600,ca9bbca8,c04148cd,caa3c618,ce209800) at ip_input+0xf0d 
0xc045cf11
ip_input_handler(caa3c618,ce209800,2,c0b61d98,ca9bbd24) at 
ip_input_handler+0x14 0xc045cf3a
netisr_handle(2,caa3c600,ca9bbcc8,0,c07bc180) at netisr_handle+0xb9 
0xc04148cd
ether_demux_oncpu(cb4bc400,caa3c600,caa3c600) at ether_demux_oncpu+0x463 
0xc040964c
ether_input_oncpu(5,c0b61dac,0,0,5) at ether_input_oncpu+0xcc 0xc040977d
ether_input_handler(caa3c618,0,c0b61d80,c0b474f8,ca9bbda0) at 
ether_input_handler+0x11a 0xc040a3e6
netmsg_service_loop(0,0,0,0,0) at netmsg_service_loop+0x73 0xc041402b
lwkt_exit() at lwkt_exit 0xc037dbaf
Debugger("panic")

CPU0 stopping CPUs: 0x00000002
  stopped
panic: from debugger
cpuid = 0
boot() called on cpu#0
Uptime: 13m6s
Physical memory: 754 MB
Dumping 110 MB: 95 79 63 47 31 15

[New pid 766/0, getty]
[New pid 760/0, getty]
[New pid 1/0, init]
[New pid 764/0, getty]
[New pid 762/0, getty]
[New pid 761/0, getty]
[New pid 754/0, httpd]
[New pid 753/0, httpd]
[New pid 335/0, syslogd]
[New kernel fq_balance_ad0]
[New kernel pagezero]
[New kernel consttyd]
[New kernel vnlru]
[New kernel bufdaemon]
[New kernel vmdaemon]
[New kernel pagedaemon]
[New kernel hammer-S2]
[New kernel hammer-S0]
[New kernel rtable_cpu 1]
[New kernel usbtask-dr]
[New kernel usb0]
[New kernel ithread197 1]
[New kernel ithread195 1]
[New kernel ithreadE 1]
[New kernel xpt_thrd]
[New kernel crypto 1]
[New kernel firmware taskq]
[New kernel taskq_cpu 1]
[New kernel ifnet 1]
[New kernel disk_msg_core]
[New kernel netisr_cpu 1]
[New kernel usched 1]
[New kernel dsched 1]
[New kernel softclock 1]
[New kernel idle_1]
[New pid 758/0, httpd]
[New pid 767/0, getty]
[New pid 765/0, getty]
[New pid 759/0, getty]
[New pid 763/0, getty]
[New pid 757/0, httpd]
[New pid 755/0, httpd]
[New pid 756/0, httpd]
[New pid 750/0, httpd]
[New pid 689/0, cron]
[New pid 674/0, sendmail]
[New pid 670/0, sendmail]
[New pid 656/0, sshd]
[New pid 541/0, dntpd]
[New kernel fq_dispatch_ad0]
[New kernel syncer]
[New kernel bufdaemon_hw]
[New kernel swapcached]
[New kernel hammer-S3]
[New kernel hammer-S1]
[New kernel hammer-M]
[New kernel rtable_cpu 0]
[New kernel random]
[New kernel ithread4 0]
[New kernel ithread192 0]
[New kernel ithread6 0]
[New kernel ithread12 0]
[New kernel ithread1 0]
[New kernel usbtask-hc]
[New kernel ithread11 0]
[New kernel ithread15 0]
[New kernel ithread14 0]
[New kernel ithread9 0]
[New kernel acpi_task]
[New pid 0/0, swapper]
[New kernel crypto returns]
[New kernel crypto 0]
[New kernel taskq_cpu 0]
[New kernel ifnet 0]
[New kernel devfs_msg_core]
[New kernel netisr_cpu 0]
[New kernel usched 0]
[New kernel dsched 0]
[New kernel ithread196 0]
[New kernel ithreadE 0]
[New kernel softclock 0]
[New kernel idle_0]
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/ehci.ko...done.
Loaded symbols for /boot/kernel/ehci.ko
_get_mycpu () at ./machine/thread.h:79
79          __asm ("movl %%fs:globaldata,%0" : "=r" (gd) : 
"m"(__mycpu__dummy));
(kgdb) backtrace
#0  _get_mycpu () at ./machine/thread.h:79
#1  md_dumpsys (di=0xc0ae59e0) at 
/usr/src/sys/platform/pc32/i386/dump_machdep.c:264
#2  0xc0372bd8 in dumpsys () at /usr/src/sys/kern/kern_shutdown.c:925
#3  0xc03731ee in boot (howto=<optimized out>) at 
/usr/src/sys/kern/kern_shutdown.c:387
#4  0xc0373457 in panic (fmt=0xc06971f2 "from debugger") at 
/usr/src/sys/kern/kern_shutdown.c:831
#5  0xc018ac02 in db_panic (addr=-1067342974, have_addr=0, count=-1, 
modif=0xca9bb898 "") at /usr/src/sys/ddb/db_command.c:445
#6  0xc018b2cf in db_command (aux_cmd_tablep_end=0xc0721114, 
aux_cmd_tablep=0xc07210f8, cmd_table=<optimized out>, 
last_cmdp=<optimized out>)
     at /usr/src/sys/ddb/db_command.c:401
#7  db_command_loop () at /usr/src/sys/ddb/db_command.c:467
#8  0xc018de2e in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:71
#9  0xc061a6f5 in kdb_trap (type=3, code=0, regs=0xca9bb9b8) at 
/usr/src/sys/platform/pc32/i386/db_interface.c:152
#10 0xc064412b in trap (frame=0xca9bb9b8) at 
/usr/src/sys/platform/pc32/i386/trap.c:843
#11 0xc061bbe7 in calltrap () at 
/usr/src/sys/platform/pc32/i386/exception.s:787
#12 0xc061a382 in breakpoint () at ./cpu/cpufunc.h:72
#13 Debugger (msg=0xc06b080d "panic") at 
/usr/src/sys/platform/pc32/i386/db_interface.c:334
#14 0xc0373438 in panic (fmt=0xc07111b0 "vm_page_wakeup: page not 
busy!!!") at /usr/src/sys/kern/kern_shutdown.c:822
#15 0xc05855b5 in vm_page_wakeup (m=0xc1e55ae8) at 
/usr/src/sys/vm/vm_page.c:838
#16 0xc03bc000 in sf_buf_mfree (arg=0xca941898) at 
/usr/src/sys/kern/uipc_syscalls.c:1419
#17 0xc03b3eb4 in m_free (m=0xcaa3f800) at 
/usr/src/sys/kern/uipc_mbuf.c:1184
#18 0xc03b3fd6 in m_freem (m=0x12) at /usr/src/sys/kern/uipc_mbuf.c:1223
#19 0xc03b624e in sbdrop (sb=0xc28b578c, len=1271) at 
/usr/src/sys/kern/uipc_sockbuf.c:494
#20 0xc0463929 in tcp_input (mp=0xca9bbbf4, offp=0xca9bbbf0, proto=6) at 
/usr/src/sys/netinet/tcp_input.c:2167
#21 0xc045b065 in transport_processing_oncpu (m=0x0, hlen=20, 
ip=0xce20980e) at /usr/src/sys/netinet/ip_input.c:395
#22 0xc045cf11 in ip_input (m=0xcaa3c600) at 
/usr/src/sys/netinet/ip_input.c:960
#23 0xc045cf3a in ip_input_handler (msg=0xcaa3c618) at 
/usr/src/sys/netinet/ip_input.c:416
#24 0xc04148cd in netisr_handle (num=2, m=0xcaa3c600) at 
/usr/src/sys/net/netisr.c:416
#25 0xc040964c in ether_demux_oncpu (ifp=0xcb4bc400, m=0xcaa3c600) at 
/usr/src/sys/net/if_ethersubr.c:1216
#26 0xc040977d in ether_input_oncpu (ifp=0xcb4bc400, m=0xcaa3c600) at 
/usr/src/sys/net/if_ethersubr.c:1287
#27 0xc040a3e6 in ether_input_handler (nmsg=0xcaa3c618) at 
/usr/src/sys/net/if_ethersubr.c:1405
#28 0xc041402b in netmsg_service_loop (arg=0x0) at 
/usr/src/sys/net/netisr.c:307
#29 0xc037dbaf in lwkt_deschedule_self (td=Cannot access memory at 
address 0x8
) at /usr/src/sys/kern/lwkt_thread.c:365
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(kgdb)


Am 23.12.2011 22:55, schrieb Jan Lenrfer:
> My nightly build box crashes (2 times in a row) on either ab/apache or
> sysbench filesystem benchmark:
>
> dfbench# kgdb kern.0 vmcore.0
> GNU gdb (GDB) 7.3
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i386-dragonfly".
> For bug reporting instructions, please see:
> <http://bugs.dragonflybsd.org/>...
> Reading symbols from /var/crash/kern.0...done.
>
> Unread portion of the kernel message buffer:
> panic: vm_page_wakeup: page not busy!!!
> cpuid = 1
> Trace beginning at frame 0xce353b0c
> panic(ffffffff,1,c07111b0,ce353b40,ca883360) at panic+0x19e 0xc0373423
> panic(c07111b0,c16e59d0,cb55a400,ce353b60,c03bc000) at panic+0x19e
> 0xc0373423
> vm_page_wakeup(c16e59d0,0,c06b3f01) at vm_page_wakeup+0x1d 0xc05855b5
> sf_buf_mfree(c266aa60,cb559800,126,ce1daab8,0) at sf_buf_mfree+0x6f
> 0xc03bc000
> m_free(cb55a600,ce353bf8,c03ba2dc,cb559800,20000) at m_free+0x279
> 0xc03b3eb4
> m_freem(cb559800,20000,ce353be8,1b32,3fe) at m_freem+0x15 0xc03b3fd6
> soreceive(ce1daab8,0,ce353c90,0,0) at soreceive+0x97a 0xc03ba2dc
> soo_read(cdf3a978,ce353c90,ca9983a8,0,c03a0af4) at soo_read+0x7a 0xc03a6e3a
> kern_preadv(3,ce353c90,0,ce353cf0,8056518,3a8,ce353c88,1,1c57,0,3a8,0,0,ca883360)
> at kern_preadv+0x148 0xc03a1a6a
> sys_read(ce353cf0,ce353d00,c,c037fc8c,c0b61d98) at sys_read+0x6d 0xc03a1c27
> syscall2(ce353d40) at syscall2+0x26e 0xc0644701
> Xint0x80_syscall() at Xint0x80_syscall+0x36 0xc061bc96
> Debugger("panic")
>
> CPU1 stopping CPUs: 0x00000001
> stopped
> panic: from debugger
> cpuid = 1
> boot() called on cpu#1
> Uptime: 1h42m16s
> Physical memory: 754 MB
> Dumping 82 MB: 67 51 35 19 3
>
> [New pid 762/0, httpd]
> [New pid 763/0, httpd]
> [New pid 765/0, httpd]
> [New pid 916/0, ab1.3]
> [New pid 734/0, bash]
> [New pid 757/0, httpd]
> [New pid 335/0, syslogd]
> [New pid 689/0, cron]
> [New pid 674/0, sendmail]
> [New pid 769/0, getty]
> [New pid 768/0, getty]
> [New pid 767/0, getty]
> [New pid 771/0, getty]
> [New pid 772/0, getty]
> [New pid 773/0, getty]
> [New pid 770/0, getty]
> [New pid 766/0, getty]
> [New pid 760/0, httpd]
> [New pid 725/0, cron]
> [New pid 656/0, sshd]
> [New kernel fq_balance_ad0]
> [New kernel pagezero]
> [New kernel consttyd]
> [New kernel vnlru]
> [New kernel bufdaemon]
> [New kernel vmdaemon]
> [New kernel pagedaemon]
> [New pid 1/0, init]
> [New kernel hammer-S2]
> [New kernel hammer-S0]
> [New kernel rtable_cpu 1]
> [New kernel usbtask-dr]
> [New kernel usb0]
> [New kernel ithread197 1]
> [New kernel ithread195 1]
> [New kernel ithreadE 1]
> [New kernel xpt_thrd]
> [New kernel crypto 1]
> [New kernel firmware taskq]
> [New kernel taskq_cpu 1]
> [New kernel ifnet 1]
> [New kernel disk_msg_core]
> [New kernel netisr_cpu 1]
> [New kernel usched 1]
> [New kernel dsched 1]
> [New kernel softclock 1]
> [New kernel idle_1]
> [New pid 764/0, httpd]
> [New pid 670/0, sendmail]
> [New pid 541/0, dntpd]
> [New pid 774/0, getty]
> [New pid 761/0, httpd]
> [New pid 754/0, sendmail]
> [New kernel fq_dispatch_ad0]
> [New kernel syncer]
> [New kernel bufdaemon_hw]
> [New kernel swapcached]
> [New kernel hammer-S3]
> [New kernel hammer-S1]
> [New kernel hammer-M]
> [New kernel rtable_cpu 0]
> [New kernel random]
> [New kernel ithread4 0]
> [New kernel ithread192 0]
> [New kernel ithread6 0]
> [New kernel ithread12 0]
> [New kernel ithread1 0]
> [New kernel usbtask-hc]
> [New kernel ithread11 0]
> [New kernel ithread15 0]
> [New kernel ithread14 0]
> [New kernel ithread9 0]
> [New kernel acpi_task]
> [New pid 0/0, swapper]
> [New kernel crypto returns]
> [New kernel crypto 0]
> [New kernel taskq_cpu 0]
> [New kernel ifnet 0]
> [New kernel devfs_msg_core]
> [New kernel netisr_cpu 0]
> [New kernel usched 0]
> [New kernel dsched 0]
> [New kernel ithread196 0]
> [New kernel ithreadE 0]
> [New kernel softclock 0]
> [New kernel idle_0]
> Reading symbols from /boot/kernel/acpi.ko...done.
> Loaded symbols for /boot/kernel/acpi.ko
> Reading symbols from /boot/kernel/ehci.ko...done.
> Loaded symbols for /boot/kernel/ehci.ko
> _get_mycpu () at ./machine/thread.h:79
> 79 __asm ("movl %%fs:globaldata,%0" : "=r" (gd) : "m"(__mycpu__dummy));
> (kgdb) backtrace
> #0 _get_mycpu () at ./machine/thread.h:79
> #1 md_dumpsys (di=0xc0ae59e0) at
> /usr/src/sys/platform/pc32/i386/dump_machdep.c:264
> #2 0xc0372bd8 in dumpsys () at /usr/src/sys/kern/kern_shutdown.c:925
> #3 0xc03731ee in boot (howto=<optimized out>) at
> /usr/src/sys/kern/kern_shutdown.c:387
> #4 0xc0373457 in panic (fmt=0xc06971f2 "from debugger") at
> /usr/src/sys/kern/kern_shutdown.c:831
> #5 0xc018ac02 in db_panic (addr=-1067342974, have_addr=0, count=-1,
> modif=0xce35399c "") at /usr/src/sys/ddb/db_command.c:445
> #6 0xc018b2cf in db_command (aux_cmd_tablep_end=0xc0721114,
> aux_cmd_tablep=0xc07210f8, cmd_table=<optimized out>,
> last_cmdp=<optimized out>)
> at /usr/src/sys/ddb/db_command.c:401
> #7 db_command_loop () at /usr/src/sys/ddb/db_command.c:467
> #8 0xc018de2e in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:71
> #9 0xc061a6f5 in kdb_trap (type=3, code=0, regs=0xce353abc) at
> /usr/src/sys/platform/pc32/i386/db_interface.c:152
> #10 0xc064412b in trap (frame=0xce353abc) at
> /usr/src/sys/platform/pc32/i386/trap.c:843
> #11 0xc061bbe7 in calltrap () at
> /usr/src/sys/platform/pc32/i386/exception.s:787
> #12 0xc061a382 in breakpoint () at ./cpu/cpufunc.h:72
> #13 Debugger (msg=0xc06b080d "panic") at
> /usr/src/sys/platform/pc32/i386/db_interface.c:334
> #14 0xc0373438 in panic (fmt=0xc07111b0 "vm_page_wakeup: page not
> busy!!!") at /usr/src/sys/kern/kern_shutdown.c:822
> #15 0xc05855b5 in vm_page_wakeup (m=0xc16e59d0) at
> /usr/src/sys/vm/vm_page.c:838
> #16 0xc03bc000 in sf_buf_mfree (arg=0xc266aa60) at
> /usr/src/sys/kern/uipc_syscalls.c:1419
> #17 0xc03b3eb4 in m_free (m=0xcb55a600) at
> /usr/src/sys/kern/uipc_mbuf.c:1184
> #18 0xc03b3fd6 in m_freem (m=0x12) at /usr/src/sys/kern/uipc_mbuf.c:1223
> #19 0xc03ba2dc in soreceive (so=0xce1daab8, psa=0x0, uio=0xce353c90,
> sio=0x0, controlp=0x0, flagsp=0xce353c18) at
> /usr/src/sys/kern/uipc_socket.c:1495
> #20 0xc03a6e3a in so_pru_soreceive (flagsp=<optimized out>,
> controlp=<optimized out>, sio=<optimized out>, uio=<optimized out>,
> paddr=<optimized out>, so=<optimized out>) at
> /usr/src/sys/sys/socketops.h:73
> #21 soo_read (fp=0xcdf3a978, uio=0xce353c90, cred=0xca9983a8, fflags=0)
> at /usr/src/sys/kern/sys_socket.c:90
> #22 0xc03a1a6a in fo_read (cred=<optimized out>, uio=<optimized out>,
> fp=<optimized out>, flags=<optimized out>) at /usr/src/sys/sys/file2.h:57
> #23 dofileread (res=<optimized out>, flags=<optimized out>,
> auio=<optimized out>, fp=<optimized out>, fd=<optimized out>)
> at /usr/src/sys/kern/sys_generic.c:305
> #24 kern_preadv (fd=3, auio=0xce353c90, flags=0, res=0xce353cf0) at
> /usr/src/sys/kern/sys_generic.c:269
> #25 0xc03a1c27 in sys_read (uap=0xce353cf0) at
> /usr/src/sys/kern/sys_generic.c:145
> #26 0xc0644701 in syscall2 (frame=0xce353d40) at
> /usr/src/sys/platform/pc32/i386/trap.c:1336
> #27 0xc061bc96 in Xint0x80_syscall () at
> /usr/src/sys/platform/pc32/i386/exception.s:878
> #28 0x0000001f in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> (kgdb) q
> dfbench# uname -a
> DragonFly dfbench.lan.net 2.13-DEVELOPMENT DragonFly
> v2.13.0.650.gea2c6-DEVELOPMENT #59: Fri Dec 23 04:28:28 CET 2011
> root@dfbench.lan.net:/usr/obj/usr/src/sys/GENERIC i386
> dfbench#
>