Re: Livelock limit engaged?

[Dave Hayes <dave@xxxxxxxxxxx>]

Matthew Dillon <dillon@apollo.backplane.com> writes:
>  run vmstat -i

# vmstat -i
interrupt                   total       rate
clk                                              281783        437
atkbd0                                                4          0
sio0                                                  0          0
fdc0                                                  0          0
ppc0                                                 18          0
acpi0/em0                                             0          0
uhci0                                                 0          0
uhci1/uhci2/uhci3/twe0/fxp0/atapci1/atapci1     6766087      10506
ata0                                              12463         19
ata1                                                  0          0
irq19                                               101          0
swi_siopoll                                           0          0
swi_crypto/swi_camnet                                 0          0
swi_cambio                                            0          0
swi_vm                                                0          0
swi_taskq                                             0          0
Total                                           7060456      10963

Ok I see the interrupt in question. (I don't suppose there is any way to
move the USB interrupts to another less used IRQ?)

>    Monitor your interrupt rate with systat -vm 1
>    If the interrupt rate is normal for the workload on the machine,

There's a good question. Define "normal". :) I have no idea what normal
is, this is a brand new box.

Here's a vmstat -i from a production machine that is doing your basic
web/email/DNS:

# vmstat -i
interrupt                   total       rate
clk                                           784643353        281
atkbd0                                                1          0
sio0                                                  0          0
ppc0                                                  1          0
acpi0/em0                                             0          0
uhci0                                                 0          0
uhci1/uhci2/uhci3/twe0/fxp0/atapci1/atapci1    75223514         26
ata0                                              71880          0
ata1                                                  0          0
irq19                                                96          0
swi_siopoll                                           0          0
swi_crypto/swi_camnet                                 0          0
swi_cambio                                            0          0
swi_vm                                                0          0
swi_taskq                                             0          0
Total                                         859938845        308

The two machines are supposed to be identical, but obviously they
aren't. (The last one was bought recently, the previous one in April.)

If I do your systat -vm 1, I hardly see a peak interrupt rate on
the production machine, but the new machine spikes to 30-40%. 

Both motherboards are D946GZIS Intel, identical as far as I can 
tell from the model and packaging. 

Attached is also a sysctl -a hw. I did a diff between the old
and new machines on this:

2c2
< hw.model: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz
---
> hw.model: Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz
5,6c5,6
< hw.physmem: 2062254080
< hw.usermem: 1860755456
---
> hw.physmem: 2061856768
> hw.usermem: 2007019520
53c53
< hw.availpages: 503514
---
> hw.availpages: 503513
55c55
< hw.i8254.timestamp: 00000000 00151b77cbbffe68
---
> hw.i8254.timestamp: 00000000 000002dde46d87d0

Anything else I can probe? 
------
Dave Hayes - Consultant - Altadena CA, USA - dave@jetcafe.org 
>>> The opinions expressed above are entirely my own <<<

The universal body of reality is so subtle that you do not
hear it when you deliberately listen for it, and you do not
see it when you look at it. As for the pure knowledge that
has no teacher, how can it be attained by thought or study?


new# sysctl -a hw
hw.machine: i386
hw.model: Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz
hw.ncpu: 1
hw.byteorder: 1234
hw.physmem: 2061856768
hw.usermem: 2007019520
hw.pagesize: 4096
hw.floatingpoint: 1
hw.machine_arch: i386
hw.platform: pc32
hw.aac.iosize_max: 65536
hw.cbb.start_memory: 2281701376
hw.cbb.start_16_io: 256
hw.cbb.start_32_io: 4096
hw.cbb.debug: 0
hw.cardbus.debug: 0
hw.cardbus.cis_debug: 0
hw.an.an_dump: off
hw.an.an_cache_mode: dbm
hw.an.an_cache_mcastonly: 0
hw.an.an_cache_iponly: 1
hw.ata.mpipe_size: 4
hw.ata.ata_dma: 1
hw.ata.wc: 1
hw.ata.tags: 0
hw.ata.atapi_dma: 0
hw.firewire.try_bmr: 1
hw.firewire.hold_count: 3
hw.firewire.fwmem.eui64_hi: 0
hw.firewire.fwmem.eui64_lo: 0
hw.firewire.fwmem.speed: 2
hw.firewire.fwe.stream_ch: 1
hw.firewire.fwe.tx_speed: 2
hw.firewire.fwe.rx_queue_len: 128
hw.firewire.sbp.auto_login: 1
hw.firewire.sbp.max_speed: -1
hw.firewire.sbp.exclusive_login: 1
hw.firewire.sbp.login_delay: 1000
hw.firewire.sbp.scan_delay: 500
hw.firewire.sbp.tags: 0
hw.fxp_rnr: 0
hw.wi.txerate: 0
hw.wi.debug: 0
hw.rtw.rfprog_fallback: 0
hw.rtw.host_rfio: 0
hw.rtw.chip_reset: 0
hw.xe.debug: 1
hw.pccard.debug: 0
hw.pccard.cis_debug: 0
hw.dc_quick: 1
hw.usb.ugen.bufsize: 16384
hw.instruction_sse: 1
hw.availpages: 503513
hw.i8254.freq: 1193182
hw.i8254.timestamp: 00000000 000002dde46d87d0
hw.pci_disable_bios_route: 0
hw.acpi.supported_sleep_state: S3 S4 S5 
hw.acpi.power_button_state: S5
hw.acpi.sleep_button_state: S3
hw.acpi.lid_switch_state: NONE
hw.acpi.standby_state: S1
hw.acpi.suspend_state: S3
hw.acpi.sleep_delay: 1
hw.acpi.s4bios: 0
hw.acpi.verbose: 0
hw.acpi.disable_on_poweroff: 0
hw.acpi.reset_video: 1
hw.acpi.cpu.throttle_max: 8
hw.acpi.cpu.throttle_state: 8
hw.acpi.cpu.performance_speed: 8
hw.acpi.cpu.economy_speed: 4
hw.acpi.cpu.cx_supported: C1/0
hw.acpi.cpu.cx_lowest: C1
hw.acpi.cpu.cx_usage: 100.00%
hw.twe0.driver_version: 1.40.01.002
hw.em0.debug_info: -1
hw.em0.stats: -1
hw.em0.rx_int_delay: 0
hw.em0.tx_int_delay: 66
hw.em0.rx_abs_int_delay: 66
hw.em0.tx_abs_int_delay: 66
hw.em0.int_throttle_ceil: 10000
hw.fxp0.int_delay: 1000
hw.fxp0.bundle_max: 6

old# systcl -a hw
hw.machine: i386
hw.model: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz
hw.ncpu: 1
hw.byteorder: 1234
hw.physmem: 2062254080
hw.usermem: 1860755456
hw.pagesize: 4096
hw.floatingpoint: 1
hw.machine_arch: i386
hw.platform: pc32
hw.aac.iosize_max: 65536
hw.cbb.start_memory: 2281701376
hw.cbb.start_16_io: 256
hw.cbb.start_32_io: 4096
hw.cbb.debug: 0
hw.cardbus.debug: 0
hw.cardbus.cis_debug: 0
hw.an.an_dump: off
hw.an.an_cache_mode: dbm
hw.an.an_cache_mcastonly: 0
hw.an.an_cache_iponly: 1
hw.ata.mpipe_size: 4
hw.ata.ata_dma: 1
hw.ata.wc: 1
hw.ata.tags: 0
hw.ata.atapi_dma: 0
hw.firewire.try_bmr: 1
hw.firewire.hold_count: 3
hw.firewire.fwmem.eui64_hi: 0
hw.firewire.fwmem.eui64_lo: 0
hw.firewire.fwmem.speed: 2
hw.firewire.fwe.stream_ch: 1
hw.firewire.fwe.tx_speed: 2
hw.firewire.fwe.rx_queue_len: 128
hw.firewire.sbp.auto_login: 1
hw.firewire.sbp.max_speed: -1
hw.firewire.sbp.exclusive_login: 1
hw.firewire.sbp.login_delay: 1000
hw.firewire.sbp.scan_delay: 500
hw.firewire.sbp.tags: 0
hw.fxp_rnr: 0
hw.wi.txerate: 0
hw.wi.debug: 0
hw.rtw.rfprog_fallback: 0
hw.rtw.host_rfio: 0
hw.rtw.chip_reset: 0
hw.xe.debug: 1
hw.pccard.debug: 0
hw.pccard.cis_debug: 0
hw.dc_quick: 1
hw.usb.ugen.bufsize: 16384
hw.instruction_sse: 1
hw.availpages: 503514
hw.i8254.freq: 1193182
hw.i8254.timestamp: 00000000 00151b77cbbffe68
hw.pci_disable_bios_route: 0
hw.acpi.supported_sleep_state: S3 S4 S5 
hw.acpi.power_button_state: S5
hw.acpi.sleep_button_state: S3
hw.acpi.lid_switch_state: NONE
hw.acpi.standby_state: S1
hw.acpi.suspend_state: S3
hw.acpi.sleep_delay: 1
hw.acpi.s4bios: 0
hw.acpi.verbose: 0
hw.acpi.disable_on_poweroff: 0
hw.acpi.reset_video: 1
hw.acpi.cpu.throttle_max: 8
hw.acpi.cpu.throttle_state: 8
hw.acpi.cpu.performance_speed: 8
hw.acpi.cpu.economy_speed: 4
hw.acpi.cpu.cx_supported: C1/0
hw.acpi.cpu.cx_lowest: C1
hw.acpi.cpu.cx_usage: 100.00%
hw.twe0.driver_version: 1.40.01.002
hw.em0.debug_info: -1
hw.em0.stats: -1
hw.em0.rx_int_delay: 0
hw.em0.tx_int_delay: 66
hw.em0.rx_abs_int_delay: 66
hw.em0.tx_abs_int_delay: 66
hw.em0.int_throttle_ceil: 10000
hw.fxp0.int_delay: 1000
hw.fxp0.bundle_max: 6