Sstrhsrtj

Environment: Linux [hostname] 3.2.0-4-amd64 #1 SMP Debian 3.2.96-2 x86_64 GNU/Linux
Hardware: AMD Opteron(tm) Processor 6344, 6x2MiB L2, 2x8MiB L3, 6-core ht (12 logical cores)

Today I got a warning in syslog:

Feb 28 09:58:53 amalthea kernel: [4367033.060016] INFO: rcu_bh detected stall on CPU 10 (t=0 jiffies)

Feb 28 09:58:53 amalthea kernel: [4367033.060018] sending NMI to all CPUs:

Followed by a dump of CPU state. There seems to be nothing "bad" leading up to this situation in the log.

The server is still running, no (apparent) stalled processes, etc., and the warning has not repeated itself in the hour or so it's been since it happened.

I've skimmed through some information on RCU's stall detector (it's far too technical for me to really understand), and I can see that:

1. My CPU was stalled for t=0 jiffies
   


2. There is no "detected by" CPU

There is a note in that document that suggests to me this may be a false-positive:

["Stall ended before state dump start"] is rare, but does happen from time to time in real life. It is also possible for a zero-jiffy stall to be flagged in this case, depending on how the stall warning and the grace-period initialization happen to interact. Please note that it is not possible to entirely eliminate this sort of false positive without resorting to things like stop_machine(), which is overkill for this sort of problem.

(Emphasis mine)

I didn't get a "Stall ended before state dump start" message, but I also didn't seem to get much else in terms of diagnostics other than the slew of CPU-dumps that came after the two log lines shown above.

I can post more information from the CPU-dump if it would be helpful. Nothing jumped-out at me, though I'm no expert.

What could have caused this situation? Is it likely to be a false-positive based solely upon the t=0 jiffies data point, plus no other diagnostic information printed to the log?

(Please note that this question is distinct from rcu_sched detected stall on CPU, which seems to have indicated a "real problem".)

Most likely, this is caused by one of three things:

1. A very hard to trigger kernel bug. I don't think this is likely though, as the RCU stuff is so core to the kernel that any bugs there are likely to be hit by just about everyone.


2. Bad RAM. Memory corruption caused by a bad memory module can quite easily cause bizarre stuff like this to happen.


3. A transient memory error caused by something other than the memory itself. Similar to the above, but not likely to show up again. This is the type of thing that ECC memory tries to protect against (but can't completely because it's fully possible for something to go wrong in the ECC logic).

Unless it' happens again, you can probably assume it's case 3 and just move on. If it does happen again, look for similarities in the surrounding kernel messages and/or check your RAM.