rcu: Make rcu_all_qs() do deferred NOCB wakeups
If a CONFIG_RCU_NOCB_CPUS kernel invokes call_rcu() with interrupts
disabled, wakeups must be deferred in order to avoid self-deadlock in
the cases where the disabled interrupts are due to scheduler locks being
held. In this case, a flag is set and is checked on entry to extended
quiescent states (usermode, idle), on exit from the RCU_SOFTIRQ handler,
when the CPU in question goes offline, and on a subsequent invocation
of call_rcu(). However, a given CPU could avoid all of those states
for a considerable length of time.
This commit therefore allows an invocation of rcu_all_qs() to do the
wakeup. It also makes the wakeup function clear the deferred-wakeup flag.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 4262446..83c920c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -520,6 +520,7 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch);
void rcu_all_qs(void)
{
unsigned long flags;
+ struct rcu_state *rsp;
barrier(); /* Avoid RCU read-side critical sections leaking down. */
if (unlikely(raw_cpu_read(rcu_sched_qs_mask))) {
@@ -542,6 +543,8 @@ void rcu_all_qs(void)
rcu_sched_qs();
preempt_enable();
}
+ for_each_rcu_flavor(rsp)
+ do_nocb_deferred_wakeup(this_cpu_ptr(rsp->rda));
this_cpu_inc(rcu_qs_ctr);
barrier(); /* Avoid RCU read-side critical sections leaking up. */
}
