x86/retpoline: Avoid uncontrolled return buffer underflows
retpoline tries to avoid uncontrolled indirect branches in the
kernel. It relies on using return instructions which use
the return buffer in the CPU to predict. When the return buffer
underflows the CPU falls back to the indirect branch predictor,
which we want to avoid because it could be poisoned.
This patch handles two scenarios where this could happen:
(1) A retpoline gets interrupted before executing the final return.
The interrupt has a deep enough (32+) call stack that it fills and
then clears the return buffer. While this should be unlikely we
cannot rule it out completely. The retpoline return would then underflow
and fall back to the indirect branch predictor.
To avoid this we always execute 32 unpaired rets on interrupt/NMI
exit to fill the return buffer with controlled content.
In principle this only needs to be done when interrupting retpoline,
but since they can be inlined there is no convenient IP range to
test. So we do it unconditionally.
This has to be done in interrupt, NMI, machine check handlers
and anything else that could in theory interrupt retpoline, such
as a single stepping or break points or in kernel vmalloc faults
for the stack. I didn't do it for double and stack faults because
if those happen you already got worse problems.
(2) We're context switching from a shallower to a deeper kernel stack.
The deeper kernel stack would eventually underflow the return buffer,
which again would fall back to the indirect branch predictor.
To guard against this again fill the return buffer with controlled
content during context switch.
This is not a problem on newer CPUs (Broadwall+) implementing SMEP,
but happens on older CPUs or if SMEP is disabled. On those
without SMEP, or those without Meltdown, all the extra code
can be skipped.
We always fill the buffer with 32 entries. That's pessimistic
because we already did some controlled kernel calls. So in theory
less dummy calls would be needed to be safe. However it's hard to
maintain such an invariant, and it may be broken with more aggressive
compilers. So err on the side of safety and always fill
completely.
Contains intentional unreachable code, which would cause objtool
warnings if it was running. Right now that is excluded with retpoline
so that's not a problem. Later on it would need one more
ASM_UNREACHABLE, once that exists for pure assembler.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
9 files changed
