README

## ras-tools

ras-tools are an excellent set of tools to inject and test RAS ability on X86 
and Arm platform through APEI EINJ interface.

## Brief Introduction

Common tools on both X86 and Arm platform:

- rep_ce_page: injects and consumes corrected errors from a single page
until either the page is taken offline (and replaced) by the OS, or
a limit of 30 tries is reached.
- mca-recover: an example recovery application shows how to setup a SIGBUS handler for recoverable machine checks.
- cmcistorm: inject a bunch of corrected errors, then trigger them all quickly.
- hornet: inject a UC memory error into some other process.
- einj_mem_uc: inject an error and then trigger it in one of a variety of ways.

Arm platform specific drivers:

- memattr: a test suit to poison specific memory attribute.
- ras-tolerance: a driver to overwrite error severity to a lower level at runtime.

Virtualization:

Injecting errors into guests is a rather manual process. You can run einj_mem_uc
inside the guest with special arguments to skip the injection, but still print
the guest physical address. Then on the host convert that to a host physical
address and inject. Finally have the process on the guest consume the error.

Detailed steps are:

- '-j': skip error injection, this step should do with host physical
  address on host which creates GPA->HPA mappings for the guest.
- '-k': kick off trigger by writing a file from remote (host).

The steps to inject guest error are:

STEP 1: start a VM with a stdio monitor which allows giving complex
commands to the QEMU emulator.

        qemu-system-aarch64  -enable-kvm \
                -cpu host \
                -M virt,gic-version=3 \
                -m 8G \
                -d guest_errors \
                -rtc base=localtime,clock=host \
                -smp cores=2,threads=2,sockets=2 \
                -object memory-backend-ram,id=mem0,size=4G \
                -object memory-backend-ram,id=mem1,size=4G \
                -numa node,memdev=mem0,cpus=0-3,nodeid=0 \
                -numa node,memdev=mem1,cpus=4-7,nodeid=1 \
                -bios /usr/share/AAVMF/AAVMF_CODE.fd \
                -drive driver=qcow2,media=disk,cache=writeback,if=virtio,id=alinu1_rootfs,file=/path/to/image.qcow2 \
                -netdev user,id=n1,hostfwd=tcp::5555-:22  \
                -serial telnet:localhost:4321,server,nowait \
                -device virtio-net-pci,netdev=n1 \
                -monitor stdio
        QEMU 7.2.0 monitor - type 'help' for more information
        (qemu) VNC server running on 127.0.0.1:5900

STEP 2: login guest and install ras-tools, then run `einj_mem_uc` to
allocate a page in userspace, dumps the virtual and physical address of the
page. The `-j` is to skip error injection and `-k` is to wait for a kick.

        $ ./einj_mem_uc single -j -k
        0: single   vaddr = 0xffffbd88c400 paddr = 151f21400

STEP 3: run command `gpa2hpa` in QEMU monitor and it will print the host
physical address at which the guest's physical address addr is mapped.

        (qemu) gpa2hpa 0x151f21400
        Host physical address for 0x151f21400 (mem1) is 0x935757400

STEP 4: inject an uncorrected error via the APEI interface to the finally
translated host physical address on host.

        echo 0x949a84400 > /sys/kernel/debug/apei/einj/param1
        echo 0xfffffffffffff000 > /sys/kernel/debug/apei/einj/param2
        echo 0x0 > /sys/kernel/debug/apei/einj/flags
        echo 0x10 > /sys/kernel/debug/apei/einj/error_type
        echo 1 > /sys/kernel/debug/apei/einj/notrigger
        echo 1 > /sys/kernel/debug/apei/einj/error_inject

STEP 5: then kick `einj_mem_uc` to trigger the error by writing
"trigger_start".  In this example, the kick is done on host.

        ssh -p 5555 root@localhost "echo trigger > ~/trigger_start"

STEP 6: We will observe that the QEMU process exit.

        (qemu) qemu-system-aarch64: Hardware memory error!