MCE Stress Test HOWTO
====================

Oct 10th, 2009

Haicheng Li


Abstract
--------

This document explains the design and structure of MCE stress test suite,
the kernel configurations and user space tools required for automated
stress testing, as well as usage guide and etc.


0. Quick Shortcut
-----------------

- Install the Linux kernel (2.6.32 or newer) with full MCA recovery support.
  Make sure following configuration options are enabled:

  CONFIG_X86_MCE=y
  CONFIG_MEMORY_FAILURE=y

  With these two options enabled, you can do stress testing thru madvise
  syscall (sec 4.1).

- Install page-types tool (sec 3.3), which is accompanied with Linux kernel
  source (2.6.32 or newer).

  # cd $KERNEL_SRC/Documentation/vm/
  # gcc -o page-types page-types.c 
  # cp page-types /usr/bin/

- Get latest LTP (Linux Test Project) image from http://ltp.sf.net. Refer
  to INSTALL of LTP to install LTP on your machine.

- Build and run stress testing

  # make
  # cd stress
  # ./hwpoison.sh -d $YOUR_PARTITION -M -o $YOUR_LTP_DIR -N

  Note here, '-d $YOUR_PARTITION' is a mandatory option. Test will create 
  all temporary files on $YOUR_PARTITION, and error injection will just
  affect the pages associated with $$YOUR_PARTITION. So you must provide a
  free disk partition to stress test driver!

  This will do the stress testing thru madvise syscall (sec 4.1). However,
  there are more advanced test methods provided (sec 4.2, 4.3).

Note, for all examples in the rest of this doc, it is supposed that $PWD is
the stress subdir.  

1. Overview
-----------

The MCE stress test suite is a collection of tools and test scripts, which
intends to achieve stress testing on Linux kernel MCA high level handlers
that include HWPosion page recovery, soft page offline, and so on.

In general, this test suite is designed to do stress testing thru various
test interfaces, i.e. madvise syscall, HWPoison page injector, and APEI
injector (see ACPI4.0 spec). And it's able to support most of popular
Linux File Systems (FS), that is, there is an option for user to specify which
FS type they want the test to be running on.

If you just want to start testing as quickly as possible, you can skip
section 2 & 3, just go to section 4 directly.


2. Design Details
-----------------

The MCE stress test suite consists of four parts: test driver, workload
controller, customized workloads, and background workloads.

The main test idea is described as below:
- Test driver launchs various customized workloads to continuously generate
  lots of pages with expected page states, Note, all of these workloads know
  about their expected results that should not be affected by Linux MCE high
  level handlers.
- Then test driver injects MCE errors to these pages thru either madvise 
  syscall or HWPoison injector or APEI injector. While Linux Kernel handling
  these MCE errors, all the workloads continue running normally,
- After long time running, test driver will collect test result of each
  workload to see if any unexpected failures happened. In such a way, it can 
  decide if any bug is found.
- If any system panics or FS corruption happens, that means there must be a
  bug. It's the bottom line to decide if test gets pass.

2.1 Test Driver

Test driver (a.k.a hwpoison.sh) drives the whole test procedure. It's
responsible for managing test environment, setting up error injection 
interface, controlling test progress, launching workloads, injecting page 
errors, as well as recording test logs and reporting test result.

For detailed usage of hwpoison.sh test driver, please refer to:
# ./hwpoison.sh -h

2.2 Workload Controller

Workload controller needs to have various test workloads running parallelly
and continuously within a required duration time. We select ltp-pan
program of Linux Test Project (LTP) as the workload controller of this
stress test suite.

Test driver (hwpoison.sh) interacts with ltp-pan in following ways:
- hwpoison.sh generates a test config file that lists the workload type
  to be launched by ltp-pan.
- hwpoison also passes test duration time and other workload specific
  parameters to ltp-pan via test config file.
- ltp-pan makes each workload run and get finished in time, then test driver
  can get the result of each workload via corresponding result files.
- finally, hwpoison.sh will decide the overall test result based on each
  workload result, and report final result out.

2.3 Customized Workloads

There are three types of customized workloads, which are intended to generate 
pages with various page state.

* Type0: page-poisoning workload, meant to cover:
  - anonymous pages operations.
  - file data operations.

* Type1: fs-metadata workload, meant to cover:
  - inode operations.

* Type2: fs_type specific workload, meant to cover:
  - extended functions of some special FS.

2.4 Background Workloads

LTP is selected as the background workload to simulate normal system
operations in background while stress testing is running.

Besides LTP, there are also some alternatives, like AIM. We might extend more
background workloads in future.

2.5 Test Result

How to determine that stress testing gets pass?
- at least no kernel panics happens during stress testing.
- fsck on the target disk at the end of stress testing should get pass.
- there is no failure found by customized workloads, especially for
  page-poisoning workload.

Where to get detailed test result?
- When stress testing is done, the general test result is recorded in
  result/hwpoison.result, and the general test log is in result/hwpoison.log.
  However, you can specify them in following way:
  # hwpoison.sh -r $YOUR_RESULT -l $YOUR_LOG
- The test result and test log of each workload are recorded as
  log/$workload/$workload.result and log/$workload/$workload.log.
  For example, for page-poisoning workload, its test result and test logs are
  log/page-poisoning/page-poisoning.result and 
  log/page-poisoning/page-poisoning.log.
- Besides, under each workload result dir, you can find other extra logs
  like pan_log, pan_output and etc. These logs are generated by ltp-pan 
  workload controller. Usually they can help you understand what has been 
  going on with ltp-pan while workload is running. Pls. refer to ltp-pan doc
  for details.


3. Tools
--------

3.1 page-poisoning

It is the page-poisoning workload. page-poisoning workload is an extension of
tinjpage test program with a multi-process model. It spawns thousands of
processes that inject HWPosion error to various pages simultaneously thru
madvise syscall. Then it checks if these errors get handled correctly, 
i.e. whether each test process receives or doesn't receive SIGBUS signal as 
expected.

For more info about page-poisoning workload, pls. read through README file
under stress/tools/page-poisoning/.

3.2 fs-metadata

It is the fs-metadata workload. fs-metadata is designed to test i-node
operations with heavy workload and make sure every i-node operation gets
the expected result. In details, it firstly generates a huge directory
hierarchy on the target disk, then it performs unlink operations on this
directory hierarchy and duplicate a copy of the directory, finally it
checks if these two directories are same as expected.

For more info about fs-metadata workload, pls. read through README file
under stress/tools/fs-metadata/.

3.3 page-types

page-types is a tool to query the page type of every memory page in the
system. We use it to filter out pages with required page types. Test will
inject error to these pages via error injector, although the page filter
of HWPosion handler in Linux Kernel will filter them out for a second
time. Note, the reason we need to use page-types to do first time filtering
is just about performance.

To install page-types on your test machine:

  # cd $KERNEL_SRC/Documentation/vm/
  # gcc -o page-types page-types.c 
  # cp page-types /usr/bin/

3.4 ltp-pan

It's the workload controller of this stress test suite. In fact, ltp-pan
is the test harness of LTP (Linux Test Project), and is included in
LTP package. For more information, please refer to ltp-pan document of LTP.


4. Usage Guide
--------------

This section is trying to show you how to conduct the stress testing thru
various test interfaces. 

As an example, we choose to run stress testing based on partition /dev/sda1 
for 1 hour. Note, we've installed LTP to /ltp.  

4.1 Stress Test thru Madvise Syscall.

To run this stress testing, you need to strictly follow below test
instructions. 

* Test instructions:

- make sure following kernel options are enabled:
  CONFIG_X86_MCE=y
  CONFIG_MEMORY_FAILURE=y

- build and run stress testing
  # make
  # ./hwpoison.sh -d $YOUR_PARTITION -M -o $YOUR_LTP_DIR

* Example: 

- launch testing
  # ./hwpoison.sh -d /dev/sda1 -M -t 3600

- general test results
  result: result/hwpoison.result
  logs: result/hwpoison.log

- detailed workload results
  result: log/page-poisoning/page-poisoning.result
  log: log/page-poisoning/page-poisoning.log

4.2 Stress Test thru HWPosion Page Injector

This is the default test method of this stress test suite.

To run this stress testing, you need to strictly follow below test
instructions. 

* Test instructions:

- make sure following kernel options are enabled:
  CONFIG_X86_MCE=y
  CONFIG_MEMORY_FAILURE=y
  CONFIG_DEBUG_KERNEL=y
  CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
  CONFIG_HWPOISON_INJECT=y

- build and run stress testing
  # make
  # ./hwpoison.sh -d $YOUR_PARTITION -o $YOUR_LTP_DIR -L 

* Example: 

- launch testing
  # ./hwpoison.sh -d /dev/sda1 -t 3600 -L

- general test results
  result: result/hwpoison.result
  logs: result/hwpoison.log

- detailed workload results
  fs-metadata result: log/fs-metadata/fs-metadata.result
  fs-metadata log: log/fs-metadata/fs-metadata.log
  ltp result: log/ltp/ltp.result
  ltp log: log/ltp/ltp.log
  fs-specific result: log/fs-specific/fs-specific.result
  fs-specific log: log/fs-specific/fs-specific.log

4.3 Stress Test thru APEI Injector

To run this stress testing, you need to follow below test instructions. 

* Test instructions:

- make sure following kernel options are enabled:
  CONFIG_X86_MCE=y
  CONFIG_X86_MCE_INTEL=y
  CONFIG_MEMORY_FAILURE=y
  CONFIG_ACPI_APEI=y
  CONFIG_ACPI_APEI_EINJ=y

- build and run stress testing
  # make
  # ./hwpoison.sh -d $YOUR_PARTITION -o $YOUR_LTP_DIR -L -A

* Example: 

- launch testing
  # ./hwpoison.sh -d /dev/sda1 -t 3600 -L -A

- general test results
  result: result/hwpoison.result
  logs: result/hwpoison.log

- detailed workload results
  fs-metadata result: log/fs-metadata/fs-metadata.result
  fs-metadata log: log/fs-metadata/fs-metadata.log
  ltp result: log/ltp/ltp.result
  ltp log: log/ltp/ltp.log
  fs-specific result: log/fs-specific/fs-specific.result
  fs-specific log: log/fs-specific/fs-specific.log


5. FAQs
-------

Here is a collection of frequently asked questions:

Q: How to tell test driver not to format my disk partition?
A: Use the option '-N'.

Q: Can three types of tests run on same sytem simultaneously?
A: No. There are limitations in Linux Kernel HWPoison page filtering.  

Q: Can I run this stress testing on multiple disks parallely?
A: Yes. But it requires updated Kernel patches for HWPosion page filtering.
   Now, it just supports one same test with same pagetype flags specified.