doc/stress-howto.txt - pub/scm/linux/kernel/git/gong.chen/mce-test - Git at Google

 MCE Stress Test HOWTO
 ====================

 Oct 10th, 2009

 Haicheng Li


 Abstract
 --------

 This document explains the design and structure of MCE stress test suite,
 the kernel configurations and user space tools required for automated
 stress testing, as well as usage guide and etc.


 0. Quick Shortcut
 -----------------

 - Install the Linux kernel (2.6.32 or newer) with full MCA recovery support.
   Make sure following configuration options are enabled:

   CONFIG_X86_MCE=y
   CONFIG_MEMORY_FAILURE=y

   With these two options enabled, you can do stress testing thru madvise
   syscall (sec 4.1).

 - Install page-types tool (sec 3.3), which is accompanied with Linux kernel
   source (2.6.32 or newer).

   # cd $KERNEL_SRC/Documentation/vm/
   # gcc -o page-types page-types.c
   # cp page-types /usr/bin/

 - Get latest LTP (Linux Test Project) image from http://ltp.sf.net. Refer
   to INSTALL of LTP to install LTP on your machine.

 - Build and run stress testing

   # make
   # cd stress
   # ./hwpoison.sh -d $YOUR_PARTITION -M -o $YOUR_LTP_DIR -N

   Note here, '-d $YOUR_PARTITION' is a mandatory option. Test will create
   all temporary files on $YOUR_PARTITION, and error injection will just
   affect the pages associated with $$YOUR_PARTITION. So you must provide a
   free disk partition to stress test driver!

   This will do the stress testing thru madvise syscall (sec 4.1). However,
   there are more advanced test methods provided (sec 4.2, 4.3).

 Note, for all examples in the rest of this doc, it is supposed that $PWD is
 the stress subdir.

 1. Overview
 -----------

 The MCE stress test suite is a collection of tools and test scripts, which
 intends to achieve stress testing on Linux kernel MCA high level handlers
 that include HWPosion page recovery, soft page offline, and so on.

 In general, this test suite is designed to do stress testing thru various
 test interfaces, i.e. madvise syscall, HWPoison page injector, and APEI
 injector (see ACPI4.0 spec). And it's able to support most of popular
 Linux File Systems (FS), that is, there is an option for user to specify which
 FS type they want the test to be running on.

 If you just want to start testing as quickly as possible, you can skip
 section 2 & 3, just go to section 4 directly.


 2. Design Details
 -----------------

 The MCE stress test suite consists of four parts: test driver, workload
 controller, customized workloads, and background workloads.

 The main test idea is described as below:
 - Test driver launchs various customized workloads to continuously generate
   lots of pages with expected page states, Note, all of these workloads know
   about their expected results that should not be affected by Linux MCE high
   level handlers.
 - Then test driver injects MCE errors to these pages thru either madvise
   syscall or HWPoison injector or APEI injector. While Linux Kernel handling
   these MCE errors, all the workloads continue running normally,
 - After long time running, test driver will collect test result of each
   workload to see if any unexpected failures happened. In such a way, it can
   decide if any bug is found.
 - If any system panics or FS corruption happens, that means there must be a
   bug. It's the bottom line to decide if test gets pass.

 2.1 Test Driver

 Test driver (a.k.a hwpoison.sh) drives the whole test procedure. It's
 responsible for managing test environment, setting up error injection
 interface, controlling test progress, launching workloads, injecting page
 errors, as well as recording test logs and reporting test result.

 For detailed usage of hwpoison.sh test driver, please refer to:
 # ./hwpoison.sh -h

 2.2 Workload Controller

 Workload controller needs to have various test workloads running parallelly
 and continuously within a required duration time. We select ltp-pan
 program of Linux Test Project (LTP) as the workload controller of this
 stress test suite.

 Test driver (hwpoison.sh) interacts with ltp-pan in following ways:
 - hwpoison.sh generates a test config file that lists the workload type
   to be launched by ltp-pan.
 - hwpoison also passes test duration time and other workload specific
   parameters to ltp-pan via test config file.
 - ltp-pan makes each workload run and get finished in time, then test driver
   can get the result of each workload via corresponding result files.
 - finally, hwpoison.sh will decide the overall test result based on each
   workload result, and report final result out.

 2.3 Customized Workloads

 There are three types of customized workloads, which are intended to generate
 pages with various page state.

 * Type0: page-poisoning workload, meant to cover:
   - anonymous pages operations.
   - file data operations.

 * Type1: fs-metadata workload, meant to cover:
   - inode operations.

 * Type2: fs_type specific workload, meant to cover:
   - extended functions of some special FS.

 2.4 Background Workloads

 LTP is selected as the background workload to simulate normal system
 operations in background while stress testing is running.

 Besides LTP, there are also some alternatives, like AIM. We might extend more
 background workloads in future.

 2.5 Test Result

 How to determine that stress testing gets pass?
 - at least no kernel panics happens during stress testing.
 - fsck on the target disk at the end of stress testing should get pass.
 - there is no failure found by customized workloads, especially for
   page-poisoning workload.

 Where to get detailed test result?
 - When stress testing is done, the general test result is recorded in
   result/hwpoison.result, and the general test log is in result/hwpoison.log.
   However, you can specify them in following way:
   # hwpoison.sh -r $YOUR_RESULT -l $YOUR_LOG
 - The test result and test log of each workload are recorded as
   log/$workload/$workload.result and log/$workload/$workload.log.
   For example, for page-poisoning workload, its test result and test logs are
   log/page-poisoning/page-poisoning.result and
   log/page-poisoning/page-poisoning.log.
 - Besides, under each workload result dir, you can find other extra logs
   like pan_log, pan_output and etc. These logs are generated by ltp-pan
   workload controller. Usually they can help you understand what has been
   going on with ltp-pan while workload is running. Pls. refer to ltp-pan doc
   for details.


 3. Tools
 --------

 3.1 page-poisoning

 It is the page-poisoning workload. page-poisoning workload is an extension of
 tinjpage test program with a multi-process model. It spawns thousands of
 processes that inject HWPosion error to various pages simultaneously thru
 madvise syscall. Then it checks if these errors get handled correctly,
 i.e. whether each test process receives or doesn't receive SIGBUS signal as
 expected.

 For more info about page-poisoning workload, pls. read through README file
 under stress/tools/page-poisoning/.

 3.2 fs-metadata

 It is the fs-metadata workload. fs-metadata is designed to test i-node
 operations with heavy workload and make sure every i-node operation gets
 the expected result. In details, it firstly generates a huge directory
 hierarchy on the target disk, then it performs unlink operations on this
 directory hierarchy and duplicate a copy of the directory, finally it
 checks if these two directories are same as expected.

 For more info about fs-metadata workload, pls. read through README file
 under stress/tools/fs-metadata/.

 3.3 page-types

 page-types is a tool to query the page type of every memory page in the
 system. We use it to filter out pages with required page types. Test will
 inject error to these pages via error injector, although the page filter
 of HWPosion handler in Linux Kernel will filter them out for a second
 time. Note, the reason we need to use page-types to do first time filtering
 is just about performance.

 To install page-types on your test machine:

   # cd $KERNEL_SRC/Documentation/vm/
   # gcc -o page-types page-types.c
   # cp page-types /usr/bin/

 3.4 ltp-pan

 It's the workload controller of this stress test suite. In fact, ltp-pan
 is the test harness of LTP (Linux Test Project), and is included in
 LTP package. For more information, please refer to ltp-pan document of LTP.


 4. Usage Guide
 --------------

 This section is trying to show you how to conduct the stress testing thru
 various test interfaces.

 As an example, we choose to run stress testing based on partition /dev/sda1
 for 1 hour. Note, we've installed LTP to /ltp.

 4.1 Stress Test thru Madvise Syscall.

 To run this stress testing, you need to strictly follow below test
 instructions.

 * Test instructions:

 - make sure following kernel options are enabled:
   CONFIG_X86_MCE=y
   CONFIG_MEMORY_FAILURE=y

 - build and run stress testing
   # make
   # ./hwpoison.sh -d $YOUR_PARTITION -M -o $YOUR_LTP_DIR

 * Example:

 - launch testing
   # ./hwpoison.sh -d /dev/sda1 -M -t 3600

 - general test results
   result: result/hwpoison.result
   logs: result/hwpoison.log

 - detailed workload results
   result: log/page-poisoning/page-poisoning.result
   log: log/page-poisoning/page-poisoning.log

 4.2 Stress Test thru HWPosion Page Injector

 This is the default test method of this stress test suite.

 To run this stress testing, you need to strictly follow below test
 instructions.

 * Test instructions:

 - make sure following kernel options are enabled:
   CONFIG_X86_MCE=y
   CONFIG_MEMORY_FAILURE=y
   CONFIG_DEBUG_KERNEL=y
   CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
   CONFIG_HWPOISON_INJECT=y

 - build and run stress testing
   # make
   # ./hwpoison.sh -d $YOUR_PARTITION -o $YOUR_LTP_DIR -L

 * Example:

 - launch testing
   # ./hwpoison.sh -d /dev/sda1 -t 3600 -L

 - general test results
   result: result/hwpoison.result
   logs: result/hwpoison.log

 - detailed workload results
   fs-metadata result: log/fs-metadata/fs-metadata.result
   fs-metadata log: log/fs-metadata/fs-metadata.log
   ltp result: log/ltp/ltp.result
   ltp log: log/ltp/ltp.log
   fs-specific result: log/fs-specific/fs-specific.result
   fs-specific log: log/fs-specific/fs-specific.log

 4.3 Stress Test thru APEI Injector

 To run this stress testing, you need to follow below test instructions.

 * Test instructions:

 - make sure following kernel options are enabled:
   CONFIG_X86_MCE=y
   CONFIG_X86_MCE_INTEL=y
   CONFIG_MEMORY_FAILURE=y
   CONFIG_ACPI_APEI=y
   CONFIG_ACPI_APEI_EINJ=y

 - build and run stress testing
   # make
   # ./hwpoison.sh -d $YOUR_PARTITION -o $YOUR_LTP_DIR -L -A

 * Example:

 - launch testing
   # ./hwpoison.sh -d /dev/sda1 -t 3600 -L -A

 - general test results
   result: result/hwpoison.result
   logs: result/hwpoison.log

 - detailed workload results
   fs-metadata result: log/fs-metadata/fs-metadata.result
   fs-metadata log: log/fs-metadata/fs-metadata.log
   ltp result: log/ltp/ltp.result
   ltp log: log/ltp/ltp.log
   fs-specific result: log/fs-specific/fs-specific.result
   fs-specific log: log/fs-specific/fs-specific.log


 5. FAQs
 -------

 Here is a collection of frequently asked questions:

 Q: How to tell test driver not to format my disk partition?
 A: Use the option '-N'.

 Q: Can three types of tests run on same sytem simultaneously?
 A: No. There are limitations in Linux Kernel HWPoison page filtering.

 Q: Can I run this stress testing on multiple disks parallely?
 A: Yes. But it requires updated Kernel patches for HWPosion page filtering.
    Now, it just supports one same test with same pagetype flags specified.
	MCE Stress Test HOWTO
	====================

	Oct 10th, 2009

	Haicheng Li


	Abstract
	--------

	This document explains the design and structure of MCE stress test suite,
	the kernel configurations and user space tools required for automated
	stress testing, as well as usage guide and etc.


	0. Quick Shortcut
	-----------------

	- Install the Linux kernel (2.6.32 or newer) with full MCA recovery support.
	Make sure following configuration options are enabled:

	CONFIG_X86_MCE=y
	CONFIG_MEMORY_FAILURE=y

	With these two options enabled, you can do stress testing thru madvise
	syscall (sec 4.1).

	- Install page-types tool (sec 3.3), which is accompanied with Linux kernel
	source (2.6.32 or newer).

	# cd $KERNEL_SRC/Documentation/vm/
	# gcc -o page-types page-types.c
	# cp page-types /usr/bin/

	- Get latest LTP (Linux Test Project) image from http://ltp.sf.net. Refer
	to INSTALL of LTP to install LTP on your machine.

	- Build and run stress testing

	# make
	# cd stress
	# ./hwpoison.sh -d $YOUR_PARTITION -M -o $YOUR_LTP_DIR -N

	Note here, '-d $YOUR_PARTITION' is a mandatory option. Test will create
	all temporary files on $YOUR_PARTITION, and error injection will just
	affect the pages associated with $$YOUR_PARTITION. So you must provide a
	free disk partition to stress test driver!

	This will do the stress testing thru madvise syscall (sec 4.1). However,
	there are more advanced test methods provided (sec 4.2, 4.3).

	Note, for all examples in the rest of this doc, it is supposed that $PWD is
	the stress subdir.

	1. Overview
	-----------

	The MCE stress test suite is a collection of tools and test scripts, which
	intends to achieve stress testing on Linux kernel MCA high level handlers
	that include HWPosion page recovery, soft page offline, and so on.

	In general, this test suite is designed to do stress testing thru various
	test interfaces, i.e. madvise syscall, HWPoison page injector, and APEI
	injector (see ACPI4.0 spec). And it's able to support most of popular
	Linux File Systems (FS), that is, there is an option for user to specify which
	FS type they want the test to be running on.

	If you just want to start testing as quickly as possible, you can skip
	section 2 & 3, just go to section 4 directly.


	2. Design Details
	-----------------

	The MCE stress test suite consists of four parts: test driver, workload
	controller, customized workloads, and background workloads.

	The main test idea is described as below:
	- Test driver launchs various customized workloads to continuously generate
	lots of pages with expected page states, Note, all of these workloads know
	about their expected results that should not be affected by Linux MCE high
	level handlers.
	- Then test driver injects MCE errors to these pages thru either madvise
	syscall or HWPoison injector or APEI injector. While Linux Kernel handling
	these MCE errors, all the workloads continue running normally,
	- After long time running, test driver will collect test result of each
	workload to see if any unexpected failures happened. In such a way, it can
	decide if any bug is found.
	- If any system panics or FS corruption happens, that means there must be a
	bug. It's the bottom line to decide if test gets pass.

	2.1 Test Driver

	Test driver (a.k.a hwpoison.sh) drives the whole test procedure. It's
	responsible for managing test environment, setting up error injection
	interface, controlling test progress, launching workloads, injecting page
	errors, as well as recording test logs and reporting test result.

	For detailed usage of hwpoison.sh test driver, please refer to:
	# ./hwpoison.sh -h

	2.2 Workload Controller

	Workload controller needs to have various test workloads running parallelly
	and continuously within a required duration time. We select ltp-pan
	program of Linux Test Project (LTP) as the workload controller of this
	stress test suite.

	Test driver (hwpoison.sh) interacts with ltp-pan in following ways:
	- hwpoison.sh generates a test config file that lists the workload type
	to be launched by ltp-pan.
	- hwpoison also passes test duration time and other workload specific
	parameters to ltp-pan via test config file.
	- ltp-pan makes each workload run and get finished in time, then test driver
	can get the result of each workload via corresponding result files.
	- finally, hwpoison.sh will decide the overall test result based on each
	workload result, and report final result out.

	2.3 Customized Workloads

	There are three types of customized workloads, which are intended to generate
	pages with various page state.

	* Type0: page-poisoning workload, meant to cover:
	- anonymous pages operations.
	- file data operations.

	* Type1: fs-metadata workload, meant to cover:
	- inode operations.

	* Type2: fs_type specific workload, meant to cover:
	- extended functions of some special FS.

	2.4 Background Workloads

	LTP is selected as the background workload to simulate normal system
	operations in background while stress testing is running.

	Besides LTP, there are also some alternatives, like AIM. We might extend more
	background workloads in future.

	2.5 Test Result

	How to determine that stress testing gets pass?
	- at least no kernel panics happens during stress testing.
	- fsck on the target disk at the end of stress testing should get pass.
	- there is no failure found by customized workloads, especially for
	page-poisoning workload.

	Where to get detailed test result?
	- When stress testing is done, the general test result is recorded in
	result/hwpoison.result, and the general test log is in result/hwpoison.log.
	However, you can specify them in following way:
	# hwpoison.sh -r $YOUR_RESULT -l $YOUR_LOG
	- The test result and test log of each workload are recorded as
	log/$workload/$workload.result and log/$workload/$workload.log.
	For example, for page-poisoning workload, its test result and test logs are
	log/page-poisoning/page-poisoning.result and
	log/page-poisoning/page-poisoning.log.
	- Besides, under each workload result dir, you can find other extra logs
	like pan_log, pan_output and etc. These logs are generated by ltp-pan
	workload controller. Usually they can help you understand what has been
	going on with ltp-pan while workload is running. Pls. refer to ltp-pan doc
	for details.


	3. Tools
	--------

	3.1 page-poisoning

	It is the page-poisoning workload. page-poisoning workload is an extension of
	tinjpage test program with a multi-process model. It spawns thousands of
	processes that inject HWPosion error to various pages simultaneously thru
	madvise syscall. Then it checks if these errors get handled correctly,
	i.e. whether each test process receives or doesn't receive SIGBUS signal as
	expected.

	For more info about page-poisoning workload, pls. read through README file
	under stress/tools/page-poisoning/.

	3.2 fs-metadata

	It is the fs-metadata workload. fs-metadata is designed to test i-node
	operations with heavy workload and make sure every i-node operation gets
	the expected result. In details, it firstly generates a huge directory
	hierarchy on the target disk, then it performs unlink operations on this
	directory hierarchy and duplicate a copy of the directory, finally it
	checks if these two directories are same as expected.

	For more info about fs-metadata workload, pls. read through README file
	under stress/tools/fs-metadata/.

	3.3 page-types

	page-types is a tool to query the page type of every memory page in the
	system. We use it to filter out pages with required page types. Test will
	inject error to these pages via error injector, although the page filter
	of HWPosion handler in Linux Kernel will filter them out for a second
	time. Note, the reason we need to use page-types to do first time filtering
	is just about performance.

	To install page-types on your test machine:

	# cd $KERNEL_SRC/Documentation/vm/
	# gcc -o page-types page-types.c
	# cp page-types /usr/bin/

	3.4 ltp-pan

	It's the workload controller of this stress test suite. In fact, ltp-pan
	is the test harness of LTP (Linux Test Project), and is included in
	LTP package. For more information, please refer to ltp-pan document of LTP.


	4. Usage Guide
	--------------

	This section is trying to show you how to conduct the stress testing thru
	various test interfaces.

	As an example, we choose to run stress testing based on partition /dev/sda1
	for 1 hour. Note, we've installed LTP to /ltp.

	4.1 Stress Test thru Madvise Syscall.

	To run this stress testing, you need to strictly follow below test
	instructions.

	* Test instructions:

	- make sure following kernel options are enabled:
	CONFIG_X86_MCE=y
	CONFIG_MEMORY_FAILURE=y

	- build and run stress testing
	# make
	# ./hwpoison.sh -d $YOUR_PARTITION -M -o $YOUR_LTP_DIR

	* Example:

	- launch testing
	# ./hwpoison.sh -d /dev/sda1 -M -t 3600

	- general test results
	result: result/hwpoison.result
	logs: result/hwpoison.log

	- detailed workload results
	result: log/page-poisoning/page-poisoning.result
	log: log/page-poisoning/page-poisoning.log

	4.2 Stress Test thru HWPosion Page Injector

	This is the default test method of this stress test suite.

	To run this stress testing, you need to strictly follow below test
	instructions.

	* Test instructions:

	- make sure following kernel options are enabled:
	CONFIG_X86_MCE=y
	CONFIG_MEMORY_FAILURE=y
	CONFIG_DEBUG_KERNEL=y
	CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
	CONFIG_HWPOISON_INJECT=y

	- build and run stress testing
	# make
	# ./hwpoison.sh -d $YOUR_PARTITION -o $YOUR_LTP_DIR -L

	* Example:

	- launch testing
	# ./hwpoison.sh -d /dev/sda1 -t 3600 -L

	- general test results
	result: result/hwpoison.result
	logs: result/hwpoison.log

	- detailed workload results
	fs-metadata result: log/fs-metadata/fs-metadata.result
	fs-metadata log: log/fs-metadata/fs-metadata.log
	ltp result: log/ltp/ltp.result
	ltp log: log/ltp/ltp.log
	fs-specific result: log/fs-specific/fs-specific.result
	fs-specific log: log/fs-specific/fs-specific.log

	4.3 Stress Test thru APEI Injector

	To run this stress testing, you need to follow below test instructions.

	* Test instructions:

	- make sure following kernel options are enabled:
	CONFIG_X86_MCE=y
	CONFIG_X86_MCE_INTEL=y
	CONFIG_MEMORY_FAILURE=y
	CONFIG_ACPI_APEI=y
	CONFIG_ACPI_APEI_EINJ=y

	- build and run stress testing
	# make
	# ./hwpoison.sh -d $YOUR_PARTITION -o $YOUR_LTP_DIR -L -A

	* Example:

	- launch testing
	# ./hwpoison.sh -d /dev/sda1 -t 3600 -L -A

	- general test results
	result: result/hwpoison.result
	logs: result/hwpoison.log

	- detailed workload results
	fs-metadata result: log/fs-metadata/fs-metadata.result
	fs-metadata log: log/fs-metadata/fs-metadata.log
	ltp result: log/ltp/ltp.result
	ltp log: log/ltp/ltp.log
	fs-specific result: log/fs-specific/fs-specific.result
	fs-specific log: log/fs-specific/fs-specific.log


	5. FAQs
	-------

	Here is a collection of frequently asked questions:

	Q: How to tell test driver not to format my disk partition?
	A: Use the option '-N'.

	Q: Can three types of tests run on same sytem simultaneously?
	A: No. There are limitations in Linux Kernel HWPoison page filtering.

	Q: Can I run this stress testing on multiple disks parallely?
	A: Yes. But it requires updated Kernel patches for HWPosion page filtering.
	Now, it just supports one same test with same pagetype flags specified.