doc/README - pub/scm/linux/kernel/git/will/aiaiai - Git at Google

 1 Overview
 ~~~~~~~~~~

 This is a set of scripts and tools for verifying Linux kernel patches.
 Most of the scripts accept "-h" or "--help" options, which you can use
 to get an idea what different scripts do.

 It is important to know that all the scripts are written so that:
  * all the debugging and verbose (-v) cruft is printed to stderr
  * all the user-friendly output is printed to stdout

 Thus, if you for example run "./aiaiai-test-patchset -v .... 2>log", then the
 "log" file will contain all the unreadable debugging cruft and the nice
 output will be visible on the console.

 Also it is useful to know that the scripts never modify the repositories
 you specify and the scripts remove all the temporary files upon exit,
 Ctrl-C interruption, or error (unless -p option is specified).

 The layout of the repository.

 * aiaiai/email/
     All the scripts related to emails handling. If you use Aiaiai locally
     you do not need these scripts.
 * aiaiai/helpers/
     Variouls programs and scripts which Aiaiai users should not usually use
     directly. Instead, Aiaiai scripts use them internally.
 * aiaiai/doc
     Some documentation.
 * aiaiai/tests/
     Unit-tests.
 * aiaiai/
     The root repository directory contains the scripts which do all the
     patch checking work. The central script is 'aiaiai-test-patchset' which
     uses all the other scripts as helpers.

 Let's briefly review the contents of the repository.


 1.1 E-mail-related scripts
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

 * email/aiaiai-email-lda
     This is Aiaiai's local delivery agent script which receives the incoming
     emails from the mail server (via a pipe), processes them (drops all e-mails
     which do not start with [PATCH*], detects and collects patch-sets), and
     puts them to the "queue" sub-directory of the work directory. This
     should, in turn, wake up the "aiaiai-email-dispatcher" script which will
     take further care of the patch or patch-set.

 * email/aiaiai-email-dispatcher
     This script waits for the incoming patch-sets in the "queue" directory and
     runs a user-defined command for every incoming patch-set. The role of this
     script is to limit the amount of patch-sets we can run at a time. You
     can configure this via command line options.

     As it was already said, in our setup 'aiaiai-email-dispatcher' runs
     'aiaiai-email-test-patchset'.

 * email/aiaiai-email-dispatcher-helper
     This is a private helper script for 'aiaiai-email-dispatcher'.

 * email/aiaiai-email-test-patchset
     This script parses the incoming e-mail (in mbox format), finds out which
     kernel tree has to be used, sends a notification to the sender, runs
     the 'aiaiai-test-patchset' script with the right command-line arguments, and
     sends the results of 'aiaiai-test-patchset' back to the patch submitter.

 * email/aiaiai.cfg
     This is an example of 'aiaiai-email-test-patchset' configuration file. It
     describes the projects Aiaiai supports and their parameters. Take a
     look inside - the file is self-descriptive.

 Here is how all these scripts work together.  When you send an e-mail to the
 server which runs Aiaiai, the MTA pipes it to 'aiaiai-email-lda'. We used the
 postfix MTA, and the 'aiaiai' Linux user had the following file in its home
 directory:

 $ cat $HOME/.forward
 |/home/aiaiai/bin/my-aiaiai-lda

 Which made sure that all e-mails coming to aiaiai@our-domain.ququ were piped
 to the my-aiaiai-lda script. The script simply ran aiaiai-email-lda with
 the options we needed:

 $ cat /home/aiaiai/bin/my-aiaiai-lda
 #!/bin/sh
 $HOME/git/email/aiaiai-email-lda -v --reap-archive=43200 --reap-incomplete=10 -- $HOME/aiaiai-workdir >> "$HOME/aiaiai-logs/email-lda.log" 2>&1


 So, aiaiai-email-lda collects patches belonging to the patch-sets, and when it
 has all of them, it squashes them into one mbox file and moves them to the
 queue directory. This will wake up the 'aiaiai-email-dispatcher' script, which
 will execute 'aiaiai-email-test-patchset' for this mbox (unless there are
 already too many jobs, in which case it will wait).

 The 'aiaiai-email-test-patchset' will parse the patch-set, find out which
 kernel it has to be tested against using the configuration file, send a
 notification e-mail to the patch sender, and execute the 'aiaiai-test-patchset'
 script.

 The 'aiaiai-test-patchset' script does all the testing work and outputs the
 test results to stdout. 'aiaiai-email-test-patchset' captures the output and
 sends it back to the patch submitter.

 A more complicated mail setup dealing with forwards and multi-part emails
 would be :
 $ cat /home/aiaiai/bin/my-aiaiai-lda
 	#!/bin/sh
 	formail_opt=""

 	mail=$(mktemp)
 	cat > $mail

 	to="$(formail -c -z -x "To:" < $mail)"
 	[ -n "$to" ] && formail_opt="$formail_opt -I \"To: $to\""

 	project="$(formail -c -z -x "X-Aiaiai-Project:" < $mail)"
 	[ -n "$project" ] && formail_opt="$formail_opt -I \"X-Aiaiai-Project: $project\""

 	commit="$(formail -c -z -x "X-Aiaiai-Commit:" < $mail)"
 	[ -n "$commit" ] && formail_opt="$formail_opt -I \"X-Aiaiai-Commit: $commit\""

 	cat "$mail" | eval formail $formail_opt -d -s "$HOME/git/email/aiaiai-email-lda" \
 	-v --reap-archive=43200 --reap-incomplete=10 -- "$HOME/aiaiai-workdir" \
 	>> "$HOME/aiaiai-logs/email-lda.log" 2>&1

 1.2 Email Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~

 aiaiai-email scripts use a common configuration file, which is documented in
 doc/email/example-aiaiai.cfg

 By default aiaiai expects email targeting a specific project to be delivered to
 a specific aiaiai recipient address which includes it, e.g:
 aiaiai+foo@domain.tld.

 In case you would want to use the same aiaiai recipient email address for all
 projects, you need to create a hook file which will be looking at the
 X-Aiaiai-Project header in the mbox file, and output it to stdout (see:
 email/aiaiai-email-test-patchset). An example hook script could look like this:

 #!/bin/sh
 CFGFILE="$1"
 MBOX="$2"
 grep "X-Aiaiai-Project" $MBOX
 ret=$?

 1.3 Non-e-mail scripts
 ~~~~~~~~~~~~~~~~~~~~~~

 * aiaiai-test-patchset
     The central script which tests a patch-set is 'aiaiai-test-patchset'. You
     should use it if you want to test your patches locally. It accepts patches
     in the mbox format. You should specify various input parameters like the
     path to the kernel to test against, the commit id to test against, which
     tests to run, etc.

 * aiaiai-make-kernel
     Low-level scripts which builds the kernel. It has several switches which
     change its behavior, e.g., you can select whether you want to run
     sparse/smatch/cppcheck/coccinelle tools or not.

     'aiaiai-make-kernel' utilizes the 'aiaiai-locker' utility to make
     sure the build logs of parallel builds are not scrambled and stay readable.

 * aiaiai-locker
     This utility was inspired by Eric Melski's blog-post I found in the Internet.
     We use this tool to make Linux kernel build logs saner than parallel build
     produces (e.g., "make -j 20"). Indeed, parallel make scrambles all the
     lines which makes build logs not comparable and not readable.

     The idea is to run all make commands via the 'aiaiai-locker' utility using
     the GNU make 'SHELL' variable re-definition. 'aiaiai-locker' intercepts the
     stdout and stderr output of commands, gathers all the output until the
     command finishes, and locks a common lockfile, and prints the intercepted
     output. So the effect of this is that the in the build log the output of
     one program goes as one big block and it is not mixed with the output of
     another command.

 * aiaiai-checker
     A helper script for 'aiaiai-make-kernel' which runs
     sparse/smatch/cppcheck/coccinelle tools. The reason we need it is because
     the kernel build system allows only one checking program at a time, but we
     want to run multiple of them.

 * aiaiai-diff-log
     Compares 2 build logs and provides the differences. This script used the
     'helpers/remap-log' tool to re-map line numbers in the warnings of the
     first build log to correspond to line numbers in the second build-log.

 * aiaiai-diff-log-helper
     A helper script for aiaiai-diff-log which sorts 2 build logs and compares
     them.  Written in Python because the shell version was too slow.

 * helpers/remap-log
     This is a small C program originally written by Al Viro which helps
     comparing 2 build logs by remapping line numbers. Roughly speaking,
     'remap-log' takes the build log of the pre-patched kernel and the patch and
     changes line numbers in the build log to match line numbers in the build
     log of the patched kernel. For example, supposed that you have the
     following warning in the pre-patched kernel:

     driver.c:100 Unused variable 'ret'

     and your patch adds a new include to the driver.c file, so the patched
     kernel will produce the following warning:

     driver.c:101 Unused variable 'ret'

     Obviously, you do not want to bug the patch author about this warning
     because it was not his patch which introduced it. And if you used a plain
     diff, it would produce something like:

     -driver.c:100 Unused variable 'ret'
     +driver.c:101 Unused variable 'ret'

     But we pass the build log for the pre-patched kernel via 'remap-log' and it
     transforms that warning into

     driver.c:101 Unused variable 'ret'

     so the build logs of the pre-patched and patched kernels will contain the
     same line and the diff won't show anything.

 * aiaiai-test-bisectability
     Accepts a patch-set as the input and checks that the patch-set does not
     break compilation at any step. This is achieved by applying every patch in
     the series one-by-one, compiling the kernel and checking that compilation
     was successful.

 * aiaiai-decode-rfc-2047
     A small helper script which decodes unreadable patch subjects encoded with
     RFC-2047 MIME encoding.

 * aiaiai-concat-mboxes
     A little helper script which concatenates multiple mbox files into one.

 * aiaiai-match-keywords
    A simple script which can be used to match certain unwelcom keywords in
    patches.

 To summarize:
  * 'aiaiai-test-patchset' is the central script
    * it runs 'checkpatch.pl'
    * it runs 'aiaiai-test-bisectability'
    * for every configuration specified by the user, it builds the kernel
      before and after the patch using the 'aiaiai-make-kernel' script, then
      compares the build log using the 'aiaiai-diff-log' script.
    * Things are done in parallel.
    * The results of all tests are collected and printed to stdout
    * The "verbose" contents is printed to stderr
    * Many of the jobs above are run in parallel

 The 'aiaiai-diff-log' script which compares 2 logs. It uses the 'remap-log'
 tool for comparing 2 build logs the smart way.


 2 Using Aiaiai locally
 ~~~~~~~~~~~~~~~~~~~~~~

 If you just want to test a patch-set locally, you should use the
 'aiaiai-test-patchset' script. Do not forget to build the remap-log.c file in
 the 'helpers' subdirectory.

 Here are some examples.

 1. Test a patch or a patch-set p.mbox against the linux-vfs kernel tree,
    against the 'origin/next' branch, and use the 'i386_defconfig'
    defconfig, the architecture is i386 (translates to 'make ARCH=i386'). Use
    16 jobs.

    cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
        --cppcheck --coccinelle -j 16 -c origin/next \
        /<path>/linux-vfs i386_defconfig,i386

 2. Test a patch or a patch-set p.mbox against the linux-omap kernel tree,
    against the master branch, use the 'omap2plus_defconfig' defconfig,
    'arm' architecture and 'arm-eabi-' cross-compiler prefix (translates to
    'make CROSS_COMPILE=arm-eabi-').

    cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
        --cppcheck --coccinelle -j 16 -c master \
        /<path>/linux-omap omap2plus_defconfig,arm,arm-eabi-

 3. If you are a maintainer you may want to test patches for more than one
    architecture. You can do this - you'll just need to maintain defconfigs.
    E.g., to test p.mbox against both i386_defconfig (Intel i386) and
    omap2plus_defconfig (ARM-based SoC), you can run:

    cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
        --cppcheck --coccinelle -j 16 -c master /<path>/linux-my \
        i386_defconfig,i386 omap2plus_defconfig,arm,arm-eabi-

    IOW, just specify defconfigs, architectures, and cross-compiler prefixes at
    the end.

 4. Test a patch or a patch-set p.mbox against the net-next kernel tree,
    against the 'origin/master' branch, using a randomly generated config, the
    architecture is i386 (translates to 'make ARCH=i386'). Use 16 jobs.

    cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
        --cppcheck --coccinelle -j 16 -c origin/master \
        /<path>/net-next randconfig,i386

 Note, the 'aiaiai-test-patchset' script accepts one mbox file via stdio or the
 '-i' option. If you have several files belonging to one patch-set and you want
 to test them all, you need to concatenate them. Use the 'aiaiai-concat-mboxes'
 script for this. E.g., if you have '01.mbox' and '02.mbox' comprising a
 patch-set, you can test do like this:

 $ aiaiai-concat-mboxes 01.mbox 02.mbox | aiaiai-test-patchset ....

 Patches generated with 'git format-patch' are also mbox files so you can test
 them with 'aiaiai-test-patchset' as well. Just remember to use the '--thread'
 option to make 'git format-patch' generate "Message-Id:" and "In-Reply-To:"
 headers. For example:

 $ git format-patch --thread HEAD~10
 $ aiaiai-concat-mboxes 00* | aiaiai-test-patchset ...

 I usually run 'aiaiai-test-patchset' with the lowest priority in order to make
 sure the machine is still usable for other things:

 $ nice -n19 ionice -c3 aiaiai-test-patchset ...

 And for the kernel trees I test frequently against, I usually have a helper
 script so that I do not have to type the 'aiaiai-test-patchset' options over
 and over again.


 3 Dependencies
 ~~~~~~~~~~~~~~

 3.1 smatch
 ~~~~~~~~~~

 Project homepage: http://smatch.sourceforge.net

 I do not think this tool is a part of Linux distributions. I found it very
 useful. To use it, you have to clone it, patch it, and install in your system.

 $ git clone git://repo.or.cz/smatch.git
 $ cd smatch
 $ make PREFIX=$HOME/programs/smatch all install
 $ export PATH="$PATH:/home/aiaiai/programs/smatch/bin"

 These commands will install smatch to "/home/aiaiai/programs/smatch". Of
 course, you should specify your path and make sure smatch is in your "PATH"
 environment variable.


 3.2 sparse
 ~~~~~~~~~~

 Similarly to smatch, you need to install it manually. This is what I did:

 $ git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git
 $ cd sparse
 $ make
 $ cp sparse $HOME/bin
 $ export PATH="$HOME/bin:$PATH"


 3.3 coccinelle
 ~~~~~~~~~~~~~~

 Coccinelle or "spatch" is a great semantic patch tool, very useful on its own.
 The Linux kernel contains a bunch of coccinelle semantic patches which catch
 various types of API abuses etc. We store a copy of these scripts in the
 "helpers" subdirectory.

 The "spatch" tool is part of distributions like Debian and Fedora, so you most
 probably just need to install the corresponding package. The binary program is
 called "spatch".

 You may find a lot of information about this awesome tool by Googling.


 3.4 Dependencies
 ~~~~~~~~~~~~~~~~

 The scripts use many external programs which are usually part of the Linux
 distribution. Here are they are, but the list is not complete - if you spot
 a missing dependency - please, let me know or send a patch against the README
 file.

 gcc
 make
 cppcheck
 formail
 perl
 python
 mutt
 sed
 grep
 awk
 git
 lockfile
 diff
 patch
 inotifywait


 3.5 Code conventions
 ~~~~~~~~~~~~~~~~~~~~

 The scripts have been written for the "dash" shell and they do not use
 any "bashizm" and should be very portable. Please, keep this in mind
 when changing them. The scripts try to follow a uniform coding style
 and conventions. Below are some of them.

 * Stdout is used only for user-friendly stuff. In many cases it is assumed that
   the higher-level scripts just capture the stdout of lower-level scripts and
   provide that to the user. For example, 'aiaiai-test-patchset' captures stdout
   of the 'aiaiai-test-bisectability' scripts and uses it for reporting to the
   user. In turn, the 'aiaiai-email-test-patchset' script captures stdout of the
   'aiaiai-test-patchset' script and sends it back to the user via e-mail.

 * Stderr is use for all the debugging, verbose and additional prints.

 * The scripts use the "-e" shell option and in case of any error (non-zero
   return code from any program) we exit immediately. This allows us simplify
   the scripts and avoid handling errors for all the commands which is very
   convenient.

   Well, the automatic exit does not work if the errorred command is in a
   pipeline, and not the last. But this is not a big deal in most cases.

   And sometimes we do not want the whole script to error out if a command
   returns non zero (e.g., grep returns 1 if it did not match the pattern),
   in which case we usually use "||:" after the command.

 * We use "-u" shell option to make sure our scripts are of good quality.

 * We try to be more secure and use the "-f" shell option whenever we can.

 * We make use of the external "libshell" library which is part of the Alt
   Linux project and is extremely useful.

 * All the temporary files should be remover upon exit or interruption
   (ctrl-C must be handled gracefully)

 * Users' git repository have to be treated as read-only and no modifications
   should be made there. All the work has to be done in a clone of the original
   repository.
	1 Overview
	~~~~~~~~~~

	This is a set of scripts and tools for verifying Linux kernel patches.
	Most of the scripts accept "-h" or "--help" options, which you can use
	to get an idea what different scripts do.

	It is important to know that all the scripts are written so that:
	* all the debugging and verbose (-v) cruft is printed to stderr
	* all the user-friendly output is printed to stdout

	Thus, if you for example run "./aiaiai-test-patchset -v .... 2>log", then the
	"log" file will contain all the unreadable debugging cruft and the nice
	output will be visible on the console.

	Also it is useful to know that the scripts never modify the repositories
	you specify and the scripts remove all the temporary files upon exit,
	Ctrl-C interruption, or error (unless -p option is specified).

	The layout of the repository.

	* aiaiai/email/
	All the scripts related to emails handling. If you use Aiaiai locally
	you do not need these scripts.
	* aiaiai/helpers/
	Variouls programs and scripts which Aiaiai users should not usually use
	directly. Instead, Aiaiai scripts use them internally.
	* aiaiai/doc
	Some documentation.
	* aiaiai/tests/
	Unit-tests.
	* aiaiai/
	The root repository directory contains the scripts which do all the
	patch checking work. The central script is 'aiaiai-test-patchset' which
	uses all the other scripts as helpers.

	Let's briefly review the contents of the repository.


	1.1 E-mail-related scripts
	~~~~~~~~~~~~~~~~~~~~~~~~~~

	* email/aiaiai-email-lda
	This is Aiaiai's local delivery agent script which receives the incoming
	emails from the mail server (via a pipe), processes them (drops all e-mails
	which do not start with [PATCH*], detects and collects patch-sets), and
	puts them to the "queue" sub-directory of the work directory. This
	should, in turn, wake up the "aiaiai-email-dispatcher" script which will
	take further care of the patch or patch-set.

	* email/aiaiai-email-dispatcher
	This script waits for the incoming patch-sets in the "queue" directory and
	runs a user-defined command for every incoming patch-set. The role of this
	script is to limit the amount of patch-sets we can run at a time. You
	can configure this via command line options.

	As it was already said, in our setup 'aiaiai-email-dispatcher' runs
	'aiaiai-email-test-patchset'.

	* email/aiaiai-email-dispatcher-helper
	This is a private helper script for 'aiaiai-email-dispatcher'.

	* email/aiaiai-email-test-patchset
	This script parses the incoming e-mail (in mbox format), finds out which
	kernel tree has to be used, sends a notification to the sender, runs
	the 'aiaiai-test-patchset' script with the right command-line arguments, and
	sends the results of 'aiaiai-test-patchset' back to the patch submitter.

	* email/aiaiai.cfg
	This is an example of 'aiaiai-email-test-patchset' configuration file. It
	describes the projects Aiaiai supports and their parameters. Take a
	look inside - the file is self-descriptive.

	Here is how all these scripts work together. When you send an e-mail to the
	server which runs Aiaiai, the MTA pipes it to 'aiaiai-email-lda'. We used the
	postfix MTA, and the 'aiaiai' Linux user had the following file in its home
	directory:

	$ cat $HOME/.forward
	\|/home/aiaiai/bin/my-aiaiai-lda

	Which made sure that all e-mails coming to aiaiai@our-domain.ququ were piped
	to the my-aiaiai-lda script. The script simply ran aiaiai-email-lda with
	the options we needed:

	$ cat /home/aiaiai/bin/my-aiaiai-lda
	#!/bin/sh
	$HOME/git/email/aiaiai-email-lda -v --reap-archive=43200 --reap-incomplete=10 -- $HOME/aiaiai-workdir >> "$HOME/aiaiai-logs/email-lda.log" 2>&1


	So, aiaiai-email-lda collects patches belonging to the patch-sets, and when it
	has all of them, it squashes them into one mbox file and moves them to the
	queue directory. This will wake up the 'aiaiai-email-dispatcher' script, which
	will execute 'aiaiai-email-test-patchset' for this mbox (unless there are
	already too many jobs, in which case it will wait).

	The 'aiaiai-email-test-patchset' will parse the patch-set, find out which
	kernel it has to be tested against using the configuration file, send a
	notification e-mail to the patch sender, and execute the 'aiaiai-test-patchset'
	script.

	The 'aiaiai-test-patchset' script does all the testing work and outputs the
	test results to stdout. 'aiaiai-email-test-patchset' captures the output and
	sends it back to the patch submitter.

	A more complicated mail setup dealing with forwards and multi-part emails
	would be :
	$ cat /home/aiaiai/bin/my-aiaiai-lda
	#!/bin/sh
	formail_opt=""

	mail=$(mktemp)
	cat > $mail

	to="$(formail -c -z -x "To:" < $mail)"
	[ -n "$to" ] && formail_opt="$formail_opt -I \"To: $to\""

	project="$(formail -c -z -x "X-Aiaiai-Project:" < $mail)"
	[ -n "$project" ] && formail_opt="$formail_opt -I \"X-Aiaiai-Project: $project\""

	commit="$(formail -c -z -x "X-Aiaiai-Commit:" < $mail)"
	[ -n "$commit" ] && formail_opt="$formail_opt -I \"X-Aiaiai-Commit: $commit\""

	cat "$mail" \| eval formail $formail_opt -d -s "$HOME/git/email/aiaiai-email-lda" \
	-v --reap-archive=43200 --reap-incomplete=10 -- "$HOME/aiaiai-workdir" \
	>> "$HOME/aiaiai-logs/email-lda.log" 2>&1

	1.2 Email Configuration
	~~~~~~~~~~~~~~~~~~~~~~~

	aiaiai-email scripts use a common configuration file, which is documented in
	doc/email/example-aiaiai.cfg

	By default aiaiai expects email targeting a specific project to be delivered to
	a specific aiaiai recipient address which includes it, e.g:
	aiaiai+foo@domain.tld.

	In case you would want to use the same aiaiai recipient email address for all
	projects, you need to create a hook file which will be looking at the
	X-Aiaiai-Project header in the mbox file, and output it to stdout (see:
	email/aiaiai-email-test-patchset). An example hook script could look like this:

	#!/bin/sh
	CFGFILE="$1"
	MBOX="$2"
	grep "X-Aiaiai-Project" $MBOX
	ret=$?

	1.3 Non-e-mail scripts
	~~~~~~~~~~~~~~~~~~~~~~

	* aiaiai-test-patchset
	The central script which tests a patch-set is 'aiaiai-test-patchset'. You
	should use it if you want to test your patches locally. It accepts patches
	in the mbox format. You should specify various input parameters like the
	path to the kernel to test against, the commit id to test against, which
	tests to run, etc.

	* aiaiai-make-kernel
	Low-level scripts which builds the kernel. It has several switches which
	change its behavior, e.g., you can select whether you want to run
	sparse/smatch/cppcheck/coccinelle tools or not.

	'aiaiai-make-kernel' utilizes the 'aiaiai-locker' utility to make
	sure the build logs of parallel builds are not scrambled and stay readable.

	* aiaiai-locker
	This utility was inspired by Eric Melski's blog-post I found in the Internet.
	We use this tool to make Linux kernel build logs saner than parallel build
	produces (e.g., "make -j 20"). Indeed, parallel make scrambles all the
	lines which makes build logs not comparable and not readable.

	The idea is to run all make commands via the 'aiaiai-locker' utility using
	the GNU make 'SHELL' variable re-definition. 'aiaiai-locker' intercepts the
	stdout and stderr output of commands, gathers all the output until the
	command finishes, and locks a common lockfile, and prints the intercepted
	output. So the effect of this is that the in the build log the output of
	one program goes as one big block and it is not mixed with the output of
	another command.

	* aiaiai-checker
	A helper script for 'aiaiai-make-kernel' which runs
	sparse/smatch/cppcheck/coccinelle tools. The reason we need it is because
	the kernel build system allows only one checking program at a time, but we
	want to run multiple of them.

	* aiaiai-diff-log
	Compares 2 build logs and provides the differences. This script used the
	'helpers/remap-log' tool to re-map line numbers in the warnings of the
	first build log to correspond to line numbers in the second build-log.

	* aiaiai-diff-log-helper
	A helper script for aiaiai-diff-log which sorts 2 build logs and compares
	them. Written in Python because the shell version was too slow.

	* helpers/remap-log
	This is a small C program originally written by Al Viro which helps
	comparing 2 build logs by remapping line numbers. Roughly speaking,
	'remap-log' takes the build log of the pre-patched kernel and the patch and
	changes line numbers in the build log to match line numbers in the build
	log of the patched kernel. For example, supposed that you have the
	following warning in the pre-patched kernel:

	driver.c:100 Unused variable 'ret'

	and your patch adds a new include to the driver.c file, so the patched
	kernel will produce the following warning:

	driver.c:101 Unused variable 'ret'

	Obviously, you do not want to bug the patch author about this warning
	because it was not his patch which introduced it. And if you used a plain
	diff, it would produce something like:

	-driver.c:100 Unused variable 'ret'
	+driver.c:101 Unused variable 'ret'

	But we pass the build log for the pre-patched kernel via 'remap-log' and it
	transforms that warning into

	driver.c:101 Unused variable 'ret'

	so the build logs of the pre-patched and patched kernels will contain the
	same line and the diff won't show anything.

	* aiaiai-test-bisectability
	Accepts a patch-set as the input and checks that the patch-set does not
	break compilation at any step. This is achieved by applying every patch in
	the series one-by-one, compiling the kernel and checking that compilation
	was successful.

	* aiaiai-decode-rfc-2047
	A small helper script which decodes unreadable patch subjects encoded with
	RFC-2047 MIME encoding.

	* aiaiai-concat-mboxes
	A little helper script which concatenates multiple mbox files into one.

	* aiaiai-match-keywords
	A simple script which can be used to match certain unwelcom keywords in
	patches.

	To summarize:
	* 'aiaiai-test-patchset' is the central script
	* it runs 'checkpatch.pl'
	* it runs 'aiaiai-test-bisectability'
	* for every configuration specified by the user, it builds the kernel
	before and after the patch using the 'aiaiai-make-kernel' script, then
	compares the build log using the 'aiaiai-diff-log' script.
	* Things are done in parallel.
	* The results of all tests are collected and printed to stdout
	* The "verbose" contents is printed to stderr
	* Many of the jobs above are run in parallel

	The 'aiaiai-diff-log' script which compares 2 logs. It uses the 'remap-log'
	tool for comparing 2 build logs the smart way.


	2 Using Aiaiai locally
	~~~~~~~~~~~~~~~~~~~~~~

	If you just want to test a patch-set locally, you should use the
	'aiaiai-test-patchset' script. Do not forget to build the remap-log.c file in
	the 'helpers' subdirectory.

	Here are some examples.

	1. Test a patch or a patch-set p.mbox against the linux-vfs kernel tree,
	against the 'origin/next' branch, and use the 'i386_defconfig'
	defconfig, the architecture is i386 (translates to 'make ARCH=i386'). Use
	16 jobs.

	cat p.mbox \| ./aiaiai-test-patchset --bisectability --sparse --smatch \
	--cppcheck --coccinelle -j 16 -c origin/next \
	/<path>/linux-vfs i386_defconfig,i386

	2. Test a patch or a patch-set p.mbox against the linux-omap kernel tree,
	against the master branch, use the 'omap2plus_defconfig' defconfig,
	'arm' architecture and 'arm-eabi-' cross-compiler prefix (translates to
	'make CROSS_COMPILE=arm-eabi-').

	cat p.mbox \| ./aiaiai-test-patchset --bisectability --sparse --smatch \
	--cppcheck --coccinelle -j 16 -c master \
	/<path>/linux-omap omap2plus_defconfig,arm,arm-eabi-

	3. If you are a maintainer you may want to test patches for more than one
	architecture. You can do this - you'll just need to maintain defconfigs.
	E.g., to test p.mbox against both i386_defconfig (Intel i386) and
	omap2plus_defconfig (ARM-based SoC), you can run:

	cat p.mbox \| ./aiaiai-test-patchset --bisectability --sparse --smatch \
	--cppcheck --coccinelle -j 16 -c master /<path>/linux-my \
	i386_defconfig,i386 omap2plus_defconfig,arm,arm-eabi-

	IOW, just specify defconfigs, architectures, and cross-compiler prefixes at
	the end.

	4. Test a patch or a patch-set p.mbox against the net-next kernel tree,
	against the 'origin/master' branch, using a randomly generated config, the
	architecture is i386 (translates to 'make ARCH=i386'). Use 16 jobs.

	cat p.mbox \| ./aiaiai-test-patchset --bisectability --sparse --smatch \
	--cppcheck --coccinelle -j 16 -c origin/master \
	/<path>/net-next randconfig,i386

	Note, the 'aiaiai-test-patchset' script accepts one mbox file via stdio or the
	'-i' option. If you have several files belonging to one patch-set and you want
	to test them all, you need to concatenate them. Use the 'aiaiai-concat-mboxes'
	script for this. E.g., if you have '01.mbox' and '02.mbox' comprising a
	patch-set, you can test do like this:

	$ aiaiai-concat-mboxes 01.mbox 02.mbox \| aiaiai-test-patchset ....

	Patches generated with 'git format-patch' are also mbox files so you can test
	them with 'aiaiai-test-patchset' as well. Just remember to use the '--thread'
	option to make 'git format-patch' generate "Message-Id:" and "In-Reply-To:"
	headers. For example:

	$ git format-patch --thread HEAD~10
	$ aiaiai-concat-mboxes 00* \| aiaiai-test-patchset ...

	I usually run 'aiaiai-test-patchset' with the lowest priority in order to make
	sure the machine is still usable for other things:

	$ nice -n19 ionice -c3 aiaiai-test-patchset ...

	And for the kernel trees I test frequently against, I usually have a helper
	script so that I do not have to type the 'aiaiai-test-patchset' options over
	and over again.


	3 Dependencies
	~~~~~~~~~~~~~~

	3.1 smatch
	~~~~~~~~~~

	Project homepage: http://smatch.sourceforge.net

	I do not think this tool is a part of Linux distributions. I found it very
	useful. To use it, you have to clone it, patch it, and install in your system.

	$ git clone git://repo.or.cz/smatch.git
	$ cd smatch
	$ make PREFIX=$HOME/programs/smatch all install
	$ export PATH="$PATH:/home/aiaiai/programs/smatch/bin"

	These commands will install smatch to "/home/aiaiai/programs/smatch". Of
	course, you should specify your path and make sure smatch is in your "PATH"
	environment variable.


	3.2 sparse
	~~~~~~~~~~

	Similarly to smatch, you need to install it manually. This is what I did:

	$ git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git
	$ cd sparse
	$ make
	$ cp sparse $HOME/bin
	$ export PATH="$HOME/bin:$PATH"


	3.3 coccinelle
	~~~~~~~~~~~~~~

	Coccinelle or "spatch" is a great semantic patch tool, very useful on its own.
	The Linux kernel contains a bunch of coccinelle semantic patches which catch
	various types of API abuses etc. We store a copy of these scripts in the
	"helpers" subdirectory.

	The "spatch" tool is part of distributions like Debian and Fedora, so you most
	probably just need to install the corresponding package. The binary program is
	called "spatch".

	You may find a lot of information about this awesome tool by Googling.


	3.4 Dependencies
	~~~~~~~~~~~~~~~~

	The scripts use many external programs which are usually part of the Linux
	distribution. Here are they are, but the list is not complete - if you spot
	a missing dependency - please, let me know or send a patch against the README
	file.

	gcc
	make
	cppcheck
	formail
	perl
	python
	mutt
	sed
	grep
	awk
	git
	lockfile
	diff
	patch
	inotifywait


	3.5 Code conventions
	~~~~~~~~~~~~~~~~~~~~

	The scripts have been written for the "dash" shell and they do not use
	any "bashizm" and should be very portable. Please, keep this in mind
	when changing them. The scripts try to follow a uniform coding style
	and conventions. Below are some of them.

	* Stdout is used only for user-friendly stuff. In many cases it is assumed that
	the higher-level scripts just capture the stdout of lower-level scripts and
	provide that to the user. For example, 'aiaiai-test-patchset' captures stdout
	of the 'aiaiai-test-bisectability' scripts and uses it for reporting to the
	user. In turn, the 'aiaiai-email-test-patchset' script captures stdout of the
	'aiaiai-test-patchset' script and sends it back to the user via e-mail.

	* Stderr is use for all the debugging, verbose and additional prints.

	* The scripts use the "-e" shell option and in case of any error (non-zero
	return code from any program) we exit immediately. This allows us simplify
	the scripts and avoid handling errors for all the commands which is very
	convenient.

	Well, the automatic exit does not work if the errorred command is in a
	pipeline, and not the last. But this is not a big deal in most cases.

	And sometimes we do not want the whole script to error out if a command
	returns non zero (e.g., grep returns 1 if it did not match the pattern),
	in which case we usually use "\|\|:" after the command.

	* We use "-u" shell option to make sure our scripts are of good quality.

	* We try to be more secure and use the "-f" shell option whenever we can.

	* We make use of the external "libshell" library which is part of the Alt
	Linux project and is extremely useful.

	* All the temporary files should be remover upon exit or interruption
	(ctrl-C must be handled gracefully)

	* Users' git repository have to be treated as read-only and no modifications
	should be made there. All the work has to be done in a clone of the original
	repository.