1 Overview
~~~~~~~~~~

This is a set of scripts and tools for verifying Linux kernel patches.
Most of the scripts accept "-h" or "--help" options, which you can use
to get an idea what different scripts do.

It is important to know that all the scripts are written so that:
 * all the debugging and verbose (-v) cruft is printed to stderr
 * all the user-friendly output is printed to stdout

Thus, if you for example run "./aiaiai-test-patchset -v .... 2>log", then the
"log" file will contain all the unreadable debugging cruft and the nice
output will be visible on the console.

Also it is useful to know that the scripts never modify the repositories
you specify and the scripts remove all the temporary files upon exit,
Ctrl-C interruption, or error (unless -p option is specified).

The layout of the repository.

* aiaiai/email/
    All the scripts related to emails handling. If you use Aiaiai locally
    you do not need these scripts.
* aiaiai/helpers/
    Variouls programs and scripts which Aiaiai users should not usually use
    directly. Instead, Aiaiai scripts use them internally.
* aiaiai/doc
    Some documentation.
* aiaiai/tests/
    Unit-tests.
* aiaiai/
    The root repository directory contains the scripts which do all the
    patch checking work. The central script is 'aiaiai-test-patchset' which
    uses all the other scripts as helpers.

Let's briefly review the contents of the repository.


1.1 E-mail-related scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~

* email/aiaiai-email-lda
    This is Aiaiai's local delivery agent script which receives the incoming
    emails from the mail server (via a pipe), processes them (drops all e-mails
    which do not start with [PATCH*], detects and collects patch-sets), and
    puts them to the "queue" sub-directory of the work directory. This
    should, in turn, wake up the "aiaiai-email-dispatcher" script which will
    take further care of the patch or patch-set.

* email/aiaiai-email-dispatcher
    This script waits for the incoming patch-sets in the "queue" directory and
    runs a user-defined command for every incoming patch-set. The role of this
    script is to limit the amount of patch-sets we can run at a time. You
    can configure this via command line options.

    As it was already said, in our setup 'aiaiai-email-dispatcher' runs
    'aiaiai-email-test-patchset'.

* email/aiaiai-email-dispatcher-helper
    This is a private helper script for 'aiaiai-email-dispatcher'.

* email/aiaiai-email-test-patchset
    This script parses the incoming e-mail (in mbox format), finds out which
    kernel tree has to be used, sends a notification to the sender, runs
    the 'aiaiai-test-patchset' script with the right command-line arguments, and
    sends the results of 'aiaiai-test-patchset' back to the patch submitter.

* email/aiaiai.cfg
    This is an example of 'aiaiai-email-test-patchset' configuration file. It
    describes the projects Aiaiai supports and their parameters. Take a
    look inside - the file is self-descriptive.

Here is how all these scripts work together.  When you send an e-mail to the
server which runs Aiaiai, the MTA pipes it to 'aiaiai-email-lda'. We used the
postfix MTA, and the 'aiaiai' Linux user had the following file in its home
directory:

$ cat $HOME/.forward
|/home/aiaiai/bin/my-aiaiai-lda

Which made sure that all e-mails coming to aiaiai@our-domain.ququ were piped
to the my-aiaiai-lda script. The script simply ran aiaiai-email-lda with
the options we needed:

$ cat /home/aiaiai/bin/my-aiaiai-lda
#!/bin/sh
$HOME/git/email/aiaiai-email-lda -v --reap-archive=43200 --reap-incomplete=10 -- $HOME/aiaiai-workdir >> "$HOME/aiaiai-logs/email-lda.log" 2>&1


So, aiaiai-email-lda collects patches belonging to the patch-sets, and when it
has all of them, it squashes them into one mbox file and moves them to the
queue directory. This will wake up the 'aiaiai-email-dispatcher' script, which
will execute 'aiaiai-email-test-patchset' for this mbox (unless there are
already too many jobs, in which case it will wait).

The 'aiaiai-email-test-patchset' will parse the patch-set, find out which
kernel it has to be tested against using the configuration file, send a
notification e-mail to the patch sender, and execute the 'aiaiai-test-patchset'
script.

The 'aiaiai-test-patchset' script does all the testing work and outputs the
test results to stdout. 'aiaiai-email-test-patchset' captures the output and
sends it back to the patch submitter.

A more complicated mail setup dealing with forwards and multi-part emails
would be :
$ cat /home/aiaiai/bin/my-aiaiai-lda
	#!/bin/sh
	formail_opt=""

	mail=$(mktemp)
	cat > $mail

	to="$(formail -c -z -x "To:" < $mail)"
	[ -n "$to" ] && formail_opt="$formail_opt -I \"To: $to\""

	project="$(formail -c -z -x "X-Aiaiai-Project:" < $mail)"
	[ -n "$project" ] && formail_opt="$formail_opt -I \"X-Aiaiai-Project: $project\""

	commit="$(formail -c -z -x "X-Aiaiai-Commit:" < $mail)"
	[ -n "$commit" ] && formail_opt="$formail_opt -I \"X-Aiaiai-Commit: $commit\""

	cat "$mail" | eval formail $formail_opt -d -s "$HOME/git/email/aiaiai-email-lda" \
	-v --reap-archive=43200 --reap-incomplete=10 -- "$HOME/aiaiai-workdir" \
	>> "$HOME/aiaiai-logs/email-lda.log" 2>&1

1.2 Email Configuration
~~~~~~~~~~~~~~~~~~~~~~~

aiaiai-email scripts use a common configuration file, which is documented in
doc/email/example-aiaiai.cfg

By default aiaiai expects email targeting a specific project to be delivered to
a specific aiaiai recipient address which includes it, e.g:
aiaiai+foo@domain.tld.

In case you would want to use the same aiaiai recipient email address for all
projects, you need to create a hook file which will be looking at the
X-Aiaiai-Project header in the mbox file, and output it to stdout (see:
email/aiaiai-email-test-patchset). An example hook script could look like this:

#!/bin/sh
CFGFILE="$1"
MBOX="$2"
grep "X-Aiaiai-Project" $MBOX
ret=$?

1.3 Non-e-mail scripts
~~~~~~~~~~~~~~~~~~~~~~

* aiaiai-test-patchset
    The central script which tests a patch-set is 'aiaiai-test-patchset'. You
    should use it if you want to test your patches locally. It accepts patches
    in the mbox format. You should specify various input parameters like the
    path to the kernel to test against, the commit id to test against, which
    tests to run, etc.

* aiaiai-make-kernel
    Low-level scripts which builds the kernel. It has several switches which
    change its behavior, e.g., you can select whether you want to run
    sparse/smatch/cppcheck/coccinelle tools or not.

    'aiaiai-make-kernel' utilizes the 'aiaiai-locker' utility to make
    sure the build logs of parallel builds are not scrambled and stay readable.

* aiaiai-locker
    This utility was inspired by Eric Melski's blog-post I found in the Internet.
    We use this tool to make Linux kernel build logs saner than parallel build
    produces (e.g., "make -j 20"). Indeed, parallel make scrambles all the
    lines which makes build logs not comparable and not readable.

    The idea is to run all make commands via the 'aiaiai-locker' utility using
    the GNU make 'SHELL' variable re-definition. 'aiaiai-locker' intercepts the
    stdout and stderr output of commands, gathers all the output until the
    command finishes, and locks a common lockfile, and prints the intercepted
    output. So the effect of this is that the in the build log the output of
    one program goes as one big block and it is not mixed with the output of
    another command.

* aiaiai-checker
    A helper script for 'aiaiai-make-kernel' which runs
    sparse/smatch/cppcheck/coccinelle tools. The reason we need it is because
    the kernel build system allows only one checking program at a time, but we
    want to run multiple of them.

* aiaiai-diff-log
    Compares 2 build logs and provides the differences. This script used the
    'helpers/remap-log' tool to re-map line numbers in the warnings of the
    first build log to correspond to line numbers in the second build-log.

* aiaiai-diff-log-helper
    A helper script for aiaiai-diff-log which sorts 2 build logs and compares
    them.  Written in Python because the shell version was too slow.

* helpers/remap-log
    This is a small C program originally written by Al Viro which helps
    comparing 2 build logs by remapping line numbers. Roughly speaking,
    'remap-log' takes the build log of the pre-patched kernel and the patch and
    changes line numbers in the build log to match line numbers in the build
    log of the patched kernel. For example, supposed that you have the
    following warning in the pre-patched kernel:

    driver.c:100 Unused variable 'ret'

    and your patch adds a new include to the driver.c file, so the patched
    kernel will produce the following warning:

    driver.c:101 Unused variable 'ret'

    Obviously, you do not want to bug the patch author about this warning
    because it was not his patch which introduced it. And if you used a plain
    diff, it would produce something like:

    -driver.c:100 Unused variable 'ret'
    +driver.c:101 Unused variable 'ret'

    But we pass the build log for the pre-patched kernel via 'remap-log' and it
    transforms that warning into

    driver.c:101 Unused variable 'ret'

    so the build logs of the pre-patched and patched kernels will contain the
    same line and the diff won't show anything.

* aiaiai-test-bisectability
    Accepts a patch-set as the input and checks that the patch-set does not
    break compilation at any step. This is achieved by applying every patch in
    the series one-by-one, compiling the kernel and checking that compilation
    was successful.

* aiaiai-decode-rfc-2047
    A small helper script which decodes unreadable patch subjects encoded with
    RFC-2047 MIME encoding.

* aiaiai-concat-mboxes
    A little helper script which concatenates multiple mbox files into one.

* aiaiai-match-keywords
   A simple script which can be used to match certain unwelcom keywords in
   patches.

To summarize:
 * 'aiaiai-test-patchset' is the central script
   * it runs 'checkpatch.pl'
   * it runs 'aiaiai-test-bisectability'
   * for every configuration specified by the user, it builds the kernel
     before and after the patch using the 'aiaiai-make-kernel' script, then
     compares the build log using the 'aiaiai-diff-log' script.
   * Things are done in parallel.
   * The results of all tests are collected and printed to stdout
   * The "verbose" contents is printed to stderr
   * Many of the jobs above are run in parallel

The 'aiaiai-diff-log' script which compares 2 logs. It uses the 'remap-log'
tool for comparing 2 build logs the smart way.


2 Using Aiaiai locally
~~~~~~~~~~~~~~~~~~~~~~

If you just want to test a patch-set locally, you should use the
'aiaiai-test-patchset' script. Do not forget to build the remap-log.c file in
the 'helpers' subdirectory.

Here are some examples.

1. Test a patch or a patch-set p.mbox against the linux-vfs kernel tree,
   against the 'origin/next' branch, and use the 'i386_defconfig'
   defconfig, the architecture is i386 (translates to 'make ARCH=i386'). Use
   16 jobs.

   cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
       --cppcheck --coccinelle -j 16 -c origin/next \
       /<path>/linux-vfs i386_defconfig,i386

2. Test a patch or a patch-set p.mbox against the linux-omap kernel tree,
   against the master branch, use the 'omap2plus_defconfig' defconfig,
   'arm' architecture and 'arm-eabi-' cross-compiler prefix (translates to
   'make CROSS_COMPILE=arm-eabi-').

   cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
       --cppcheck --coccinelle -j 16 -c master \
       /<path>/linux-omap omap2plus_defconfig,arm,arm-eabi-

3. If you are a maintainer you may want to test patches for more than one
   architecture. You can do this - you'll just need to maintain defconfigs.
   E.g., to test p.mbox against both i386_defconfig (Intel i386) and
   omap2plus_defconfig (ARM-based SoC), you can run:

   cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
       --cppcheck --coccinelle -j 16 -c master /<path>/linux-my \
       i386_defconfig,i386 omap2plus_defconfig,arm,arm-eabi-

   IOW, just specify defconfigs, architectures, and cross-compiler prefixes at
   the end.

4. Test a patch or a patch-set p.mbox against the net-next kernel tree,
   against the 'origin/master' branch, using a randomly generated config, the
   architecture is i386 (translates to 'make ARCH=i386'). Use 16 jobs.

   cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
       --cppcheck --coccinelle -j 16 -c origin/master \
       /<path>/net-next randconfig,i386

Note, the 'aiaiai-test-patchset' script accepts one mbox file via stdio or the
'-i' option. If you have several files belonging to one patch-set and you want
to test them all, you need to concatenate them. Use the 'aiaiai-concat-mboxes'
script for this. E.g., if you have '01.mbox' and '02.mbox' comprising a
patch-set, you can test do like this:

$ aiaiai-concat-mboxes 01.mbox 02.mbox | aiaiai-test-patchset ....

Patches generated with 'git format-patch' are also mbox files so you can test
them with 'aiaiai-test-patchset' as well. Just remember to use the '--thread'
option to make 'git format-patch' generate "Message-Id:" and "In-Reply-To:"
headers. For example:

$ git format-patch --thread HEAD~10
$ aiaiai-concat-mboxes 00* | aiaiai-test-patchset ...

I usually run 'aiaiai-test-patchset' with the lowest priority in order to make
sure the machine is still usable for other things:

$ nice -n19 ionice -c3 aiaiai-test-patchset ...

And for the kernel trees I test frequently against, I usually have a helper
script so that I do not have to type the 'aiaiai-test-patchset' options over
and over again.


3 Dependencies
~~~~~~~~~~~~~~

3.1 smatch
~~~~~~~~~~

Project homepage: http://smatch.sourceforge.net

I do not think this tool is a part of Linux distributions. I found it very
useful. To use it, you have to clone it, patch it, and install in your system.

$ git clone git://repo.or.cz/smatch.git
$ cd smatch
$ make PREFIX=$HOME/programs/smatch all install
$ export PATH="$PATH:/home/aiaiai/programs/smatch/bin"

These commands will install smatch to "/home/aiaiai/programs/smatch". Of
course, you should specify your path and make sure smatch is in your "PATH"
environment variable.


3.2 sparse
~~~~~~~~~~

Similarly to smatch, you need to install it manually. This is what I did:

$ git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git
$ cd sparse
$ make
$ cp sparse $HOME/bin
$ export PATH="$HOME/bin:$PATH"


3.3 coccinelle
~~~~~~~~~~~~~~

Coccinelle or "spatch" is a great semantic patch tool, very useful on its own.
The Linux kernel contains a bunch of coccinelle semantic patches which catch
various types of API abuses etc. We store a copy of these scripts in the
"helpers" subdirectory.

The "spatch" tool is part of distributions like Debian and Fedora, so you most
probably just need to install the corresponding package. The binary program is
called "spatch".

You may find a lot of information about this awesome tool by Googling.


3.4 Dependencies
~~~~~~~~~~~~~~~~

The scripts use many external programs which are usually part of the Linux
distribution. Here are they are, but the list is not complete - if you spot
a missing dependency - please, let me know or send a patch against the README
file.

gcc
make
cppcheck
formail
perl
python
mutt
sed
grep
awk
git
lockfile
diff
patch
inotifywait


3.5 Code conventions
~~~~~~~~~~~~~~~~~~~~

The scripts have been written for the "dash" shell and they do not use
any "bashizm" and should be very portable. Please, keep this in mind
when changing them. The scripts try to follow a uniform coding style
and conventions. Below are some of them.

* Stdout is used only for user-friendly stuff. In many cases it is assumed that
  the higher-level scripts just capture the stdout of lower-level scripts and
  provide that to the user. For example, 'aiaiai-test-patchset' captures stdout
  of the 'aiaiai-test-bisectability' scripts and uses it for reporting to the
  user. In turn, the 'aiaiai-email-test-patchset' script captures stdout of the
  'aiaiai-test-patchset' script and sends it back to the user via e-mail.

* Stderr is use for all the debugging, verbose and additional prints.

* The scripts use the "-e" shell option and in case of any error (non-zero
  return code from any program) we exit immediately. This allows us simplify
  the scripts and avoid handling errors for all the commands which is very
  convenient.

  Well, the automatic exit does not work if the errorred command is in a
  pipeline, and not the last. But this is not a big deal in most cases.

  And sometimes we do not want the whole script to error out if a command
  returns non zero (e.g., grep returns 1 if it did not match the pattern),
  in which case we usually use "||:" after the command.

* We use "-u" shell option to make sure our scripts are of good quality.

* We try to be more secure and use the "-f" shell option whenever we can.

* We make use of the external "libshell" library which is part of the Alt
  Linux project and is extremely useful.

* All the temporary files should be remover upon exit or interruption
  (ctrl-C must be handled gracefully)

* Users' git repository have to be treated as read-only and no modifications
  should be made there. All the work has to be done in a clone of the original
  repository.