README - pub/scm/linux/kernel/git/dborkman/pw - Git at Google

 Kernel patch/review workflow
 ============================

 Minimal howto for the tools involved in my daily patch handling and kernel
 mailing list interaction. It may be not perfect and the workflow as well
 as tooling/scripts can be improved for sure, but I thought I'd document it
 here for myself or others in case it's found useful. The example config
 is very much bpf and netdev centric and needs to be adapted for other
 subsystems.

 Please send patches to Daniel Borkmann <daniel@iogearbox.net>.

 Used tooling:
 -------------

 I mainly use git, git-pw, git-send-email, l2md, pw-*, quilttools, msmtp,
 mutt and patchwork UI for all my daily list workflow. The command-line
 tools themselves can be obtained via:

   - git, git-send-email, msmtp, mutt (available as distro packages)
   - git://git.kernel.org/pub/scm/linux/kernel/git/tglx/quilttools.git
   - git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
   - git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/pw.git
   - https://github.com/getpatchwork/git-pw

 Base setup:
 -----------

 Example configs for various tools are in this repository as well.

 1. git-pw:
 ~~~~~~~~~~

 Configuration of git-pw is straight forward, simply follow the below
 steps. The authentication token can be generated from the patchwork
 UI from https://patchwork.ozlabs.org/user/. Below example is related
 to netdev:

   $ git config --global pw.server 'https://patchwork.ozlabs.org/'
   $ git config --global pw.project 'netdev'
   $ git config --global pw.token 'API-TOKEN'

 Note that git-pw must be at least version 1.7.0 or higher as earlier
 versions have issues to decode received mboxes in some cases:

   https://github.com/getpatchwork/git-pw/issues/52

 2. quilttools aka mb2q:
 ~~~~~~~~~~~~~~~~~~~~~~~

 Example .mb2q.yaml in this repo, needs to be in home directory. More
 information on the options can be found in the quilttools repository.

 quilttools as well as git-pw is needed for the pw-apply tool in this
 repository. Although I don't use quilt, I really like mb2q's patch
 normalization.

 3. l2md:
 ~~~~~~~~

 I'm using l2md for fetching new emails via lore.kernel.org git archives
 and msmtp for sending them. My l2md feeds into mutt, and msmtp is used
 for mutt as well as git-send-email.

 Example .l2mdconfig for lists I follow in this repo, needs to be in
 home directory. The l2md repository also has additional information
 with regards to mutt and procmail integration and a service unit file
 for letting it run in the background automatically.

 Patch handling:
 ---------------

 Main tools are pw-apply and pw-request-pull, both minimal, quick and dirty
 scripts to get the job done. pw-check is called from pw-apply. pw-backport
 for the backports to the stable tree. Patches for improving them are very
 welcome.

 1. Initial patch triage:
 ~~~~~~~~~~~~~~~~~~~~~~~~

 Patches from netdev and bpf lists land under netdev patchwork project, hence
 first manual step is to delegate them accordingly such that they land in the
 delegate's todo:

   https://patchwork.ozlabs.org/project/netdev/list/

 Todo list:

   https://patchwork.ozlabs.org/user/todo/netdev/

 From public view, the current BPF todo list is accessible via:

   https://patchwork.ozlabs.org/project/netdev/list/?series=&submitter=&state=&q=&archive=&delegate=77147

 All delegation and setting patches into 'Changes requested' is done through
 the web UI. This remains to be improved for more automation.

 2. Applying patches:
 ~~~~~~~~~~~~~~~~~~~~

 Applying patches from patchwork, add Tested-by, and Acked-by from others.
 Patches are always identified by the series id even if it's just a single
 patch. Use -t/-a to propagate tags from the cover letter to individual
 patches, multiple people separated by comma:

   $ pw-apply -s 132626 -- -t 'Joe Hacker <j@hack.er>' -a 'Joe Hacker <jh@ack.er>, Acker Jon <ack@er.jon>'

 The pw-apply tool can also handle mboxes directly compressed in gzip or
 uncompressed format:

   $ pw-apply -b https://patchwork.ozlabs.org/series/132626/mbox/ -- -t 'Joe Hacker <j@hack.er>'

 Another example without adding optional tags, this time from lore:

   $ pw-apply -b https://lore.kernel.org/bpf/20190925203745.3173184-1-andriin@fb.com/t.mbox.gz

 Similar example with adding an explicit merge commit with branch name
 bpf-foo-bar:

   $ pw-apply -s 132626 -m bpf-foo-bar

 The pw-apply tool normalizes the patches through mb2q, sorts and groups
 tags, adds permanent links to the lore list archive and merges them into
 the master branch with the maintainer's SOB added to the chain.

 After they have been applied and sanity checked successfully, mark patches
 as accepted in patchwork. The -a before the optional "--" will tell pw-apply
 to mark the series as accepted, it does _not_ re-apply the series or do
 anything else:

   $ pw-apply -s 132626 -a

 Note that this does not work in combination with the -b parameter since
 the stand-alone mailbox does not have any context of patchwork.

 Full workflow:

   $ pw-apply -s 132626
   # Double check if all looks good in the git log
   $ pw-apply -s 132626 -a

 Currently, the acceptance email must be sent manually, but this will be
 automated in near term and integrated into -a.

 Side-note, manual sanity checking:

   $ pw-check -s e3439af4a339acd7fddbd6d59b8ecefaac07a611 -e d21b06927c8bf8083c64a95f272bc5491682ae24
   Commit: d21b06927c8b ("bpf: Fix bpf_event_output re-entry issue")
         Fixes tag: Fixes: ae5a3a828cd0 ("bpf: add perf event notificaton support for sock_ops")
         Has these problem(s):
               - Target SHA1 does not exist

 Note that pw-check is called automatically for all applied patches through
 pw-apply, so above example output is just to demonstrate a manual workflow.

 3. Pull requests:
 ~~~~~~~~~~~~~~~~~

 Pull requests together with a consistent format are created as described in
 below example. I do like consistency, hence even the pre-cooked template.
 The commit sha is the merge base, meaning the last commit that both trees
 had before patches have been applied. This could be automated but my own
 preference is to explicitly state it here as both {bpf,bpf-next} trees don't
 have too many commits piling up. Again, this workflow is mostly bpf specific
 and the scripts would need to be adapted for other subsystems.

   $ pw-request-pull -t bpf -s aef70a1f44c0b570e6345c02c2d240471859f0a4
   PR: pr-bpf-2019-09-26.patch

 Currently the -t option only accepts: bpf, bpf-next

 The -s option is optional and if not specified then pw-request-pull will
 determine the merge base automatically. In order for this to work, it requires
 that there is a net (for -t bpf) or net-next (for -t bpf-next) remote in the
 repository. To generate the PR, simply do:

   $ pw-request-pull -t bpf
   PR: pr-bpf-2019-09-26.patch

 The resulting file is then edited to describe the changes that the pull-request
 contains. It's then sent out same way as other patches through git-send-email:

   $ git send-email --to davem@davemloft.net --cc kuba@kernel.org --cc pabeni@redhat.com --cc edumazet@google.com --cc daniel@iogearbox.net --cc ast@kernel.org --cc andrii@kernel.org --cc martin.lau@linux.dev --cc netdev@vger.kernel.org --cc bpf@vger.kernel.org pr-bpf-2019-09-26.patch

 The PR will then show up the usual way in patchwork and assigned to the higher
 level maintainer delegate for pulling.

 4. Backporting to stable:
 ~~~~~~~~~~~~~~~~~~~~~~~~~

 Backporting is done through the pw-backport tool which automatically adds the
 SOB and upstream commit sha to the commit message.

 It's tailored to https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/.
 In the local git clone, the following workflow can be used:

   $ git pull origin master
   $ git checkout -b linux-5.3.y origin/linux-5.3.y
   $ pw-backport d895a0f16fadb26d22ab531c49768f7642ae5c3e
   Applying: d895a0f16fad bpf: fix accessing bpf_sysctl.file_pos on s390

 The pw-backport can also take multiple shas as command line arguments.

 5. Creating review schedule:
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 The pw-schedule tool creates a csv schedule which can then be imported into
 the BPF review calendar. By default, the tool creates a schedule for two full
 rotations:

   $ pw-schedule
   $ cat schedule.csv
   Subject,Start Date,End Date
   BPF Reviewer: Daniel,2022-06-27,2022-06-30
   BPF Reviewer: Alexei,2022-06-30,2022-07-02
   BPF Reviewer: Martin,2022-07-04,2022-07-07
   BPF Reviewer: Song,2022-07-07,2022-07-09
   BPF Reviewer: Jiri,2022-07-11,2022-07-14
   BPF Reviewer: Yonghong,2022-07-14,2022-07-16
   BPF Reviewer: Hao,2022-07-18,2022-07-21
   BPF Reviewer: KP,2022-07-21,2022-07-23
   BPF Reviewer: John,2022-07-25,2022-07-28
   BPF Reviewer: Stanislav,2022-07-28,2022-07-30
   BPF Reviewer: Andrii,2022-08-01,2022-08-04
   BPF Reviewer: Daniel,2022-08-04,2022-08-06
   BPF Reviewer: Alexei,2022-08-08,2022-08-11
   BPF Reviewer: Martin,2022-08-11,2022-08-13
   BPF Reviewer: Song,2022-08-15,2022-08-18
   BPF Reviewer: Jiri,2022-08-18,2022-08-20
   BPF Reviewer: Yonghong,2022-08-22,2022-08-25
   BPF Reviewer: Hao,2022-08-25,2022-08-27
   BPF Reviewer: KP,2022-08-29,2022-09-01
   BPF Reviewer: John,2022-09-01,2022-09-03
   BPF Reviewer: Stanislav,2022-09-05,2022-09-08
   BPF Reviewer: Andrii,2022-09-08,2022-09-10
   $

 The csv can be imported into the following way: In Google Calendar
 click on 'Add other calendars' under the left 'Other calendars'
 menu. Then, click on 'Import'. Select the schedule.csv and select
 'BPF upstream' for 'Add to calendar'. Click 'Import' and that's it.

 Creating a schedule starting from a specific date works as follows:

   $ pw-schedule -n 11 -s $(date -I --date="2022-04-08")
   $ cat schedule.csv
   Subject,Start Date,End Date
   BPF Reviewer: Hao,2022-08-04,2022-08-06
   BPF Reviewer: Martin,2022-08-08,2022-08-11
   BPF Reviewer: Song,2022-08-11,2022-08-13
   BPF Reviewer: Andrii,2022-08-15,2022-08-18
   BPF Reviewer: Jiri,2022-08-18,2022-08-20
   BPF Reviewer: Daniel,2022-08-22,2022-08-25
   BPF Reviewer: John,2022-08-25,2022-08-27
   BPF Reviewer: KP,2022-08-29,2022-09-01
   BPF Reviewer: Yonghong,2022-09-01,2022-09-03
   BPF Reviewer: Alexei,2022-09-05,2022-09-08
   BPF Reviewer: Stanislav,2022-09-08,2022-09-10
   $

 6. Signing Tags for PRs:
 ~~~~~~~~~~~~~~~~~~~~~~~~

 GPG base setup:

   $ gpg --full-generate-key
   $ gpg --list-keys
   [...]
   pub   rsa4096 2022-10-07 [SC] [expires: 2026-10-06]
         21BB8B2D82228D9CC0049A193AC5D24571557BB1
   uid           [ultimate] Foo Bar <foo@bar.com>
   sub   rsa4096 2022-10-07 [E] [expires: 2026-10-06]
   [...]

   $ gpg --quick-addkey 21BB8B2D82228D9CC0049A193AC5D24571557BB1 ed25519 sign
   $ gpg --list-keys
   [...]
   pub   rsa4096 2022-10-07 [SC] [expires: 2026-10-06]
         21BB8B2D82228D9CC0049A193AC5D24571557BB1
   uid           [ultimate] Foo Bar <foo@bar.com>
   sub   rsa4096 2022-10-07 [E] [expires: 2026-10-06]
   sub   ed25519 2022-10-07 [S]
   [...]

   $ gpg --keyserver pgp.surfnet.nl --send-keys 21BB8B2D82228D9CC0049A193AC5D24571557BB1
   $ gpg --keyserver pgp.mit.edu --send-keys 21BB8B2D82228D9CC0049A193AC5D24571557BB1

   https://www.kernel.org/doc/html/latest/process/maintainer-pgp-guide.html#back-up-your-master-key-for-disaster-recovery

 GPG signing, etc:

   Bob's key being AABB8B2D82228D9CC0049A193AC5D24571557BB1

   $ gpg --recv-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1
   $ gpg --sign-key AABB8B2D82228D9CC0049A193AC5D24571557BB1

   Option A:

   $ gpg -a --export AABB8B2D82228D9CC0049A193AC5D24571557BB1 |
     gpg -se -r AABB8B2D82228D9CC0049A193AC5D24571557BB1 >
     ~/tmp/AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc.pgp

   [ email AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc.pgp to Bob ]

   Bob:

   $ gpg --decrypt AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc.pgp
   $ gpg --import AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc

   $ gpg --keyserver pgp.surfnet.nl --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1
   $ gpg --keyserver pgp.mit.edu --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1

   Option B (not recommended):

   Alice:

   $ gpg --keyserver pgp.surfnet.nl --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1
   $ gpg --keyserver pgp.mit.edu --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1

   Bob can double check Alice's signature on keyservers.

 Git base setup:

   $ git config --global gpg.program /usr/bin/gpg

 For bpf & bpf-next tree dirs:

   $ git config user.signingkey 21BB8B2D82228D9CC0049A193AC5D24571557BB1

 Git signing/pushing/deleting tags for PR:

   For BPF trees, it's a 3 step process:

   NOTE: you might need to add `GPG_TTY=$(tty)` to below `pw-request-pull`
   invocations to let gpg ask for your password instead of silently failing.

   1) Create and push the signed tag to the tree:

     $ pw-request-pull -t bpf-next -S -c

   2) Create the pull-request template:

     $ pw-request-pull -t bpf-next -S

   3) After merge, remove the signed tag from tree:

     $ pw-request-pull -t bpf-next -S -d

   Manual steps would look like:

   $ git tag -s bpf-for-net
   $ git verify-tag bpf-for-net
   $ git push origin bpf-for-net
   # after pull got merged

   $ git push --delete origin bpf-for-net
   $ git tag -d bpf-for-net

 Links:

   https://www.kernel.org/doc/html/latest/process/maintainer-pgp-guide.html
   https://gist.github.com/F21/b0e8c62c49dfab267ff1d0c6af39ab84

 Other workflow examples
 -----------------------

 Jakub:
 ~~~~~~

 mb2q reshuffles tags so I disable it. I always apply patches based
 on series with "auto-merge" and edit on:

   $ pw-apply -e -N -M -s 211734

 If series doesn't have a cover letter patches just get applied.
 If there is a merge it will do an automatic git rebase -i and wait for the
 rebase to finish. This lets me edit the commits without dealing with
 the --rebase-merges magic. It will also automatically git commit --amend
 the merge commit, because there are usually things that need to be removed
 from the text, like "Hi!", signatures, etc.

 pw-pull works great for pulling, again using patchwork ID, but of the
 particular PR "patch", PRs don't have series IDs:

   $ pw-pull -p 1393469

 When author posted the patches alongside the PR (and patchwork grouped them
 correctly which doesn't always happen) the script can apply those patches
 to a 'tmp' branch and check the contents of series and PR match:

   $ pw-pull -p 1393469 -s 12345

 pw-pull can also pull from PRs which patchwork does not recognize as PRs
 at all, using -c (AKA --pull-from-cover):

   $ pw-pull -s 12345 -c

 Last but not least it can pull from an explicit URL with -P which can be
 useful as well (it will at least provide the merge commit template).

 I don't mark as applied from the command line (yet) because I double
 check the tests passed in the UI, anyway.

 I use this rather ugly script in applypatch-msg to strip CCs and
 semi-manually propagate tags from the cover letter:

   perl -pi -e 's|^Message-Id:\s*<?([^>]+)>?$|Link: https://lore.kernel.org/r/$1|g;' "$1"

   # drop the pointless CCs from the commit message
   tag_names_emails=$(echo "@vger.kernel.org";
 		   echo "jakub kicinski";
 		   echo "davemloft.net";
 		   cat "$1" |
 		       sed -n 's/^[ARST][A-Za-z-]*: \(.* <\).*/\1/p';
 		   cat "$1" |
 		       sed -n 's/^[ARST][A-Za-z-]*:.*\(<.*>\)/\1/p')

   while IFS= read -r ident; do
     ccs=`sed -n "/cc:.*$ident.*/Ip" "$1"`
     [ ! -z "$ccs" ] && echo -e Remove tag: '\e[36m'$ccs'\e[0m'
     sed -i "/cc:.*$ident.*/Id" "$1"
   done <<< "$tag_names_emails"

   while IFS= read -r tag; do
     echo -e Adding tag: '\e[35m'$tag'\e[0m'
       git interpret-trailers --in-place \
           --if-exists=addIfDifferent \
           --trailer "$tag" \
           "$1"
   done <<< "$ADD_TAGS"

   # Move my tag last (and remove duplicates)
   git interpret-trailers --in-place \
     --if-exists=replace \
     --trailer "Signed-off-by: Jakub Kicinski <kuba@kernel.org>" \
     "$1"
	Kernel patch/review workflow
	============================

	Minimal howto for the tools involved in my daily patch handling and kernel
	mailing list interaction. It may be not perfect and the workflow as well
	as tooling/scripts can be improved for sure, but I thought I'd document it
	here for myself or others in case it's found useful. The example config
	is very much bpf and netdev centric and needs to be adapted for other
	subsystems.

	Please send patches to Daniel Borkmann <daniel@iogearbox.net>.

	Used tooling:
	-------------

	I mainly use git, git-pw, git-send-email, l2md, pw-*, quilttools, msmtp,
	mutt and patchwork UI for all my daily list workflow. The command-line
	tools themselves can be obtained via:

	- git, git-send-email, msmtp, mutt (available as distro packages)
	- git://git.kernel.org/pub/scm/linux/kernel/git/tglx/quilttools.git
	- git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
	- git://git.kernel.org/pub/scm/linux/kernel/git/dborkman/pw.git
	- https://github.com/getpatchwork/git-pw

	Base setup:
	-----------

	Example configs for various tools are in this repository as well.

	1. git-pw:
	~~~~~~~~~~

	Configuration of git-pw is straight forward, simply follow the below
	steps. The authentication token can be generated from the patchwork
	UI from https://patchwork.ozlabs.org/user/. Below example is related
	to netdev:

	$ git config --global pw.server 'https://patchwork.ozlabs.org/'
	$ git config --global pw.project 'netdev'
	$ git config --global pw.token 'API-TOKEN'

	Note that git-pw must be at least version 1.7.0 or higher as earlier
	versions have issues to decode received mboxes in some cases:

	https://github.com/getpatchwork/git-pw/issues/52

	2. quilttools aka mb2q:
	~~~~~~~~~~~~~~~~~~~~~~~

	Example .mb2q.yaml in this repo, needs to be in home directory. More
	information on the options can be found in the quilttools repository.

	quilttools as well as git-pw is needed for the pw-apply tool in this
	repository. Although I don't use quilt, I really like mb2q's patch
	normalization.

	3. l2md:
	~~~~~~~~

	I'm using l2md for fetching new emails via lore.kernel.org git archives
	and msmtp for sending them. My l2md feeds into mutt, and msmtp is used
	for mutt as well as git-send-email.

	Example .l2mdconfig for lists I follow in this repo, needs to be in
	home directory. The l2md repository also has additional information
	with regards to mutt and procmail integration and a service unit file
	for letting it run in the background automatically.

	Patch handling:
	---------------

	Main tools are pw-apply and pw-request-pull, both minimal, quick and dirty
	scripts to get the job done. pw-check is called from pw-apply. pw-backport
	for the backports to the stable tree. Patches for improving them are very
	welcome.

	1. Initial patch triage:
	~~~~~~~~~~~~~~~~~~~~~~~~

	Patches from netdev and bpf lists land under netdev patchwork project, hence
	first manual step is to delegate them accordingly such that they land in the
	delegate's todo:

	https://patchwork.ozlabs.org/project/netdev/list/

	Todo list:

	https://patchwork.ozlabs.org/user/todo/netdev/

	From public view, the current BPF todo list is accessible via:

	https://patchwork.ozlabs.org/project/netdev/list/?series=&submitter=&state=&q=&archive=&delegate=77147

	All delegation and setting patches into 'Changes requested' is done through
	the web UI. This remains to be improved for more automation.

	2. Applying patches:
	~~~~~~~~~~~~~~~~~~~~

	Applying patches from patchwork, add Tested-by, and Acked-by from others.
	Patches are always identified by the series id even if it's just a single
	patch. Use -t/-a to propagate tags from the cover letter to individual
	patches, multiple people separated by comma:

	$ pw-apply -s 132626 -- -t 'Joe Hacker <j@hack.er>' -a 'Joe Hacker <jh@ack.er>, Acker Jon <ack@er.jon>'

	The pw-apply tool can also handle mboxes directly compressed in gzip or
	uncompressed format:

	$ pw-apply -b https://patchwork.ozlabs.org/series/132626/mbox/ -- -t 'Joe Hacker <j@hack.er>'

	Another example without adding optional tags, this time from lore:

	$ pw-apply -b https://lore.kernel.org/bpf/20190925203745.3173184-1-andriin@fb.com/t.mbox.gz

	Similar example with adding an explicit merge commit with branch name
	bpf-foo-bar:

	$ pw-apply -s 132626 -m bpf-foo-bar

	The pw-apply tool normalizes the patches through mb2q, sorts and groups
	tags, adds permanent links to the lore list archive and merges them into
	the master branch with the maintainer's SOB added to the chain.

	After they have been applied and sanity checked successfully, mark patches
	as accepted in patchwork. The -a before the optional "--" will tell pw-apply
	to mark the series as accepted, it does _not_ re-apply the series or do
	anything else:

	$ pw-apply -s 132626 -a

	Note that this does not work in combination with the -b parameter since
	the stand-alone mailbox does not have any context of patchwork.

	Full workflow:

	$ pw-apply -s 132626
	# Double check if all looks good in the git log
	$ pw-apply -s 132626 -a

	Currently, the acceptance email must be sent manually, but this will be
	automated in near term and integrated into -a.

	Side-note, manual sanity checking:

	$ pw-check -s e3439af4a339acd7fddbd6d59b8ecefaac07a611 -e d21b06927c8bf8083c64a95f272bc5491682ae24
	Commit: d21b06927c8b ("bpf: Fix bpf_event_output re-entry issue")
	Fixes tag: Fixes: ae5a3a828cd0 ("bpf: add perf event notificaton support for sock_ops")
	Has these problem(s):
	- Target SHA1 does not exist

	Note that pw-check is called automatically for all applied patches through
	pw-apply, so above example output is just to demonstrate a manual workflow.

	3. Pull requests:
	~~~~~~~~~~~~~~~~~

	Pull requests together with a consistent format are created as described in
	below example. I do like consistency, hence even the pre-cooked template.
	The commit sha is the merge base, meaning the last commit that both trees
	had before patches have been applied. This could be automated but my own
	preference is to explicitly state it here as both {bpf,bpf-next} trees don't
	have too many commits piling up. Again, this workflow is mostly bpf specific
	and the scripts would need to be adapted for other subsystems.

	$ pw-request-pull -t bpf -s aef70a1f44c0b570e6345c02c2d240471859f0a4
	PR: pr-bpf-2019-09-26.patch

	Currently the -t option only accepts: bpf, bpf-next

	The -s option is optional and if not specified then pw-request-pull will
	determine the merge base automatically. In order for this to work, it requires
	that there is a net (for -t bpf) or net-next (for -t bpf-next) remote in the
	repository. To generate the PR, simply do:

	$ pw-request-pull -t bpf
	PR: pr-bpf-2019-09-26.patch

	The resulting file is then edited to describe the changes that the pull-request
	contains. It's then sent out same way as other patches through git-send-email:

	$ git send-email --to davem@davemloft.net --cc kuba@kernel.org --cc pabeni@redhat.com --cc edumazet@google.com --cc daniel@iogearbox.net --cc ast@kernel.org --cc andrii@kernel.org --cc martin.lau@linux.dev --cc netdev@vger.kernel.org --cc bpf@vger.kernel.org pr-bpf-2019-09-26.patch

	The PR will then show up the usual way in patchwork and assigned to the higher
	level maintainer delegate for pulling.

	4. Backporting to stable:
	~~~~~~~~~~~~~~~~~~~~~~~~~

	Backporting is done through the pw-backport tool which automatically adds the
	SOB and upstream commit sha to the commit message.

	It's tailored to https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/.
	In the local git clone, the following workflow can be used:

	$ git pull origin master
	$ git checkout -b linux-5.3.y origin/linux-5.3.y
	$ pw-backport d895a0f16fadb26d22ab531c49768f7642ae5c3e
	Applying: d895a0f16fad bpf: fix accessing bpf_sysctl.file_pos on s390

	The pw-backport can also take multiple shas as command line arguments.

	5. Creating review schedule:
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	The pw-schedule tool creates a csv schedule which can then be imported into
	the BPF review calendar. By default, the tool creates a schedule for two full
	rotations:

	$ pw-schedule
	$ cat schedule.csv
	Subject,Start Date,End Date
	BPF Reviewer: Daniel,2022-06-27,2022-06-30
	BPF Reviewer: Alexei,2022-06-30,2022-07-02
	BPF Reviewer: Martin,2022-07-04,2022-07-07
	BPF Reviewer: Song,2022-07-07,2022-07-09
	BPF Reviewer: Jiri,2022-07-11,2022-07-14
	BPF Reviewer: Yonghong,2022-07-14,2022-07-16
	BPF Reviewer: Hao,2022-07-18,2022-07-21
	BPF Reviewer: KP,2022-07-21,2022-07-23
	BPF Reviewer: John,2022-07-25,2022-07-28
	BPF Reviewer: Stanislav,2022-07-28,2022-07-30
	BPF Reviewer: Andrii,2022-08-01,2022-08-04
	BPF Reviewer: Daniel,2022-08-04,2022-08-06
	BPF Reviewer: Alexei,2022-08-08,2022-08-11
	BPF Reviewer: Martin,2022-08-11,2022-08-13
	BPF Reviewer: Song,2022-08-15,2022-08-18
	BPF Reviewer: Jiri,2022-08-18,2022-08-20
	BPF Reviewer: Yonghong,2022-08-22,2022-08-25
	BPF Reviewer: Hao,2022-08-25,2022-08-27
	BPF Reviewer: KP,2022-08-29,2022-09-01
	BPF Reviewer: John,2022-09-01,2022-09-03
	BPF Reviewer: Stanislav,2022-09-05,2022-09-08
	BPF Reviewer: Andrii,2022-09-08,2022-09-10
	$

	The csv can be imported into the following way: In Google Calendar
	click on 'Add other calendars' under the left 'Other calendars'
	menu. Then, click on 'Import'. Select the schedule.csv and select
	'BPF upstream' for 'Add to calendar'. Click 'Import' and that's it.

	Creating a schedule starting from a specific date works as follows:

	$ pw-schedule -n 11 -s $(date -I --date="2022-04-08")
	$ cat schedule.csv
	Subject,Start Date,End Date
	BPF Reviewer: Hao,2022-08-04,2022-08-06
	BPF Reviewer: Martin,2022-08-08,2022-08-11
	BPF Reviewer: Song,2022-08-11,2022-08-13
	BPF Reviewer: Andrii,2022-08-15,2022-08-18
	BPF Reviewer: Jiri,2022-08-18,2022-08-20
	BPF Reviewer: Daniel,2022-08-22,2022-08-25
	BPF Reviewer: John,2022-08-25,2022-08-27
	BPF Reviewer: KP,2022-08-29,2022-09-01
	BPF Reviewer: Yonghong,2022-09-01,2022-09-03
	BPF Reviewer: Alexei,2022-09-05,2022-09-08
	BPF Reviewer: Stanislav,2022-09-08,2022-09-10
	$

	6. Signing Tags for PRs:
	~~~~~~~~~~~~~~~~~~~~~~~~

	GPG base setup:

	$ gpg --full-generate-key
	$ gpg --list-keys
	[...]
	pub rsa4096 2022-10-07 [SC] [expires: 2026-10-06]
	21BB8B2D82228D9CC0049A193AC5D24571557BB1
	uid [ultimate] Foo Bar <foo@bar.com>
	sub rsa4096 2022-10-07 [E] [expires: 2026-10-06]
	[...]

	$ gpg --quick-addkey 21BB8B2D82228D9CC0049A193AC5D24571557BB1 ed25519 sign
	$ gpg --list-keys
	[...]
	pub rsa4096 2022-10-07 [SC] [expires: 2026-10-06]
	21BB8B2D82228D9CC0049A193AC5D24571557BB1
	uid [ultimate] Foo Bar <foo@bar.com>
	sub rsa4096 2022-10-07 [E] [expires: 2026-10-06]
	sub ed25519 2022-10-07 [S]
	[...]

	$ gpg --keyserver pgp.surfnet.nl --send-keys 21BB8B2D82228D9CC0049A193AC5D24571557BB1
	$ gpg --keyserver pgp.mit.edu --send-keys 21BB8B2D82228D9CC0049A193AC5D24571557BB1

	https://www.kernel.org/doc/html/latest/process/maintainer-pgp-guide.html#back-up-your-master-key-for-disaster-recovery

	GPG signing, etc:

	Bob's key being AABB8B2D82228D9CC0049A193AC5D24571557BB1

	$ gpg --recv-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1
	$ gpg --sign-key AABB8B2D82228D9CC0049A193AC5D24571557BB1

	Option A:

	$ gpg -a --export AABB8B2D82228D9CC0049A193AC5D24571557BB1 \|
	gpg -se -r AABB8B2D82228D9CC0049A193AC5D24571557BB1 >
	~/tmp/AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc.pgp

	[ email AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc.pgp to Bob ]

	Bob:

	$ gpg --decrypt AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc.pgp
	$ gpg --import AABB8B2D82228D9CC0049A193AC5D24571557BB1.asc

	$ gpg --keyserver pgp.surfnet.nl --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1
	$ gpg --keyserver pgp.mit.edu --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1

	Option B (not recommended):

	Alice:

	$ gpg --keyserver pgp.surfnet.nl --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1
	$ gpg --keyserver pgp.mit.edu --send-keys AABB8B2D82228D9CC0049A193AC5D24571557BB1

	Bob can double check Alice's signature on keyservers.

	Git base setup:

	$ git config --global gpg.program /usr/bin/gpg

	For bpf & bpf-next tree dirs:

	$ git config user.signingkey 21BB8B2D82228D9CC0049A193AC5D24571557BB1

	Git signing/pushing/deleting tags for PR:

	For BPF trees, it's a 3 step process:

	NOTE: you might need to add `GPG_TTY=$(tty)` to below `pw-request-pull`
	invocations to let gpg ask for your password instead of silently failing.

	1) Create and push the signed tag to the tree:

	$ pw-request-pull -t bpf-next -S -c

	2) Create the pull-request template:

	$ pw-request-pull -t bpf-next -S

	3) After merge, remove the signed tag from tree:

	$ pw-request-pull -t bpf-next -S -d

	Manual steps would look like:

	$ git tag -s bpf-for-net
	$ git verify-tag bpf-for-net
	$ git push origin bpf-for-net
	# after pull got merged

	$ git push --delete origin bpf-for-net
	$ git tag -d bpf-for-net

	Links:

	https://www.kernel.org/doc/html/latest/process/maintainer-pgp-guide.html
	https://gist.github.com/F21/b0e8c62c49dfab267ff1d0c6af39ab84

	Other workflow examples
	-----------------------

	Jakub:
	~~~~~~

	mb2q reshuffles tags so I disable it. I always apply patches based
	on series with "auto-merge" and edit on:

	$ pw-apply -e -N -M -s 211734

	If series doesn't have a cover letter patches just get applied.
	If there is a merge it will do an automatic git rebase -i and wait for the
	rebase to finish. This lets me edit the commits without dealing with
	the --rebase-merges magic. It will also automatically git commit --amend
	the merge commit, because there are usually things that need to be removed
	from the text, like "Hi!", signatures, etc.

	pw-pull works great for pulling, again using patchwork ID, but of the
	particular PR "patch", PRs don't have series IDs:

	$ pw-pull -p 1393469

	When author posted the patches alongside the PR (and patchwork grouped them
	correctly which doesn't always happen) the script can apply those patches
	to a 'tmp' branch and check the contents of series and PR match:

	$ pw-pull -p 1393469 -s 12345

	pw-pull can also pull from PRs which patchwork does not recognize as PRs
	at all, using -c (AKA --pull-from-cover):

	$ pw-pull -s 12345 -c

	Last but not least it can pull from an explicit URL with -P which can be
	useful as well (it will at least provide the merge commit template).

	I don't mark as applied from the command line (yet) because I double
	check the tests passed in the UI, anyway.

	I use this rather ugly script in applypatch-msg to strip CCs and
	semi-manually propagate tags from the cover letter:

	perl -pi -e 's\|^Message-Id:\s*<?([^>]+)>?$\|Link: https://lore.kernel.org/r/$1\|g;' "$1"

	# drop the pointless CCs from the commit message
	tag_names_emails=$(echo "@vger.kernel.org";
	echo "jakub kicinski";
	echo "davemloft.net";
	cat "$1" \|
	sed -n 's/^[ARST][A-Za-z-]: \(. <\).*/\1/p';
	cat "$1" \|
	sed -n 's/^[ARST][A-Za-z-]:.\(<.*>\)/\1/p')

	while IFS= read -r ident; do
	ccs=`sed -n "/cc:.$ident./Ip" "$1"`
	[ ! -z "$ccs" ] && echo -e Remove tag: '\e[36m'$ccs'\e[0m'
	sed -i "/cc:.$ident./Id" "$1"
	done <<< "$tag_names_emails"

	while IFS= read -r tag; do
	echo -e Adding tag: '\e[35m'$tag'\e[0m'
	git interpret-trailers --in-place \
	--if-exists=addIfDifferent \
	--trailer "$tag" \
	"$1"
	done <<< "$ADD_TAGS"

	# Move my tag last (and remove duplicates)
	git interpret-trailers --in-place \
	--if-exists=replace \
	--trailer "Signed-off-by: Jakub Kicinski <kuba@kernel.org>" \
	"$1"