Documentation/public-inbox-tuning.pod

=head1 NAME

public-inbox-tuning - tuning public-inbox

=head1 DESCRIPTION

public-inbox intends to support a wide variety of hardware.  While
we strive to provide the best out-of-the-box performance possible,
tuning knobs are an unfortunate necessity in some cases.

=over 4

=item 1

New inboxes: public-inbox-init -V2

=item 2

Optional Inline::C use

=item 3

Performance on rotational hard disk drives

=item 4

Btrfs (and possibly other copy-on-write filesystems)

=item 5

Performance on solid state drives

=item 6

Read-only daemons

=item 7

Other OS tuning knobs

=item 8

Scalability to many inboxes

=item 9

public-inbox-cindex --join performance

=item 10

public-inbox-clone with shared object stores

=back

=head2 New inboxes: public-inbox-init -V2

If you're starting a new inbox (and not mirroring an existing one),
the L<-V2|public-inbox-v2-format(5)> requires L<DBD::SQLite>, but is
orders of magnitude more scalable than the original C<-V1> format.

=head2 Optional Inline::C use

Our optional use of L<Inline::C> speeds up subprocess spawning from
large daemon processes.

To enable L<Inline::C>, either set the C<PERL_INLINE_DIRECTORY>
environment variable to point to a writable directory, or create
C<~/.cache/public-inbox/inline-c> for any user(s) running
public-inbox processes.

If libgit2 development files are installed and L<Inline::C>
is enabled (described above), per-inbox C<git cat-file --batch>
processes are replaced with a single L<perl(1)> process running
C<PublicInbox::Gcf2::loop> in read-only daemons.  libgit2 use
will be available in public-inbox 1.7.0+

More (optional) L<Inline::C> use will be introduced in the future
to lower memory use and improve scalability.

Note: L<Inline::C> is required for L<lei(1)>, but not public-inbox-*

=head2 Performance on rotational hard disk drives

Random I/O performance is poor on rotational HDDs.  Xapian indexing
performance degrades significantly as DBs grow larger than available
RAM.  Attempts to parallelize random I/O on HDDs leads to pathological
slowdowns as inboxes grow.

While C<-V2> introduced Xapian shards as a parallelization
mechanism for SSDs, enabling C<publicInbox.indexSequentialShard>
repurposes sharding as a mechanism to reduce the kernel page cache
footprint when indexing on HDDs.

Initializing a mirror with a high C<--jobs> count to create more
shards (in C<-V2> inboxes) will keep each shard smaller and
reduce its kernel page cache footprint.  Keep in mind excessive
sharding imposes a performance penalty for read-only queries.

Users with large amounts of RAM are advised to set a large value
for C<publicinbox.indexBatchSize> as documented in
L<public-inbox-index(1)>.

C<dm-crypt> users on Linux 4.0+ are advised to try the
C<--perf-same_cpu_crypt> C<--perf-submit_from_crypt_cpus>
switches of L<cryptsetup(8)> to reduce I/O contention from
kernel workqueue threads.

=head2 Btrfs (and possibly other copy-on-write filesystems)

L<btrfs(5)> performance degrades from fragmentation when using
large databases and random writes.  The Xapian + SQLite indices
used by public-inbox are no exception to that.

public-inbox 1.6.0+ disables copy-on-write (CoW) on Xapian and SQLite
indices on btrfs to achieve acceptable performance (even on SSD).
Disabling copy-on-write also disables checksumming, thus C<raid1>
(or higher) configurations may be corrupt after unsafe shutdowns.

Fortunately, these SQLite and Xapian indices are designed to be
recoverable from git if missing.

Disabling CoW does not prevent all fragmentation.  Large values
of C<publicInbox.indexBatchSize> also limit fragmentation during
the initial index.

Avoid snapshotting subvolumes containing Xapian and/or SQLite indices.
Snapshots use CoW despite our efforts to disable it, resulting
in fragmentation.

L<filefrag(8)> can be used to monitor fragmentation, and
C<btrfs filesystem defragment -fr $INBOX_DIR> may be necessary.

Large filesystems benefit significantly from the C<space_cache=v2>
mount option documented in L<btrfs(5)>.

Older, non-CoW filesystems generally work well out of the box
for our Xapian and SQLite indices.

=head2 Performance on solid state drives

While SSD read performance is generally good, SSD write performance
degrades as the drive ages and/or gets full.  Issuing C<TRIM> commands
via L<fstrim(8)> or similar is required to sustain write performance.

Users of the Flash-Friendly File System
L<F2FS|https://en.wikipedia.org/wiki/F2FS> may benefit from
optimizations found in SQLite 3.21.0+.  Benchmarks are greatly
appreciated.

=head2 Read-only daemons

L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, and
L<public-inbox-nntpd(1)> are all designed for C10K (or higher)
levels of concurrency from a single process.  SMP systems may
use C<--worker-processes=NUM> as documented in L<public-inbox-daemon(8)>
for parallelism.

The open file descriptor limit (C<RLIMIT_NOFILE>, C<ulimit -n> in L<sh(1)>,
C<LimitNOFILE=> in L<systemd.exec(5)>) may need to be raised to
accommodate many concurrent clients.

Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly
increases memory use of client sockets, be sure to account for that in
capacity planning.

Bursts of small object allocations late in process life contribute to
fragmentation of the heap due to arenas (slabs) used internally by Perl.
glibc malloc users should use C<MALLOC_MMAP_THRESHOLD_=131072> to reduce
fragmentation from the sliding mmap window.  On 64-bit systems, jemalloc
(tested as an LD_PRELOAD on GNU/Linux) reduces fragmentation at the
expense of VM space.  32-bit systems may be better off sticking with
glibc and MALLOC_MMAP_THRESHOLD_.

=head2 Other OS tuning knobs

Linux users: the C<sys.vm.max_map_count> sysctl may need to be increased if
handling thousands of inboxes (with L<public-inbox-extindex(1)>) to avoid
out-of-memory errors from git.

Other OSes may have similar tuning knobs (patches appreciated).

=head2 Scalability to many inboxes

L<public-inbox-extindex(1)> allows any number of public-inboxes
to share the same Xapian indices.

git 2.33+ startup time is orders of magnitude faster and uses
less memory when dealing with thousands of alternates required
for thousands of inboxes with L<public-inbox-extindex(1)>.

Frequent packing (via L<git-gc(1)>) both improves performance
and reduces the need to increase C<sys.vm.max_map_count>.

=head2 public-inbox-cindex --join performance

A C++ compiler and the Xapian development files makes C<--join> or
C<--join=aggressive> orders of magnitude faster in L<public-inbox-cindex(1)>.
On Debian-based systems this is C<libxapian-dev>.  RPM-based distros have
these in C<xapian-core-devel> or C<xapian14-core-libs>.  *BSDs typically
package development files together with runtime libraries, so the C<xapian>
or C<xapian-core> package will already have the development files.

=head2 public-inbox-clone with shared object stores

When mirroring manifests with many forks using the same objstore,
git 2.41+ is highly recommended for performance as we automatically
use the C<fetch.hideRefs> feature to speed up negotiation.

=head1 CONTACT

Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org>

Information for *BSDs and non-traditional filesystems especially
welcome.

Our archives are hosted at L<https://public-inbox.org/meta/>,
L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>, and other places

=head1 COPYRIGHT

Copyright all contributors L<mailto:meta@public-inbox.org>

License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>