=head1 NAME

public-inbox-extindex - create and update external search indices

=head1 SYNOPSIS

public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...

public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all

=head1 DESCRIPTION

public-inbox-extindex creates and updates an external search and
overview database used by the read-only public-inbox PSGI (HTTP),
NNTP, and IMAP interfaces.  This requires either the
L<Xapian> SWIG bindings OR or L<Search::Xapian> XS bindings
along with L<DBD::SQLite> and L<DBI> Perl modules.

=head1 OPTIONS

=over

=item -j JOBS

=item --jobs=JOBS

=item --no-fsync

=item --dangerous

=item --rethread

=item --max-size SIZE

=item --batch-size SIZE

These switches behave as they do for L<public-inbox-index(1)>

=item --all

Index all C<publicinbox> entries in C<PI_CONFIG>.

C<publicinbox> entries indexed by C<public-inbox-extindex> can
have full Xapian searching abilities with the per-C<publicinbox>
C<indexlevel> set to C<basic> and their respective Xapian
(C<xap15> or C<xapian15>) directories removed.  For multiple
public-inboxes where cross-posting is common, this allows
significant space savings on Xapian indices.

=item --dedupe=MSGID

=item --dedupe

Rerun deduplication on messages with the given Message-ID or
all messages if no Message-ID is specified.  Deduplication rules may
change and evolve over time, especially if filters are involved.

C<--dedupe=MSGID> may be specified multiple times to deduplicate
multiple Message-IDs.

Use this if you see C<W: BUG? $MSGID not deduplicated properly>
warnings from WWW logs.

=item --gc

Perform garbage collection instead of indexing.  Use this if
inboxes are removed from the extindex, a newsgroup name is
set or changed, or if messages are purged or removed from
some inboxes.

=item --reindex

Forces a re-index of all messages in the extindex.  This can be
used for in-place upgrades and bugfixes while read-only server
processes are utilizing the index.  Keep in mind this roughly
doubles the size of the already-large Xapian database.

=item --fast

Used with C<--reindex>, it will only look for new and stale
entries and not touch already-indexed messages.

=item --no-multi-pack-index

Disable writing a L<git-multi-pack-index(1)> file to save memory.
Normally, enabling multi-pack-index speeds up startup time of
subsequent L<git-cat-file(1)> processes by 3-4%, but generating
this file requires several GB of memory with large repos.

Unlike the C<core.multiPackIndex> directive in git, it's still
possible to read existing multi-pack-index files if they are
created elsewhere.

Available in public-inbox 2.0.0+

=back

=head1 FILES

L<public-inbox-extindex-format(5)>

=head1 CONFIGURATION

public-inbox-extindex does not write to the L<public-inbox-config(5)>
file, it must be entered manually.
The extindex name of C<all> is a special case which
corresponds to indexing C<--all> inboxes.  An example for
C<--all> is as follows:

	[extindex "all"]
		topdir = /path/to/extindex_dir
		url = all
		coderepo = foo
		coderepo = bar

Putting an C<extindex> entry in the config allows L<PublicInbox::WWW>.
You can have any number of C<extentry.$NAME> sections where C<$NAME>
is something other than C<all> to display a union of several inboxes.

It is strongly recommended any public inboxes indexed by this command
have a stable C<publicinbox.$NAME.newsgroup> entry (regardless of
the presence of an NNTP or IMAP server).  Otherwise, public-inbox-extindex
will use C<publicinbox.$NAME.inboxdir> as an internal key which can
cause needless reindexing and require L<--gc> if inboxes are relocated.

See L<public-inbox-config(5)> for more details.

=head1 ENVIRONMENT

=over 8

=item PI_CONFIG

Used to override the default "~/.public-inbox/config" value.

=item XAPIAN_FLUSH_THRESHOLD

The number of documents to update before committing changes to
disk.  This environment is handled directly by Xapian, refer to
Xapian API documentation for more details.

Setting C<XAPIAN_FLUSH_THRESHOLD> or
C<publicinbox.indexBatchSize> for a large C<--reindex> may cause
L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and
L<public-inbox-watch(1)> tasks to wait long and unpredictable
periods of time during C<--reindex>.

Default: none, uses C<publicinbox.indexBatchSize>

=back

=head1 UPGRADING

Occasionally, public-inbox will update its schema version and
require a full index by running this command.

=head1 LOCKING

It is safe to use C<--dedupe>, C<--gc> and C<--reindex> while
other processes are writing to covered inboxes or extindex.
The extindex locks will be released roughly every 10s to
allow L<public-inbox-mda(1)> and L<public-inbox-watch(1)>
processes to write to the extindex.

=head1 CONTACT

Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>

The mail archives are hosted at L<https://public-inbox.org/meta/> and
L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>

=head1 COPYRIGHT

Copyright all contributors L<mailto:meta@public-inbox.org>

License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>

=head1 SEE ALSO

L<Search::Xapian>, L<DBD::SQLite>