| GROKMIRROR |
| ========== |
| -------------------------------------------- |
| Framework to smartly mirror git repositories |
| -------------------------------------------- |
| |
| :Author: konstantin@linuxfoundation.org |
| :Date: 2020-09-18 |
| :Copyright: The Linux Foundation and contributors |
| :License: GPLv3+ |
| :Version: 2.0.0 |
| |
| DESCRIPTION |
| ----------- |
| Grokmirror was written to make replicating large git repository |
| collections more efficient. Grokmirror uses the manifest file published |
| by the origin server in order to figure out which repositories to clone, |
| and to track which repositories require updating. The process is |
| lightweight and efficient both for the primary and for the replicas. |
| |
| CONCEPTS |
| -------- |
| The origin server publishes a json-formatted manifest file containing |
| information about all git repositories that it carries. The format of |
| the manifest file is as follows:: |
| |
| { |
| "/path/to/bare/repository.git": { |
| "description": "Repository description", |
| "head": "ref: refs/heads/branchname", |
| "reference": "/path/to/reference/repository.git", |
| "forkgroup": "forkgroup-guid", |
| "modified": timestamp, |
| "fingerprint": sha1sum(git show-ref), |
| "symlinks": [ |
| "/location/to/symlink", |
| ... |
| ], |
| } |
| ... |
| } |
| |
| The manifest file is usually gzip-compressed to preserve bandwidth. |
| |
| Each time a commit is made to one of the git repositories, it |
| automatically updates the manifest file using an appropriate git hook, |
| so the manifest.js file should always contain the most up-to-date |
| information about the state of all repositories. |
| |
| The mirroring clients will poll the manifest.js file and download the |
| updated manifest if it is newer than the locally stored copy (using |
| ``Last-Modified`` and ``If-Modified-Since`` http headers). After |
| downloading the updated manifest.js file, the mirrors will parse it to |
| find out which repositories have been updated and which new repositories |
| have been added. |
| |
| Object Storage Repositories |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| Grokmirror 2.0 introduces the concept of "object storage repositories", |
| which aims to optimize how repository forks are stored on disk and |
| served to the cloning clients. |
| |
| When grok-fsck runs, it will automatically recognize related |
| repositories by analyzing their root commits. If it finds two or more |
| related repositories, it will set up a unified "object storage" repo and |
| fetch all refs from each related repository into it. |
| |
| For example, you can have two forks of linux.git: |
| torvalds/linux.git: |
| refs/heads/master |
| refs/tags/v5.0-rc3 |
| ... |
| |
| and its fork: |
| |
| maintainer/linux.git: |
| refs/heads/master |
| refs/heads/devbranch |
| refs/tags/v5.0-rc3 |
| ... |
| |
| Grok-fsck will set up an object storage repository and fetch all refs from |
| both repositories: |
| |
| objstore/[random-guid-name].git |
| refs/virtual/[sha1-of-torvalds/linux.git:12]/heads/master |
| refs/virtual/[sha1-of-torvalds/linux.git:12]/tags/v5.0-rc3 |
| ... |
| refs/virtual/[sha1-of-maintainer/linux.git:12]/heads/master |
| refs/virtual/[sha1-of-maintainer/linux.git:12]/heads/devbranch |
| refs/virtual/[sha1-of-maintainer/linux.git:12]/tags/v5.0-rc3 |
| ... |
| |
| Then both torvalds/linux.git and maintainer/linux.git with be configured |
| to use objstore/[random-guid-name].git via objects/info/alternates |
| and repacked to just contain metadata and no objects. |
| |
| The alternates repository will be repacked with "delta islands" enabled, |
| which should help optimize clone operations for each "sibling" |
| repository. |
| |
| Please see the example grokmirror.conf for more details about |
| configuring objstore repositories. |
| |
| |
| ORIGIN SETUP |
| ------------ |
| Install grokmirror on the origin server using your preferred way. |
| |
| **IMPORTANT: Only bare git repositories are supported.** |
| |
| You will need to add a hook to each one of your repositories that would |
| update the manifest upon repository modification. This can either be a |
| post-receive hook, or a post-update hook. The hook must call the |
| following command:: |
| |
| /usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \ |
| -t /var/lib/gitolite3/repositories -n `pwd` |
| |
| The **-m** flag is the path to the manifest.js file. The git process |
| must be able to write to it and to the directory the file is in (it |
| creates a manifest.js.randomstring file first, and then moves it in |
| place of the old one for atomicity). |
| |
| The **-t** flag is to help grokmirror trim the irrelevant toplevel disk |
| path, so it is trimmed from the top. |
| |
| The **-n** flag tells grokmirror to use the current timestamp instead of |
| the exact timestamp of the commit (much faster this way). |
| |
| Before enabling the hook, you will need to generate the manifest.js of |
| all your git repositories. In order to do that, run the same command, |
| but omit the -n and the \`pwd\` argument. E.g.:: |
| |
| /usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \ |
| -t /var/lib/gitolite3/repositories |
| |
| The last component you need to set up is to automatically purge deleted |
| repositories from the manifest. As this can't be added to a git hook, |
| you can either run the ``--purge`` command from cron:: |
| |
| /usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \ |
| -t /var/lib/gitolite3/repositories -p |
| |
| Or add it to your gitolite's ``D`` command using the ``--remove`` flag:: |
| |
| /usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \ |
| -t /var/lib/gitolite3/repositories -x $repo.git |
| |
| If you would like grok-manifest to honor the ``git-daemon-export-ok`` |
| magic file and only add to the manifest those repositories specifically |
| marked as exportable, pass the ``--check-export-ok`` flag. See |
| ``git-daemon(1)`` for more info on ``git-daemon-export-ok`` file. |
| |
| You will need to have some kind of httpd server to serve the manifest |
| file. |
| |
| REPLICA SETUP |
| ------------- |
| Install grokmirror on the replica using your preferred way. |
| |
| Locate grokmirror.conf and modify it to reflect your needs. The default |
| configuration file is heavily commented to explain what each option |
| does. |
| |
| Make sure the user "mirror" (or whichever user you specified) is able to |
| write to the toplevel and log locations specified in grokmirror.conf. |
| |
| You can either run grok-pull manually, from cron, or as a |
| systemd-managed daemon (see contrib). If you do it more frequently than |
| once every few hours, you should definitely run it as a daemon in order |
| to improve performance. |
| |
| GROK-FSCK |
| --------- |
| Git repositories should be routinely repacked and checked for |
| corruption. This utility will perform the necessary optimizations and |
| report any problems to the email defined via fsck.report_to ('root' by |
| default). It should run weekly from cron or from the systemd timer (see |
| contrib). |
| |
| Please examine the example grokmirror.conf file for various things you |
| can tweak. |
| |
| FAQ |
| --- |
| Why is it called "grok mirror"? |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| Because it's developed at kernel.org and "grok" is a mirror of "korg". |
| Also, because it groks git mirroring. |
| |
| Why not just use rsync? |
| ~~~~~~~~~~~~~~~~~~~~~~~ |
| Rsync is extremely inefficient for the purpose of mirroring git trees |
| that mostly consist of a lot of small files that very rarely change. |
| Since rsync must calculate checksums on each file during each run, it |
| mostly results in a lot of disk thrashing. |
| |
| Additionally, if several repositories share objects between each-other, |
| unless the disk paths are exactly the same on both the remote and local |
| mirror, this will result in broken git repositories. |
| |
| It is also a bit silly, considering git provides its own extremely |
| efficient mechanism for specifying what changed between revision X and |
| revision Y. |
