maintenance: add incremental-repack task
The previous change cleaned up loose objects using the
'loose-objects' that can be run safely in the background. Add a
similar job that performs similar cleanups for pack-files.
One issue with running 'git repack' is that it is designed to
repack all pack-files into a single pack-file. While this is the
most space-efficient way to store object data, it is not time or
memory efficient. This becomes extremely important if the repo is
so large that a user struggles to store two copies of the pack on
their disk.
Instead, perform an "incremental" repack by collecting a few small
pack-files into a new pack-file. The multi-pack-index facilitates
this process ever since 'git multi-pack-index expire' was added in
19575c7 (multi-pack-index: implement 'expire' subcommand,
2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1
(midx: implement midx_repack(), 2019-06-10).
The 'incremental-repack' task runs the following steps:
1. 'git multi-pack-index write' creates a multi-pack-index file if
one did not exist, and otherwise will update the multi-pack-index
with any new pack-files that appeared since the last write. This
is particularly relevant with the background fetch job.
When the multi-pack-index sees two copies of the same object, it
stores the offset data into the newer pack-file. This means that
some old pack-files could become "unreferenced" which I will use
to mean "a pack-file that is in the pack-file list of the
multi-pack-index but none of the objects in the multi-pack-index
reference a location inside that pack-file."
2. 'git multi-pack-index expire' deletes any unreferenced pack-files
and updaes the multi-pack-index to drop those pack-files from the
list. This is safe to do as concurrent Git processes will see the
multi-pack-index and not open those packs when looking for object
contents. (Similar to the 'loose-objects' job, there are some Git
commands that open pack-files regardless of the multi-pack-index,
but they are rarely used. Further, a user that self-selects to
use background operations would likely refrain from using those
commands.)
3. 'git multi-pack-index repack --bacth-size=<size>' collects a set
of pack-files that are listed in the multi-pack-index and creates
a new pack-file containing the objects whose offsets are listed
by the multi-pack-index to be in those objects. The set of pack-
files is selected greedily by sorting the pack-files by modified
time and adding a pack-file to the set if its "expected size" is
smaller than the batch size until the total expected size of the
selected pack-files is at least the batch size. The "expected
size" is calculated by taking the size of the pack-file divided
by the number of objects in the pack-file and multiplied by the
number of objects from the multi-pack-index with offset in that
pack-file. The expected size approximates how much data from that
pack-file will contribute to the resulting pack-file size. The
intention is that the resulting pack-file will be close in size
to the provided batch size.
The next run of the incremental-repack task will delete these
repacked pack-files during the 'expire' step.
In this version, the batch size is set to "0" which ignores the
size restrictions when selecting the pack-files. It instead
selects all pack-files and repacks all packed objects into a
single pack-file. This will be updated in the next change, but
it requires doing some calculations that are better isolated to
a separate change.
These steps are based on a similar background maintenance step in
Scalar (and VFS for Git) [1]. This was incredibly effective for
users of the Windows OS repository. After using the same VFS for Git
repository for over a year, some users had _thousands_ of pack-files
that combined to up to 250 GB of data. We noticed a few users were
running into the open file descriptor limits (due in part to a bug
in the multi-pack-index fixed by af96fe3 (midx: add packs to
packed_git linked list, 2019-04-29).
These pack-files were mostly small since they contained the commits
and trees that were pushed to the origin in a given hour. The GVFS
protocol includes a "prefetch" step that asks for pre-computed pack-
files containing commits and trees by timestamp. These pack-files
were grouped into "daily" pack-files once a day for up to 30 days.
If a user did not request prefetch packs for over 30 days, then they
would get the entire history of commits and trees in a new, large
pack-file. This led to a large number of pack-files that had poor
delta compression.
By running this pack-file maintenance step once per day, these repos
with thousands of packs spanning 200+ GB dropped to dozens of pack-
files spanning 30-50 GB. This was done all without removing objects
from the system and using a constant batch size of two gigabytes.
Once the work was done to reduce the pack-files to small sizes, the
batch size of two gigabytes means that not every run triggers a
repack operation, so the following run will not expire a pack-file.
This has kept these repos in a "clean" state.
[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt
index fc95eb5..3f5d894 100644
--- a/Documentation/git-maintenance.txt
+++ b/Documentation/git-maintenance.txt
@@ -85,6 +85,24 @@
advisable to enable both the `loose-objects` and `gc` tasks at the
same time.
+incremental-repack::
+ The `incremental-repack` job repacks the object directory
+ using the `multi-pack-index` feature. In order to prevent race
+ conditions with concurrent Git commands, it follows a two-step
+ process. First, it calls `git multi-pack-index expire` to delete
+ pack-files unreferenced by the `multi-pack-index` file. Second, it
+ calls `git multi-pack-index repack` to select several small
+ pack-files and repack them into a bigger one, and then update the
+ `multi-pack-index` entries that refer to the small pack-files to
+ refer to the new pack-file. This prepares those small pack-files
+ for deletion upon the next run of `git multi-pack-index expire`.
+ The selection of the small pack-files is such that the expected
+ size of the big pack-file is at least the batch size; see the
+ `--batch-size` option for the `repack` subcommand in
+ linkgit:git-multi-pack-index[1]. The default batch-size is zero,
+ which is a special case that attempts to repack all pack-files
+ into a single pack-file.
+
OPTIONS
-------
--auto::
diff --git a/builtin/gc.c b/builtin/gc.c
index 4403827..5f877b0 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -1001,6 +1001,77 @@ static int maintenance_task_loose_objects(struct maintenance_run_opts *opts)
return prune_packed(opts) || pack_loose(opts);
}
+static int multi_pack_index_write(struct maintenance_run_opts *opts)
+{
+ struct child_process child = CHILD_PROCESS_INIT;
+
+ child.git_cmd = 1;
+ strvec_pushl(&child.args, "multi-pack-index", "write", NULL);
+
+ if (opts->quiet)
+ strvec_push(&child.args, "--no-progress");
+
+ if (run_command(&child))
+ return error(_("failed to write multi-pack-index"));
+
+ return 0;
+}
+
+static int multi_pack_index_expire(struct maintenance_run_opts *opts)
+{
+ struct child_process child = CHILD_PROCESS_INIT;
+
+ child.git_cmd = 1;
+ strvec_pushl(&child.args, "multi-pack-index", "expire", NULL);
+
+ if (opts->quiet)
+ strvec_push(&child.args, "--no-progress");
+
+ close_object_store(the_repository->objects);
+
+ if (run_command(&child))
+ return error(_("'git multi-pack-index expire' failed"));
+
+ return 0;
+}
+
+static int multi_pack_index_repack(struct maintenance_run_opts *opts)
+{
+ struct child_process child = CHILD_PROCESS_INIT;
+
+ child.git_cmd = 1;
+ strvec_pushl(&child.args, "multi-pack-index", "repack", NULL);
+
+ if (opts->quiet)
+ strvec_push(&child.args, "--no-progress");
+
+ strvec_push(&child.args, "--batch-size=0");
+
+ close_object_store(the_repository->objects);
+
+ if (run_command(&child))
+ return error(_("'git multi-pack-index repack' failed"));
+
+ return 0;
+}
+
+static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts)
+{
+ prepare_repo_settings(the_repository);
+ if (!the_repository->settings.core_multi_pack_index) {
+ warning(_("skipping incremental-repack task because core.multiPackIndex is disabled"));
+ return 0;
+ }
+
+ if (multi_pack_index_write(opts))
+ return 1;
+ if (multi_pack_index_expire(opts))
+ return 1;
+ if (multi_pack_index_repack(opts))
+ return 1;
+ return 0;
+}
+
typedef int maintenance_task_fn(struct maintenance_run_opts *opts);
/*
@@ -1023,6 +1094,7 @@ struct maintenance_task {
enum maintenance_task_label {
TASK_PREFETCH,
TASK_LOOSE_OBJECTS,
+ TASK_INCREMENTAL_REPACK,
TASK_GC,
TASK_COMMIT_GRAPH,
@@ -1040,6 +1112,10 @@ static struct maintenance_task tasks[] = {
maintenance_task_loose_objects,
loose_object_auto_condition,
},
+ [TASK_INCREMENTAL_REPACK] = {
+ "incremental-repack",
+ maintenance_task_incremental_repack,
+ },
[TASK_GC] = {
"gc",
maintenance_task_gc,
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index ec87f61..2f942ee 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -3,6 +3,7 @@
test_description='multi-pack-indexes'
. ./test-lib.sh
+GIT_TEST_MULTI_PACK_INDEX=0
objdir=.git/objects
midx_read_expect () {
diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
index 27565c5..a2db229 100755
--- a/t/t7900-maintenance.sh
+++ b/t/t7900-maintenance.sh
@@ -5,6 +5,7 @@
. ./test-lib.sh
GIT_TEST_COMMIT_GRAPH=0
+GIT_TEST_MULTI_PACK_INDEX=0
test_expect_success 'help text' '
test_expect_code 129 git maintenance -h 2>err &&
@@ -149,4 +150,41 @@
test_subcommand git prune-packed --quiet <trace-loC
'
+test_expect_success 'incremental-repack task' '
+ packDir=.git/objects/pack &&
+ for i in $(test_seq 1 5)
+ do
+ test_commit $i || return 1
+ done &&
+
+ # Create three disjoint pack-files with size BIG, small, small.
+ echo HEAD~2 | git pack-objects --revs $packDir/test-1 &&
+ test_tick &&
+ git pack-objects --revs $packDir/test-2 <<-\EOF &&
+ HEAD~1
+ ^HEAD~2
+ EOF
+ test_tick &&
+ git pack-objects --revs $packDir/test-3 <<-\EOF &&
+ HEAD
+ ^HEAD~1
+ EOF
+ rm -f $packDir/pack-* &&
+ rm -f $packDir/loose-* &&
+ ls $packDir/*.pack >packs-before &&
+ test_line_count = 3 packs-before &&
+
+ # the job repacks the two into a new pack, but does not
+ # delete the old ones.
+ git maintenance run --task=incremental-repack &&
+ ls $packDir/*.pack >packs-between &&
+ test_line_count = 4 packs-between &&
+
+ # the job deletes the two old packs, and does not write
+ # a new one because only one pack remains.
+ git maintenance run --task=incremental-repack &&
+ ls .git/objects/pack/*.pack >packs-after &&
+ test_line_count = 1 packs-after
+'
+
test_done
