blob: 40ba4aa3e65b4c33886ba4334e5d3c84e979853e [file] [log] [blame]
Johannes Schindelinc401b332007-07-04 00:41:55 +01001git-filter-branch(1)
2====================
3
4NAME
5----
6git-filter-branch - Rewrite branches
7
8SYNOPSIS
9--------
10[verse]
David Glasser07c49842017-10-17 09:45:15 +000011'git filter-branch' [--setup <command>] [--subdirectory-filter <directory>]
12 [--env-filter <command>] [--tree-filter <command>]
13 [--index-filter <command>] [--parent-filter <command>]
14 [--msg-filter <command>] [--commit-filter <command>]
15 [--tag-name-filter <command>] [--prune-empty]
Giuseppe Bilotta54332352007-08-30 19:10:42 +020016 [--original <namespace>] [-d <directory>] [-f | --force]
Ian Campbellbd2c79f2017-09-21 08:49:31 +010017 [--state-branch <branch>] [--] [<rev-list options>...]
Johannes Schindelinc401b332007-07-04 00:41:55 +010018
Elijah Newren9df53c52019-09-04 15:32:38 -070019WARNING
20-------
21'git filter-branch' has a plethora of pitfalls that can produce non-obvious
22manglings of the intended history rewrite (and can leave you with little
23time to investigate such problems since it has such abysmal performance).
24These safety and performance issues cannot be backward compatibly fixed and
25as such, its use is not recommended. Please use an alternative history
26filtering tool such as https://github.com/newren/git-filter-repo/[git
27filter-repo]. If you still need to use 'git filter-branch', please
28carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land
29mines of filter-branch, and then vigilantly avoid as many of the hazards
30listed there as reasonably possible.
31
Johannes Schindelinc401b332007-07-04 00:41:55 +010032DESCRIPTION
33-----------
Thomas Ackermann2de9b712013-01-21 20:17:53 +010034Lets you rewrite Git revision history by rewriting the branches mentioned
Johannes Schindelin08203662007-08-31 17:42:33 +010035in the <rev-list options>, applying custom filters on each revision.
Johannes Schindelinc401b332007-07-04 00:41:55 +010036Those filters can modify each tree (e.g. removing a file or running
37a perl rewrite on all files) or information about each commit.
38Otherwise, all information (including original commit times or merge
39information) will be preserved.
40
Johannes Schindelin08203662007-08-31 17:42:33 +010041The command will only rewrite the _positive_ refs mentioned in the
Ralf Wildenhuesbf7c9022008-03-20 22:30:32 +010042command line (e.g. if you pass 'a..b', only 'b' will be rewritten).
Johannes Schindelin08203662007-08-31 17:42:33 +010043If you specify no filters, the commits will be recommitted without any
44changes, which would normally have no effect. Nevertheless, this may be
Thomas Ackermann2de9b712013-01-21 20:17:53 +010045useful in the future for compensating for some Git bugs or such,
Johannes Schindelin08203662007-08-31 17:42:33 +010046therefore such a usage is permitted.
Johannes Schindelinc401b332007-07-04 00:41:55 +010047
Junio C Hamano831e61f2012-08-06 13:36:47 -070048*NOTE*: This command honors `.git/info/grafts` file and refs in
49the `refs/replace/` namespace.
Peter Collingbourne0dc310e2011-07-21 16:10:52 +010050If you have any grafts or replacement refs defined, running this command
51will make them permanent.
Daniel Cheng (aka SDiZ)c6d8f762009-04-10 14:26:49 +080052
Johannes Schindelin73616fd2007-07-04 15:50:45 +010053*WARNING*! The rewritten history will have different object names for all
Johannes Schindelinc401b332007-07-04 00:41:55 +010054the objects and will not converge with the original branch. You will not
55be able to easily push and distribute the rewritten branch on top of the
56original branch. Please do not use this command if you do not know the
57full implications, and avoid using it anyway, if a simple single commit
Thomas Rast97c33c62008-09-13 18:11:01 +020058would suffice to fix your problem. (See the "RECOVERING FROM UPSTREAM
59REBASE" section in linkgit:git-rebase[1] for further information about
60rewriting published history.)
Johannes Schindelinc401b332007-07-04 00:41:55 +010061
Johannes Schindelindfd05e32007-07-23 18:34:13 +010062Always verify that the rewritten version is correct: The original refs,
63if different from the rewritten ones, will be stored in the namespace
64'refs/original/'.
Johannes Schindelinc401b332007-07-04 00:41:55 +010065
Ralf Wildenhuesbf7c9022008-03-20 22:30:32 +010066Note that since this operation is very I/O expensive, it might
Johannes Schindelin08203662007-08-31 17:42:33 +010067be a good idea to redirect the temporary directory off-disk with the
Matthieu Moy23f82392016-06-28 13:40:10 +020068`-d` option, e.g. on tmpfs. Reportedly the speedup is very noticeable.
Johannes Schindelinc401b332007-07-04 00:41:55 +010069
70
71Filters
72~~~~~~~
73
74The filters are applied in the order as listed below. The <command>
Ralf Wildenhuesbf7c9022008-03-20 22:30:32 +010075argument is always evaluated in the shell context using the 'eval' command
76(with the notable exception of the commit filter, for technical reasons).
Tom Russello47d81b52016-06-08 00:35:07 +020077Prior to that, the `$GIT_COMMIT` environment variable will be set to contain
Johannes Schindelinc401b332007-07-04 00:41:55 +010078the id of the commit being rewritten. Also, GIT_AUTHOR_NAME,
79GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL,
Tadeusz Andrzej Kadłubowskibee3eb02013-02-21 21:22:50 +010080and GIT_COMMITTER_DATE are taken from the current commit and exported to
81the environment, in order to affect the author and committer identities of
82the replacement commit created by linkgit:git-commit-tree[1] after the
83filters have run.
84
Ralf Wildenhuesbf7c9022008-03-20 22:30:32 +010085If any evaluation of <command> returns a non-zero exit status, the whole
86operation will be aborted.
Johannes Schindelinc401b332007-07-04 00:41:55 +010087
88A 'map' function is available that takes an "original sha1 id" argument
89and outputs a "rewritten sha1 id" if the commit has been already
Johannes Sixt32c37c12007-07-04 09:32:47 +020090rewritten, and "original sha1 id" otherwise; the 'map' function can
91return several ids on separate lines if your commit filter emitted
92multiple commits.
Johannes Schindelinc401b332007-07-04 00:41:55 +010093
94
95OPTIONS
96-------
97
Andreas Heiduk3b117f72017-06-10 10:54:44 +020098--setup <command>::
99 This is not a real filter executed for each commit but a one
100 time setup just before the loop. Therefore no commit-specific
101 variables are defined yet. Functions or variables defined here
102 can be used or modified in the following filter steps except
103 the commit filter, for technical reasons.
104
David Glasser07c49842017-10-17 09:45:15 +0000105--subdirectory-filter <directory>::
106 Only look at the history which touches the given subdirectory.
107 The result will contain that directory (and only that) as its
108 project root. Implies <<Remap_to_ancestor>>.
109
Johannes Schindelinc401b332007-07-04 00:41:55 +0100110--env-filter <command>::
Ralf Wildenhuesbf7c9022008-03-20 22:30:32 +0100111 This filter may be used if you only need to modify the environment
112 in which the commit will be performed. Specifically, you might
113 want to rewrite the author/committer name/email/time environment
Andreas Heidukba746ff2017-05-26 19:36:54 +0200114 variables (see linkgit:git-commit-tree[1] for details).
Johannes Schindelinc401b332007-07-04 00:41:55 +0100115
116--tree-filter <command>::
117 This is the filter for rewriting the tree and its contents.
118 The argument is evaluated in shell with the working
119 directory set to the root of the checked out tree. The new tree
120 is then used as-is (new files are auto-added, disappeared files
121 are auto-removed - neither .gitignore files nor any other ignore
Johannes Schindelin73616fd2007-07-04 15:50:45 +0100122 rules *HAVE ANY EFFECT*!).
Johannes Schindelinc401b332007-07-04 00:41:55 +0100123
124--index-filter <command>::
125 This is the filter for rewriting the index. It is similar to the
126 tree filter but does not check out the tree, which makes it much
Jeff King6cf378f2012-04-26 04:51:57 -0400127 faster. Frequently used with `git rm --cached
128 --ignore-unmatch ...`, see EXAMPLES below. For hairy
Thomas Rast3bc427e2009-03-12 00:00:56 +0100129 cases, see linkgit:git-update-index[1].
Johannes Schindelinc401b332007-07-04 00:41:55 +0100130
131--parent-filter <command>::
132 This is the filter for rewriting the commit's parent list.
133 It will receive the parent string on stdin and shall output
134 the new parent string on stdout. The parent string is in
Jonathan Nieder483bc4f2008-06-30 13:56:34 -0500135 the format described in linkgit:git-commit-tree[1]: empty for
Johannes Schindelinc401b332007-07-04 00:41:55 +0100136 the initial commit, "-p parent" for a normal commit and
137 "-p parent1 -p parent2 -p parent3 ..." for a merge commit.
138
139--msg-filter <command>::
140 This is the filter for rewriting the commit messages.
141 The argument is evaluated in the shell with the original
142 commit message on standard input; its standard output is
143 used as the new commit message.
144
145--commit-filter <command>::
146 This is the filter for performing the commit.
147 If this filter is specified, it will be called instead of the
Thomas Rast0b444cd2010-01-10 00:33:00 +0100148 'git commit-tree' command, with arguments of the form
Štěpán Němec0adda932010-10-08 19:31:17 +0200149 "<TREE_ID> [(-p <PARENT_COMMIT_ID>)...]" and the log message on
Johannes Schindelinc401b332007-07-04 00:41:55 +0100150 stdin. The commit id is expected on stdout.
151+
152As a special extension, the commit filter may emit multiple
Kevin Ballardc5833f62008-05-30 14:43:40 -0700153commit ids; in that case, the rewritten children of the original commit will
Johannes Schindelinc401b332007-07-04 00:41:55 +0100154have all of them as parents.
Johannes Schindelinf95eef12007-08-31 20:06:27 +0100155+
156You can use the 'map' convenience function in this filter, and other
157convenience functions, too. For example, calling 'skip_commit "$@"'
158will leave out the current commit (but not its changes! If you want
Thomas Rast0b444cd2010-01-10 00:33:00 +0100159that, use 'git rebase' instead).
Pierre Habouzitd3240d92008-10-31 10:12:21 +0100160+
Thomas Rastca768282010-01-07 17:49:12 +0100161You can also use the `git_commit_non_empty_tree "$@"` instead of
162`git commit-tree "$@"` if you don't wish to keep commits with a single parent
Pierre Habouzitd3240d92008-10-31 10:12:21 +0100163and that makes no change to the tree.
Johannes Schindelinc401b332007-07-04 00:41:55 +0100164
165--tag-name-filter <command>::
166 This is the filter for rewriting tag names. When passed,
167 it will be called for every tag ref that points to a rewritten
168 object (or to a tag object which points to a rewritten object).
169 The original tag name is passed via standard input, and the new
170 tag name is expected on standard output.
171+
172The original tags are not deleted, but can be overwritten;
Brian Gernhardt5876b8e2007-08-17 19:13:04 -0400173use "--tag-name-filter cat" to simply update the tags. In this
Johannes Schindelinc401b332007-07-04 00:41:55 +0100174case, be very careful and make sure you have the old tags
175backed up in case the conversion has run afoul.
176+
Brandon Casey1bf65512008-03-26 10:47:09 -0500177Nearly proper rewriting of tag objects is supported. If the tag has
178a message attached, a new tag object will be created with the same message,
179author, and timestamp. If the tag has a signature attached, the
180signature will be stripped. It is by definition impossible to preserve
181signatures. The reason this is "nearly" proper, is because ideally if
182the tag did not change (points to the same object, has the same name, etc.)
183it should retain any signature. That is not the case, signatures will always
184be removed, buyer beware. There is also no support for changing the
185author or timestamp (or the tag message for that matter). Tags which point
186to other tags will be rewritten to point to the underlying commit.
Johannes Schindelinc401b332007-07-04 00:41:55 +0100187
Pierre Habouzitd3240d92008-10-31 10:12:21 +0100188--prune-empty::
Devin J. Pohlya582a822017-02-23 02:27:35 -0600189 Some filters will generate empty commits that leave the tree untouched.
190 This option instructs git-filter-branch to remove such commits if they
191 have exactly one or zero non-pruned parents; merge commits will
192 therefore remain intact. This option cannot be used together with
193 `--commit-filter`, though the same effect can be achieved by using the
194 provided `git_commit_non_empty_tree` function in a commit filter.
Pierre Habouzitd3240d92008-10-31 10:12:21 +0100195
Giuseppe Bilotta54332352007-08-30 19:10:42 +0200196--original <namespace>::
197 Use this option to set the namespace where the original commits
198 will be stored. The default value is 'refs/original'.
199
Johannes Schindelinc401b332007-07-04 00:41:55 +0100200-d <directory>::
201 Use this option to set the path to the temporary directory used for
202 rewriting. When applying a tree filter, the command needs to
Ralf Wildenhuesbf7c9022008-03-20 22:30:32 +0100203 temporarily check out the tree to some directory, which may consume
Johannes Schindelinc401b332007-07-04 00:41:55 +0100204 considerable space in case of large projects. By default it
Corentin BOMPARD68ed71b2019-03-06 14:04:46 +0100205 does this in the `.git-rewrite/` directory but you can override
Johannes Schindelinc401b332007-07-04 00:41:55 +0100206 that choice by this parameter.
207
Stephan Beyer32402402008-06-08 03:36:09 +0200208-f::
209--force::
Thomas Rast0b444cd2010-01-10 00:33:00 +0100210 'git filter-branch' refuses to start with an existing temporary
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100211 directory or when there are already refs starting with
212 'refs/original/', unless forced.
213
Ian Campbellbd2c79f2017-09-21 08:49:31 +0100214--state-branch <branch>::
215 This option will cause the mapping from old to new objects to
216 be loaded from named branch upon startup and saved as a new
217 commit to that branch upon exit, enabling incremental of large
218 trees. If '<branch>' does not exist it will be created.
219
Abhijit Menon-Senf448e242008-07-30 15:03:43 +0530220<rev-list options>...::
Thomas Rast0b444cd2010-01-10 00:33:00 +0100221 Arguments for 'git rev-list'. All positive refs included by
Thomas Rast8afa4212008-08-07 16:16:03 +0200222 these options are rewritten. You may also specify options
Matthieu Moy04b125d2016-06-28 13:40:12 +0200223 such as `--all`, but you must use `--` to separate them from
Csaba Henk7ec344d2010-08-27 20:44:56 +0000224 the 'git filter-branch' options. Implies <<Remap_to_ancestor>>.
225
226
227[[Remap_to_ancestor]]
228Remap to ancestor
229~~~~~~~~~~~~~~~~~
230
Junio C Hamano1cca17d2016-05-04 10:36:24 -0700231By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the
Csaba Henk7ec344d2010-08-27 20:44:56 +0000232set of revisions which get rewritten. However, positive refs on the command
233line are distinguished: we don't let them be excluded by such limiters. For
234this purpose, they are instead rewritten to point at the nearest ancestor that
235was not excluded.
Johannes Schindelinc401b332007-07-04 00:41:55 +0100236
237
Michele Locati0a0eb2e2018-03-15 18:09:18 +0100238EXIT STATUS
239-----------
240
241On success, the exit status is `0`. If the filter can't find any commits to
242rewrite, the exit status is `2`. On any other error, the exit status may be
243any other non-zero value.
244
245
Nguyễn Thái Ngọc Duy76a87882018-04-30 17:35:33 +0200246EXAMPLES
Johannes Schindelinc401b332007-07-04 00:41:55 +0100247--------
248
249Suppose you want to remove a file (containing confidential information
250or copyright violation) from all commits:
251
252-------------------------------------------------------
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100253git filter-branch --tree-filter 'rm filename' HEAD
Johannes Schindelinc401b332007-07-04 00:41:55 +0100254-------------------------------------------------------
255
Jon Loeligere4d594c2008-05-16 14:43:50 -0500256However, if the file is absent from the tree of some commit,
257a simple `rm filename` will fail for that tree and commit.
258Thus you may instead want to use `rm -f filename` as the script.
259
Jeff King6cf378f2012-04-26 04:51:57 -0400260Using `--index-filter` with 'git rm' yields a significantly faster
Thomas Rast3bc427e2009-03-12 00:00:56 +0100261version. Like with using `rm filename`, `git rm --cached filename`
262will fail if the file is absent from the tree of a commit. If you
263want to "completely forget" a file, it does not matter when it entered
Jeff King6cf378f2012-04-26 04:51:57 -0400264history, so we also add `--ignore-unmatch`:
Johannes Schindelinc401b332007-07-04 00:41:55 +0100265
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100266--------------------------------------------------------------------------
Thomas Rast3bc427e2009-03-12 00:00:56 +0100267git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100268--------------------------------------------------------------------------
Johannes Schindelinc401b332007-07-04 00:41:55 +0100269
Johannes Schindelin8ef44512007-10-17 03:22:25 +0100270Now, you will get the rewritten history saved in HEAD.
Johannes Schindelinc401b332007-07-04 00:41:55 +0100271
Thomas Rast8afa4212008-08-07 16:16:03 +0200272To rewrite the repository to look as if `foodir/` had been its project
273root, and discard all other history:
274
275-------------------------------------------------------
276git filter-branch --subdirectory-filter foodir -- --all
277-------------------------------------------------------
278
279Thus you can, e.g., turn a library subdirectory into a repository of
Jeff King6cf378f2012-04-26 04:51:57 -0400280its own. Note the `--` that separates 'filter-branch' options from
281revision options, and the `--all` to rewrite all branches and tags.
Thomas Rast8afa4212008-08-07 16:16:03 +0200282
Johannes Sixt32c37c12007-07-04 09:32:47 +0200283To set a commit (which typically is at the tip of another
284history) to be the parent of the current initial commit, in
285order to paste the other history behind the current history:
Johannes Schindelinc401b332007-07-04 00:41:55 +0100286
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100287-------------------------------------------------------------------
288git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD
289-------------------------------------------------------------------
Johannes Schindelinc401b332007-07-04 00:41:55 +0100290
Johannes Schindelin08203662007-08-31 17:42:33 +0100291(if the parent string is empty - which happens when we are dealing with
292the initial commit - add graftcommit as a parent). Note that this assumes
Johannes Schindelinc401b332007-07-04 00:41:55 +0100293history with a single root (that is, no merge without common ancestors
294happened). If this is not the case, use:
295
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100296--------------------------------------------------------------------------
Johannes Schindelinc401b332007-07-04 00:41:55 +0100297git filter-branch --parent-filter \
Caio Marcelo de Oliveira Filho41e86a32008-02-25 23:14:31 -0300298 'test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>" || cat' HEAD
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100299--------------------------------------------------------------------------
Johannes Schindelinc401b332007-07-04 00:41:55 +0100300
Johannes Sixt32c37c12007-07-04 09:32:47 +0200301or even simpler:
302
303-----------------------------------------------
Johannes Schindeline2d65c12018-04-29 00:44:53 +0200304git replace --graft $commit-id $graft-id
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100305git filter-branch $graft-id..HEAD
Johannes Sixt32c37c12007-07-04 09:32:47 +0200306-----------------------------------------------
307
Johannes Schindelinc401b332007-07-04 00:41:55 +0100308To remove commits authored by "Darl McBribe" from the history:
309
310------------------------------------------------------------------------------
311git filter-branch --commit-filter '
312 if [ "$GIT_AUTHOR_NAME" = "Darl McBribe" ];
313 then
Johannes Schindelinf95eef12007-08-31 20:06:27 +0100314 skip_commit "$@";
Johannes Schindelinc401b332007-07-04 00:41:55 +0100315 else
316 git commit-tree "$@";
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100317 fi' HEAD
Johannes Schindelinc401b332007-07-04 00:41:55 +0100318------------------------------------------------------------------------------
319
Sergei Organov8451c562007-11-01 16:24:11 +0300320The function 'skip_commit' is defined as follows:
Johannes Schindelinf95eef12007-08-31 20:06:27 +0100321
322--------------------------
323skip_commit()
324{
325 shift;
326 while [ -n "$1" ];
327 do
328 shift;
329 map "$1";
330 shift;
331 done;
332}
333--------------------------
334
Johannes Schindelinc401b332007-07-04 00:41:55 +0100335The shift magic first throws away the tree id and then the -p
336parameters. Note that this handles merges properly! In case Darl
337committed a merge between P1 and P2, it will be propagated properly
338and all children of the merge will become merge commits with P1,P2
339as their parents instead of the merge commit.
340
Andreas Schwab8093ae82012-09-18 17:55:08 +0200341*NOTE* the changes introduced by the commits, and which are not reverted
342by subsequent commits, will still be in the rewritten branch. If you want
343to throw out _changes_ together with the commits, you should use the
344interactive mode of 'git rebase'.
345
Florian Ragwitza1748892008-04-30 09:47:43 +0200346You can rewrite the commit log messages using `--msg-filter`. For
Thomas Rast0b444cd2010-01-10 00:33:00 +0100347example, 'git svn-id' strings in a repository created by 'git svn' can
Miklos Vajnaed10d9a2008-02-25 15:43:53 +0100348be removed this way:
349
350-------------------------------------------------------
Florian Ragwitza1748892008-04-30 09:47:43 +0200351git filter-branch --msg-filter '
Miklos Vajnaed10d9a2008-02-25 15:43:53 +0100352 sed -e "/^git-svn-id:/d"
353'
354-------------------------------------------------------
Johannes Schindelinf95eef12007-08-31 20:06:27 +0100355
Johannes Schindelinb8f42332009-08-17 22:38:46 +0200356If you need to add 'Acked-by' lines to, say, the last 10 commits (none
357of which is a merge), use this command:
358
359--------------------------------------------------------
360git filter-branch --msg-filter '
361 cat &&
362 echo "Acked-by: Bugs Bunny <bunny@bugzilla.org>"
363' HEAD~10..HEAD
364--------------------------------------------------------
365
Tadeusz Andrzej Kadłubowski21b6e4f2013-02-21 21:23:38 +0100366The `--env-filter` option can be used to modify committer and/or author
367identity. For example, if you found out that your commits have the wrong
368identity due to a misconfigured user.email, you can make a correction,
369before publishing the project, like this:
370
371--------------------------------------------------------
372git filter-branch --env-filter '
373 if test "$GIT_AUTHOR_EMAIL" = "root@localhost"
374 then
375 GIT_AUTHOR_EMAIL=john@example.com
Tadeusz Andrzej Kadłubowski21b6e4f2013-02-21 21:23:38 +0100376 fi
377 if test "$GIT_COMMITTER_EMAIL" = "root@localhost"
378 then
379 GIT_COMMITTER_EMAIL=john@example.com
Tadeusz Andrzej Kadłubowski21b6e4f2013-02-21 21:23:38 +0100380 fi
381' -- --all
382--------------------------------------------------------
383
Andreas Schwab8093ae82012-09-18 17:55:08 +0200384To restrict rewriting to only part of the history, specify a revision
385range in addition to the new branch name. The new branch name will
386point to the top-most revision that a 'git rev-list' of this range
387will print.
Johannes Schindelin08203662007-08-31 17:42:33 +0100388
Johannes Schindelinc401b332007-07-04 00:41:55 +0100389Consider this history:
390
391------------------
392 D--E--F--G--H
393 / /
394A--B-----C
395------------------
396
397To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use:
398
399--------------------------------
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100400git filter-branch ... C..H
Johannes Schindelinc401b332007-07-04 00:41:55 +0100401--------------------------------
402
403To rewrite commits E,F,G,H, use one of these:
404
405----------------------------------------
Johannes Schindelindfd05e32007-07-23 18:34:13 +0100406git filter-branch ... C..H --not D
407git filter-branch ... D..H --not C
Johannes Schindelinc401b332007-07-04 00:41:55 +0100408----------------------------------------
409
410To move the whole tree into a subdirectory, or remove it from there:
411
412---------------------------------------------------------------
413git filter-branch --index-filter \
Jeff Kingd2d66f12010-02-01 07:43:45 -0500414 'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
Johannes Schindelinc401b332007-07-04 00:41:55 +0100415 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
416 git update-index --index-info &&
Jeff King6cb01862011-04-01 10:46:27 -0400417 mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD
Johannes Schindelinc401b332007-07-04 00:41:55 +0100418---------------------------------------------------------------
419
420
Thomas Rastd0268de2009-02-14 21:56:51 +0100421
Nguyễn Thái Ngọc Duy76a87882018-04-30 17:35:33 +0200422CHECKLIST FOR SHRINKING A REPOSITORY
Thomas Rastd0268de2009-02-14 21:56:51 +0100423------------------------------------
424
Roberto Tyley615b8f12013-12-18 14:25:16 +0000425git-filter-branch can be used to get rid of a subset of files,
Jeff King6cf378f2012-04-26 04:51:57 -0400426usually with some combination of `--index-filter` and
427`--subdirectory-filter`. People expect the resulting repository to
Thomas Rastd0268de2009-02-14 21:56:51 +0100428be smaller than the original, but you need a few more steps to
Thomas Ackermann2de9b712013-01-21 20:17:53 +0100429actually make it smaller, because Git tries hard not to lose your
Thomas Rastd0268de2009-02-14 21:56:51 +0100430objects until you tell it to. First make sure that:
431
432* You really removed all variants of a filename, if a blob was moved
Jeff King6cf378f2012-04-26 04:51:57 -0400433 over its lifetime. `git log --name-only --follow --all -- filename`
434 can help you find renames.
Thomas Rastd0268de2009-02-14 21:56:51 +0100435
Jeff King6cf378f2012-04-26 04:51:57 -0400436* You really filtered all refs: use `--tag-name-filter cat -- --all`
437 when calling git-filter-branch.
Thomas Rastd0268de2009-02-14 21:56:51 +0100438
439Then there are two ways to get a smaller repository. A safer way is
440to clone, that keeps your original intact.
441
Jeff King6cf378f2012-04-26 04:51:57 -0400442* Clone it with `git clone file:///path/to/repo`. The clone
Thomas Rastd0268de2009-02-14 21:56:51 +0100443 will not have the removed objects. See linkgit:git-clone[1]. (Note
444 that cloning with a plain path just hardlinks everything!)
445
446If you really don't want to clone it, for whatever reasons, check the
447following points instead (in this order). This is a very destructive
448approach, so *make a backup* or go back to cloning it. You have been
449warned.
450
451* Remove the original refs backed up by git-filter-branch: say `git
Jeff King6cf378f2012-04-26 04:51:57 -0400452 for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git
Thomas Rastd0268de2009-02-14 21:56:51 +0100453 update-ref -d`.
454
Jeff King6cf378f2012-04-26 04:51:57 -0400455* Expire all reflogs with `git reflog expire --expire=now --all`.
Thomas Rastd0268de2009-02-14 21:56:51 +0100456
Jeff King6cf378f2012-04-26 04:51:57 -0400457* Garbage collect all unreferenced objects with `git gc --prune=now`
Thomas Rastd0268de2009-02-14 21:56:51 +0100458 (or if your git-gc is not new enough to support arguments to
Jeff King6cf378f2012-04-26 04:51:57 -0400459 `--prune`, use `git repack -ad; git prune` instead).
Thomas Rastd0268de2009-02-14 21:56:51 +0100460
Elijah Newren9df53c52019-09-04 15:32:38 -0700461[[PERFORMANCE]]
462PERFORMANCE
463-----------
Roberto Tyley615b8f12013-12-18 14:25:16 +0000464
Elijah Newren9df53c52019-09-04 15:32:38 -0700465The performance of git-filter-branch is glacially slow; its design makes it
466impossible for a backward-compatible implementation to ever be fast:
Roberto Tyley615b8f12013-12-18 14:25:16 +0000467
Elijah Newren9df53c52019-09-04 15:32:38 -0700468* In editing files, git-filter-branch by design checks out each and
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100469 every commit as it existed in the original repo. If your repo has
Martin Ågrene469afe2020-02-03 21:36:50 +0100470 `10^5` files and `10^5` commits, but each commit only modifies five
471 files, then git-filter-branch will make you do `10^10` modifications,
472 despite only having (at most) `5*10^5` unique blobs.
Roberto Tyley615b8f12013-12-18 14:25:16 +0000473
Elijah Newren9df53c52019-09-04 15:32:38 -0700474* If you try and cheat and try to make git-filter-branch only work on
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100475 files modified in a commit, then two things happen
Roberto Tyley615b8f12013-12-18 14:25:16 +0000476
Elijah Newren9df53c52019-09-04 15:32:38 -0700477 ** you run into problems with deletions whenever the user is simply
478 trying to rename files (because attempting to delete files that
479 don't exist looks like a no-op; it takes some chicanery to remap
480 deletes across file renames when the renames happen via arbitrary
481 user-provided shell)
482
483 ** even if you succeed at the map-deletes-for-renames chicanery, you
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100484 still technically violate backward compatibility because users
485 are allowed to filter files in ways that depend upon topology of
486 commits instead of filtering solely based on file contents or
487 names (though this has not been observed in the wild).
Elijah Newren9df53c52019-09-04 15:32:38 -0700488
489* Even if you don't need to edit files but only want to e.g. rename or
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100490 remove some and thus can avoid checking out each file (i.e. you can
491 use --index-filter), you still are passing shell snippets for your
492 filters. This means that for every commit, you have to have a
493 prepared git repo where those filters can be run. That's a
494 significant setup.
Elijah Newren9df53c52019-09-04 15:32:38 -0700495
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100496* Further, several additional files are created or updated per commit
497 by git-filter-branch. Some of these are for supporting the
498 convenience functions provided by git-filter-branch (such as map()),
499 while others are for keeping track of internal state (but could have
500 also been accessed by user filters; one of git-filter-branch's
501 regression tests does so). This essentially amounts to using the
502 filesystem as an IPC mechanism between git-filter-branch and the
503 user-provided filters. Disks tend to be a slow IPC mechanism, and
504 writing these files also effectively represents a forced
505 synchronization point between separate processes that we hit with
506 every commit.
Elijah Newren9df53c52019-09-04 15:32:38 -0700507
508* The user-provided shell commands will likely involve a pipeline of
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100509 commands, resulting in the creation of many processes per commit.
510 Creating and running another process takes a widely varying amount
511 of time between operating systems, but on any platform it is very
512 slow relative to invoking a function.
Elijah Newren9df53c52019-09-04 15:32:38 -0700513
514* git-filter-branch itself is written in shell, which is kind of slow.
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100515 This is the one performance issue that could be backward-compatibly
516 fixed, but compared to the above problems that are intrinsic to the
517 design of git-filter-branch, the language of the tool itself is a
518 relatively minor issue.
Elijah Newren9df53c52019-09-04 15:32:38 -0700519
520 ** Side note: Unfortunately, people tend to fixate on the
521 written-in-shell aspect and periodically ask if git-filter-branch
522 could be rewritten in another language to fix the performance
523 issues. Not only does that ignore the bigger intrinsic problems
524 with the design, it'd help less than you'd expect: if
525 git-filter-branch itself were not shell, then the convenience
526 functions (map(), skip_commit(), etc) and the `--setup` argument
527 could no longer be executed once at the beginning of the program
528 but would instead need to be prepended to every user filter (and
529 thus re-executed with every commit).
530
531The https://github.com/newren/git-filter-repo/[git filter-repo] tool is
532an alternative to git-filter-branch which does not suffer from these
533performance problems or the safety problems (mentioned below). For those
534with existing tooling which relies upon git-filter-branch, 'git
535repo-filter' also provides
536https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],
537a drop-in git-filter-branch replacement (with a few caveats). While
538filter-lamely suffers from all the same safety issues as
Elijah Newren031fd4b2019-11-05 17:07:20 +0000539git-filter-branch, it at least ameliorates the performance issues a
Elijah Newren9df53c52019-09-04 15:32:38 -0700540little.
541
542[[SAFETY]]
543SAFETY
544------
545
546git-filter-branch is riddled with gotchas resulting in various ways to
547easily corrupt repos or end up with a mess worse than what you started
548with:
549
550* Someone can have a set of "working and tested filters" which they
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100551 document or provide to a coworker, who then runs them on a different
552 OS where the same commands are not working/tested (some examples in
553 the git-filter-branch manpage are also affected by this).
554 BSD vs. GNU userland differences can really bite. If lucky, error
555 messages are spewed. But just as likely, the commands either don't
556 do the filtering requested, or silently corrupt by making some
557 unwanted change. The unwanted change may only affect a few commits,
558 so it's not necessarily obvious either. (The fact that problems
559 won't necessarily be obvious means they are likely to go unnoticed
560 until the rewritten history is in use for quite a while, at which
561 point it's really hard to justify another flag-day for another
562 rewrite.)
Elijah Newren9df53c52019-09-04 15:32:38 -0700563
564* Filenames with spaces are often mishandled by shell snippets since
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100565 they cause problems for shell pipelines. Not everyone is familiar
566 with find -print0, xargs -0, git-ls-files -z, etc. Even people who
567 are familiar with these may assume such flags are not relevant
568 because someone else renamed any such files in their repo back
569 before the person doing the filtering joined the project. And
570 often, even those familiar with handling arguments with spaces may
571 not do so just because they aren't in the mindset of thinking about
572 everything that could possibly go wrong.
Elijah Newren9df53c52019-09-04 15:32:38 -0700573
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100574* Non-ascii filenames can be silently removed despite being in a
575 desired directory. Keeping only wanted paths is often done using
576 pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.
577 ls-files will only quote filenames if needed, so folks may not
578 notice that one of the files didn't match the regex (at least not
579 until it's much too late). Yes, someone who knows about
580 core.quotePath can avoid this (unless they have other special
581 characters like \t, \n, or "), and people who use ls-files -z with
582 something other than grep can avoid this, but that doesn't mean they
583 will.
Elijah Newren9df53c52019-09-04 15:32:38 -0700584
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100585* Similarly, when moving files around, one can find that filenames
586 with non-ascii or special characters end up in a different
587 directory, one that includes a double quote character. (This is
588 technically the same issue as above with quoting, but perhaps an
589 interesting different way that it can and has manifested as a
590 problem.)
Elijah Newren9df53c52019-09-04 15:32:38 -0700591
592* It's far too easy to accidentally mix up old and new history. It's
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100593 still possible with any tool, but git-filter-branch almost
594 invites it. If lucky, the only downside is users getting frustrated
595 that they don't know how to shrink their repo and remove the old
596 stuff. If unlucky, they merge old and new history and end up with
597 multiple "copies" of each commit, some of which have unwanted or
598 sensitive files and others which don't. This comes about in
599 multiple different ways:
Elijah Newren9df53c52019-09-04 15:32:38 -0700600
601 ** the default to only doing a partial history rewrite ('--all' is not
602 the default and few examples show it)
603
604 ** the fact that there's no automatic post-run cleanup
605
606 ** the fact that --tag-name-filter (when used to rename tags) doesn't
607 remove the old tags but just adds new ones with the new name
608
609 ** the fact that little educational information is provided to inform
610 users of the ramifications of a rewrite and how to avoid mixing old
611 and new history. For example, this man page discusses how users
612 need to understand that they need to rebase their changes for all
613 their branches on top of new history (or delete and reclone), but
614 that's only one of multiple concerns to consider. See the
615 "DISCUSSION" section of the git filter-repo manual page for more
616 details.
617
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100618* Annotated tags can be accidentally converted to lightweight tags,
619 due to either of two issues:
Elijah Newren9df53c52019-09-04 15:32:38 -0700620
621 ** Someone can do a history rewrite, realize they messed up, restore
622 from the backups in refs/original/, and then redo their
623 git-filter-branch command. (The backup in refs/original/ is not a
624 real backup; it dereferences tags first.)
625
626 ** Running git-filter-branch with either --tags or --all in your
627 <rev-list options>. In order to retain annotated tags as
628 annotated, you must use --tag-name-filter (and must not have
629 restored from refs/original/ in a previously botched rewrite).
630
631* Any commit messages that specify an encoding will become corrupted
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100632 by the rewrite; git-filter-branch ignores the encoding, takes the
633 original bytes, and feeds it to commit-tree without telling it the
634 proper encoding. (This happens whether or not --msg-filter is
635 used.)
Elijah Newren9df53c52019-09-04 15:32:38 -0700636
637* Commit messages (even if they are all UTF-8) by default become
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100638 corrupted due to not being updated -- any references to other commit
639 hashes in commit messages will now refer to no-longer-extant
640 commits.
Elijah Newren9df53c52019-09-04 15:32:38 -0700641
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100642* There are no facilities for helping users find what unwanted crud
643 they should delete, which means they are much more likely to have
644 incomplete or partial cleanups that sometimes result in confusion
645 and people wasting time trying to understand. (For example, folks
646 tend to just look for big files to delete instead of big directories
647 or extensions, and once they do so, then sometime later folks using
648 the new repository who are going through history will notice a build
649 artifact directory that has some files but not others, or a cache of
650 dependencies (node_modules or similar) which couldn't have ever been
651 functional since it's missing some files.)
Elijah Newren9df53c52019-09-04 15:32:38 -0700652
653* If --prune-empty isn't specified, then the filtering process can
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100654 create hoards of confusing empty commits
Elijah Newren9df53c52019-09-04 15:32:38 -0700655
656* If --prune-empty is specified, then intentionally placed empty
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100657 commits from before the filtering operation are also pruned instead
658 of just pruning commits that became empty due to filtering rules.
Elijah Newren9df53c52019-09-04 15:32:38 -0700659
Elijah Newrena6d39f22019-11-05 23:31:30 +0000660* If --prune-empty is specified, sometimes empty commits are missed
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100661 and left around anyway (a somewhat rare bug, but it happens...)
Elijah Newren9df53c52019-09-04 15:32:38 -0700662
663* A minor issue, but users who have a goal to update all names and
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100664 emails in a repository may be led to --env-filter which will only
665 update authors and committers, missing taggers.
Elijah Newren9df53c52019-09-04 15:32:38 -0700666
667* If the user provides a --tag-name-filter that maps multiple tags to
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100668 the same name, no warning or error is provided; git-filter-branch
669 simply overwrites each tag in some undocumented pre-defined order
670 resulting in only one tag at the end. (A git-filter-branch
671 regression test requires this surprising behavior.)
Elijah Newren9df53c52019-09-04 15:32:38 -0700672
673Also, the poor performance of git-filter-branch often leads to safety
674issues:
675
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100676* Coming up with the correct shell snippet to do the filtering you
677 want is sometimes difficult unless you're just doing a trivial
678 modification such as deleting a couple files. Unfortunately, people
679 often learn if the snippet is right or wrong by trying it out, but
680 the rightness or wrongness can vary depending on special
681 circumstances (spaces in filenames, non-ascii filenames, funny
682 author names or emails, invalid timezones, presence of grafts or
683 replace objects, etc.), meaning they may have to wait a long time,
684 hit an error, then restart. The performance of git-filter-branch is
685 so bad that this cycle is painful, reducing the time available to
686 carefully re-check (to say nothing about what it does to the
687 patience of the person doing the rewrite even if they do technically
688 have more time available). This problem is extra compounded because
689 errors from broken filters may not be shown for a long time and/or
690 get lost in a sea of output. Even worse, broken filters often just
691 result in silent incorrect rewrites.
Elijah Newren9df53c52019-09-04 15:32:38 -0700692
Jean-Noël Aviladf5be012019-12-12 21:46:55 +0100693* To top it all off, even when users finally find working commands,
694 they naturally want to share them. But they may be unaware that
695 their repo didn't have some special cases that someone else's does.
696 So, when someone else with a different repository runs the same
697 commands, they get hit by the problems above. Or, the user just
698 runs commands that really were vetted for special cases, but they
699 run it on a different OS where it doesn't work, as noted above.
Roberto Tyley615b8f12013-12-18 14:25:16 +0000700
Johannes Schindelinc401b332007-07-04 00:41:55 +0100701GIT
702---
Christian Couder9e1f0a82008-06-06 09:07:32 +0200703Part of the linkgit:git[1] suite