Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 1 | git-filter-branch(1) |
| 2 | ==================== |
| 3 | |
| 4 | NAME |
| 5 | ---- |
| 6 | git-filter-branch - Rewrite branches |
| 7 | |
| 8 | SYNOPSIS |
| 9 | -------- |
| 10 | [verse] |
David Glasser | 07c4984 | 2017-10-17 09:45:15 +0000 | [diff] [blame] | 11 | 'git filter-branch' [--setup <command>] [--subdirectory-filter <directory>] |
| 12 | [--env-filter <command>] [--tree-filter <command>] |
| 13 | [--index-filter <command>] [--parent-filter <command>] |
| 14 | [--msg-filter <command>] [--commit-filter <command>] |
| 15 | [--tag-name-filter <command>] [--prune-empty] |
Giuseppe Bilotta | 5433235 | 2007-08-30 19:10:42 +0200 | [diff] [blame] | 16 | [--original <namespace>] [-d <directory>] [-f | --force] |
Ian Campbell | bd2c79f | 2017-09-21 08:49:31 +0100 | [diff] [blame] | 17 | [--state-branch <branch>] [--] [<rev-list options>...] |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 18 | |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 19 | WARNING |
| 20 | ------- |
| 21 | 'git filter-branch' has a plethora of pitfalls that can produce non-obvious |
| 22 | manglings of the intended history rewrite (and can leave you with little |
| 23 | time to investigate such problems since it has such abysmal performance). |
| 24 | These safety and performance issues cannot be backward compatibly fixed and |
| 25 | as such, its use is not recommended. Please use an alternative history |
| 26 | filtering tool such as https://github.com/newren/git-filter-repo/[git |
| 27 | filter-repo]. If you still need to use 'git filter-branch', please |
| 28 | carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land |
| 29 | mines of filter-branch, and then vigilantly avoid as many of the hazards |
| 30 | listed there as reasonably possible. |
| 31 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 32 | DESCRIPTION |
| 33 | ----------- |
Thomas Ackermann | 2de9b71 | 2013-01-21 20:17:53 +0100 | [diff] [blame] | 34 | Lets you rewrite Git revision history by rewriting the branches mentioned |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 35 | in the <rev-list options>, applying custom filters on each revision. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 36 | Those filters can modify each tree (e.g. removing a file or running |
| 37 | a perl rewrite on all files) or information about each commit. |
| 38 | Otherwise, all information (including original commit times or merge |
| 39 | information) will be preserved. |
| 40 | |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 41 | The command will only rewrite the _positive_ refs mentioned in the |
Ralf Wildenhues | bf7c902 | 2008-03-20 22:30:32 +0100 | [diff] [blame] | 42 | command line (e.g. if you pass 'a..b', only 'b' will be rewritten). |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 43 | If you specify no filters, the commits will be recommitted without any |
| 44 | changes, which would normally have no effect. Nevertheless, this may be |
Thomas Ackermann | 2de9b71 | 2013-01-21 20:17:53 +0100 | [diff] [blame] | 45 | useful in the future for compensating for some Git bugs or such, |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 46 | therefore such a usage is permitted. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 47 | |
Junio C Hamano | 831e61f | 2012-08-06 13:36:47 -0700 | [diff] [blame] | 48 | *NOTE*: This command honors `.git/info/grafts` file and refs in |
| 49 | the `refs/replace/` namespace. |
Peter Collingbourne | 0dc310e | 2011-07-21 16:10:52 +0100 | [diff] [blame] | 50 | If you have any grafts or replacement refs defined, running this command |
| 51 | will make them permanent. |
Daniel Cheng (aka SDiZ) | c6d8f76 | 2009-04-10 14:26:49 +0800 | [diff] [blame] | 52 | |
Johannes Schindelin | 73616fd | 2007-07-04 15:50:45 +0100 | [diff] [blame] | 53 | *WARNING*! The rewritten history will have different object names for all |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 54 | the objects and will not converge with the original branch. You will not |
| 55 | be able to easily push and distribute the rewritten branch on top of the |
| 56 | original branch. Please do not use this command if you do not know the |
| 57 | full implications, and avoid using it anyway, if a simple single commit |
Thomas Rast | 97c33c6 | 2008-09-13 18:11:01 +0200 | [diff] [blame] | 58 | would suffice to fix your problem. (See the "RECOVERING FROM UPSTREAM |
| 59 | REBASE" section in linkgit:git-rebase[1] for further information about |
| 60 | rewriting published history.) |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 61 | |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 62 | Always verify that the rewritten version is correct: The original refs, |
| 63 | if different from the rewritten ones, will be stored in the namespace |
| 64 | 'refs/original/'. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 65 | |
Ralf Wildenhues | bf7c902 | 2008-03-20 22:30:32 +0100 | [diff] [blame] | 66 | Note that since this operation is very I/O expensive, it might |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 67 | be a good idea to redirect the temporary directory off-disk with the |
Matthieu Moy | 23f8239 | 2016-06-28 13:40:10 +0200 | [diff] [blame] | 68 | `-d` option, e.g. on tmpfs. Reportedly the speedup is very noticeable. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 69 | |
| 70 | |
| 71 | Filters |
| 72 | ~~~~~~~ |
| 73 | |
| 74 | The filters are applied in the order as listed below. The <command> |
Ralf Wildenhues | bf7c902 | 2008-03-20 22:30:32 +0100 | [diff] [blame] | 75 | argument is always evaluated in the shell context using the 'eval' command |
| 76 | (with the notable exception of the commit filter, for technical reasons). |
Tom Russello | 47d81b5 | 2016-06-08 00:35:07 +0200 | [diff] [blame] | 77 | Prior to that, the `$GIT_COMMIT` environment variable will be set to contain |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 78 | the id of the commit being rewritten. Also, GIT_AUTHOR_NAME, |
| 79 | GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, |
Tadeusz Andrzej Kadłubowski | bee3eb0 | 2013-02-21 21:22:50 +0100 | [diff] [blame] | 80 | and GIT_COMMITTER_DATE are taken from the current commit and exported to |
| 81 | the environment, in order to affect the author and committer identities of |
| 82 | the replacement commit created by linkgit:git-commit-tree[1] after the |
| 83 | filters have run. |
| 84 | |
Ralf Wildenhues | bf7c902 | 2008-03-20 22:30:32 +0100 | [diff] [blame] | 85 | If any evaluation of <command> returns a non-zero exit status, the whole |
| 86 | operation will be aborted. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 87 | |
| 88 | A 'map' function is available that takes an "original sha1 id" argument |
| 89 | and outputs a "rewritten sha1 id" if the commit has been already |
Johannes Sixt | 32c37c1 | 2007-07-04 09:32:47 +0200 | [diff] [blame] | 90 | rewritten, and "original sha1 id" otherwise; the 'map' function can |
| 91 | return several ids on separate lines if your commit filter emitted |
| 92 | multiple commits. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 93 | |
| 94 | |
| 95 | OPTIONS |
| 96 | ------- |
| 97 | |
Andreas Heiduk | 3b117f7 | 2017-06-10 10:54:44 +0200 | [diff] [blame] | 98 | --setup <command>:: |
| 99 | This is not a real filter executed for each commit but a one |
| 100 | time setup just before the loop. Therefore no commit-specific |
| 101 | variables are defined yet. Functions or variables defined here |
| 102 | can be used or modified in the following filter steps except |
| 103 | the commit filter, for technical reasons. |
| 104 | |
David Glasser | 07c4984 | 2017-10-17 09:45:15 +0000 | [diff] [blame] | 105 | --subdirectory-filter <directory>:: |
| 106 | Only look at the history which touches the given subdirectory. |
| 107 | The result will contain that directory (and only that) as its |
| 108 | project root. Implies <<Remap_to_ancestor>>. |
| 109 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 110 | --env-filter <command>:: |
Ralf Wildenhues | bf7c902 | 2008-03-20 22:30:32 +0100 | [diff] [blame] | 111 | This filter may be used if you only need to modify the environment |
| 112 | in which the commit will be performed. Specifically, you might |
| 113 | want to rewrite the author/committer name/email/time environment |
Andreas Heiduk | ba746ff | 2017-05-26 19:36:54 +0200 | [diff] [blame] | 114 | variables (see linkgit:git-commit-tree[1] for details). |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 115 | |
| 116 | --tree-filter <command>:: |
| 117 | This is the filter for rewriting the tree and its contents. |
| 118 | The argument is evaluated in shell with the working |
| 119 | directory set to the root of the checked out tree. The new tree |
| 120 | is then used as-is (new files are auto-added, disappeared files |
| 121 | are auto-removed - neither .gitignore files nor any other ignore |
Johannes Schindelin | 73616fd | 2007-07-04 15:50:45 +0100 | [diff] [blame] | 122 | rules *HAVE ANY EFFECT*!). |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 123 | |
| 124 | --index-filter <command>:: |
| 125 | This is the filter for rewriting the index. It is similar to the |
| 126 | tree filter but does not check out the tree, which makes it much |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 127 | faster. Frequently used with `git rm --cached |
| 128 | --ignore-unmatch ...`, see EXAMPLES below. For hairy |
Thomas Rast | 3bc427e | 2009-03-12 00:00:56 +0100 | [diff] [blame] | 129 | cases, see linkgit:git-update-index[1]. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 130 | |
| 131 | --parent-filter <command>:: |
| 132 | This is the filter for rewriting the commit's parent list. |
| 133 | It will receive the parent string on stdin and shall output |
| 134 | the new parent string on stdout. The parent string is in |
Jonathan Nieder | 483bc4f | 2008-06-30 13:56:34 -0500 | [diff] [blame] | 135 | the format described in linkgit:git-commit-tree[1]: empty for |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 136 | the initial commit, "-p parent" for a normal commit and |
| 137 | "-p parent1 -p parent2 -p parent3 ..." for a merge commit. |
| 138 | |
| 139 | --msg-filter <command>:: |
| 140 | This is the filter for rewriting the commit messages. |
| 141 | The argument is evaluated in the shell with the original |
| 142 | commit message on standard input; its standard output is |
| 143 | used as the new commit message. |
| 144 | |
| 145 | --commit-filter <command>:: |
| 146 | This is the filter for performing the commit. |
| 147 | If this filter is specified, it will be called instead of the |
Thomas Rast | 0b444cd | 2010-01-10 00:33:00 +0100 | [diff] [blame] | 148 | 'git commit-tree' command, with arguments of the form |
Štěpán Němec | 0adda93 | 2010-10-08 19:31:17 +0200 | [diff] [blame] | 149 | "<TREE_ID> [(-p <PARENT_COMMIT_ID>)...]" and the log message on |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 150 | stdin. The commit id is expected on stdout. |
| 151 | + |
| 152 | As a special extension, the commit filter may emit multiple |
Kevin Ballard | c5833f6 | 2008-05-30 14:43:40 -0700 | [diff] [blame] | 153 | commit ids; in that case, the rewritten children of the original commit will |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 154 | have all of them as parents. |
Johannes Schindelin | f95eef1 | 2007-08-31 20:06:27 +0100 | [diff] [blame] | 155 | + |
| 156 | You can use the 'map' convenience function in this filter, and other |
| 157 | convenience functions, too. For example, calling 'skip_commit "$@"' |
| 158 | will leave out the current commit (but not its changes! If you want |
Thomas Rast | 0b444cd | 2010-01-10 00:33:00 +0100 | [diff] [blame] | 159 | that, use 'git rebase' instead). |
Pierre Habouzit | d3240d9 | 2008-10-31 10:12:21 +0100 | [diff] [blame] | 160 | + |
Thomas Rast | ca76828 | 2010-01-07 17:49:12 +0100 | [diff] [blame] | 161 | You can also use the `git_commit_non_empty_tree "$@"` instead of |
| 162 | `git commit-tree "$@"` if you don't wish to keep commits with a single parent |
Pierre Habouzit | d3240d9 | 2008-10-31 10:12:21 +0100 | [diff] [blame] | 163 | and that makes no change to the tree. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 164 | |
| 165 | --tag-name-filter <command>:: |
| 166 | This is the filter for rewriting tag names. When passed, |
| 167 | it will be called for every tag ref that points to a rewritten |
| 168 | object (or to a tag object which points to a rewritten object). |
| 169 | The original tag name is passed via standard input, and the new |
| 170 | tag name is expected on standard output. |
| 171 | + |
| 172 | The original tags are not deleted, but can be overwritten; |
Brian Gernhardt | 5876b8e | 2007-08-17 19:13:04 -0400 | [diff] [blame] | 173 | use "--tag-name-filter cat" to simply update the tags. In this |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 174 | case, be very careful and make sure you have the old tags |
| 175 | backed up in case the conversion has run afoul. |
| 176 | + |
Brandon Casey | 1bf6551 | 2008-03-26 10:47:09 -0500 | [diff] [blame] | 177 | Nearly proper rewriting of tag objects is supported. If the tag has |
| 178 | a message attached, a new tag object will be created with the same message, |
| 179 | author, and timestamp. If the tag has a signature attached, the |
| 180 | signature will be stripped. It is by definition impossible to preserve |
| 181 | signatures. The reason this is "nearly" proper, is because ideally if |
| 182 | the tag did not change (points to the same object, has the same name, etc.) |
| 183 | it should retain any signature. That is not the case, signatures will always |
| 184 | be removed, buyer beware. There is also no support for changing the |
| 185 | author or timestamp (or the tag message for that matter). Tags which point |
| 186 | to other tags will be rewritten to point to the underlying commit. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 187 | |
Pierre Habouzit | d3240d9 | 2008-10-31 10:12:21 +0100 | [diff] [blame] | 188 | --prune-empty:: |
Devin J. Pohly | a582a82 | 2017-02-23 02:27:35 -0600 | [diff] [blame] | 189 | Some filters will generate empty commits that leave the tree untouched. |
| 190 | This option instructs git-filter-branch to remove such commits if they |
| 191 | have exactly one or zero non-pruned parents; merge commits will |
| 192 | therefore remain intact. This option cannot be used together with |
| 193 | `--commit-filter`, though the same effect can be achieved by using the |
| 194 | provided `git_commit_non_empty_tree` function in a commit filter. |
Pierre Habouzit | d3240d9 | 2008-10-31 10:12:21 +0100 | [diff] [blame] | 195 | |
Giuseppe Bilotta | 5433235 | 2007-08-30 19:10:42 +0200 | [diff] [blame] | 196 | --original <namespace>:: |
| 197 | Use this option to set the namespace where the original commits |
| 198 | will be stored. The default value is 'refs/original'. |
| 199 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 200 | -d <directory>:: |
| 201 | Use this option to set the path to the temporary directory used for |
| 202 | rewriting. When applying a tree filter, the command needs to |
Ralf Wildenhues | bf7c902 | 2008-03-20 22:30:32 +0100 | [diff] [blame] | 203 | temporarily check out the tree to some directory, which may consume |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 204 | considerable space in case of large projects. By default it |
Corentin BOMPARD | 68ed71b | 2019-03-06 14:04:46 +0100 | [diff] [blame] | 205 | does this in the `.git-rewrite/` directory but you can override |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 206 | that choice by this parameter. |
| 207 | |
Stephan Beyer | 3240240 | 2008-06-08 03:36:09 +0200 | [diff] [blame] | 208 | -f:: |
| 209 | --force:: |
Thomas Rast | 0b444cd | 2010-01-10 00:33:00 +0100 | [diff] [blame] | 210 | 'git filter-branch' refuses to start with an existing temporary |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 211 | directory or when there are already refs starting with |
| 212 | 'refs/original/', unless forced. |
| 213 | |
Ian Campbell | bd2c79f | 2017-09-21 08:49:31 +0100 | [diff] [blame] | 214 | --state-branch <branch>:: |
| 215 | This option will cause the mapping from old to new objects to |
| 216 | be loaded from named branch upon startup and saved as a new |
| 217 | commit to that branch upon exit, enabling incremental of large |
| 218 | trees. If '<branch>' does not exist it will be created. |
| 219 | |
Abhijit Menon-Sen | f448e24 | 2008-07-30 15:03:43 +0530 | [diff] [blame] | 220 | <rev-list options>...:: |
Thomas Rast | 0b444cd | 2010-01-10 00:33:00 +0100 | [diff] [blame] | 221 | Arguments for 'git rev-list'. All positive refs included by |
Thomas Rast | 8afa421 | 2008-08-07 16:16:03 +0200 | [diff] [blame] | 222 | these options are rewritten. You may also specify options |
Matthieu Moy | 04b125d | 2016-06-28 13:40:12 +0200 | [diff] [blame] | 223 | such as `--all`, but you must use `--` to separate them from |
Csaba Henk | 7ec344d | 2010-08-27 20:44:56 +0000 | [diff] [blame] | 224 | the 'git filter-branch' options. Implies <<Remap_to_ancestor>>. |
| 225 | |
| 226 | |
| 227 | [[Remap_to_ancestor]] |
| 228 | Remap to ancestor |
| 229 | ~~~~~~~~~~~~~~~~~ |
| 230 | |
Junio C Hamano | 1cca17d | 2016-05-04 10:36:24 -0700 | [diff] [blame] | 231 | By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the |
Csaba Henk | 7ec344d | 2010-08-27 20:44:56 +0000 | [diff] [blame] | 232 | set of revisions which get rewritten. However, positive refs on the command |
| 233 | line are distinguished: we don't let them be excluded by such limiters. For |
| 234 | this purpose, they are instead rewritten to point at the nearest ancestor that |
| 235 | was not excluded. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 236 | |
| 237 | |
Michele Locati | 0a0eb2e | 2018-03-15 18:09:18 +0100 | [diff] [blame] | 238 | EXIT STATUS |
| 239 | ----------- |
| 240 | |
| 241 | On success, the exit status is `0`. If the filter can't find any commits to |
| 242 | rewrite, the exit status is `2`. On any other error, the exit status may be |
| 243 | any other non-zero value. |
| 244 | |
| 245 | |
Nguyễn Thái Ngọc Duy | 76a8788 | 2018-04-30 17:35:33 +0200 | [diff] [blame] | 246 | EXAMPLES |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 247 | -------- |
| 248 | |
| 249 | Suppose you want to remove a file (containing confidential information |
| 250 | or copyright violation) from all commits: |
| 251 | |
| 252 | ------------------------------------------------------- |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 253 | git filter-branch --tree-filter 'rm filename' HEAD |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 254 | ------------------------------------------------------- |
| 255 | |
Jon Loeliger | e4d594c | 2008-05-16 14:43:50 -0500 | [diff] [blame] | 256 | However, if the file is absent from the tree of some commit, |
| 257 | a simple `rm filename` will fail for that tree and commit. |
| 258 | Thus you may instead want to use `rm -f filename` as the script. |
| 259 | |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 260 | Using `--index-filter` with 'git rm' yields a significantly faster |
Thomas Rast | 3bc427e | 2009-03-12 00:00:56 +0100 | [diff] [blame] | 261 | version. Like with using `rm filename`, `git rm --cached filename` |
| 262 | will fail if the file is absent from the tree of a commit. If you |
| 263 | want to "completely forget" a file, it does not matter when it entered |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 264 | history, so we also add `--ignore-unmatch`: |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 265 | |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 266 | -------------------------------------------------------------------------- |
Thomas Rast | 3bc427e | 2009-03-12 00:00:56 +0100 | [diff] [blame] | 267 | git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 268 | -------------------------------------------------------------------------- |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 269 | |
Johannes Schindelin | 8ef4451 | 2007-10-17 03:22:25 +0100 | [diff] [blame] | 270 | Now, you will get the rewritten history saved in HEAD. |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 271 | |
Thomas Rast | 8afa421 | 2008-08-07 16:16:03 +0200 | [diff] [blame] | 272 | To rewrite the repository to look as if `foodir/` had been its project |
| 273 | root, and discard all other history: |
| 274 | |
| 275 | ------------------------------------------------------- |
| 276 | git filter-branch --subdirectory-filter foodir -- --all |
| 277 | ------------------------------------------------------- |
| 278 | |
| 279 | Thus you can, e.g., turn a library subdirectory into a repository of |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 280 | its own. Note the `--` that separates 'filter-branch' options from |
| 281 | revision options, and the `--all` to rewrite all branches and tags. |
Thomas Rast | 8afa421 | 2008-08-07 16:16:03 +0200 | [diff] [blame] | 282 | |
Johannes Sixt | 32c37c1 | 2007-07-04 09:32:47 +0200 | [diff] [blame] | 283 | To set a commit (which typically is at the tip of another |
| 284 | history) to be the parent of the current initial commit, in |
| 285 | order to paste the other history behind the current history: |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 286 | |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 287 | ------------------------------------------------------------------- |
| 288 | git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD |
| 289 | ------------------------------------------------------------------- |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 290 | |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 291 | (if the parent string is empty - which happens when we are dealing with |
| 292 | the initial commit - add graftcommit as a parent). Note that this assumes |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 293 | history with a single root (that is, no merge without common ancestors |
| 294 | happened). If this is not the case, use: |
| 295 | |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 296 | -------------------------------------------------------------------------- |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 297 | git filter-branch --parent-filter \ |
Caio Marcelo de Oliveira Filho | 41e86a3 | 2008-02-25 23:14:31 -0300 | [diff] [blame] | 298 | 'test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>" || cat' HEAD |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 299 | -------------------------------------------------------------------------- |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 300 | |
Johannes Sixt | 32c37c1 | 2007-07-04 09:32:47 +0200 | [diff] [blame] | 301 | or even simpler: |
| 302 | |
| 303 | ----------------------------------------------- |
Johannes Schindelin | e2d65c1 | 2018-04-29 00:44:53 +0200 | [diff] [blame] | 304 | git replace --graft $commit-id $graft-id |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 305 | git filter-branch $graft-id..HEAD |
Johannes Sixt | 32c37c1 | 2007-07-04 09:32:47 +0200 | [diff] [blame] | 306 | ----------------------------------------------- |
| 307 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 308 | To remove commits authored by "Darl McBribe" from the history: |
| 309 | |
| 310 | ------------------------------------------------------------------------------ |
| 311 | git filter-branch --commit-filter ' |
| 312 | if [ "$GIT_AUTHOR_NAME" = "Darl McBribe" ]; |
| 313 | then |
Johannes Schindelin | f95eef1 | 2007-08-31 20:06:27 +0100 | [diff] [blame] | 314 | skip_commit "$@"; |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 315 | else |
| 316 | git commit-tree "$@"; |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 317 | fi' HEAD |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 318 | ------------------------------------------------------------------------------ |
| 319 | |
Sergei Organov | 8451c56 | 2007-11-01 16:24:11 +0300 | [diff] [blame] | 320 | The function 'skip_commit' is defined as follows: |
Johannes Schindelin | f95eef1 | 2007-08-31 20:06:27 +0100 | [diff] [blame] | 321 | |
| 322 | -------------------------- |
| 323 | skip_commit() |
| 324 | { |
| 325 | shift; |
| 326 | while [ -n "$1" ]; |
| 327 | do |
| 328 | shift; |
| 329 | map "$1"; |
| 330 | shift; |
| 331 | done; |
| 332 | } |
| 333 | -------------------------- |
| 334 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 335 | The shift magic first throws away the tree id and then the -p |
| 336 | parameters. Note that this handles merges properly! In case Darl |
| 337 | committed a merge between P1 and P2, it will be propagated properly |
| 338 | and all children of the merge will become merge commits with P1,P2 |
| 339 | as their parents instead of the merge commit. |
| 340 | |
Andreas Schwab | 8093ae8 | 2012-09-18 17:55:08 +0200 | [diff] [blame] | 341 | *NOTE* the changes introduced by the commits, and which are not reverted |
| 342 | by subsequent commits, will still be in the rewritten branch. If you want |
| 343 | to throw out _changes_ together with the commits, you should use the |
| 344 | interactive mode of 'git rebase'. |
| 345 | |
Florian Ragwitz | a174889 | 2008-04-30 09:47:43 +0200 | [diff] [blame] | 346 | You can rewrite the commit log messages using `--msg-filter`. For |
Thomas Rast | 0b444cd | 2010-01-10 00:33:00 +0100 | [diff] [blame] | 347 | example, 'git svn-id' strings in a repository created by 'git svn' can |
Miklos Vajna | ed10d9a | 2008-02-25 15:43:53 +0100 | [diff] [blame] | 348 | be removed this way: |
| 349 | |
| 350 | ------------------------------------------------------- |
Florian Ragwitz | a174889 | 2008-04-30 09:47:43 +0200 | [diff] [blame] | 351 | git filter-branch --msg-filter ' |
Miklos Vajna | ed10d9a | 2008-02-25 15:43:53 +0100 | [diff] [blame] | 352 | sed -e "/^git-svn-id:/d" |
| 353 | ' |
| 354 | ------------------------------------------------------- |
Johannes Schindelin | f95eef1 | 2007-08-31 20:06:27 +0100 | [diff] [blame] | 355 | |
Johannes Schindelin | b8f4233 | 2009-08-17 22:38:46 +0200 | [diff] [blame] | 356 | If you need to add 'Acked-by' lines to, say, the last 10 commits (none |
| 357 | of which is a merge), use this command: |
| 358 | |
| 359 | -------------------------------------------------------- |
| 360 | git filter-branch --msg-filter ' |
| 361 | cat && |
| 362 | echo "Acked-by: Bugs Bunny <bunny@bugzilla.org>" |
| 363 | ' HEAD~10..HEAD |
| 364 | -------------------------------------------------------- |
| 365 | |
Tadeusz Andrzej Kadłubowski | 21b6e4f | 2013-02-21 21:23:38 +0100 | [diff] [blame] | 366 | The `--env-filter` option can be used to modify committer and/or author |
| 367 | identity. For example, if you found out that your commits have the wrong |
| 368 | identity due to a misconfigured user.email, you can make a correction, |
| 369 | before publishing the project, like this: |
| 370 | |
| 371 | -------------------------------------------------------- |
| 372 | git filter-branch --env-filter ' |
| 373 | if test "$GIT_AUTHOR_EMAIL" = "root@localhost" |
| 374 | then |
| 375 | GIT_AUTHOR_EMAIL=john@example.com |
Tadeusz Andrzej Kadłubowski | 21b6e4f | 2013-02-21 21:23:38 +0100 | [diff] [blame] | 376 | fi |
| 377 | if test "$GIT_COMMITTER_EMAIL" = "root@localhost" |
| 378 | then |
| 379 | GIT_COMMITTER_EMAIL=john@example.com |
Tadeusz Andrzej Kadłubowski | 21b6e4f | 2013-02-21 21:23:38 +0100 | [diff] [blame] | 380 | fi |
| 381 | ' -- --all |
| 382 | -------------------------------------------------------- |
| 383 | |
Andreas Schwab | 8093ae8 | 2012-09-18 17:55:08 +0200 | [diff] [blame] | 384 | To restrict rewriting to only part of the history, specify a revision |
| 385 | range in addition to the new branch name. The new branch name will |
| 386 | point to the top-most revision that a 'git rev-list' of this range |
| 387 | will print. |
Johannes Schindelin | 0820366 | 2007-08-31 17:42:33 +0100 | [diff] [blame] | 388 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 389 | Consider this history: |
| 390 | |
| 391 | ------------------ |
| 392 | D--E--F--G--H |
| 393 | / / |
| 394 | A--B-----C |
| 395 | ------------------ |
| 396 | |
| 397 | To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use: |
| 398 | |
| 399 | -------------------------------- |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 400 | git filter-branch ... C..H |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 401 | -------------------------------- |
| 402 | |
| 403 | To rewrite commits E,F,G,H, use one of these: |
| 404 | |
| 405 | ---------------------------------------- |
Johannes Schindelin | dfd05e3 | 2007-07-23 18:34:13 +0100 | [diff] [blame] | 406 | git filter-branch ... C..H --not D |
| 407 | git filter-branch ... D..H --not C |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 408 | ---------------------------------------- |
| 409 | |
| 410 | To move the whole tree into a subdirectory, or remove it from there: |
| 411 | |
| 412 | --------------------------------------------------------------- |
| 413 | git filter-branch --index-filter \ |
Jeff King | d2d66f1 | 2010-02-01 07:43:45 -0500 | [diff] [blame] | 414 | 'git ls-files -s | sed "s-\t\"*-&newsubdir/-" | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 415 | GIT_INDEX_FILE=$GIT_INDEX_FILE.new \ |
| 416 | git update-index --index-info && |
Jeff King | 6cb0186 | 2011-04-01 10:46:27 -0400 | [diff] [blame] | 417 | mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 418 | --------------------------------------------------------------- |
| 419 | |
| 420 | |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 421 | |
Nguyễn Thái Ngọc Duy | 76a8788 | 2018-04-30 17:35:33 +0200 | [diff] [blame] | 422 | CHECKLIST FOR SHRINKING A REPOSITORY |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 423 | ------------------------------------ |
| 424 | |
Roberto Tyley | 615b8f1 | 2013-12-18 14:25:16 +0000 | [diff] [blame] | 425 | git-filter-branch can be used to get rid of a subset of files, |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 426 | usually with some combination of `--index-filter` and |
| 427 | `--subdirectory-filter`. People expect the resulting repository to |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 428 | be smaller than the original, but you need a few more steps to |
Thomas Ackermann | 2de9b71 | 2013-01-21 20:17:53 +0100 | [diff] [blame] | 429 | actually make it smaller, because Git tries hard not to lose your |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 430 | objects until you tell it to. First make sure that: |
| 431 | |
| 432 | * You really removed all variants of a filename, if a blob was moved |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 433 | over its lifetime. `git log --name-only --follow --all -- filename` |
| 434 | can help you find renames. |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 435 | |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 436 | * You really filtered all refs: use `--tag-name-filter cat -- --all` |
| 437 | when calling git-filter-branch. |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 438 | |
| 439 | Then there are two ways to get a smaller repository. A safer way is |
| 440 | to clone, that keeps your original intact. |
| 441 | |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 442 | * Clone it with `git clone file:///path/to/repo`. The clone |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 443 | will not have the removed objects. See linkgit:git-clone[1]. (Note |
| 444 | that cloning with a plain path just hardlinks everything!) |
| 445 | |
| 446 | If you really don't want to clone it, for whatever reasons, check the |
| 447 | following points instead (in this order). This is a very destructive |
| 448 | approach, so *make a backup* or go back to cloning it. You have been |
| 449 | warned. |
| 450 | |
| 451 | * Remove the original refs backed up by git-filter-branch: say `git |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 452 | for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 453 | update-ref -d`. |
| 454 | |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 455 | * Expire all reflogs with `git reflog expire --expire=now --all`. |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 456 | |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 457 | * Garbage collect all unreferenced objects with `git gc --prune=now` |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 458 | (or if your git-gc is not new enough to support arguments to |
Jeff King | 6cf378f | 2012-04-26 04:51:57 -0400 | [diff] [blame] | 459 | `--prune`, use `git repack -ad; git prune` instead). |
Thomas Rast | d0268de | 2009-02-14 21:56:51 +0100 | [diff] [blame] | 460 | |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 461 | [[PERFORMANCE]] |
| 462 | PERFORMANCE |
| 463 | ----------- |
Roberto Tyley | 615b8f1 | 2013-12-18 14:25:16 +0000 | [diff] [blame] | 464 | |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 465 | The performance of git-filter-branch is glacially slow; its design makes it |
| 466 | impossible for a backward-compatible implementation to ever be fast: |
Roberto Tyley | 615b8f1 | 2013-12-18 14:25:16 +0000 | [diff] [blame] | 467 | |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 468 | * In editing files, git-filter-branch by design checks out each and |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 469 | every commit as it existed in the original repo. If your repo has |
Martin Ågren | e469afe | 2020-02-03 21:36:50 +0100 | [diff] [blame] | 470 | `10^5` files and `10^5` commits, but each commit only modifies five |
| 471 | files, then git-filter-branch will make you do `10^10` modifications, |
| 472 | despite only having (at most) `5*10^5` unique blobs. |
Roberto Tyley | 615b8f1 | 2013-12-18 14:25:16 +0000 | [diff] [blame] | 473 | |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 474 | * If you try and cheat and try to make git-filter-branch only work on |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 475 | files modified in a commit, then two things happen |
Roberto Tyley | 615b8f1 | 2013-12-18 14:25:16 +0000 | [diff] [blame] | 476 | |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 477 | ** you run into problems with deletions whenever the user is simply |
| 478 | trying to rename files (because attempting to delete files that |
| 479 | don't exist looks like a no-op; it takes some chicanery to remap |
| 480 | deletes across file renames when the renames happen via arbitrary |
| 481 | user-provided shell) |
| 482 | |
| 483 | ** even if you succeed at the map-deletes-for-renames chicanery, you |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 484 | still technically violate backward compatibility because users |
| 485 | are allowed to filter files in ways that depend upon topology of |
| 486 | commits instead of filtering solely based on file contents or |
| 487 | names (though this has not been observed in the wild). |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 488 | |
| 489 | * Even if you don't need to edit files but only want to e.g. rename or |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 490 | remove some and thus can avoid checking out each file (i.e. you can |
| 491 | use --index-filter), you still are passing shell snippets for your |
| 492 | filters. This means that for every commit, you have to have a |
| 493 | prepared git repo where those filters can be run. That's a |
| 494 | significant setup. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 495 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 496 | * Further, several additional files are created or updated per commit |
| 497 | by git-filter-branch. Some of these are for supporting the |
| 498 | convenience functions provided by git-filter-branch (such as map()), |
| 499 | while others are for keeping track of internal state (but could have |
| 500 | also been accessed by user filters; one of git-filter-branch's |
| 501 | regression tests does so). This essentially amounts to using the |
| 502 | filesystem as an IPC mechanism between git-filter-branch and the |
| 503 | user-provided filters. Disks tend to be a slow IPC mechanism, and |
| 504 | writing these files also effectively represents a forced |
| 505 | synchronization point between separate processes that we hit with |
| 506 | every commit. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 507 | |
| 508 | * The user-provided shell commands will likely involve a pipeline of |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 509 | commands, resulting in the creation of many processes per commit. |
| 510 | Creating and running another process takes a widely varying amount |
| 511 | of time between operating systems, but on any platform it is very |
| 512 | slow relative to invoking a function. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 513 | |
| 514 | * git-filter-branch itself is written in shell, which is kind of slow. |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 515 | This is the one performance issue that could be backward-compatibly |
| 516 | fixed, but compared to the above problems that are intrinsic to the |
| 517 | design of git-filter-branch, the language of the tool itself is a |
| 518 | relatively minor issue. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 519 | |
| 520 | ** Side note: Unfortunately, people tend to fixate on the |
| 521 | written-in-shell aspect and periodically ask if git-filter-branch |
| 522 | could be rewritten in another language to fix the performance |
| 523 | issues. Not only does that ignore the bigger intrinsic problems |
| 524 | with the design, it'd help less than you'd expect: if |
| 525 | git-filter-branch itself were not shell, then the convenience |
| 526 | functions (map(), skip_commit(), etc) and the `--setup` argument |
| 527 | could no longer be executed once at the beginning of the program |
| 528 | but would instead need to be prepended to every user filter (and |
| 529 | thus re-executed with every commit). |
| 530 | |
| 531 | The https://github.com/newren/git-filter-repo/[git filter-repo] tool is |
| 532 | an alternative to git-filter-branch which does not suffer from these |
| 533 | performance problems or the safety problems (mentioned below). For those |
| 534 | with existing tooling which relies upon git-filter-branch, 'git |
| 535 | repo-filter' also provides |
| 536 | https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely], |
| 537 | a drop-in git-filter-branch replacement (with a few caveats). While |
| 538 | filter-lamely suffers from all the same safety issues as |
Elijah Newren | 031fd4b | 2019-11-05 17:07:20 +0000 | [diff] [blame] | 539 | git-filter-branch, it at least ameliorates the performance issues a |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 540 | little. |
| 541 | |
| 542 | [[SAFETY]] |
| 543 | SAFETY |
| 544 | ------ |
| 545 | |
| 546 | git-filter-branch is riddled with gotchas resulting in various ways to |
| 547 | easily corrupt repos or end up with a mess worse than what you started |
| 548 | with: |
| 549 | |
| 550 | * Someone can have a set of "working and tested filters" which they |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 551 | document or provide to a coworker, who then runs them on a different |
| 552 | OS where the same commands are not working/tested (some examples in |
| 553 | the git-filter-branch manpage are also affected by this). |
| 554 | BSD vs. GNU userland differences can really bite. If lucky, error |
| 555 | messages are spewed. But just as likely, the commands either don't |
| 556 | do the filtering requested, or silently corrupt by making some |
| 557 | unwanted change. The unwanted change may only affect a few commits, |
| 558 | so it's not necessarily obvious either. (The fact that problems |
| 559 | won't necessarily be obvious means they are likely to go unnoticed |
| 560 | until the rewritten history is in use for quite a while, at which |
| 561 | point it's really hard to justify another flag-day for another |
| 562 | rewrite.) |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 563 | |
| 564 | * Filenames with spaces are often mishandled by shell snippets since |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 565 | they cause problems for shell pipelines. Not everyone is familiar |
| 566 | with find -print0, xargs -0, git-ls-files -z, etc. Even people who |
| 567 | are familiar with these may assume such flags are not relevant |
| 568 | because someone else renamed any such files in their repo back |
| 569 | before the person doing the filtering joined the project. And |
| 570 | often, even those familiar with handling arguments with spaces may |
| 571 | not do so just because they aren't in the mindset of thinking about |
| 572 | everything that could possibly go wrong. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 573 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 574 | * Non-ascii filenames can be silently removed despite being in a |
| 575 | desired directory. Keeping only wanted paths is often done using |
| 576 | pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. |
| 577 | ls-files will only quote filenames if needed, so folks may not |
| 578 | notice that one of the files didn't match the regex (at least not |
| 579 | until it's much too late). Yes, someone who knows about |
| 580 | core.quotePath can avoid this (unless they have other special |
| 581 | characters like \t, \n, or "), and people who use ls-files -z with |
| 582 | something other than grep can avoid this, but that doesn't mean they |
| 583 | will. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 584 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 585 | * Similarly, when moving files around, one can find that filenames |
| 586 | with non-ascii or special characters end up in a different |
| 587 | directory, one that includes a double quote character. (This is |
| 588 | technically the same issue as above with quoting, but perhaps an |
| 589 | interesting different way that it can and has manifested as a |
| 590 | problem.) |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 591 | |
| 592 | * It's far too easy to accidentally mix up old and new history. It's |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 593 | still possible with any tool, but git-filter-branch almost |
| 594 | invites it. If lucky, the only downside is users getting frustrated |
| 595 | that they don't know how to shrink their repo and remove the old |
| 596 | stuff. If unlucky, they merge old and new history and end up with |
| 597 | multiple "copies" of each commit, some of which have unwanted or |
| 598 | sensitive files and others which don't. This comes about in |
| 599 | multiple different ways: |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 600 | |
| 601 | ** the default to only doing a partial history rewrite ('--all' is not |
| 602 | the default and few examples show it) |
| 603 | |
| 604 | ** the fact that there's no automatic post-run cleanup |
| 605 | |
| 606 | ** the fact that --tag-name-filter (when used to rename tags) doesn't |
| 607 | remove the old tags but just adds new ones with the new name |
| 608 | |
| 609 | ** the fact that little educational information is provided to inform |
| 610 | users of the ramifications of a rewrite and how to avoid mixing old |
| 611 | and new history. For example, this man page discusses how users |
| 612 | need to understand that they need to rebase their changes for all |
| 613 | their branches on top of new history (or delete and reclone), but |
| 614 | that's only one of multiple concerns to consider. See the |
| 615 | "DISCUSSION" section of the git filter-repo manual page for more |
| 616 | details. |
| 617 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 618 | * Annotated tags can be accidentally converted to lightweight tags, |
| 619 | due to either of two issues: |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 620 | |
| 621 | ** Someone can do a history rewrite, realize they messed up, restore |
| 622 | from the backups in refs/original/, and then redo their |
| 623 | git-filter-branch command. (The backup in refs/original/ is not a |
| 624 | real backup; it dereferences tags first.) |
| 625 | |
| 626 | ** Running git-filter-branch with either --tags or --all in your |
| 627 | <rev-list options>. In order to retain annotated tags as |
| 628 | annotated, you must use --tag-name-filter (and must not have |
| 629 | restored from refs/original/ in a previously botched rewrite). |
| 630 | |
| 631 | * Any commit messages that specify an encoding will become corrupted |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 632 | by the rewrite; git-filter-branch ignores the encoding, takes the |
| 633 | original bytes, and feeds it to commit-tree without telling it the |
| 634 | proper encoding. (This happens whether or not --msg-filter is |
| 635 | used.) |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 636 | |
| 637 | * Commit messages (even if they are all UTF-8) by default become |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 638 | corrupted due to not being updated -- any references to other commit |
| 639 | hashes in commit messages will now refer to no-longer-extant |
| 640 | commits. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 641 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 642 | * There are no facilities for helping users find what unwanted crud |
| 643 | they should delete, which means they are much more likely to have |
| 644 | incomplete or partial cleanups that sometimes result in confusion |
| 645 | and people wasting time trying to understand. (For example, folks |
| 646 | tend to just look for big files to delete instead of big directories |
| 647 | or extensions, and once they do so, then sometime later folks using |
| 648 | the new repository who are going through history will notice a build |
| 649 | artifact directory that has some files but not others, or a cache of |
| 650 | dependencies (node_modules or similar) which couldn't have ever been |
| 651 | functional since it's missing some files.) |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 652 | |
| 653 | * If --prune-empty isn't specified, then the filtering process can |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 654 | create hoards of confusing empty commits |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 655 | |
| 656 | * If --prune-empty is specified, then intentionally placed empty |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 657 | commits from before the filtering operation are also pruned instead |
| 658 | of just pruning commits that became empty due to filtering rules. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 659 | |
Elijah Newren | a6d39f2 | 2019-11-05 23:31:30 +0000 | [diff] [blame] | 660 | * If --prune-empty is specified, sometimes empty commits are missed |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 661 | and left around anyway (a somewhat rare bug, but it happens...) |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 662 | |
| 663 | * A minor issue, but users who have a goal to update all names and |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 664 | emails in a repository may be led to --env-filter which will only |
| 665 | update authors and committers, missing taggers. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 666 | |
| 667 | * If the user provides a --tag-name-filter that maps multiple tags to |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 668 | the same name, no warning or error is provided; git-filter-branch |
| 669 | simply overwrites each tag in some undocumented pre-defined order |
| 670 | resulting in only one tag at the end. (A git-filter-branch |
| 671 | regression test requires this surprising behavior.) |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 672 | |
| 673 | Also, the poor performance of git-filter-branch often leads to safety |
| 674 | issues: |
| 675 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 676 | * Coming up with the correct shell snippet to do the filtering you |
| 677 | want is sometimes difficult unless you're just doing a trivial |
| 678 | modification such as deleting a couple files. Unfortunately, people |
| 679 | often learn if the snippet is right or wrong by trying it out, but |
| 680 | the rightness or wrongness can vary depending on special |
| 681 | circumstances (spaces in filenames, non-ascii filenames, funny |
| 682 | author names or emails, invalid timezones, presence of grafts or |
| 683 | replace objects, etc.), meaning they may have to wait a long time, |
| 684 | hit an error, then restart. The performance of git-filter-branch is |
| 685 | so bad that this cycle is painful, reducing the time available to |
| 686 | carefully re-check (to say nothing about what it does to the |
| 687 | patience of the person doing the rewrite even if they do technically |
| 688 | have more time available). This problem is extra compounded because |
| 689 | errors from broken filters may not be shown for a long time and/or |
| 690 | get lost in a sea of output. Even worse, broken filters often just |
| 691 | result in silent incorrect rewrites. |
Elijah Newren | 9df53c5 | 2019-09-04 15:32:38 -0700 | [diff] [blame] | 692 | |
Jean-Noël Avila | df5be01 | 2019-12-12 21:46:55 +0100 | [diff] [blame] | 693 | * To top it all off, even when users finally find working commands, |
| 694 | they naturally want to share them. But they may be unaware that |
| 695 | their repo didn't have some special cases that someone else's does. |
| 696 | So, when someone else with a different repository runs the same |
| 697 | commands, they get hit by the problems above. Or, the user just |
| 698 | runs commands that really were vetted for special cases, but they |
| 699 | run it on a different OS where it doesn't work, as noted above. |
Roberto Tyley | 615b8f1 | 2013-12-18 14:25:16 +0000 | [diff] [blame] | 700 | |
Johannes Schindelin | c401b33 | 2007-07-04 00:41:55 +0100 | [diff] [blame] | 701 | GIT |
| 702 | --- |
Christian Couder | 9e1f0a8 | 2008-06-06 09:07:32 +0200 | [diff] [blame] | 703 | Part of the linkgit:git[1] suite |