Documentation/howto/coordinate-embargoed-releases.txt - jrn/git - Git at Google

 Content-type: text/asciidoc
 Abstract: When a vulnerability is reported, we follow these guidelines to
  assess the vulnerability, create and review a fix, and coordinate embargoed
  security releases.

 How we coordinate embargoed releases
 ------------------------------------

 To protect Git users from critical vulnerabilities, we do not just release
 fixed versions like regular maintenance releases. Instead, we coordinate
 releases with packagers, keeping the fixes under an embargo until the release
 date. That way, users will have a chance to upgrade on that date, no matter
 what Operating System or distribution they run.

 The `git-security` mailing list
 -------------------------------

 Responsible disclosures of vulnerabilities, analysis, proposed fixes as
 well as the orchestration of coordinated embargoed releases all happen on the
 `git-security` mailing list at <git-security@googlegroups.com>.

 In this context, the term "embargo" refers to the time period that information
 about a vulnerability is kept under wraps and only shared on a need-to-know
 basis. This is necessary to protect Git's users from bad actors who would
 otherwise be made aware of attack vectors that could be exploited. "Lifting the
 embargo" refers to publishing the version that fixes the vulnerabilities.

 Audience of the `git-security` mailing list
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Anybody may contact the `git-security` mailing list by sending an email
 to <git-security@googlegroups.com>, though the archive is closed to the
 public and only accessible to subscribed members.

 There are a few dozen subscribed members: core Git developers who are trusted
 with addressing vulnerabilities, and stakeholders (i.e. owners of products
 affected by security vulnerabilities in Git).

 Most of the discussions revolve around assessing the severity of the reported
 issue (including the decision whether the report is security-relevant or can be
 redirected to the public mailing list), how to remediate the issue, determining
 the timeline of the disclosure as well as aligning priorities and
 requirements.

 Communications
 ~~~~~~~~~~~~~~

 If you are a stakeholder, it is a good idea to pay close attention to the
 discussions, as pertinent information may be buried in the middle of a lively
 conversation that might not look relevant to your interests. For example, the
 tentative timeline might be agreed upon in the middle of discussing code
 comment formatting in one of the patches and whether or not to combine fixes
 for multiple, separate vulnerabilities into the same embargoed release. Most
 mail threads are not usually structured specifically to communicate
 agreements, assessments or timelines.

 Typical timeline
 ----------------

 - A potential vulnerability is reported to the `git-security` mailing list.

 - The members of the git-security list start a discussion to give an initial
   assessment of the severity of the reported potential vulnerability.
   We aspire to do so within a few days.

 - After discussion, if consensus is reached that it is not critical enough
   to warrant any embargo, the reporter is redirected to the public Git mailing
   list. This ends the reporter's interaction with the `git-security` list.

 - If it is deemed critical enough for an embargo, ideas are presented on how to
   address the vulnerability.

 - Usually around that time, the Git maintainer or their delegate(s) open a draft
   security advisory in the `git/git` repository on GitHub (see below for more
   details).

 - Code review can take place in a variety of different locations,
   depending on context. These are: patches sent inline on the git-security list,
   a private fork on GitHub associated with the draft security advisory, or the
   git/cabal repository.

 - Contributors working on a fix should consider beginning by sending
   patches to the git-security list (inline with the original thread), since they
   are accessible to all subscribers, along with the original reporter.

 - Once the review has settled and everyone involved in the review agrees that
   the patches are nearing the finish line, the Git maintainer, and others
   determine a release date as well as the release trains that are serviced. The
   decision regarding which versions need a backported fix is based on input from
   the reporter, the contributor who worked on the patches, and from
   stakeholders. Operators of hosting sites who may want to analyze whether the
   given issue is exploited via any of the repositories they host, and binary
   packagers who want to make sure their product gets patched adequately against
   the vulnerability, for example, may want to give their input at this stage.

 - While the Git community does its best to accommodate the specific timeline
   requests of the various binary packagers, the nature of the issue may preclude
   a prolonged release schedule. For fixes deemed urgent, it may be in the best
   interest of the Git users community to shorten the disclosure and release
   timeline, and packagers may need to adapt accordingly.

 - Subsequently, branches with the fixes are pushed to the git/cabal repository.

 - The tags are created by the Git maintainer and pushed to the same repository.

 - The Git for Windows, Git for macOS, BSD, Debian, etc. maintainers prepare the
   corresponding release artifacts, based on the tags created that have been
   prepared by the Git maintainer.

 - The release artifacts prepared by various binary packagers can be
   made available to stakeholders under embargo via a mail to the
   `git-security` list.

 - Less than a week before the release, a mail with the relevant information is
   sent to <distros@vs.openwall.org> (see below), a list used to pre-announce
   embargoed releases of open source projects to the stakeholders of all major
   distributions of Linux as well as other OSes.

 - Public communication is then prepared in advance of the release date. This
   includes blog posts and mails to the Git and Git for Windows mailing lists.

 - On the day of the release, at around 10am Pacific Time, the Git maintainer
   pushes the tag and the `master` branch to the public repository, then sends
   out an announcement mail.

 - Once the tag is pushed, the Git for Windows maintainer publishes the
   corresponding tag and creates a GitHub Release with the associated release
   artifacts (Git for Windows installer, Portable Git, MinGit, etc).

 - Git for Windows release is then announced via a mail to the public Git and
   Git for Windows mailing lists as well as via a tweet.

 - Ditto for distribution packagers for Linux and other platforms:
   their releases are announced via their preferred channels.

 - A mail to <oss-security@lists.openwall.org> (see below for details) is sent
   as a follow-up to the <distros@vs.openwall.org> one, describing the
   vulnerability in detail, often including a proof of concept of an exploit.

 Note: The Git project makes no guarantees about timelines, but aims to keep
 embargoes reasonably short in the interest of keeping Git's users safe.

 Opening a Security Advisory draft
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 The first step is to https://github.com/git/git/security/advisories/new[open
 an advisory]. Technically, this is not necessary. However, it is the most
 convenient way to obtain the CVE number and it gives us a private repository
 associated with it that can be used to collaborate on a fix.

 Notifying the Linux distributions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 At most two weeks before release date, we need to send a notification to
 <distros@vs.openwall.org>, preferably less than 7 days before the release date.
 This will reach most (all?) Linux distributions. See an example below, and the
 guidelines for this mailing list at
 https://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists[here].

 Once the version has been published, we send a note about that to oss-security.
 As an example, see https://www.openwall.com/lists/oss-security/2019/12/13/1[the
 v2.24.1 mail];
 https://oss-security.openwall.org/wiki/mailing-lists/oss-security[Here] are
 their guidelines.

 The mail to oss-security should also describe the exploit, and give credit to
 the reporter(s): security researchers still receive too little respect for the
 invaluable service they provide, and public credit goes a long way to keep them
 paid by their respective organizations.

 Technically, describing any exploit can be delayed up to 7 days, but we usually
 refrain from doing that, including it right away.

 As a courtesy we typically attach a Git bundle (as `.tar.xz` because the list
 will drop `.bundle` attachments) in the mail to distros@ so that the involved
 parties can take care of integrating/backporting them. This bundle is typically
 created using a command like this:

 	git bundle create cve-xxx.bundle ^origin/master vA.B.C vD.E.F
 	tar cJvf cve-xxx.bundle.tar.xz cve-xxx.bundle

 Example mail to distros@vs.openwall.org
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 ....
 To: distros@vs.openwall.org
 Cc: git-security@googlegroups.com, <other people involved in the report/fix>
 Subject: [vs] Upcoming Git security fix release

 Team,

 The Git project will release new versions on <date> at 10am Pacific Time or
 soon thereafter. I have attached a Git bundle (embedded in a `.tar.xz` to avoid
 it being dropped) which you can fetch into a clone of
 https://github.com/git/git via `git fetch --tags /path/to/cve-xxx.bundle`,
 containing the tags for versions <versions>.

 You can verify with `git tag -v <tag>` that the versions were signed by
 the Git maintainer, using the same GPG key as e.g. v2.24.0.

 Please use these tags to prepare `git` packages for your various
 distributions, using the appropriate tagged versions. The added test cases
 help verify the correctness.

 The addressed issues are:

 <list of CVEs with a short description, typically copy/pasted from Git's
 release notes, usually demo exploit(s), too>

 Credit for finding the vulnerability goes to <reporter>, credit for fixing
 it goes to <developer>.

 Thanks,
 <name>

 ....

 Example mail to oss-security@lists.openwall.com
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 ....
 To: oss-security@lists.openwall.com
 Cc: git-security@googlegroups.com, <other people involved in the report/fix>
 Subject: git: <copy from security advisory>

 Team,

 The Git project released new versions on <date>, addressing <CVE>.

 All supported platforms are affected in one way or another, and all Git
 versions all the way back to <version> are affected. The fixed versions are:
 <versions>.

 Link to the announcement: <link to lore.kernel.org/git>

 We highly recommend to upgrade.

 The addressed issues are:
 * <list of CVEs and their explanations, along with demo exploits>

 Credit for finding the vulnerability goes to <reporter>, credit for fixing
 it goes to <developer>.

 Thanks,
 <name>
 ....
	Content-type: text/asciidoc
	Abstract: When a vulnerability is reported, we follow these guidelines to
	assess the vulnerability, create and review a fix, and coordinate embargoed
	security releases.

	How we coordinate embargoed releases
	------------------------------------

	To protect Git users from critical vulnerabilities, we do not just release
	fixed versions like regular maintenance releases. Instead, we coordinate
	releases with packagers, keeping the fixes under an embargo until the release
	date. That way, users will have a chance to upgrade on that date, no matter
	what Operating System or distribution they run.

	The `git-security` mailing list
	-------------------------------

	Responsible disclosures of vulnerabilities, analysis, proposed fixes as
	well as the orchestration of coordinated embargoed releases all happen on the
	`git-security` mailing list at <git-security@googlegroups.com>.

	In this context, the term "embargo" refers to the time period that information
	about a vulnerability is kept under wraps and only shared on a need-to-know
	basis. This is necessary to protect Git's users from bad actors who would
	otherwise be made aware of attack vectors that could be exploited. "Lifting the
	embargo" refers to publishing the version that fixes the vulnerabilities.

	Audience of the `git-security` mailing list
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Anybody may contact the `git-security` mailing list by sending an email
	to <git-security@googlegroups.com>, though the archive is closed to the
	public and only accessible to subscribed members.

	There are a few dozen subscribed members: core Git developers who are trusted
	with addressing vulnerabilities, and stakeholders (i.e. owners of products
	affected by security vulnerabilities in Git).

	Most of the discussions revolve around assessing the severity of the reported
	issue (including the decision whether the report is security-relevant or can be
	redirected to the public mailing list), how to remediate the issue, determining
	the timeline of the disclosure as well as aligning priorities and
	requirements.

	Communications
	~~~~~~~~~~~~~~

	If you are a stakeholder, it is a good idea to pay close attention to the
	discussions, as pertinent information may be buried in the middle of a lively
	conversation that might not look relevant to your interests. For example, the
	tentative timeline might be agreed upon in the middle of discussing code
	comment formatting in one of the patches and whether or not to combine fixes
	for multiple, separate vulnerabilities into the same embargoed release. Most
	mail threads are not usually structured specifically to communicate
	agreements, assessments or timelines.

	Typical timeline
	----------------

	- A potential vulnerability is reported to the `git-security` mailing list.

	- The members of the git-security list start a discussion to give an initial
	assessment of the severity of the reported potential vulnerability.
	We aspire to do so within a few days.

	- After discussion, if consensus is reached that it is not critical enough
	to warrant any embargo, the reporter is redirected to the public Git mailing
	list. This ends the reporter's interaction with the `git-security` list.

	- If it is deemed critical enough for an embargo, ideas are presented on how to
	address the vulnerability.

	- Usually around that time, the Git maintainer or their delegate(s) open a draft
	security advisory in the `git/git` repository on GitHub (see below for more
	details).

	- Code review can take place in a variety of different locations,
	depending on context. These are: patches sent inline on the git-security list,
	a private fork on GitHub associated with the draft security advisory, or the
	git/cabal repository.

	- Contributors working on a fix should consider beginning by sending
	patches to the git-security list (inline with the original thread), since they
	are accessible to all subscribers, along with the original reporter.

	- Once the review has settled and everyone involved in the review agrees that
	the patches are nearing the finish line, the Git maintainer, and others
	determine a release date as well as the release trains that are serviced. The
	decision regarding which versions need a backported fix is based on input from
	the reporter, the contributor who worked on the patches, and from
	stakeholders. Operators of hosting sites who may want to analyze whether the
	given issue is exploited via any of the repositories they host, and binary
	packagers who want to make sure their product gets patched adequately against
	the vulnerability, for example, may want to give their input at this stage.

	- While the Git community does its best to accommodate the specific timeline
	requests of the various binary packagers, the nature of the issue may preclude
	a prolonged release schedule. For fixes deemed urgent, it may be in the best
	interest of the Git users community to shorten the disclosure and release
	timeline, and packagers may need to adapt accordingly.

	- Subsequently, branches with the fixes are pushed to the git/cabal repository.

	- The tags are created by the Git maintainer and pushed to the same repository.

	- The Git for Windows, Git for macOS, BSD, Debian, etc. maintainers prepare the
	corresponding release artifacts, based on the tags created that have been
	prepared by the Git maintainer.

	- The release artifacts prepared by various binary packagers can be
	made available to stakeholders under embargo via a mail to the
	`git-security` list.

	- Less than a week before the release, a mail with the relevant information is
	sent to <distros@vs.openwall.org> (see below), a list used to pre-announce
	embargoed releases of open source projects to the stakeholders of all major
	distributions of Linux as well as other OSes.

	- Public communication is then prepared in advance of the release date. This
	includes blog posts and mails to the Git and Git for Windows mailing lists.

	- On the day of the release, at around 10am Pacific Time, the Git maintainer
	pushes the tag and the `master` branch to the public repository, then sends
	out an announcement mail.

	- Once the tag is pushed, the Git for Windows maintainer publishes the
	corresponding tag and creates a GitHub Release with the associated release
	artifacts (Git for Windows installer, Portable Git, MinGit, etc).

	- Git for Windows release is then announced via a mail to the public Git and
	Git for Windows mailing lists as well as via a tweet.

	- Ditto for distribution packagers for Linux and other platforms:
	their releases are announced via their preferred channels.

	- A mail to <oss-security@lists.openwall.org> (see below for details) is sent
	as a follow-up to the <distros@vs.openwall.org> one, describing the
	vulnerability in detail, often including a proof of concept of an exploit.

	Note: The Git project makes no guarantees about timelines, but aims to keep
	embargoes reasonably short in the interest of keeping Git's users safe.

	Opening a Security Advisory draft
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	The first step is to https://github.com/git/git/security/advisories/new[open
	an advisory]. Technically, this is not necessary. However, it is the most
	convenient way to obtain the CVE number and it gives us a private repository
	associated with it that can be used to collaborate on a fix.

	Notifying the Linux distributions
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	At most two weeks before release date, we need to send a notification to
	<distros@vs.openwall.org>, preferably less than 7 days before the release date.
	This will reach most (all?) Linux distributions. See an example below, and the
	guidelines for this mailing list at
	https://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists[here].

	Once the version has been published, we send a note about that to oss-security.
	As an example, see https://www.openwall.com/lists/oss-security/2019/12/13/1[the
	v2.24.1 mail];
	https://oss-security.openwall.org/wiki/mailing-lists/oss-security[Here] are
	their guidelines.

	The mail to oss-security should also describe the exploit, and give credit to
	the reporter(s): security researchers still receive too little respect for the
	invaluable service they provide, and public credit goes a long way to keep them
	paid by their respective organizations.

	Technically, describing any exploit can be delayed up to 7 days, but we usually
	refrain from doing that, including it right away.

	As a courtesy we typically attach a Git bundle (as `.tar.xz` because the list
	will drop `.bundle` attachments) in the mail to distros@ so that the involved
	parties can take care of integrating/backporting them. This bundle is typically
	created using a command like this:

	git bundle create cve-xxx.bundle ^origin/master vA.B.C vD.E.F
	tar cJvf cve-xxx.bundle.tar.xz cve-xxx.bundle

	Example mail to distros@vs.openwall.org
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	....
	To: distros@vs.openwall.org
	Cc: git-security@googlegroups.com, <other people involved in the report/fix>
	Subject: [vs] Upcoming Git security fix release

	Team,

	The Git project will release new versions on <date> at 10am Pacific Time or
	soon thereafter. I have attached a Git bundle (embedded in a `.tar.xz` to avoid
	it being dropped) which you can fetch into a clone of
	https://github.com/git/git via `git fetch --tags /path/to/cve-xxx.bundle`,
	containing the tags for versions <versions>.

	You can verify with `git tag -v <tag>` that the versions were signed by
	the Git maintainer, using the same GPG key as e.g. v2.24.0.

	Please use these tags to prepare `git` packages for your various
	distributions, using the appropriate tagged versions. The added test cases
	help verify the correctness.

	The addressed issues are:

	<list of CVEs with a short description, typically copy/pasted from Git's
	release notes, usually demo exploit(s), too>

	Credit for finding the vulnerability goes to <reporter>, credit for fixing
	it goes to <developer>.

	Thanks,
	<name>

	....

	Example mail to oss-security@lists.openwall.com
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	....
	To: oss-security@lists.openwall.com
	Cc: git-security@googlegroups.com, <other people involved in the report/fix>
	Subject: git: <copy from security advisory>

	Team,

	The Git project released new versions on <date>, addressing <CVE>.

	All supported platforms are affected in one way or another, and all Git
	versions all the way back to <version> are affected. The fixed versions are:
	<versions>.

	Link to the announcement: <link to lore.kernel.org/git>

	We highly recommend to upgrade.

	The addressed issues are:
	* <list of CVEs and their explanations, along with demo exploits>

	Credit for finding the vulnerability goes to <reporter>, credit for fixing
	it goes to <developer>.

	Thanks,
	<name>
	....