README - willemb/test - Git at Google

 Congratulations. You managed to clone

   https://kernel.googlesource.com/users/willemb/test


 Below is a copy of the README for this repository service.
 For the latest version, visit

   https://sites.google.com/a/google.com/prodkernel/for-kernel-developers/kernel-team-n00b/publicwikiandgit

 --------------------
 PublicWikiAndGit
 As of September 2011, we own the http://code.google.com/p/kernel public project hosting site and the https://kernel.googlesource.com public git server. This page explains how to make use of these.


 Contents
 1 Git server
 1.1  A note about reliability and performance
 1.2 Access Control
 1.3 Clone a repository
 1.4 Push to a repository
 1.5 Create a new repository
 1.5.1 Server Directory Layout
 2 Project server
 2.1 Access Control
 2.1.1 Git Access
 2.1.2 Spotting content
 2.2 Download
 2.3 Git
 3 Tips and further information

 Git server


 The git server at googlesource.com is under active development by the gerritcodereview team, so there is some dogfooding involved at this stage.
  A note about reliability and performance

 Reliability is already high, with 5x replication as of September 2011, but there are still some performance issues. A push of a fresh Linux kernel may take a few minutes to write data to disk. Git does not show any feedback, so just be patient. Do not hit Ctrl^C thinking that the process crashed. A few additional things to consider:

 Git returns once data has been written to the first replica. It takes some time for data to make it to its four mirrors.
 Warming up the RAM cache for a Linux-sized project takes seconds, so response time will similarly be at least a few seconds for every operation.
 The server can write to disk at 3MB/s. Writing hundreds of Megabyte projects will thus take up to minutes before Git returns feedback. This is the source for the initial warning at the top of this section.
 Fail-over from the main server to a replica is current a manual process, so while downtime is unlikely, it is not impossible.
 Data is not yet backed up on GTape, although this is a work in progress.
 All this will probably change rapidly for the better. Only a month ago, the servers were not multi-homed, for instance.

 Access Control

 All developers in the kernel-developers mdb group have WRITER permissions on the kernel.googlesource.com. This allows you to create new repositories over which you have full OWNER control. If you additionally want OWNER privileges on the entire server, email me and I will update your status.

 For git access, you will not use gaia or mdb credentials to authenticate to the server. The first thing to do is to acquire gitserver credentials. Ask to create a new password at https://kernel.googlesource.com/new-password (for the time being, this URL does not work. instead, use https://chromium.googlesource.com/new-password). This will generate an incomprehensible username and password. Instead of trying to remember it, do as the webpage suggests and add it to your ~/.netrc file (if you do not have a netrc file yet, first create it with 0600 permissions).

 Clone a repository

 Without authentication, users can only clone public repositories. I have a created a test-public repository exactly for this purpose. Try cloning it using
 git clone http://kernel.googlesource.com/users/willemb/test-public

 Once your permissions are set up, you should also be able to clone my test repository in read-only mode. Try issuing
 git clone https://kernel.googlesource.com/users/willemb/test
 Push to a repository

 I have given no one, but myself write privileges on the above repositories. Once you have write privileges on a repository, access it using the alternative URL writable-kernel.googlesource.com. Thus, clone as
 git clone https://writable-kernel.googlesource.com/users/willemb/test
 Create a new repository

 To create new repositories, contact the server over stubby. This relies on your MDB credentials. In particular, you have to be a member of the kernel-developers group. You create commands as text files that hold what I think is protocol buffer syntax and send these to the server using

 stubby --proto2 call \
   /bns/ic/borg/ic/bns/gerritcodereview/prd.repository-control/0 \
   RepositoryControlService.Execute < $file

 I wrapped that statement in a little shell script send_rpc.sh, so that I don't have to remember it. Detailed instructions on how to communicate with the server can be found on the gerritcodereview site. In short, this is the format to create a new repository:

 create {
   repository_name: "kernel/users/willemb/test-public"
   grant { gaia_email: "willemb@google.com" role: OWNER }
   grant { mdb_user: "willemb" role: OWNER }
   grant { mdb_group: "kernel-developers" role: WRITER }
   grant { all_users: true role: READER }
 }

 A few notes about this example:
 unauthenticated users will only be able to clone a repository over plain http if the last grant line is present.
 the repository name starts with kernel/ but this prefix is dropped when accessing the git server over http(s).
 my actual test-public does not have the mdb_user or mdb_group lines, so you do not actually have write access.
 Server Directory Layout

 I've granted every kernel developer repository creation privileges. When creating new repositories, please keep to these rules:
 projects repos are named /kernel/projects/<project>/<repo>
 user repos are named /kernel/users/<gaia name>/<repo>
 See my test and test-public repositories as example. Contact me if you have repositories that fit neither of these categories.

 Project server

 The project at http://code.google.com/p/kernel is a standard Google project hosting site. It offers a blog, bugtracker, forums, a download area and a wiki. Initially, only the latter two have been set up.
 Access Control


 The very first thing to do if you want to use the project server is to acquire administrator privileges. Email me or anyone else listed as owner on the project homepage and ask them to add you to the owners list..

 Wiki

 Publish any non-confidential content that you have in the wiki (after getting all required sign-offs, of course). The syntax is "more or less a subset of" MoinMoin wiki, one of the most popular markups. More info in the FAQ.
 Git Access

 The wiki can also be fetched and pushed using git. To fetch the repository,
 get a Google Project Hosting password from https://code.google.com/hosting/settings
 save this in your ~/.netrc as a line machine code.google.com login $gaia password $pass
 git clone https://code.google.com/p/kernel.wiki
 Spotting content

 If you see content on internal wiki or sites pages, in email threads or elsewhere that can help developers outside Google and that has no reason to be kept confidential, let me know. I'll get the proper sign-offs and convert the content to the Google wiki format. Of course, if you want to help out and do this ourself, that would be even better. The open source team has a wiki that explains the sign-off procedures. Any code or documentation that you normally produce as kernel coder and can send to Linux mailing lists without the overhead, can naturally also be published here.

 Download

 All downloads combined have a per-project quota. Do not attach too much value to this: we can ask for a quota increase if you have large files to share (think of virtual machine images and the like).
 Git

 As "Git Access" in the wiki section showed, the project server also allows creation of git repositories. This is discouraged for all but the smallest repositories. Git repositories on Google Project Hosting have two drawbacks: they are relatively slow and they only accept the access permissions of the entire project. That said, you can create and add repositories if you are in the project owners list. On the website, in "Administer/Source/Repositories", click on "New Repository". The clone URL will show up.
 Tips and further information

 Kernel developers are used to sending patches upstream. For other types of content, you may need to have the opensource team to sign off, before you make it public. This is a very low overhead process, discussed here.
 Technically, there is nothing that binds the git and project server. Access permissions are completely separate. In particular, the project server is a public service, so does not rely on google credentials, OTP, or similar.
 Since Google Project Hosting is a mature product, there is ample documentation.
 The project server is maintained by the gerritcodereview-team. More info at their internal site or their mailing list.
 Email me at willemb@google.com if you get stuck, find an error on this page or have comments.
	Congratulations. You managed to clone

	https://kernel.googlesource.com/users/willemb/test


	Below is a copy of the README for this repository service.
	For the latest version, visit

	https://sites.google.com/a/google.com/prodkernel/for-kernel-developers/kernel-team-n00b/publicwikiandgit

	--------------------
	PublicWikiAndGit
	As of September 2011, we own the http://code.google.com/p/kernel public project hosting site and the https://kernel.googlesource.com public git server. This page explains how to make use of these.


	Contents
	1 Git server
	1.1 A note about reliability and performance
	1.2 Access Control
	1.3 Clone a repository
	1.4 Push to a repository
	1.5 Create a new repository
	1.5.1 Server Directory Layout
	2 Project server
	2.1 Access Control
	2.1.1 Git Access
	2.1.2 Spotting content
	2.2 Download
	2.3 Git
	3 Tips and further information

	Git server


	The git server at googlesource.com is under active development by the gerritcodereview team, so there is some dogfooding involved at this stage.
	A note about reliability and performance

	Reliability is already high, with 5x replication as of September 2011, but there are still some performance issues. A push of a fresh Linux kernel may take a few minutes to write data to disk. Git does not show any feedback, so just be patient. Do not hit Ctrl^C thinking that the process crashed. A few additional things to consider:

	Git returns once data has been written to the first replica. It takes some time for data to make it to its four mirrors.
	Warming up the RAM cache for a Linux-sized project takes seconds, so response time will similarly be at least a few seconds for every operation.
	The server can write to disk at 3MB/s. Writing hundreds of Megabyte projects will thus take up to minutes before Git returns feedback. This is the source for the initial warning at the top of this section.
	Fail-over from the main server to a replica is current a manual process, so while downtime is unlikely, it is not impossible.
	Data is not yet backed up on GTape, although this is a work in progress.
	All this will probably change rapidly for the better. Only a month ago, the servers were not multi-homed, for instance.

	Access Control

	All developers in the kernel-developers mdb group have WRITER permissions on the kernel.googlesource.com. This allows you to create new repositories over which you have full OWNER control. If you additionally want OWNER privileges on the entire server, email me and I will update your status.

	For git access, you will not use gaia or mdb credentials to authenticate to the server. The first thing to do is to acquire gitserver credentials. Ask to create a new password at https://kernel.googlesource.com/new-password (for the time being, this URL does not work. instead, use https://chromium.googlesource.com/new-password). This will generate an incomprehensible username and password. Instead of trying to remember it, do as the webpage suggests and add it to your ~/.netrc file (if you do not have a netrc file yet, first create it with 0600 permissions).

	Clone a repository

	Without authentication, users can only clone public repositories. I have a created a test-public repository exactly for this purpose. Try cloning it using
	git clone http://kernel.googlesource.com/users/willemb/test-public

	Once your permissions are set up, you should also be able to clone my test repository in read-only mode. Try issuing
	git clone https://kernel.googlesource.com/users/willemb/test
	Push to a repository

	I have given no one, but myself write privileges on the above repositories. Once you have write privileges on a repository, access it using the alternative URL writable-kernel.googlesource.com. Thus, clone as
	git clone https://writable-kernel.googlesource.com/users/willemb/test
	Create a new repository

	To create new repositories, contact the server over stubby. This relies on your MDB credentials. In particular, you have to be a member of the kernel-developers group. You create commands as text files that hold what I think is protocol buffer syntax and send these to the server using

	stubby --proto2 call \
	/bns/ic/borg/ic/bns/gerritcodereview/prd.repository-control/0 \
	RepositoryControlService.Execute < $file

	I wrapped that statement in a little shell script send_rpc.sh, so that I don't have to remember it. Detailed instructions on how to communicate with the server can be found on the gerritcodereview site. In short, this is the format to create a new repository:

	create {
	repository_name: "kernel/users/willemb/test-public"
	grant { gaia_email: "willemb@google.com" role: OWNER }
	grant { mdb_user: "willemb" role: OWNER }
	grant { mdb_group: "kernel-developers" role: WRITER }
	grant { all_users: true role: READER }
	}

	A few notes about this example:
	unauthenticated users will only be able to clone a repository over plain http if the last grant line is present.
	the repository name starts with kernel/ but this prefix is dropped when accessing the git server over http(s).
	my actual test-public does not have the mdb_user or mdb_group lines, so you do not actually have write access.
	Server Directory Layout

	I've granted every kernel developer repository creation privileges. When creating new repositories, please keep to these rules:
	projects repos are named /kernel/projects/<project>/<repo>
	user repos are named /kernel/users/<gaia name>/<repo>
	See my test and test-public repositories as example. Contact me if you have repositories that fit neither of these categories.

	Project server

	The project at http://code.google.com/p/kernel is a standard Google project hosting site. It offers a blog, bugtracker, forums, a download area and a wiki. Initially, only the latter two have been set up.
	Access Control


	The very first thing to do if you want to use the project server is to acquire administrator privileges. Email me or anyone else listed as owner on the project homepage and ask them to add you to the owners list..

	Wiki

	Publish any non-confidential content that you have in the wiki (after getting all required sign-offs, of course). The syntax is "more or less a subset of" MoinMoin wiki, one of the most popular markups. More info in the FAQ.
	Git Access

	The wiki can also be fetched and pushed using git. To fetch the repository,
	get a Google Project Hosting password from https://code.google.com/hosting/settings
	save this in your ~/.netrc as a line machine code.google.com login $gaia password $pass
	git clone https://code.google.com/p/kernel.wiki
	Spotting content

	If you see content on internal wiki or sites pages, in email threads or elsewhere that can help developers outside Google and that has no reason to be kept confidential, let me know. I'll get the proper sign-offs and convert the content to the Google wiki format. Of course, if you want to help out and do this ourself, that would be even better. The open source team has a wiki that explains the sign-off procedures. Any code or documentation that you normally produce as kernel coder and can send to Linux mailing lists without the overhead, can naturally also be published here.

	Download

	All downloads combined have a per-project quota. Do not attach too much value to this: we can ask for a quota increase if you have large files to share (think of virtual machine images and the like).
	Git

	As "Git Access" in the wiki section showed, the project server also allows creation of git repositories. This is discouraged for all but the smallest repositories. Git repositories on Google Project Hosting have two drawbacks: they are relatively slow and they only accept the access permissions of the entire project. That said, you can create and add repositories if you are in the project owners list. On the website, in "Administer/Source/Repositories", click on "New Repository". The clone URL will show up.
	Tips and further information

	Kernel developers are used to sending patches upstream. For other types of content, you may need to have the opensource team to sign off, before you make it public. This is a very low overhead process, discussed here.
	Technically, there is nothing that binds the git and project server. Access permissions are completely separate. In particular, the project server is a public service, so does not rely on google credentials, OTP, or similar.
	Since Google Project Hosting is a mature product, there is ample documentation.
	The project server is maintained by the gerritcodereview-team. More info at their internal site or their mailing list.
	Email me at willemb@google.com if you get stuck, find an error on this page or have comments.