| 2: HOW THE DEVELOPMENT PROCESS WORKS |
| |
| Linux kernel development in the early 1990's was a pretty loose affair, |
| with relatively small numbers of users and developers involved. With a |
| user base in the millions and with some 2,000 developers involved over the |
| course of one year, the kernel has since had to evolve a number of |
| processes to keep development happening smoothly. A solid understanding of |
| how the process works is required in order to be an effective part of it. |
| |
| |
| 2.1: THE BIG PICTURE |
| |
| The kernel developers use a loosely time-based release process, with a new |
| major kernel release happening every two or three months. The recent |
| release history looks like this: |
| |
| 2.6.26 July 13, 2008 |
| 2.6.25 April 16, 2008 |
| 2.6.24 January 24, 2008 |
| 2.6.23 October 9, 2007 |
| 2.6.22 July 8, 2007 |
| 2.6.21 April 25, 2007 |
| 2.6.20 February 4, 2007 |
| |
| Every 2.6.x release is a major kernel release with new features, internal |
| API changes, and more. A typical 2.6 release can contain over 10,000 |
| changesets with changes to several hundred thousand lines of code. 2.6 is |
| thus the leading edge of Linux kernel development; the kernel uses a |
| rolling development model which is continually integrating major changes. |
| |
| A relatively straightforward discipline is followed with regard to the |
| merging of patches for each release. At the beginning of each development |
| cycle, the "merge window" is said to be open. At that time, code which is |
| deemed to be sufficiently stable (and which is accepted by the development |
| community) is merged into the mainline kernel. The bulk of changes for a |
| new development cycle (and all of the major changes) will be merged during |
| this time, at a rate approaching 1,000 changes ("patches," or "changesets") |
| per day. |
| |
| (As an aside, it is worth noting that the changes integrated during the |
| merge window do not come out of thin air; they have been collected, tested, |
| and staged ahead of time. How that process works will be described in |
| detail later on). |
| |
| The merge window lasts for two weeks. At the end of this time, Linus |
| Torvalds will declare that the window is closed and release the first of |
| the "rc" kernels. For the kernel which is destined to be 2.6.26, for |
| example, the release which happens at the end of the merge window will be |
| called 2.6.26-rc1. The -rc1 release is the signal that the time to merge |
| new features has passed, and that the time to stabilize the next kernel has |
| begun. |
| |
| Over the next six to ten weeks, only patches which fix problems should be |
| submitted to the mainline. On occasion a more significant change will be |
| allowed, but such occasions are rare; developers who try to merge new |
| features outside of the merge window tend to get an unfriendly reception. |
| As a general rule, if you miss the merge window for a given feature, the |
| best thing to do is to wait for the next development cycle. (An occasional |
| exception is made for drivers for previously-unsupported hardware; if they |
| touch no in-tree code, they cannot cause regressions and should be safe to |
| add at any time). |
| |
| As fixes make their way into the mainline, the patch rate will slow over |
| time. Linus releases new -rc kernels about once a week; a normal series |
| will get up to somewhere between -rc6 and -rc9 before the kernel is |
| considered to be sufficiently stable and the final 2.6.x release is made. |
| At that point the whole process starts over again. |
| |
| As an example, here is how the 2.6.25 development cycle went (all dates in |
| 2008): |
| |
| January 24 2.6.24 stable release |
| February 10 2.6.25-rc1, merge window closes |
| February 15 2.6.25-rc2 |
| February 24 2.6.25-rc3 |
| March 4 2.6.25-rc4 |
| March 9 2.6.25-rc5 |
| March 16 2.6.25-rc6 |
| March 25 2.6.25-rc7 |
| April 1 2.6.25-rc8 |
| April 11 2.6.25-rc9 |
| April 16 2.6.25 stable release |
| |
| How do the developers decide when to close the development cycle and create |
| the stable release? The most significant metric used is the list of |
| regressions from previous releases. No bugs are welcome, but those which |
| break systems which worked in the past are considered to be especially |
| serious. For this reason, patches which cause regressions are looked upon |
| unfavorably and are quite likely to be reverted during the stabilization |
| period. |
| |
| The developers' goal is to fix all known regressions before the stable |
| release is made. In the real world, this kind of perfection is hard to |
| achieve; there are just too many variables in a project of this size. |
| There comes a point where delaying the final release just makes the problem |
| worse; the pile of changes waiting for the next merge window will grow |
| larger, creating even more regressions the next time around. So most 2.6.x |
| kernels go out with a handful of known regressions though, hopefully, none |
| of them are serious. |
| |
| Once a stable release is made, its ongoing maintenance is passed off to the |
| "stable team," currently comprised of Greg Kroah-Hartman and Chris Wright. |
| The stable team will release occasional updates to the stable release using |
| the 2.6.x.y numbering scheme. To be considered for an update release, a |
| patch must (1) fix a significant bug, and (2) already be merged into the |
| mainline for the next development kernel. Continuing our 2.6.25 example, |
| the history (as of this writing) is: |
| |
| May 1 2.6.25.1 |
| May 6 2.6.25.2 |
| May 9 2.6.25.3 |
| May 15 2.6.25.4 |
| June 7 2.6.25.5 |
| June 9 2.6.25.6 |
| June 16 2.6.25.7 |
| June 21 2.6.25.8 |
| June 24 2.6.25.9 |
| |
| Stable updates for a given kernel are made for approximately six months; |
| after that, the maintenance of stable releases is solely the responsibility |
| of the distributors which have shipped that particular kernel. |
| |
| |
| 2.2: THE LIFECYCLE OF A PATCH |
| |
| Patches do not go directly from the developer's keyboard into the mainline |
| kernel. There is, instead, a somewhat involved (if somewhat informal) |
| process designed to ensure that each patch is reviewed for quality and that |
| each patch implements a change which is desirable to have in the mainline. |
| This process can happen quickly for minor fixes, or, in the case of large |
| and controversial changes, go on for years. Much developer frustration |
| comes from a lack of understanding of this process or from attempts to |
| circumvent it. |
| |
| In the hopes of reducing that frustration, this document will describe how |
| a patch gets into the kernel. What follows below is an introduction which |
| describes the process in a somewhat idealized way. A much more detailed |
| treatment will come in later sections. |
| |
| The stages that a patch goes through are, generally: |
| |
| - Design. This is where the real requirements for the patch - and the way |
| those requirements will be met - are laid out. Design work is often |
| done without involving the community, but it is better to do this work |
| in the open if at all possible; it can save a lot of time redesigning |
| things later. |
| |
| - Early review. Patches are posted to the relevant mailing list, and |
| developers on that list reply with any comments they may have. This |
| process should turn up any major problems with a patch if all goes |
| well. |
| |
| - Wider review. When the patch is getting close to ready for mainline |
| inclusion, it will be accepted by a relevant subsystem maintainer - |
| though this acceptance is not a guarantee that the patch will make it |
| all the way to the mainline. The patch will show up in the maintainer's |
| subsystem tree and into the staging trees (described below). When the |
| process works, this step leads to more extensive review of the patch and |
| the discovery of any problems resulting from the integration of this |
| patch with work being done by others. |
| |
| - Merging into the mainline. Eventually, a successful patch will be |
| merged into the mainline repository managed by Linus Torvalds. More |
| comments and/or problems may surface at this time; it is important that |
| the developer be responsive to these and fix any issues which arise. |
| |
| - Stable release. The number of users potentially affected by the patch |
| is now large, so, once again, new problems may arise. |
| |
| - Long-term maintenance. While it is certainly possible for a developer |
| to forget about code after merging it, that sort of behavior tends to |
| leave a poor impression in the development community. Merging code |
| eliminates some of the maintenance burden, in that others will fix |
| problems caused by API changes. But the original developer should |
| continue to take responsibility for the code if it is to remain useful |
| in the longer term. |
| |
| One of the largest mistakes made by kernel developers (or their employers) |
| is to try to cut the process down to a single "merging into the mainline" |
| step. This approach invariably leads to frustration for everybody |
| involved. |
| |
| |
| 2.3: HOW PATCHES GET INTO THE KERNEL |
| |
| There is exactly one person who can merge patches into the mainline kernel |
| repository: Linus Torvalds. But, of the over 12,000 patches which went |
| into the 2.6.25 kernel, only 250 (around 2%) were directly chosen by Linus |
| himself. The kernel project has long since grown to a size where no single |
| developer could possibly inspect and select every patch unassisted. The |
| way the kernel developers have addressed this growth is through the use of |
| a lieutenant system built around a chain of trust. |
| |
| The kernel code base is logically broken down into a set of subsystems: |
| networking, specific architecture support, memory management, video |
| devices, etc. Most subsystems have a designated maintainer, a developer |
| who has overall responsibility for the code within that subsystem. These |
| subsystem maintainers are the gatekeepers (in a loose way) for the portion |
| of the kernel they manage; they are the ones who will (usually) accept a |
| patch for inclusion into the mainline kernel. |
| |
| Subsystem maintainers each manage their own version of the kernel source |
| tree, usually (but certainly not always) using the git source management |
| tool. Tools like git (and related tools like quilt or mercurial) allow |
| maintainers to track a list of patches, including authorship information |
| and other metadata. At any given time, the maintainer can identify which |
| patches in his or her repository are not found in the mainline. |
| |
| When the merge window opens, top-level maintainers will ask Linus to "pull" |
| the patches they have selected for merging from their repositories. If |
| Linus agrees, the stream of patches will flow up into his repository, |
| becoming part of the mainline kernel. The amount of attention that Linus |
| pays to specific patches received in a pull operation varies. It is clear |
| that, sometimes, he looks quite closely. But, as a general rule, Linus |
| trusts the subsystem maintainers to not send bad patches upstream. |
| |
| Subsystem maintainers, in turn, can pull patches from other maintainers. |
| For example, the networking tree is built from patches which accumulated |
| first in trees dedicated to network device drivers, wireless networking, |
| etc. This chain of repositories can be arbitrarily long, though it rarely |
| exceeds two or three links. Since each maintainer in the chain trusts |
| those managing lower-level trees, this process is known as the "chain of |
| trust." |
| |
| Clearly, in a system like this, getting patches into the kernel depends on |
| finding the right maintainer. Sending patches directly to Linus is not |
| normally the right way to go. |
| |
| |
| 2.4: STAGING TREES |
| |
| The chain of subsystem trees guides the flow of patches into the kernel, |
| but it also raises an interesting question: what if somebody wants to look |
| at all of the patches which are being prepared for the next merge window? |
| Developers will be interested in what other changes are pending to see |
| whether there are any conflicts to worry about; a patch which changes a |
| core kernel function prototype, for example, will conflict with any other |
| patches which use the older form of that function. Reviewers and testers |
| want access to the changes in their integrated form before all of those |
| changes land in the mainline kernel. One could pull changes from all of |
| the interesting subsystem trees, but that would be a big and error-prone |
| job. |
| |
| The answer comes in the form of staging trees, where subsystem trees are |
| collected for testing and review. The older of these trees, maintained by |
| Andrew Morton, is called "-mm" (for memory management, which is how it got |
| started). The -mm tree integrates patches from a long list of subsystem |
| trees; it also has some patches aimed at helping with debugging. |
| |
| Beyond that, -mm contains a significant collection of patches which have |
| been selected by Andrew directly. These patches may have been posted on a |
| mailing list, or they may apply to a part of the kernel for which there is |
| no designated subsystem tree. As a result, -mm operates as a sort of |
| subsystem tree of last resort; if there is no other obvious path for a |
| patch into the mainline, it is likely to end up in -mm. Miscellaneous |
| patches which accumulate in -mm will eventually either be forwarded on to |
| an appropriate subsystem tree or be sent directly to Linus. In a typical |
| development cycle, approximately 10% of the patches going into the mainline |
| get there via -mm. |
| |
| The current -mm patch can always be found from the front page of |
| |
| http://kernel.org/ |
| |
| Those who want to see the current state of -mm can get the "-mm of the |
| moment" tree, found at: |
| |
| http://userweb.kernel.org/~akpm/mmotm/ |
| |
| Use of the MMOTM tree is likely to be a frustrating experience, though; |
| there is a definite chance that it will not even compile. |
| |
| The other staging tree, started more recently, is linux-next, maintained by |
| Stephen Rothwell. The linux-next tree is, by design, a snapshot of what |
| the mainline is expected to look like after the next merge window closes. |
| Linux-next trees are announced on the linux-kernel and linux-next mailing |
| lists when they are assembled; they can be downloaded from: |
| |
| http://www.kernel.org/pub/linux/kernel/people/sfr/linux-next/ |
| |
| Some information about linux-next has been gathered at: |
| |
| http://linux.f-seidel.de/linux-next/pmwiki/ |
| |
| How the linux-next tree will fit into the development process is still |
| changing. As of this writing, the first full development cycle involving |
| linux-next (2.6.26) is coming to an end; thus far, it has proved to be a |
| valuable resource for finding and fixing integration problems before the |
| beginning of the merge window. See http://lwn.net/Articles/287155/ for |
| more information on how linux-next has worked to set up the 2.6.27 merge |
| window. |
| |
| Some developers have begun to suggest that linux-next should be used as the |
| target for future development as well. The linux-next tree does tend to be |
| far ahead of the mainline and is more representative of the tree into which |
| any new work will be merged. The downside to this idea is that the |
| volatility of linux-next tends to make it a difficult development target. |
| See http://lwn.net/Articles/289013/ for more information on this topic, and |
| stay tuned; much is still in flux where linux-next is involved. |
| |
| |
| 2.5: TOOLS |
| |
| As can be seen from the above text, the kernel development process depends |
| heavily on the ability to herd collections of patches in various |
| directions. The whole thing would not work anywhere near as well as it |
| does without suitably powerful tools. Tutorials on how to use these tools |
| are well beyond the scope of this document, but there is space for a few |
| pointers. |
| |
| By far the dominant source code management system used by the kernel |
| community is git. Git is one of a number of distributed version control |
| systems being developed in the free software community. It is well tuned |
| for kernel development, in that it performs quite well when dealing with |
| large repositories and large numbers of patches. It also has a reputation |
| for being difficult to learn and use, though it has gotten better over |
| time. Some sort of familiarity with git is almost a requirement for kernel |
| developers; even if they do not use it for their own work, they'll need git |
| to keep up with what other developers (and the mainline) are doing. |
| |
| Git is now packaged by almost all Linux distributions. There is a home |
| page at |
| |
| http://git.or.cz/ |
| |
| That page has pointers to documentation and tutorials. One should be |
| aware, in particular, of the Kernel Hacker's Guide to git, which has |
| information specific to kernel development: |
| |
| http://linux.yyz.us/git-howto.html |
| |
| Among the kernel developers who do not use git, the most popular choice is |
| almost certainly Mercurial: |
| |
| http://www.selenic.com/mercurial/ |
| |
| Mercurial shares many features with git, but it provides an interface which |
| many find easier to use. |
| |
| The other tool worth knowing about is Quilt: |
| |
| http://savannah.nongnu.org/projects/quilt/ |
| |
| Quilt is a patch management system, rather than a source code management |
| system. It does not track history over time; it is, instead, oriented |
| toward tracking a specific set of changes against an evolving code base. |
| Some major subsystem maintainers use quilt to manage patches intended to go |
| upstream. For the management of certain kinds of trees (-mm, for example), |
| quilt is the best tool for the job. |
| |
| |
| 2.6: MAILING LISTS |
| |
| A great deal of Linux kernel development work is done by way of mailing |
| lists. It is hard to be a fully-functioning member of the community |
| without joining at least one list somewhere. But Linux mailing lists also |
| represent a potential hazard to developers, who risk getting buried under a |
| load of electronic mail, running afoul of the conventions used on the Linux |
| lists, or both. |
| |
| Most kernel mailing lists are run on vger.kernel.org; the master list can |
| be found at: |
| |
| http://vger.kernel.org/vger-lists.html |
| |
| There are lists hosted elsewhere, though; a number of them are at |
| lists.redhat.com. |
| |
| The core mailing list for kernel development is, of course, linux-kernel. |
| This list is an intimidating place to be; volume can reach 500 messages per |
| day, the amount of noise is high, the conversation can be severely |
| technical, and participants are not always concerned with showing a high |
| degree of politeness. But there is no other place where the kernel |
| development community comes together as a whole; developers who avoid this |
| list will miss important information. |
| |
| There are a few hints which can help with linux-kernel survival: |
| |
| - Have the list delivered to a separate folder, rather than your main |
| mailbox. One must be able to ignore the stream for sustained periods of |
| time. |
| |
| - Do not try to follow every conversation - nobody else does. It is |
| important to filter on both the topic of interest (though note that |
| long-running conversations can drift away from the original subject |
| without changing the email subject line) and the people who are |
| participating. |
| |
| - Do not feed the trolls. If somebody is trying to stir up an angry |
| response, ignore them. |
| |
| - When responding to linux-kernel email (or that on other lists) preserve |
| the Cc: header for all involved. In the absence of a strong reason (such |
| as an explicit request), you should never remove recipients. Always make |
| sure that the person you are responding to is in the Cc: list. This |
| convention also makes it unnecessary to explicitly ask to be copied on |
| replies to your postings. |
| |
| - Search the list archives (and the net as a whole) before asking |
| questions. Some developers can get impatient with people who clearly |
| have not done their homework. |
| |
| - Avoid top-posting (the practice of putting your answer above the quoted |
| text you are responding to). It makes your response harder to read and |
| makes a poor impression. |
| |
| - Ask on the correct mailing list. Linux-kernel may be the general meeting |
| point, but it is not the best place to find developers from all |
| subsystems. |
| |
| The last point - finding the correct mailing list - is a common place for |
| beginning developers to go wrong. Somebody who asks a networking-related |
| question on linux-kernel will almost certainly receive a polite suggestion |
| to ask on the netdev list instead, as that is the list frequented by most |
| networking developers. Other lists exist for the SCSI, video4linux, IDE, |
| filesystem, etc. subsystems. The best place to look for mailing lists is |
| in the MAINTAINERS file packaged with the kernel source. |
| |
| |
| 2.7: GETTING STARTED WITH KERNEL DEVELOPMENT |
| |
| Questions about how to get started with the kernel development process are |
| common - from both individuals and companies. Equally common are missteps |
| which make the beginning of the relationship harder than it has to be. |
| |
| Companies often look to hire well-known developers to get a development |
| group started. This can, in fact, be an effective technique. But it also |
| tends to be expensive and does not do much to grow the pool of experienced |
| kernel developers. It is possible to bring in-house developers up to speed |
| on Linux kernel development, given the investment of a bit of time. Taking |
| this time can endow an employer with a group of developers who understand |
| the kernel and the company both, and who can help to train others as well. |
| Over the medium term, this is often the more profitable approach. |
| |
| Individual developers are often, understandably, at a loss for a place to |
| start. Beginning with a large project can be intimidating; one often wants |
| to test the waters with something smaller first. This is the point where |
| some developers jump into the creation of patches fixing spelling errors or |
| minor coding style issues. Unfortunately, such patches create a level of |
| noise which is distracting for the development community as a whole, so, |
| increasingly, they are looked down upon. New developers wishing to |
| introduce themselves to the community will not get the sort of reception |
| they wish for by these means. |
| |
| Andrew Morton gives this advice for aspiring kernel developers |
| |
| The #1 project for all kernel beginners should surely be "make sure |
| that the kernel runs perfectly at all times on all machines which |
| you can lay your hands on". Usually the way to do this is to work |
| with others on getting things fixed up (this can require |
| persistence!) but that's fine - it's a part of kernel development. |
| |
| (http://lwn.net/Articles/283982/). |
| |
| In the absence of obvious problems to fix, developers are advised to look |
| at the current lists of regressions and open bugs in general. There is |
| never any shortage of issues in need of fixing; by addressing these issues, |
| developers will gain experience with the process while, at the same time, |
| building respect with the rest of the development community. |