Tales from Linux Kernel 3.11 Development – The Kernel Column
Jon Masters summarises the happenings in the Linux kernel community around the release of the 3.11-rc1 kernel
(Ed’s note – this column was published in Linux User & Developer 130)
Linus torvalds announced the release of the 3.11-rc1 kernel, noting that it had been two weeks and “the merge window [period of time early in a release cycle during which disruptive kernel changes are permitted] has closed”.
According to Linus, “[this] merge window was smaller in terms of number of commits than the 3.10 merge window, but we actually have more new lines [of code]. Most of that seems to be in staging [a part of the kernel for new driver code] – a full third of all changes by line-count is staging, and merging in Lustre is the bulk of that”. This was followed by an RC2 that added “unnamed temporary files” (O_TMPFILE) – we’ll write about this in a later issue – and (a late change) new handling for backlights on systems designed to run Windows 8. Those systems are intended not to use direct ACPI method calls to control the backlight but instead directly drive the backlight control in the graphics driver.
Linus requested special attention be paid to testing these late features. Unfortunately, the new backlight code wasn’t quite working and so it was reverted in RC3, which also came with a plea for developers to stop sending additional patches for 3.11 other than bug fixes. Linus noted, “It’s summer. It’s nice outside. Take the kids to the pool or something. Send me just regression fixes. Otherwise I’ll have to start shouting at people again” (read on for the context of this remark). linux-next author Stephen Rothwell maintains a graphical representation of merges added to his tree as compared with Linus’s tree (showing a nice progression of patches flowing out of his tree and into Linus’s for the 3.11 merge window).
In his 3.11-rc1 announcement, Linus specifically called out the merging of initial Lustre support into ‘staging’ (the kernel subdirectory for experimental new non-core driver code). Lustre is a distributed file system popular in large distributed computing installations, which is finally beginning the process of merging into the core kernel after more than a decade living independently as an add-on external driver. Lustre has a long and storied history of ownership at various levels, including academic researchers, independent firms, Sun Microsystems, Oracle and Intel. Since Lustre is in use by many TOP500 supercomputing installations, it’ll be interesting to see whether the recent new investments in development and upstreaming of the code lead to a broader adoption in the marketplace.
Two other features of particular note that have been merged into 3.11 are the user-space ‘soft-dirty’ page tracking support, and the ‘Low Latency’ network polling patches. The soft-dirty patches add a new /proc (procfs) file-system API through which a user (or program) can request that the kernel begin to keep explicit records of all future changes to the pages (chunks of memory that contain program code and data for running programs, known as ‘processes’ or ‘tasks’ within the kernel) of a user-space process. The kernel already does this, of course, as part of its page table handling. The implementation leverages the existing page table code, setting all tracked pages as ‘read only’ in the physical memory structures representing those tables, then relying on a subsequent page fault occurring on any future write. At that time, the page fault handler is able to determine that it does not actually have to handle a real page fault (pulling in data from disk etc) but only has to update the soft-dirty metadata. So the overhead is light, but the effect is to provide a mechanism that the ‘user-space save/restore’ code can use in determining what memory a given program has modified recently.
Low Latency network polling will be another great Linux 3.11 feature for a certain subset of user. Those who have particular latency concerns in their networking that outweigh their interest in throughput or overhead (for example, stock markets) can trade some loss in performance for much lower latency by having the kernel busy-loop, polling for newly arrived network packets, rather than waiting for its conventional split approach: Linux usually utilises interrupt- driven processing for small amounts of network traffic (waiting for a hardware device to issue an interrupt request), switching to a fairly frequent polling algorithm for very high rates of traffic where interrupt-driven I/O would actually be a performance hit. Still, the conventional polling might not be frequent enough, and the new interface allows application code to specifically request (via, for instance, the poll() system call) that the kernel check for newly arrived packets for immediate processing.
Well-known kernel developer (and original USB3 support author) Sarah Sharp kicked off an extremely lengthy discussion on kernel mailing list behaviour and professionalism in response to posts from several high-profile kernel developers. Those developers were themselves responding to a fairly innocuously titled mailing list thread on ‘3.10.1-stable review’, intended to review patches destined for Greg Kroah-Hartman’s ‘stable’ Linux kernel tree. They were participating in a discussion around why patches seem to land in the stable kernel prior to Linus’s own tree. Steven Rostedt (who was not the target of Sarah’s remarks) was not alone in suggesting that Linus “scare[s] me more than Greg does”, echoing a general sentiment that Linus tends to respond very vigorously to ill-timed merge requests while Greg is often see as more accommodating. That in itself was a lengthy side discussion.
The discussion took a turn for the worse when several posters suggested that Greg needed to “create some real threat: be frank with contributors and sometimes swear a bit. That will cut your mailqueue in half, promise!”, and stop being a seen as a ‘doormat’. Linus suggested that Greg, “may need to learn to shout at people”. All of this went too far for Sarah, who responded, “Seriously, guys? Is this what we need in order to get improve -stable?… Not f***ing cool. Violence, whether it be physical intimidation, verbal threats or verbal abuse is not acceptable. Keep it professional on the mailing lists”. It would seem clear that, while the original posters were not necessarily advocating actual violence, there are concerns (not only from Sarah) around the manner (or perceived manner) in which certain topics are discussed on the kernel mailing list.
Sarah called out Linus in particular, although took pains not to make it too personal and tried to keep her comments objective. However, on one occasion she did note that Linus’s assertion that he “simply [doesn’t] believe in being polite or politically correct” was “bulls**t”. According to Sarah, “I’ve seen you be polite, and explain to clueless maintainers why there’s no way you can revert their merge that caused regressions, and ask them to [fix] it without resorting to tearing them down emotionally”. Rusty Russell noted elsewhere that, “You have to be harsh with code: people mistake politeness for uncertainty. Whenever I said ‘I prefer if you XYZ’, some proportion didn’t realise I meant ‘Don’t argue unless you have new facts: do XYZ or go away.’ This wastes my time, so I started being explicit. But be gentle with people. You’ve already called their baby ugly”. Many liked Rusty’s style of separating code review from personal commentary.
More broadly, the problem may be that over the years a tight inner circle of core contributors have formed, who are very pally with one another and don’t think twice about exchanging friendly insults. Unfortunately, it can be hard on newcomers or outsiders, who may not know the various personalities well on a one-on-one basis. Indeed, other communities have occasionally referred to this in rebuking rudeness on their own lists with “this isn’t the Linux kernel mailing list”. Eventually, there were calls to wind down the discussion and take it to the upcoming Kernel Summit, which promises to be a lively event. Linus himself ultimately issued a “back to work” request, in which he noted, “We’re over halfway through the first week after the merge window closed, and normally I’d be complaining to people about how much stuff you’re all sending me, and chewing you out for having clearly [sent] me some seriously half-baked crap during the merge window, that should probably have waited until the next release. But no. Everybody has been gossiping around the water cooler instead”.
Theodore Ts’o noted that the annual Linux Kernel Summit for 2013 “will be held October 23rd through the 25th in Edinburgh, overlapping with LinuxCon Europe (which will be held October 21[st] through the 23rd) at the Edinburgh International Conference Centre in the United Kingdom”. A general call for proposals has been issued, which also means that the exact make-up of the attendees may be more broad once again this year – not just the core kernel community, but also those with interesting and novel ideas for topics that would make a value contribution to the discussion will be invited.
Dave Jones has announced an updated release of his Trinity ‘Fuzzer’, a tool which generates random system calls using unfriendly and intentionally bad parameters, to detect potential kernel bugs such as ‘off by one’ errors, security problems and random crashes. Dave routinely posts some really nasty backtraces that have been discovered by his mischievous tool.