The kernel column by Jon Masters #82

A new OOM killer:
The Linux out-of-memory (OOM) killer has been a feature for as long as Linux has supported modern virtual memory. In the world in which we live, an operating system such as Linux will typically (unless specially disabled via a provided sysctl) ‘overcommit’ on memory allocations – just as a bank will lend more money than it actually has in its reserves on the grounds that it won’t likely all be needed at once. But if the assumption is incorrect, the system can run out of memory and trigger the OOM killer, a built-in kernel feature, to sort it out. The traditional OOM killer is undergoing an overhaul since it has been observed that it doesn’t handle modern desktop systems in which desktop apps are linked with many libraries – seeming bigger than they really are to the existing algorithm. In the future the OOM killer is much more likely to factor in actual resident memory and swap space usage.

It wasn’t all good news this month. Once again, there was a major security bug affecting a number of current and former kernels (going back at least as far as 2.6.21), and once again, it was a NULL pointer exploit. This time, though, the exploit was in so-called ‘core kernel’ code. That is code that is provided by the guts of the kernel and not in a loadable module or device driver – in this case within the pipe code.
The problem is thus very easy to trigger in a wide range of production kernels, given appropriate local-level system access, and is a good reason to upgrade all systems to the latest stable kernel releases.

Security issues are of course annoying, but one of the more interesting issues here is that modern Linux kernels include support for a mmap_min_addr sysctl configuration setting. This will determine the minimum address that can be mapped by user software. The idea is that setting this to anything above the first (typically) 4K
page of virtual memory will prevent a malicious application from planting exploit code at the NULL page, where it could be executed if a subsequent kernel system call referenced the NULL page through a NULL pointer exception.
Unfortunately, however, some legacy applications rely upon being able to map the NULL page, so a number of ‘Enterprise’ Linux systems were shipping with a more vulnerable configuration for the sakeof compatibility.

October also saw the 2009 annual Kernel Summit, which truly went global this year with its first visit to Asia. In an affront to global travel budget declines, and as a sign of the growing influence of Japanese kernel developers, KS was held in Tokyo on October 19-20. A number of topics were discussed, many of which were covered by a typically excellent write-up by Jon Corbet on LWN (www.lwn.net/). Among them were performance events, the volume of traffic on the LKML, and highly specific issues such as I/O bandwidth management. On the latter, I was very pleased to see that Vivek Goyal and a number of other developers managed to agree upon a common approach to controlling application I/O bandwidth that should see several different (but generally related) efforts join forces.

That’s all for now. Don’t forget to stay tuned between now and next month by signing up to my free daily kernel podcast at kernelpodcast.org. This article originally appeared in issue 82 of Linux User & Developer Magazine.