Keep it simple
“Things should be as simple as possible, but no simpler.” (Einstein)

Do one thing well
The UNIX Philosophy certainly applies here.

Don’t fret!
Don’t fret too much about performance – plan to optimise later when needed.

Go with it
Don’t fight the environment and go with the flow.

No-one’s Perfect
Don’t try for perfection because ‘good enough’ is often just that.

Minor shortcut
(Hence) it’s okay to cut corners sometimes, especially if you plan to optimise later.

Learn from the masters
Borrow ideas from elsewhere whenever it makes sense.

Think Cross-platform
The Python implementation should not be tied to a particular platform. It’s okay if some functionality is not always available, but the core should work everywhere.

Stop bugging me (that’s Window’s job)
Don’t bother users with details that the machine can handle.

Extensibility
A large complex system should have multiple levels of extensibility. This maximizes the opportunities for users, sophisticated or not, to help themselves.

Senseless fatalities
Errors should not be fatal. That is, user code should be able to recover from error conditions as long as the virtual machine is still functional. At the same time, errors should not pass silently.

Bug off
A bug in the user’s Python code should not be allowed to lead to undefined behavior of the Python interpreter; a core dump is never the user’s fault.

You can find Linux User & Developer’s Python Development Masterclass here.

Get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the Apple App Store now!

Think about migrating your data and services into the cloud. This sounds dangerous to you? Think about the horror stories. All your data is lost and the services are not available. Fairy tales you might think.

In February 2008 Amazon S3 crashed and the whole of Amazon S3 stopped for a few hours. In March 2009 a bug inside Google Docs had allowed unintended access to some private documents. Some people with cloud concerns ask, “What if my documents, stored by the provider of the web office (eg Google Docs) are lost?” Different question: what if your laptop is stolen or your hard disk crashes? If you are using cloud services or not, it’s always a good advice to have a backup of your data.

The tools and best practices to secure your data and improve the level of availability of your services are the same when using non-cloud services:

1. The data inside your virtual infrastructure inside a cloud IaaS can be secured when storing them inside TrueCrypt (www.truecrypt.org) containers.
2. Data transfer between you and your cloud services should be secured via SSH.
3. If you want to migrate your complete IT infrastructure into EC2, or a similar IaaS such as Rackspace (www.rackspace.com) or
GoGrid (www.gogrid.com), it’s a good advice to have the important services redundant. It’s easy to start more machines and use virtual load balancers.
4. Keep your data redundant too. Storage in the cloud can be combined to a virtual RAID 1 to improve the level of availability.
5. Try to keep your virtual machines secure, the same as you would do at home. Have the latest security patches installed. Use an up-to-date kernel. Keep the ports closed and stop the internet services you don’t really need.
6. When using a cloud platform (PaaS), back up your source code locally and keep your code. This way you could run your software at a different place at any time.
7. Have a (local) backup of your important data.
8. When using existing images inside an IaaS, look who built this image. Is the image from a trustable source (eg the provider of the IaaS services)? Otherwise, a rootkit or backdoor could be installed.
9. Keep your credentials (access key and secret access key) secure – otherwise anybody could utilise cloud resources and you have to pay them.

These hints can be used exactly the same when not using cloud services and they prove that it’s not very dangerous to use cloud. Or do you install non-open source software from sources you can’t trust? And you keep your credit card numbers secure all the time? And you do backups of your local data, don’t you?

You can find more opinion columns from the writers of Linux User & Developer here, or click here to see what else was in issue 89…

Get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the Apple App Store now!

Last month saw the opening and subsequent closing of the 2.6.35 kernel’s merge window, the period of time during which all of the exciting new features that have been waiting in the wings (and in linux-next nightly kernels provided by Stephen Rothwell) are considered for merging into the official ‘mainline’ kernel source tree by Linus Torvalds. Recent releases have often added a new file system (or perhaps two), but 2.6.35 does not add any new file systems. It does add many other new features, including support for profiling virtual machines from the host machine using ‘kvm perf’, the KDB in-kernel debugger that has augmented the existing KGDB support for remote debugging, the memory compaction patches, and memory hotplug support in the SLAB memory allocator.

A number of other features were of course not added in this release, including the ‘suspend blockers’ from Google’s Android that have been debated heavily on the LKML (Linux Kernel Mailing List) over the past month. Overall, a good set of features and accompanying discussion are on their way to a solid release. “The bulk should be there. And please, let’s try to make the merge window mean something this time – don’t send me any new pull requests unless they are for real regressions for major bugs, okay?” stated Linus Torvalds, on 2.6.35.

Linus Torvalds, continuing in his quest to reduce the post-merge window churn that traditionally saw large numbers of late and last-minute patches flying onto mailing lists, reminded everyone ahead of time that he would be strict in enforcing that only bug (and also regression) fixes, as well as special exceptions, would be taken in after the 2.6.35 merge window closed. That was signalled with the release of 2.6.35-rc1, which Linus introduced almost biblically: “and thus endeth the merge window”. He included a number of useful statistics alongside the release, such as the fact that it contained around 8,500 individual code ‘commits’ (units of source code change typically encompassing a few lines of source, perhaps spread between several files) and that there were about a thousand individual developers involved this time. He also added, in the subsequent rc2 update, that his kids were soon getting out of school for the summer and that he might be busy for a while, so he had an additional reason to not want to see horrible post-merge window breakage.

The new features in 2.6.35 are, of course, typically many and varied. Among them are the memory compaction patches from Mel Gorman that could perhaps also be described as providing support for memory defragmentation (and in fact, briefly, were so). The code here literally attempts to reorganise existing physical memory pages such that they are contiguous and such that free memory within a zone (region of memory) exists mostly toward the end. This avoids the slow physical page fragmentation that over time can make it hard for the kernel to find large regions (orders) of physically contiguous memory. Of course, user applications run in virtual memory wherein the kernel can simply manipulate some page tables that specify their mapping onto the physical pages of RAM, so memory compaction isn’t intended to benefit user applications very directly. It is instead intended for uses such as finding large regions of DMA-safe physical memory that PCI hardware can access directly since such hardware devices do not generally (IO-MMU specifics excepted) have the luxury of using the CPU’s page tables directly.

Turn the page for more including: KMV, Mauro Carvalho Chehab and Android’s suspend blocksers…

Get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the Apple App Store now!

Ubuntu 10.10 (codename Maverick Meerkat) is well into the development cycle, with the planned celebrity release date of 10 October 2010 (10/10/10). The development time frame of Ubuntu is quite condensed, and this is managed with a number of stages. These stages tend to symbolise milestones of the schedule, and usually refer to a restriction on what activity can happen during this time.

We have long passed the Alpha 1 CD image stage and I anticipate the release of Alpha 2 by the time you read this. You will no doubt be eager to test Alpha 3, which is currently scheduled for 5 August. Alpha 3 will be the last CD image created before the development of Maverick starts entering the second half of the ‘freeze’ stages.

The freeze stages we have already passed include ‘Feature Definition Freeze’, which was on 17 June. This is defined as packages in the ‘main’ repository that will have new features in this cycle and will need to be defined by this date. One of the primary reasons for this is that changes in the ‘main’ archive tend to have larger consequential issues for others, as this archive constitutes the bones of everything else. Additionally, the packages are often used as a dependency for other packages in the archives, or packages maintained by others. Therefore, following this stage the work of developers becomes implemented and fixing the current features often constitutes particular versions of particular packages.

Shortly after this, the ‘Debian Import Freeze’ passed on 24 June. Prior to this milestone, newer packages untouched by Ubuntu (solely maintained by Debian) are automatically synchronised into the Ubuntu development version. If a new package is introduced to Debian that hasn’t previously been in Ubuntu, it is automatically copied over as well. Only the source (code) package is copied over – the actual compiled binary package (end result) is built by Ubuntu build servers. This is to ensure that they do actually build with the current Ubuntu archives, and don’t have build conflicts or other issues with the packages already in Ubuntu.

Additionally, by this date Ubuntu developers should have also manually merged packages that have Ubuntu specific changes at least once, which normally brings them to a new upstream version. This means checking that the delta (or difference) between the two versions still needs to exist (removing that patch if it doesn’t), pushing patches to Debian if suitable or continuing to carry them. This can be a rewarding task for community members wishing to get involved in Ubuntu development.

Prior to this date, non-developers (ie anybody without upload access to the archive, or the specific package) can request additional imports if they notice a change in the Debian package and an Ubuntu Archive Administrator will act on it. After this date, it needs to be approved or initiated from an Ubuntu Developer (someone with upload access to that component of the archive).

Continue to page two to learn about the ‘feature freeze’, currently planned for 12 August…

The concept of hosted services was born in the 70’s when businesses used to share computers to avoid the expense of buying their own. In the 80’s businesses had bought their own computers and hired in experts to keep them working. In the 90’s some companies outsourced their entire IT departments whilst others had a combination of their own infrastructure and worked with Application Service Providers (ASPs) who provided access to shared infrastructure, introducing the multi-tenancy model.

This model suffered in the 90’s due in part to the fact that broadband was dial- up, mobile data hadn’t arrived and people were paranoid about security. To get around security fears, some ASP’s gave each customer their own applications, resulting in a huge expense of running individual not multi-tenancy solutions.

The progression went from hosted website, hosted email to hosted applications. You needed the functionality but it wasn’t a business success factor to do it yourself. It just needed to be done right.

In the Noughties, Salseforce.com cracked the ASP model, with a solution that had broad appeal but that didn’t need to be integrated with other applications. This success proved that when the price was right, functionality relevant and internet access was affordable, hosted solutions could work. Salesforce.com also got its own sales right, with heavy investment in sales and marketing that could deploy outside of the IT department or sales prevention department as IT is known be those in SaaS sales teams. Timing played a crucial role, as people were keen to take advantage of CRM technologies; they had the budget but no solution that really delivered. In addition, the rise of hosted email such as Hotmail proved to individuals that hosted solutions could work well.

At the same time, ISP’s began to extend their traditional connectivity and web hosting businesses to offer hosted servers, allowing customers to rent access to dedicated or virtual servers. These servers were loaded with Microsft Windows or Linux and software such as databases, web servers, e-commerce or development tools. All of this was happening before the term “Cloud” was coined, and before big boys like Amazon, Google or Microsoft muscled in.

SME’s today grapple with a range of widely different services that fall under the “Cloud” banner. Typically these can be defined as services that incorporate on-demand payments, self service provisioning and shared service pools. Combined, these create an “elastic” environment that allows resources to be scaled according to need.

This flexibility has appeal, but it comes at a price. The utility is only one component of the price and therefore the cost saving is often dwarfed by the additional costs that a vendor will face in terms of marketing and churn. For example hosting with Amazon’s cloud is twice as expensive as buying the equivalent dedicated service from ServerBeach (www.serverbeach.co.uk). Also Amazon’s system is proprietary. You start with it, you scale with it,and then you’re stuck with it, paying more than you need forever and a day. For many SME’s, requirements are actually relatively stable and predictable. As such the expense of cloud solutions may be a luxury that is not needed. And using standard open source or Microsoft technology means you can move providers if required.

At a recent Tier 1 Research conference, a panel session of 4 businesses “using” cloud revealed the diverse routes that businesses were taking toward the cloud. One business was using Google email, another using VMwarevirtualisation to run 10 physical servers instead of 100 in both the UK and USA rather than just in the UK. The other two included a bank playing internally with virtualization, and a vendor about to supply a public cloud.

There is no doubt that the cloud is relevant for some businesses and some applications, but this is a market that is still evolving. A suppler who can offer a hybrid can give the best advice and make sure you only pay for what you need without being locked in.

Dominic Monkhouse
Dominic is the Managing Director of PEER 1 Hosting, one of the world’s leading IT hosting providers. The company is founded on a high performance 10GB SuperNetwork connected by 17 state-of-the-art data centres, 21 points-of-presence and 10 colocation facilities throughout North America and Europe.

Get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the Apple App Store now!

Need more storage? You could buy a new hard disk, SSD or USB drive. But why not migrate to the cloud and make use of Amazon Web Services (AWS)?

One of the most popular options in this collection of could services is the Simple Storage Service (S3) that provides web-service-based storage.  Users enjoy pay-as-you-go; the cost depends on the region used and the amount of data stored. Pricing starts at $0.15 per GB for the first 50TB per month, $0.01 per 1,000 put, copy, post or list queries, and $0.01 per 10,000 get queries. Above 50TB usage, the price decreases.

The ‘Simple’ in Simple Storage Service doesn’t mean that using this service is simple for the user; it means that there are just a few features provided. S3 is a simple key-based object store. Each object stores up to 5GB, each accompanied by up to 2KB of metadata. The S3 SOAP interface and especially the REST interface are quite useful for web applications, but make it difficult to use S3 as a replacement for a local storage device. Objects are organised into buckets. Because S3 has a flat namespace, every bucket’s name has to be unique. Bucket names and keys are addressable using HTTP URLs.

Requests are authorised using an access control list. Since the start of the S3 service in 2006, lots of applications have been developed with the aim of making working with S3 easier.
Open source applications like s3cmd, S3cp, s3-bash and yas3b allow to upload, retrieve and manage data in Amazon S3 via the command shell.

You enjoy working with a GUI? Try S3Fox, an extension for the Firefox browser that allows you to work with S3 in an easy way. If you want to store files natively and transparently in S3 the same way you work with any local storage device, s3fs is the open source solution. s3fs is a fuse file system that lets you mount an Amazon S3 bucket as a local file system.

Popular file hosting and online backup services like Dropbox, Jungle Disk, ElephantDrive and ExEasy NetCDP use S3 to store the user’s data. The same is true for websites like SmugMug and Slideshare.
S3 comes with a 99.9% monthly uptime guarantee – far more than a single hard disk or NAS can provide. The price makes it worth a try and lots of handy applications exist that help when using S3 as an everyday tool. Give it a try.

Click here to see what else featured in issue 88 of Linux User & Developer magazine…

Get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the Apple App Store now!

Last month Linus Torvalds  released the final 2.6.34 kernel, following a bumpy few weeks that saw a major virtual memory (the subsystem responsible for memory management in-kernel) glitch, the usual round of regressions, and a power outage that knocked vger.kernel.org – the LKML list server – offline for a couple of days. The latest kernel includes the two new file systems mentioned the previous month: Ceph (a distributed file system) and LogFS (used especially in embedded devices, potentially including future mobile phones running Google’s Android); support for asynchronous suspend and resume (also mentioned previously); and faster virtualised networking in KVM and other VMs (virtual machines) thanks to the great work that has gone into vhost net support. You can see a very detailed breakdown of new features on the 2.6.34 kernel on the kernelnewbies.org website.

With the release of 2.6.34 came the opening of the merge window for 2.6.35, during which time we can expect a lot of very interesting patches, which will have have gone through the daily linux-next testing development source tree already. The new features in 2.6.34 and beyond are of course interesting, and there will be more to say about these in future issues, but perhaps the most important thing to come out of this most recent development cycle was a tremendous feeling of satisfaction from many core kernel developers that a particularly nasty VM (virtual memory) glitch was tracked down and fixed as quickly as it was. Linus himself was likely particularly happy to take a break from maintaining the tree to do a lot of very heavy thought and reasoning about VM behaviour.

The problem reports first began to show up a matter of weeks ago, in particular from Borislav Petkov, who would find that every time he resumed his laptop from a suspend-to-disk operation, there would be a nasty kernel crash. He didn’t have a lot to go on at first, and it took a few days of instrumenting kernel code with diagnostic messages, and rebooting over and over to figure out roughly what was happening, but not the why of the matter. The what requires that you understand anonymous VMAs (virtual memory areas). VMAs are the kernel’s means to represent regions of allocated memory in tasks (known as processes to users) and they are what you see in /proc/pid/maps (where pid is any normal user process ID for some running application). The anonymous part refers to VMAs that are not representing a memory mapped file – created with a call to a kernel feature such as mmap – but are instead representing pure ‘anonymous’ memory used by the running process for some data structure, stack, etc.

What was happening during suspend on Borislav’s laptop was that the tracking structures – known as anon_vma_chains – that link related anon_vma structs within the kernel were becoming corrupted such that they were linked in the wrong order. Usually, anon_vma_chains should be wired together from the oldest parent task to the youngest child so that a child process that terminates can release its link in the chain. Instead, they were wired backwards during the resume operation, such that when a child process terminated, the anon_vma its parent was referencing would cease to exist and silent corruption of kernel memory would ensue. The reasons for this corruption stemmed from the new introduction of anon_vma_chains into recent kernels, but the problem case here had been overlooked by everyone involved (virtual memory code is tricky).

After a few days of rolling up his sleeves and getting dirty, Linus was able to track down the problem, give a reason, and thank Borislav as well as the many others in the very long discussion on the LKML for their assistance. While we’re on the subject of virtual memory, the issue of kernel stack size raised its ugly head again last month. The Linux kernel uses a fixed size stack for its own purposes – with a separate stack for interrupt handling on some systems – because automatically growing the kernel stack (as would happen with regular user applications without them even noticing) from within the kernel can be very tricky (especially in low-memory situations). This isn’t usually a problem, but today’s complex uses of the kernel often see very deeply nested unction calls within the kernel – especially when using layered file systems on top of LVM, iSCSI and so forth – and these can push the limits of the fixed size stack, especially if the system is otherwise under what is known as memory pressure as it runs low on available RAM and needs to enter a phase of ‘direct reclaim’ during one of these nested function calls. The usual suggestions have been made to grow the kernel stack, and others (including Mel Gorman) have been working on patches to reduce kernel stack use in general, but it seems that this discussion is not yet over.

Five years of Git
Christian Ludwig noted that it’s now been five years since Linus Torvalds – frustrated by the fallout from use of the proprietary BitKeeper software – wrote the guts of the Git distributed revision control system in the space of about a week. A lot more effort has gone in since then, and Junio C Hamano (as well as many others) continue to do an excellent job further developing and maintaining Linus’s original invention. Git (now at version 1.7.1) is used by a vast number of different open source projects, and tools like gitweb, github and others make collaboration between developers easier than ever before. Christian Ludwig has made a fun video showing Git kernel history, here.

Jon Masters is a Linux kernel hacker who has been working on Linux for almost 14 years, since he first attended university at the age of 13. Jon lives in Cambridge, Massachusetts, and works for a large enterprise Linux vendor. He publishes a daily Linux kernel mailing list summary at kernelpodcast.org.

Click here to see what else featured in issue 88 of Linux User & Developer magazine…

Get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the Apple App Store now!

The first three months of the year were defined, in the technology sector, by some very scary numbers. Just feast your eyes on some of these. Apple, we learned,  pulled in profits in just three months of over $3bn. That’s not in a year – that’s just in a quarter. That number doesn’t account for sales of the iPad either, its latest bizarre contraption, for which the numbers are due to hit in the next round of figures.

Microsoft is happy to play the numbers game, too. It’s been buoyed by the fact that Windows 7 is the most successful iteration of its operating system to date, and that its numbers are trouncing those of Windows Vista. As such, how much has Microsoft – a company that was pruning its staff numbers on more than one occasion last year – brought in in profit for the first three months of the year? Ah, that’ll be a cool $4bn. Four. Billion. Dollars. You could cut VAT with that. By anyone’s measure, they’re scary, scary figures. It gets more staggering if you look at straight revenue before profit. Here, Apple snared $15.68bn. Microsoft’s revenue was $14.5bn.

What also becomes clear from these numbers is that Apple and Microsoft support an ecosystem of firms that rely on them for much of their income. Take the profits being reported at chip-maker ARM, which makes technology for Apple products. Look at the knock-on software purchases that tend to go with a copy of Microsoft Windows. Bluntly, when Apple and Microsoft are bringing in the loot – over $10bn a month between them – there are financial ramifications right down the chain.

It’s little surprise then that there’s not much incentive for retailers and e-tailers to challenge the status quo. We saw this most clearly with the first wave of netbook computers, which infamously arrived with Linux, rather than Windows, installed upon them. And while there’s a degree of smoke and mirrors about the scare stories that followed – higher return rates on machines with Linux installed, for instance – the bottom line became that retail couldn’t wait for Microsoft to sort out its operating system for netbook suitability.

After all, look again at the ecosystem. As has been pointed out by better people than me, a sales rep in PC World will be acutely aware that if they sell a machine with Linux on it, then that’s the last software sale said store is likely to make to that customer. Notwithstanding the fact that open source software is freely and widely available, there’s the fact that you simply wouldn’t need to go back to buy a copy of McAfee Anti-Virus or something of that ilk. You just wouldn’t need to buy software solutions for problems that simply don’t exist in the Linux world.

As such, from a short-term business perspective – and retailers would argue that it’s a long term one as well – there’s little point pushing open source in any kind of environment where one eye is always going to be on what they can sell a person next.

And thus you come back to the kind of retail ecosystem that’s now got to such an advanced stage that two individual companies are controlling massive, massive amounts of business. Bringing in the kind of profits that could chop a solid percentage off the national debt of this fair nation in just a year.
In spite of being a passionate believer in open source, I can’t help feeling that that kind of big battle, fighting the control that Apple and Microsoft exert on their respective markets, is not a winnable one. That’s not an ideal scenario, and that doesn’t mean that open source can’t make and isn’t making major inroads. But it’s one of those instances where a bit of expectation management won’t do much harm…

Click here to see what else featured in issue 88 of Linux User & Developer magazine…

Don’t forget to follow us on Twitter or get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the app store now!

Image: Kenneth Wimer; Licence: cc-by-sa-2.0

As each new developmental cycle commences, there is an Ubuntu Developers Summit (UDS) – it’s a biannual event where many of the developers meet to plan, and create, the goals for the next release of Ubuntu. The attendees are a mixture of Canonical employees, community members (many sponsored by Canonical) and external project representatives.

The summit is always held in a different location, and usually switched between Europe and the USA each time. The latest one was held in La Hulpe (near Brussels), Belgium, to plan the next Ubuntu release Maverick Meerkat.

The intensity of the week coupled with the sessions is hard, serious work. This one had 18 session rooms, with around ten concurrent sessions running during each slot. When I first attended a UDS, I found I attended sessions that I thought would be interesting, but as each summit passes I’m increasingly finding that I start to attend sessions that I am of most use in. This means that I found myself mainly attending the Cloud and Server track.

The design of the summit has ‘tracks’, which this time included Cloud and Server, Community, Design, Desktop, Foundations, Kernel, QA (Quality Assurance), Security, and Ubuntu on ARM. Each session is attached to one track, sometimes more.

However, it is also thoroughly rewarding. There is a great social feel, even though many attendees start to feel like zombies towards the end of the week. The hard work coupled with the social evenings is fantastic.

One thing that really stands out to me is quite how far-reaching the development is distributed. There are attendees from so many countries, I couldn’t possibly keep count. Working, and socialising with these people is very enjoyable. As I arrived on the Sunday I felt a great sense of joy seeing people who I haven’t seen for six months, at the last summit. Catching up on some of the really exciting things is what helps build our community often talking to people with interests that would never have been introduced to me if it were not for this event.

This summit kicked off with a highlight from Mark Shuttleworth giving us a warm welcome and outlining some of the ideas this UDS would bring us. One of the more alarming things he announced was that the 10.10 release will be launched on 10 October 2010 (10.10.10). Our development cycle is already very tight with the amount of work that needs to be achieved, and moving this in by a couple of weeks created a sense of worry in the room. However, at the end of the week, Robbie Williamson (normally the Foundations lead) did some number crunching, some pretty graphs and demonstrated that it really is an obtainable goal.

Each day there are plenary sessions, which are usually in a theatre style where all attendees gather in a large room, rather than the small rooms that each session occupies. We have a presentation from various projects outlining some of the work they have been doing, with one of the plenary sessions being occupied with lightning talks where each talker has an allocation of five minutes; these can be very interesting.

Continued over the page…

Linux User & Developer, one of the nation’s favourite Linux and Open Source publications, is now part of the award winning Imagine Publishing family. Readers can subscribe and save more than 30% and receive our exclusive money back guarantee – click here to find out more.

Don’t forget to follow us on Twitter or get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the app store now!

Google uses MapReduce everywhere for processing huge datasets using a large number of nodes, eg to run PageRank. The MapReduce programming model borrows from functional programming and implements two basic functions: Map and Reduce. Before the map phase can start, the input data is structured in (key, value) pairs.

Map: The master node splits the input into smaller sub-problems and distributes these to the worker nodes. The worker nodes process the smaller problems and pass them back to their master node.
Reduce: The master node takes the answers of the sub-problems and combines them in a way that the original problem is answered.

The Map functions, creating different intermediate values from different input data sets, run in parallel. The Reduce functions also run in parallel, each working on a different output key. All values are processed independently. There is just a single bottleneck. The reduce phase can’t start until the map phase is completely finished.

While MapReduce is the name of the patented software framework and algorithm used by Google, various open source implementations exist. Hadoop is a re-implementation of Google’s proprietary software infrastructure. The Hadoop project was started in 2004 by Doug Cutting and is written in Java. Hadoop re-implements the Google software infrastructure as an open source ecosystem of services to process large-scale data.

Hadoop implements the parallel programming model MapReduce as open source. Hadoop includes its own file system, Hadoop Distributed File System (HDFS). But various file systems are supported, including the Amazon S3 file system and CloudStore. Hadoop also includes a BigTable-model database called HBase, the programming environment Pig and much more. Hadoop can be used for lots of applications, such as distributed grep and sort, document clustering, log file analysis and various kinds of statistics.

The largest Hadoop cluster is run by Yahoo! and consists of 4.000 nodes with 32.000 cores with 64TB of RAM and 16PB of storage. Also, Amazon with its Amazon Web Services, is involved in the MapReduce topic. With Amazon Elastic MapReduce the user can easily start a Hadoop cluster (master and slaves) inside Amazon Elastic Compute Cloud (EC2). The Amazon Simple Storage Service (S3) serves as the source for the data being analysed, and the results are also copied over to S3.

You want your own Hadoop installation at home? Try Cloudera. Its free Hadoop distribution is easy to install and run on top of all up-to-date Linux distributions, inside VMware or inside the Amazon Web Services.

Click here to see what else featured in issue 87 of Linux User & Developer magazine…

Linux User & Developer, one of the nation’s favourite Linux and Open Source publications, is now part of the award winning Imagine Publishing family. Readers can subscribe and save more than 30% and receive our exclusive money back guarantee – click here to find out more.

Don’t forget to follow us on Twitter or get your first digital copy of the magazine for iPhone and iPad free – just search for ‘Linux User’ on the app store!

The past month saw steady progress toward the final 2.6.34 kernel release, including the announcement of initial Release Candidate kernels 2.6.34-rc1 through 2.6.34-rc4. The latter had an interesting virtual memory bug that added a week of delay (I will cover that in a future issue), and of course there was already an incompatible release of the nouveau graphics driver that was covered in last month’s column. But such issues aside, the 2.6.34 kernel is otherwise shaping up to be a good release, including a number of new features of some note as well as fixes for various performance regressions that have affected some of the more recent comparative benchmarks against older releases.

Amongst the new features in 2.6.34 are the LogFS log-based flash file system that has been in the works for the past few years as a replacement for JFFS2/3 (or YAFFS2 even), and support for asynchronous suspend and resume of devices, which should considerably improve system suspend and resume times because it allows for different subsystems to suspend in parallel rather than sequentially. The inclusion of LogFS support will be of particular interest to many embedded developers, since it provides a very scalable journaled flash file system with built-in file versioning support of the kind needed to really support today’s large flash devices, while async suspend and resume will benefit laptop users and those working on lower power devices that need to frequently use suspend.

Regressions have continued to be of particular interest of late as several of the larger vendors gear up for new enterprise Linux releases. Many comparative studies on the LKML allude to developers eager to compare older distribution kernel versions with the latest and greatest upstream releases. A number of such benchmarks – such as one from desktop-focused Phoronix – have shown an overall good progression from 2.6.24 through 2.6.30 (770% improvement overall in the aforementioned Phoronix benchmark), but a much more noticeable regression in 2.6.33 with “PostSQL performance atop the ext3 file system [falling] off a cliff” in that release. In mitigation, Rafael J Wysocki continues to do an excellent job assisting in collecting information on such regressions and nagging the developers involved to post fixes for these bugs via email.

Beyond 2.6.34, current kernel development includes a merging of Intel e820 BIOS provided hardware table parsing support into the more generic LMB – logical memory block – code. This should eventually allow for one single implementation of processing vendor-supplied descriptions of the locations of various parts of physical system memory and devices, rather than the two implementations in common use today (e820 on Intel x86 systems, and LMB on POWER/PowerPC, SPARC and other systems). This work came about following a number of (at times heated) discussions about the need to have a common approach.

There is also more work going into Big Kernel Lock elimination (the formerly giant, coarse-grained locking mechanism from the early SMP days), nested use of VM within KVM, an ability to effectively disable the page cache so that virtualised guest machines don’t need to provide a second page cache in addition to that of their host (which saves memory), and good progress toward a generic union file system mounts option – allowing for overlays of parts of one file system (for example a read-only DVD) with another (for example a writable memory-based) file system. There were of course the usual rants, this month including one started by Ingo Molnar and concerning the relative merits of SELinux as compared with AppArmor or TOMOYO, with the former being compared to a heavy fire door that provides great protection but often needs to be propped open to avoid becoming an unwieldy obstruction for its varied users.

Last month saw the release of the final 2.6.33 Linux kernel, following several months of development (and some last-minute patches – including the network namespace bug this author discovered and wrote about in the previous month). In his release announcement, Linus Torvalds specifically drew attention to newly added support for Nvidia graphics acceleration through the ‘upstreaming’ (a colloquial term for inclusion of code in the official kernel) of the Nouveau 3D graphics driver, as well as the final upstreaming of the DRBD ‘Distributed Replicated Block Device’ driver that has existed for years in a separate project (and has been shipped by at least one enterprise Linux).

There were other changes contained within 2.6.33 though too – of course – including the removal of the AS (anticipatory) I/O scheduler that has long since been replaced by CFQ (the Completely Fair Queuing I/O scheduler), a number of performance counters (perf) improvements, and many other smaller items that we don’t have room to mention here. You can find an excellent community effort to document the new features online here.

Nouveau 3D graphics
Perhaps the biggest end-user visible change in 2.6.33 was the inclusion of the Nouveau accelerated graphics driver for Nvidia chipsets. Nouveau is a community effort to produce an open source alternative driver to Nvidia’s proprietary graphics driver, and work has been ongoing since 2006. The driver at this point is quite capable and in use by a large number of users (more on that in a moment) already, although some issues certainly do remain in spite of the developers’ efforts. Not least amongst these is the lack of a stable ABI (application binary interface) between parts of the driver and the non-kernel ‘user space’ X.Org graphics drivers that actually run your desktop. It might seem curious that the Nouveau developers have not (yet) made an effort to handle versioning of their interfaces, but there is at least one good reason for this: the Nouveau developers never requested that their driver be included in the 2.6.33 kernel. That came from Linus.

To understand why Linus would specifically request a driver be merged, it is necessary to have some historical perspective. Linus is a Fedora user (F12 as of this writing), and Fedora has been shipping support for the Nouveau graphics driver for some time in its own kernels. They are, of course, free to do this – most distributions contain some patches that are not yet in the official kernel, for various reasons – but Linus had specifically raised the lack of this driver being upstream as an item of concern amongst those present at the 2009 Kernel Summit back in the autumn of last year. He seemed frustrated at what he saw as a lack of movement to get the driver upstream once it was being shipped by a large distribution, especially since the maintainer of the 3D graphics subsystem (DRM) had contributed to the Fedora kernel work. All of this meant that when Nouveau was not included in the regular patches sent in for 2.6.33, Linus put his foot down and forced the issue, refusing to take further 3D graphics patches unless they also included Nouveau.

That Nouveau inclusion came back to bite this month as a 2.6.34 patch posting explicitly broke the binary interface between the latest version of the kernel driver and the X.Org user‑space pieces. This would mean that users would have to upgrade both pieces at the same time, rendering it impossible to run older development kernels once the new work was included. Linus posted a series of rants about how he hadn’t been made aware of this fact (which was mentioned in one of the many patches, but not in the customary introductory email used to request their inclusion) that the updates would break compatibility, especially on his own system.
Needless to say, it is never a good idea to break Linus’s own system.

Quoting Linus’s response to the patches, “We can’t just go around breaking people’s setups. This driver is, like it or not, used by Fedora 12 (and probably other distros). It may say ‘staging’ [referring to unstable drivers marked as such in the kernel source], but that doesn’t change the fact that it’s in production use by huge distributions.” Therein began a kernel flamefest in which various people argued about ‘upstream first’ (ensuring code is in the official kernel before it is released in distributions) as a philosophy, and various other progressively less relevant posts to the topic at hand. Perhaps the only topics not explicitly covered were the weather or the global economy.

Over the page we talk about the 2.6.34 merge window…