Operational Dynamics
Technology, Strategy, and IT Operations Consulting   |   Open Source Research and Development   |   Blogs


Can Linux development in the embedded industry be improved? We evaluate the suitability of the Long Term Support Initiative in light of current engineering practices.


A research study undertaken for the Linux Foundation's Consumer Electronics Working Group by Operational Dynamics. Linux Foundation

Project Background


Corporate interest in Linux and Open Source is evolving. While the use of Linux has gained support, one of the major open questions is how best to approach development of products using the Linux platform. On the one hand upstream (the global Linux hacker community) is vibrant and actively working on the core of the Linux kernel. On the other hand, embedded systems often feature unusual devices or need fixes to problems that arise because of hardware configurations not yet commonly available to the developers in the wider community. The temptation is thus to do development for specific products privately within individual organizations, but this vastly complicates future support of the devices.

The Consumer Electronics Working Group of the Linux Foundation has undertaken a study titled “the Kernel Yaminabe” research, an initiative to gather information about the nature of the changes being made to the kernel by the production teams of different vendors using Linux for their various products. Interest has arisen as to what work was done by these vendors to the original “upstream” code to create the trees being used, and what can be learnt about the practises employed.

To this end, Operational Dynamics has carried out this project on behalf of the CEWG community. Our results are discussed here.

Methodology, Approach and Results

The premise was straight forward:

In order to improve development practises for embedded systems, compare and contrast several existing Linux kernel trees with a view to understanding the major themes in how they have diverged from 'mainline' and from each other.

A yaminabe in Japanese is a pot dish with many ingredients each brought by different people mixed in. And so our yaminabe was made with code from a variety of sources. The study was specifically restricted to devices running the 2.6.35 Linux kernel as used in the Android Gingerbread platform. Thus we considered code from publicly available kernels from currently shipping devices.

Our review began with a surface inspection of trends within the codebases provided, followed iteratively by reviews at increasing levels of detail. On each pass we will considered a variety of factors including the areas of the kernel being changed, whether the work done appeared to have been motivated by general feature development, adding drivers for new hardware, diagnostics for troubleshooting, or perhaps fixing platform specific bugs.

As we identified the provenance of the code we were working with, we were able to undertake a range of different analytic techniques.

What lessons can we learn? While we present the raw information backing our analysis on this site, the broader themes of what practises were being employed by the people working on different areas of the tree and consideration as to what the effectiveness of these practises has been over time gives us findings of hopefully wider interest.

The Linux kernel is, of course, a huge piece of software; rather than attempting shallow comprehension of the whole, we concentrated on exploring specific issues in greater depth.

As the point of this project is to highlight specific practises without drawing unnecessary focus to the identity of the individual companies’ engineering teams — after all, the inefficiencies are endemic to the industry as a whole and everyone is struggling with similar frictions. Our results are therefore anonymized and we used codenames for the individual vendors’ products.

These documents copyright © 2010-2012 Operational Dynamics Consulting, Pty Ltd unless otherwise noted. Copyright of work by other authors is retained by those authors. Most material on this site is made available under an Open Source or licence; see the top level COPYING file in that project's source repository for details

This page was generated from a text document! We use John Gruber's Markdown syntax as ported to PHP by Michel Fortin

Files on this site (notably program code) will be delivered as Content-Type: text/plain unless binary. All times UTC