Corporate interest in Linux and Open Source is evolving. While the use of Linux has gained support, one of the major open questions is how best to approach development of products using the Linux platform. On the one hand upstream (the global Linux hacker community) is vibrant and actively working on the core of the Linux kernel. On the other hand, embedded systems often feature unusual devices or need fixes to problems that arise because of hardware configurations not yet commonly available to the developers in the wider community. The temptation is thus to do development for specific products privately within individual organizations, but this vastly complicates future support of the devices.
The Consumer Electronics Working Group of the Linux Foundation has undertaken a study titled “the Kernel Yaminabe” research, an initiative to gather information about the nature of the changes being made to the kernel by the production teams of different vendors using Linux for their various products. Interest has arisen as to what work was done by these vendors to the original “upstream” code to create the trees being used, and what can be learnt about the practises employed.
To this end, Operational Dynamics has carried out this project on behalf of the CEWG community. Our results are discussed here.
Methodology, Approach and Results
The premise was straight forward:
A yaminabe in Japanese is a pot dish with many ingredients each brought by different people mixed in. And so our yaminabe was made with code from a variety of sources. The study was specifically restricted to devices running the 2.6.35 Linux kernel as used in the Android Gingerbread platform. Thus we considered code from publicly available kernels from currently shipping devices.
Our review began with a surface inspection of trends within the codebases provided, followed iteratively by reviews at increasing levels of detail. On each pass we will considered a variety of factors including the areas of the kernel being changed, whether the work done appeared to have been motivated by general feature development, adding drivers for new hardware, diagnostics for troubleshooting, or perhaps fixing platform specific bugs.
As we identified the provenance of the code we were working with, we were able to undertake a range of different analytic techniques.
What lessons can we learn? While we present the raw information backing our analysis on this site, the broader themes of what practises were being employed by the people working on different areas of the tree and consideration as to what the effectiveness of these practises has been over time gives us findings of hopefully wider interest.
The Linux kernel is, of course, a huge piece of software; rather than attempting shallow comprehension of the whole, we concentrated on exploring specific issues in greater depth.
As the point of this project is to highlight specific practises without drawing unnecessary focus to the identity of the individual companies’ engineering teams — after all, the inefficiencies are endemic to the industry as a whole and everyone is struggling with similar frictions. Our results are therefore anonymized and we used codenames for the individual vendors’ products.