Everyone has heard the rhetorical parenting phrase “If all your friends jumped off a bridge, would you jump too?” This is how it feels when considering a switch from Buildroot, or a similar pared down Linux build system, to Yocto. More and more chip vendors are switching to Yocto based distributions, and while the benefits for them are clear enough to justify the steep Yocto learning curve, it is a harder sell for smaller companies like DornerWorks. When pressed even Yocto zealots fail to come up with convincing arguments and fall back to, “Trust us, it will make maintenance easier.” or, “It’s really simple once you understand it.” In other words, “Trust us. The ground is soft so just go ahead and jump off the bridge.”
DornerWorks decided to take this leap with our Xen Zynq Distribution, and while DornerWorks has had experience customizing images with Yocto before, this was our first time creating our own custom image and distribution. This blog details some of the background, benefits, and problems with the change, which hopefully will provide better justification for others thinking about switching to Yocto.
Up until the switch, we had been using a custom build framework composed of several Makefiles to compile, test, and build the distribution. Among other things, this required coordinating Petalinux, Buildroot, and a bunch of other tools. It was a complex process that worked fairly well, but required a lot of work and testing in order to make any changes or customizations. Still, it was manageable, and we could have continued to use it for the foreseeable future without too much hassle.
However, with Xilinx’s Petalinux and Yocto integration reaching a mature state in 2016.4, it seemed like the time to take that leap. Additional motivation was provided by the recent addition of an i.MX8 based Xen distribution to our portfolio. NXP’s Linux build system is also Yocto-based, and while we could have adapted our custom build system to handle this additional platform without converting it to Yocto, doing so would have taken a lot of effort just to avoid the inevitable.
Starting the conversion was rough. Just doing things that were simply a shell command in our old build system took hours to figure out how to replicate in Yocto, but after two weeks the pace picked up and the rest of the basic XZD Yocto layer was quickly completed. The initial pace was frustrating, but the overall speed of the conversion was refreshing.
One of the biggest headaches with the old build system was how it handled device trees. This poor device tree handling meant many one-off device trees were created through copy and paste, which resulted in a lot of work when a device tree needed to be updated. Comparing multiple copy and pasted several hundred line device tree files with small differences to the Yocto version of all these device trees, which are a couple device tree fragments, each 20 lines or fewer long, the advantage is clear.
Another thing that our old build system handled less elegantly was fixing errors in upstream repositories. There were two main methods used depending on how the engineer was feeling that day:
Since Yocto seamlessly supports either option, converting these individual upstream fixes was easy. Moreover, if we ever decide to change the method used for an individual fix, switching from patches to a forked repo or vice versa is relatively simple.
One final big benefit of switching to Yocto is the turn-around time. With the old build system changes to the root filesystem essentially required a clean build, or a painful list of manual commands. Yocto takes care of all this work and even allows us to change the output format of the filesystem with a small change to
One of the biggest complaints we have with Yocto is the lack of any kind of real standard for recipes. There are multiple ways to do things with more being added every day, and every recipe seems to choose their own special way of doing it. This makes finding a recipe to use, extend, or base your work from difficult, and prevents understanding of the build process. The latter being especially important if you want to have assurance that you are building the right thing. A lot of developers seem to deal with this by just blindly copying and pasting a recipe and then making a few small tweaks to get it to work for their purposes, but this results in bad habits propagated throughout custom layers.
The next complaint is the often unhelpful error messages followed by screens of unhelpful output that Yocto produces when things go wrong. Unhelpful error messages are a pretty common complaint for build tools, but Yocto takes these error messages to the next level. Occasionally, other information will be available in the output or log file to clear up the confusion, but other times it will only add to it.
The last complaint we wanted to mention goes hand in hand with the above. When getting a strange error and looking for help, there often is not an answer that matches the specific issue in question, or the answer assumes a certain level of Yocto knowledge, so additional research is needed to even understand the answer. In these cases, it usually was helpful to take a step back, find a recipe doing something similar, and look for subtle differences. Frequently, this reveals the source of the problem, but painful trial and error are sometimes necessary.
It has been an interesting process, and while we have only converted part of the distribution over to Yocto so far, we can definitely see how it will improve maintenance and use of the distribution in the future. What’s more, it took less time than we initially thought it would based on our previous exposure to Yocto!
However, some of the more complex operations performed by our build system currently are not supported in Yocto, such as consolidating guests and Dom0 into a completed system package. While we hope to add support for these operations to Yocto someday, they are currently still being handled by a trimmed down version of our original Make based system, which due to the inclusion of Yocto has gone on an extreme diet. Additionally, the psychological benefit to the engineers should not go unmentioned, as deleting entire gigantic ugly Makefile target recipes and replacing them with a simple three-line Yocto recipe was extremely cathartic to those who had fought with these Makefiles before.
The biggest benefit to DornerWorks as a service company is the ease of customization that Yocto provides. In the past if a user wanted to make changes to the distribution, it would require expert knowledge on the user’s part or would require DornerWorks to provide extensive and sometimes problematic instructions. With the switch to Yocto we may still have to provide detailed Yocto explanations instead, but at least the Yocto recipes themselves are somewhat self-documenting and the overall wealth of information available on Yocto is much greater than the expert knowledge previously required.
All our friends were doing it, so we decided we might as well too. It was a big jump, but the landing did indeed turn out to be soft.
Try it out. You know you want to: