Modern Development Process for Embedded/IoT Devices

Savoir-faire Linux
Savoir-faire Linux
Developing with open source based products

Conception and development of embedded devices is never a trivial task, especially in a constantly changing world in which any object interacts, either physically or remotely, with multiple human-driven or autonomous systems.

When designing a product, you have to define:

Added ValueThe added value the product is bringing
User ExperienceThe user experience
FeaturesThe product’s main features
TechnologiesThe relevant technologies and methods to achieve a maintainable system
Software StackThe software stacks adapted to your team members’ skills

The scope is still wide even if we focus only on the technical side.

And that breakdown is applicable for each aspect of the product design: from manufacturing to production going through maintenance and customer support without forgetting certifications, compliance and Q/A.

Fortunately, modern HW and SW platforms such as Toradex modules and the Torizon platform help engineers and designers in worrying less about a significant part of these topics, hence saving precious months of work by using “on-the-shelf” components.

Product, Process Management

But, despite all the existing components, the “heart” of the product value, which is the business logic and what problems / needs the product is attempting to address are still to be implemented.

This part is still the most challenging one and probably the most complicated.

Dealing with product development complexity

Many studies in the Engineering field have been done to identify development challenges. Professor Fred Brooks, famous for his book “The Mythical Man-Month”, coined the following terms to quantify them in software product development:

Product Development Complexity
  • Essential complexity
    This is the minimal complexity you have to deal with to develop your product. This the primary focus on product development and summarize basically as the core of the problems your product is trying to address.
  • Accidental complexity
    Accidental complexity is introduced all along with the project by external factors and our own mistakes, shortcuts we took to face unexpected issues or to match with a deadline. Inappropriate processes or unskilled staff can also feed the growth of accidental complexity.

Unlike essential complexity, accidental complexity is a never-ending phenomenon. After a while, it gets even bigger than the essential one if no measure is taken or process in place to limit it.

By letting it grow uncontrolled, each release will deliver lower and lower value than the former one as the effort and the investment required to counteract the side-effects of “accidental complexity” gets more and more important.

Maximizing the value on each release

Minimizing system complexity is a way to release many features at a constant resource amount. As essential complexity is the product core, reducing essential complexity is limited as it has a direct impact on the product core.

Mitigating accidental complexity has a better return on investment. The following chart shows that the cost of change is kept constant by having accidental complexity in check after a while whereas having no mitigation method induces an exponential cost increase.

Cost of Change - Development Time

A few decades of experience in software development have brought some tools and methods to help us. The focus on each item is :

  • Having short feedback loops
    Short development loops give rapid feedback where the absence of it will have an impact down the chain. The sooner the better.
  • Improving team strategies
    What can be done inside the team to minimize the growth of accidental complexity

Based on those two items, we will go through the main points that are beneficial for keeping the “accidental complexity” under control.

Code Review

Having a second perspective limits the developer blind spots and as a consequence limits defects introductions in the early phase.

For sure, this can be done during regular meetings but having a streamlined code review tool will ease and make the process almost effortless for the team. There are many open and non open source alternatives. Tools such as Crucible, Gerrit or Gitlab provide good results.

Code review using Gerrit and Sonarcube Source: Code review using Gerrit and Sonarcube

Reviewing is also a way to share the knowledge and reduce knowledge silos. Sharing also helps developers in synchronizing their perception on the product design and the actual user experience that is being implemented. This will eventually be leading to brainstorms and discussions that can guide them towards improving the product’s quality.

In a second step, some repetitive code review checks can be automated. If you have coding guidelines to fulfill such as MISRA or Linux coding style, those steps can be done by a code analysis tool. The team would re-orient its effort on the higher value tasks and delegate these checks to the tools. Sonarcube, Code sonar or Polyspace are some of the industrial grade alternative.

Source code Versioning
Source code VersioningSource: Code source history using git

Software engineering is a long and complex ongoing process. Design choices are done at every step and it is almost impossible to keep all data in mind, especially after a 6 months context switch.

Get rid of the project history and context will often lead to reinvent the wheel after a while. Former uses cases may be forgotten which would induce complexity in the project maintenance.

Tools and processes from leading open source projects (Linux, Git, etc... ) address those challenges. An adequate use of code versioning tool helps the team to keep in mind the background of every single code commit.

Integrating testing the build environment

Modern CI/CI/CD platforms allow deploying and running the test suite on your hardware target for each commit. This process will spot regressions and warn the developer.

Many tests frameworks are available. Most of them offer mock-ups and testing features that can be integrated in a build environment. For C application, Cmocka offers test capabilities that can be integrated in the application codebase.

If you are using another language, there are other alternative (Robot framework, Cukinia, etc ...)that are more relevant to your context.

Build environment with integrated tests using JenkinsSource: Build environment with integrated tests using Jenkins
Agile development approaches

The aim of this chapter is not to give a comprehensive overview. Having processes will help your team to provide quality outcomes and reduce the general technical debt.

The intend is to let the reader know that Agile approaches (Kanban, XP, Scrum etc.) provides some “ready to use” processes. Some might not be relevant to your specific context but you’d rather have a look at it before inventing a new process. Our perspective would be to remind their intend:

  • Build the product right and build the right one
  • Deliver value on time
  • Keep motivation and productivity high
Agile development approaches
What's Next

There is unfortunately no magic spell. Setting up new tools and new processes requires effort and resources. Some methods require less effort than others but even using a single one will benefit the team outcomes.

Those changes are often a trade-off between the expected project velocity, available resources (either financial, human or both) and deadlines.

Having external support boosts the training process and helps to avoid the common pitfalls in setting up new development processes.

Training Process


Frederic Brooks publications

Yohann Desiles
, Savoir-faire Linux, Montréal (QC)


Please login to leave a comment!
Have a Question?