In a recent conversation I used the term “modern embedded development” and immediately got called out: What does this even mean? Fair question. In the past I used the phrase “agile software development” a lot. However, the word “agile” has been overused so much that nobody can tell anymore what people mean by it. But, yes, I have replaced one abstract, meaningless word with another. So, let’s clarify.
The way software is developed for devices and vehicles reached its limit. The complexity is so high that the traditional full-upfront design approach is no longer possible. Iterating over the Taylorism-based processes is slow and expensive. This is not unique, but it is especially true in the automotive industry. In addition, over-the-air updates bring a fundamentally changing aspect: software is never done.
The embedded software development in automotive needs to become more iterative, and more incremental. The software practices that have been established in the cloud and web world over the last two decades are exactly that. However, whenever I bring this topic up I get confronted with arguments about real-time, functional safety and regulations. “Your agile schmu doesn’t work here.” As you will see, the following practices and approaches not only do not contradict critical software, but actually promote quality and safety.
T-shaped capabilities
I wish I could say that writing software is an art or a science, but with a few exceptions, it’s mostly a craft. A craft you need to learn and practice. Acquiring experience with different practices and tools is essential to succeed in software development.
In the automotive industry we often see people with I-shaped capabilities, meaning they are experts in a specific technology or domain. To handle the complexity of embedded software development it needs T-shaped capabilities – still expert in one niche, but also having decent knowledge in general software development skills. On an individual level these are a lot of practices that originated from XP like test-driven development, refactoring, clean code, but also technologies like using git on a command line, creating docker containers, writing some Python glue code and being able to evolve the pipelines. Like any craft, it takes time and the right environment to practice. We need to deliberately create environments for this.
Continuous integration and continuous delivery
For most people developing embedded software, doing continuous integration is a less controversial practice. Almost all teams I have worked with have some kind of automation executed in a pipeline. However, continuous integration is about merging all source code changes frequently into a single branch. Now, we can argue about whether frequently means multiple times per day or once per day, but having long-lived feature branches, tens of pull requests dangling for days, is clearly not continuous integration. Every single unmerged change increases the complexity, makes refactoring harder and grows the likeliness of conflicts.
When it comes to continuous delivery, the discussion is very different. In the embedded world, it is often not feasible or even desirable to deploy with the same frequency we are used to from web services. However, continuous delivery is not about deploying everything, it’s about being able to deploy at every point in time with confidence. That is too hard? Maybe, but that should not stop teams from moving into that direction. Processes become more automated, tests are enhanced, configuration management becomes more efficient. Every step taken to get to this state will improve the robustness, the speed and the quality of the involved processes.
Architecture – The Great Decoupling
A common pattern that can be overserved in embedded automotive systems is a tight tangling in the architecture. On the whiteboard the boxes and arrows look so reasonable and clean, but when diving deeper into the actual code, often a very monolithic structure is revealed. While this is clearly not ideal for maintaining and extending software, the biggest issue with this is the thigh coupling to a specific underlying hardware. With a new generation of hardware, most of the time the only way to reuse existing code bases is to “clone and own” them. This means the existing code base gets copied and all places that are tied to the specific hardware are tweaked. As software complexity increases, this becomes increasingly expensive. Architectures should have a hard cut between the hardware-bound and feature software.
But let’s not stop there, most OEMs are because of their size, number of developers and projects in a position where they can make use of an internal open source model, where software components, modules, and libraries are extracted, documented and maintained like open source to be used in projects within the corporation.
Cross-functional collaboration
In order to create software in an iterative and incremental way, we need a setup where all roles needed can collaborate as seamlessly as possible. An established practice in other domains is to have cross-functional teams where all capabilities needed are in a single team. The common established practice in automotive is often the exact opposite, where the people involved in architecture, design, implementation and testing are in separate teams and artifacts are handed over. Discussions and collaboration are often limited to setup meetings. To be fair, it might not be feasible to have all roles involved in the V-Model in every team, but this shouldn’t stop to have a good amount of direct collaboration between different roles, co-located team setup, and pairing.
Shift-left!
Fortunately, the term “shift-left” is discussed more and more in the context of automotive embedded development. The idea is to involve testing and quality assurance early in the software development process. By catching issues sooner, developers can fix them more quickly and reduce costs. It’s about “building quality in” from the start, rather than fixing problems later. In an iterative development process this includes having different levels of automated tests in the development flow. Starting from unit to integration tests to full system tests. While there will always be a need for a certain amount of manual work, the goal is to achieve the same quality with as little effort as possible. To do this, we need to constantly ask what is the fastest way to identify defects and conflicts as early in the development flow as possible. Virtual environments and remote hardware are clearly a significant enabler for this. However, what is often overlooked is that we need to create an intrinsic motivation to put effort into shifting things left. If there is a team that develops an in-vehicle component and the system tests are done by a different team: How can we create a setup where these two collaborate in a way that drives everyone to push for this shift left?
I’m fully aware that “modern embedded development” involves more aspects than the ones mentioned above and everyone will have their own set of what they consider most important. These are mine. I do believe that adapting practices we know from cloud and web development can significantly boost the engineering effectiveness of embedded development and will play a fundamental role as we are moving in the automotive industry into the era of software-defined vehicles.
