In my last post I presented the case for developing embedded systems in small, robust modules by breaking up functionality among several processes or even separate circuit boards. Once logic has been separated into isolated silos, the subsystems must still work together as a whole via some form of communication.
In this post I’ll cover some of the things to consider when designing or choosing inter-process communication protocols based on my experience on a couple different projects.
It’s generally considered bad programming practice to use global variables because it can make code quickly spiral into a spaghetti mess. Why then do some developers often share info between embedded processes with shared memory buffers or files that are accessible whenever and by whomever? In certain constrained embedded devices, this may be the only viable option, but in general, it’s far better and more maintainable to have a clean interface that code calls into to interact with another area of the system.
It’s easier to develop (and far easier to test) against common interfaces as defined by an interface control document. When designing a system, after separating out subsystems by functionality, it’s crucial to define clean interfaces that each subsystem or process will use. This applies to both processes on the same board and different boards communicating over a bus. In general, there are three main areas to consider when developing or selecting a shared communication strategy: serialization, routing, and transport layers.
Serialization is the process of converting an internal data structure of one system to a message format that can be communicated over a channel to another system in a way that other system can understand it. In order to have one subsystem understand another they need to speak the same “language.” Some considerations here are:
On one project we developed our own custom streamed header/payload format that used code generation scripts to create serialize/deserialize functions (similar to Google’s Protocol Buffers project). For another, we used JSON as a fast, flexible and human readable format that was easy to prototype quickly. For some projects we’ve also used XML (for persistent storage, not message communication), which is human readable but can be validated against a schema. Lastly, on another project we’ve used C#’s reflection and Marshalling plus COBS encoding to serialize Bluetooth data.
In each project, once the serialization scheme was designed and in place, it was trivial to add or update messages as development progressed. This allowed us to rapidly prototype each subsystem without needing to constantly fight over changing internal memory layouts of each subsystem, since a common area of code defined the shared serialization language of the various subsystems.
With multiple subsystems that a message could go to, the next thing to consider with message protocols is the routing. Routing defines those message paths. In this realm, consider:
In one project, we defined the subsystem routing defaults as a property of the message type (though the sender could override this) and copies of the messages were sent directly to each application by the sender. On another project we used MQTT’s publish/subscribe framework so receivers subscribed to topics of interest and senders broadcast to the listening receivers via a broker.
By pruning routes so that only certain messages were sent and received by each subsystem, we were able to decouple subsystems’ interdependence and reduce the required amount of testing. Adapting our routing definitions, we could easily and dynamically re-route messages between test applications and the subsystems they tested, ensuring the inputs and outputs of each subsystem behaved correctly.
At this point our subsystem has a message serialized and knows where it’s going, but in order to actually get there, the system needs some sort of transport layer. This layer can be an existing bus like UART, CAN, Bluetooth or Ethernet, or something internal or custom. Some points to consider when choosing a transport layer are:
In Linux, it’s common to use sockets, file pointers, or pipes for inter-process communication. However, using TCP or UDP packets on a local network interface can work well too and can make development easier by being able to sniff traffic or inject new messages dynamically on the target from another PC on the network (though make sure to disable any remote access in the release build for security reasons). One of my projects used UDP packets under Linux while using the same serialization for RS-232 serial communication between different boards: a proxy application acted as a bridge that retransmitted messages from one transport layer to the other.
A less traditional approach that worked well on a few of projects I’ve been on was to use MQTT as implemented by the program “mosquitto.” (see this post for more details). In addition to the publish/subscribe routing it provided, it added several extra bonus features automatically such as delivering retained messages to processes that started listening only after the sender already sent the message, and notifying other applications that an app exited via a last-will message.
Modular embedded applications require each subsystem to communicate well with each other to get the job done. If communication is done right, the advantages of such systems over traditional monolithic applications are immense. Not only can a developer or tester monitor or inject messages between subsystems dynamically, but entire sessions of inter-process communication can be recorded and played back for reliable test case automation. Modular embedded architectures help cut development and testing costs while improving maintainability and reliability.
Break your large applications down into bite-sized apps then glue them back together with clear communication protocols. They will be stronger than if they stayed in one piece.