Back in 2007 I came up with a really simple but horribly inefficient message bus for passing data between computers in the support fleet. This bus was one of the many pieces baked into the infamous monolithic process known as VehicleDataCenter. Fast forward to 2014 and the current team is still using the same message bus protocol!
For fun I decided to design a new message bus and overall telemetry framework for the Abu Dhabi Solar Challenge.
First I started out with a design document. (Ok, I started out with a prototype and then wrote the design document. :-P) The Data Bus is built on zeromq which itself is built on top of TCP. It provides short-term reliabilty in the form of TCP retransmission and long-term reliability in the form of a chorent record-based data structure. Individual processes and computers can be linked together in a tree configuration and data efficiently (compared to the VDC!) communicated between all nodes.
Actually using the Data Bus is relatively simple. Nodes can create immutable records, and once a record is created a node can send mutable data associated with that record.
The Data Bus provides a mechanism to communicate binary blobs of data. The Wire protocol actually provides meaning to the binary blobs. In this context, a wire refers to one named value to be communicated. Multiple wires can be sent together in what's called a harness.
The record is broken up into two fields, delimited by a null character. The first field consists of a human readable identifier. The identifier can optionally include periods which are taken to describe namespaces.
Example identifiers: "motvel", "can.rx./dev/ttyUSB0", "chat.jbrogers"
The second field consists of a JSON encoded annotation containing metadata and enough structural information to parse any received values.
I broke up the functionality of the Vehicle Data Center into several smaller processes.
A standalone Data Bus process that acts as the upstream-most Data Bus for all other nodes to connect to.
Talks to the telemetry-bridge and dumps raw CAN frames to a known record.
Picks up the CAN frames from the CAN bridge and extracts their fields into Wire records based on a database of CAN messages.
The data viewer is a generic GUI for viewing all the data on the Data Bus. It uses the Wire protocol when possible to make the data human viewable.
A simple chat program that uses the Data Bus for reliable text based communication.
A client that dumps all data to a text file using the old VDC logging format.
A client that converts Data Bus data to the old VDC message bus protocol. This allows existing tools to use the new framework without modification.