Handling Data on Devices

[The following article is authored by one of the Windows Embedded MVPs (Most Valuable Professionals). Our MVPs have a heavy background in Embedded systems and are a great repository of information on Windows Embedded products. We’re providing this space on our team blog as a service to our readers by allowing MVPs to share some of their knowledge with the rest of the community.]

It is not an easy task to find the right data handling strategy on an embedded device. The sweet spot between resource usage and efficient data management from an application perspective is unique to every project. There are a lot of environmental parameters, ranging from hardware capabilities to the availability of network connections, as well the amount of data to be handled.

Stateless, dumb devices

The simplest approach to this topic is to have a device that does not hold any data at all. Very often these devices just serve as a data collection node in a network that provides services to persist data remotely in a database or on a network share. In reality complete stateless devices are not often possible, because every system needs to retain at least configuration information. It starts to get more complicated when devices in the field experience intermittent network connections and therefore have to cope with stateful information locally, until network is available again.

Robust, simple devices

With intermittent network connectivity, the loss of data must be avoided and therefore information needs to be persisted on the device. This can be in a file, in a local database or in a message queue. Writing data to a file may look like an easy solution at first glance, but it should be treated with care. A file is a good place to store e.g. configuration information in XML format, but provides significant disadvantages when using larger amounts of data. There can also be synchronization issues when connections are up again. It is important to recognize that pure persistence of data is not enough to solve the problem of on-/off-line scenarios. Furthermore, synchronization mechanisms are required, which guarantee data consistency, while interacting with available backend services (nowadays called “the cloud”).
Be aware that handling XML files is a very intense operation requiring memory and CPU resources, and should be avoided on lower-end systems.

Of all the solutions listed above, a message queue provides the most comprehensive set of functionalities to solve the problem from an application perspective, as long the data being transported is not too large and “being online” is the standard operation mode for the device.If large amounts of data need to be handled and the system is offline on a regular basis, database storage should be the option of choice. One should be aware that the implementation of synchronization functionality may be required in this case.

Robust, smart devices

It is no secret that in a real world project the ability to make the right compromises is key to success. This leads to application architectures where parts of an overall system workload get distributed onto embedded devices, be it to save bandwidth or to avoid race conditions in the backend. Due to this, some logic as well as data moves from the backend to a device and the data needs to be persisted. In this case, a local database should be the chosen storage approach, because it is the most accessible technology that is also easy to share between different applications running on the device.

Solution building blocks

Looking at the Microsoft solution platform there is a range of different products available, of which some are great fits in one or the other of the described scenarios.

XML handling
is available for native as well as managed code. The XML support includes all popular APIs such as the Document Object Model (DOM) or the Simple API for XML (SAX). As mentioned above, developers should use this infrastructure with the limitations of embedded devices in mind.
Message Queuing

is not available as a separate product, but comes as part of COM+ Services. One needs to include the component with the same name in Target Designer to make this part of a Windows Embedded Standard image.

SQL Server 2008 R2
The full server product fits best into the remote scenario only, because running a full-fledged SQL Server installation exceeds the resources of many embedded devices.
SQL Server 2008 R2 Express Edition
Although originally not designed for the embedded market, Express Edition is very applicable in most embedded scenarios. It provides a large subset of SQL Server`s functionality, but does not scale to be used in multi-user scenarios. It provides replication mechanisms that can be used to update a “big” SQL Server in the backend to enable data consistency even in on-/off-line scenarios. A very compelling “feature” is its price. It is free to be used locally on the embedded device. However, accessing a backend SQL Server, e.g. using replication, requires a CAL (client access license) for this server.
SQL Server 2008 R2 Compact Edition
SSCE is a true embedded, file based, in-process database developed for Windows CE and Windows Mobile devices. It provides sophisticated data synchronization und replication features to a SQL Server backend, which is optimized for intermittent as well as slow network connections. With its small impact on footprint, it is a perfect fit on devices targeting low functionality scenarios. Similar to the Express Edition, the use on the device is free and replication mechanisms require CALs for accessing backend servers. Using a web farm in the backend, architectures using SCCE are able to scale into the range of millions of devices.

If you want to drill into more detail about which functionality is available in which edition of SQL Server, have a look at the informative comparison table located at the Microsoft product web site.

- Alexander

Alexander Wechsler

Wechsler Consulting