In this article, NSSN Engineers Dr Ayu Saraswati and Jimmy Tran describe how the data gathered by smart sensors is transferred to artificial intelligence (AI) applications for processing.

Smart sensors are often battery-powered and are designed to stay in operation for an extended period by consuming the lowest amount of energy possible. This energy efficiency requirement allows the sensors to perform a number of simple computing tasks such as simple additions, subtractions, multiplications and divisions. Complex data operations such as artificial intelligence (AI) are implemented on more powerful processing units that can be found in desktops and small computing systems. These systems are usually servers called edge computing units or fog computing units and have resources such as unrestricted power supply and physical storage space that smart sensors lack.

Let’s take a closer look at how the data is transferred from smart sensors to the more capable computing systems. In most cases, the data is delivered via a communication link (wired or wireless). Before the delivery, smart sensors do some “packing” or data modulation to convert the data from one form to another. The “packing” function ensures a reliable and efficient delivery experience throughout the data flow process. Figure 1 shows the flow process of the data from the environment to the AI and more complex data analytic tasks.

Figure 1: Data flow from the environment to the computing unit where AI and complex data analytic tasks are performed.

After data packing, the computing system (edge, fog or server computing) unpacks the data as it receives and arranges it in sets (matrices, lists of data or dictionary of data) before processing it with an AI application. Depending on the application, the amount of data could range from a few hundred bytes (a few hundred integer numbers) to millions of millions of bytes. For instance, in visual pattern recognition, each image frame could be in the range of millions of bytes and the arrangement might take multiple frames per analysis. Therefore, CPUs and more recently, GPUs (graphical processing units) are employed for ultra-fast processing tasks where real-time detection and processing might be required.

Intermediate nodes as either edge computing or fog computing run inferences to label the sensors’ processed data using machine learning (ML) and AI algorithms. Major ML framework providers often release libraries to run inferences in such intermediate nodes.

There are multiple open-sourced, pre-trained model repositories on the web that can be used in processing the sensors’ data. Most commonly, these models are trained in object detection, face recognition, and speech recognition. The models are typically trained over a long period of time using large datasets and a few hefty computers. The object detection models are often trained using an open-source database. For instance, COCO image dataset consists of 80 different common objects labels that can identify anything from animals and vehicles to people (see Figure 2).

The pre-trained models are a great place to plug in your sensors’ data to AI straight away for a plug and play implementation. Pre-trained models require little investment and can be customised by using transfer learning. However, the limitation of such trained models can propagate to the sensors. For instance, most face recognition models are found to be biassed when it comes to certain facial features.

Figure 2: YOLOV3 Object detection sourced from an open source library.

Training ML models could be a worthwhile investment when dealing with complex problems such as diseases in plants, cancer cells, or cracks in pipes. Whichever models you choose, the capabilities and limitations of ML models can affect the efficacy of smart sensors and the accurate interpretation of their readings. For instance, an object detection model may not cover every existing object (Figure 3, the COCO dataset lacks images of drones or tablets, and the model has mislabelled them as aeroplane and laptop.).

Figure 3: The COCO dataset does not include drone or tablet images so that the model mislabeled the objects.

Once the trained model is exported to an intermediate node or server, the inference process is fast and negligible depending on the complexity of the model and the computing power of the intermediary nodes employed.

Finally, technological advancement in smart sensing, machine learning, and AI may re-define the data flow process from smart sensors to AI. Hardware advances may lead to AI algorithms implemented within the sensors, or the models could evolve to be trained while operational. To learn more about how smart sensors function, please see “Anatomy of a smart sensor” by Jimmy Tran.

The inference time for YOLOV3 object detection model is in milliseconds (ms).

From smart sensors to AI: a data flow journey