AI in embedded systems - how it works!

Artificial Intelligence (AI) is currently a hot topic and is being used more and more frequently by both individuals and businesses. With the emergence of Large Language Models (LLMs), AI has entered a new era in recent years. Tasks that seemed unthinkable just five years ago can now be handled effortlessly.

Another type of AI has even defeated the world's best Go player in his own game – a game so complex that it has more possible moves than there are atoms in the universe. Such complexity makes it impossible to calculate all possible variations in advance.

Another impressive example is the development of an AI capable of predicting the folding of proteins – a physics problem that had remained unsolved despite decades of research.

These advancements could contribute to the development of entirely new drugs and revolutionize medicine.

However, these high-performance AI systems require enormous computing power and typically run on supercomputers or powerful graphics cards. But such resources are not always available or practical.

Sometimes, AI applications need to run on small, cost-effective computers or even microcontrollers – devices that often have limited processing power and memory and need to operate on a battery for months or even years.

In recent years, numerous solutions have emerged for such applications. While it is (still) not possible to run a Large Language Model on a microcontroller, many other AI applications have now become feasible.

This article explores the processes involved in AI systems and how they can be implemented on embedded devices.

Both the advantages and disadvantages, as well as the associated challenges, will be discussed.

Why AI makes sense on an embedded device

The use of AI, such as a neural network, on an embedded device offers numerous advantages.

One key benefit is the automatic identification of relevant features from the given data.

Manual extraction is not only time-consuming but also requires deep expertise in both the data and potential algorithms.

Here, AI can assist by efficiently identifying the right features and even recognizing unexpected patterns that might elude human experts.

Running AI directly on an embedded device, instead of a networked application, also provides additional advantages:

• No internet connection required, ensuring privacy protection
• Operates on battery-powered systems
• Real-time capability
• Potentially lower hardware costs

However, there are also some disadvantages:

• Limited computing power: Large models do not run in real time
• Limited memory: Only the smallest models can be used
• Accuracy is often reduced due to smaller models

Development steps of an embedded AI project

1. Collecting Training Data

Collecting high-quality training data is a crucial step in developing an AI model. This process can be quite resource-intensive but directly impacts the model's performance. The data must be carefully selected and representative of all categories the model needs to learn. Poor or insufficient data can make training ineffective and lead to unsatisfactory results.

In supervised learning, which is often applied in embedded systems, data also needs to be annotated (labeling).

In practice, training data is often collected manually. However, there are numerous publicly available datasets and libraries that can be used to save time and resources. These pre-existing datasets are particularly useful for accelerating the development process.

Alternatively, data can also be artificially generated or expanded through data augmentation. This involves artificially multiplying existing data by applying transformations such as rotation, scaling, or mirroring to increase dataset diversity.

The simplest solution is to use a pre-trained model whenever possible. Such models can either be used directly or further improved through additional training with custom data. Another efficient method is transfer learning, where a large, pre-trained model is adapted to specific requirements. This not only saves time but also reduces the need for large amounts of custom training data.

2. Model Selection and Design

In this step, a suitable network type is chosen based on the application. Examples include Convolutional Neural Networks (CNNs) for image processing or Recurrent Neural Networks (RNNs) for time series analysis. The model size must be adapted to the target device’s resources to ensure efficient execution.

3. Training the Model

Model training is typically performed on a high-performance PC or server equipped with GPUs or TPUs. During training, hyperparameters such as learning rate, batch size, and the number of layers are optimized to achieve the best possible performance. The model’s performance is then evaluated using an unseen validation dataset to prevent overfitting and ensure good generalization capabilities.

4. Model Optimization for the Target Device

To adapt the model to the limited resources of an embedded system, various optimization techniques can be applied:

• Quantization: Reducing the precision of weights, for example, from 32-bit floating-point numbers to 8-bit integer values. This saves memory and computing power.
• Pruning: Removing unnecessary neurons or connections to make the model smaller and more efficient.
• Additional Methods: Numerous other techniques exist to reduce network size and complexity, which can be implemented using specialized frameworks.

5. Deployment and Testing on the Target Device

The optimized model is integrated into the embedded system's firmware or software. It must be ensured that the model operates within the system’s memory and processing constraints. Finally, the model undergoes extensive testing to verify its functionality and performance under real-world conditions.

Development Steps of an Embedded AI Project

What happens on the target computer?

As already mentioned, the calculations mainly consist of multiplications and additions. The data is prepared in such a way that it can be processed using matrix operations. Most calculations involve matrix multiplications (the weights of the synapses of a layer) with a vector (the inputs of the neurons in that layer).

Fortunately, GPUs (graphics processing units) are ideal for such calculations, as they perform similar operations in image processing. They allow many calculations to be executed in parallel, significantly increasing efficiency. Over time, specialized hardware solutions such as the TPU (Tensor Processing Unit) and later the NPU (Neuromorphic Processing Unit) were developed, specifically optimized for deep learning.

A lot has also happened in the embedded world. There are more and more dedicated hardware solutions tailored to the computation of neural networks.

The following table presents some of these products in more detail.

Conclusion

AI is no longer limited to powerful PCs or large data centers.

It can also be deployed on embedded systems or even microcontrollers, enabling a wide range of applications.

There are now numerous tools that simplify the implementation of such projects, along with many new hardware developments featuring dedicated AI accelerators. This specialized hardware allows AI computations to be performed efficiently and with low energy consumption.

Well-known models for object recognition have been optimized for use on small devices – often with only minimal precision loss. This makes it possible today to detect or track objects in real time on a microcontroller – all while consuming only a few hundred milliwatts.

These advancements open up completely new possibilities for AI applications in areas such as transportation, Industry 4.0, the Internet of Things, and smart cities.

Our experts are happy to support you with your next embedded AI project, so don’t hesitate to contact us. 

Get in touch now and start your AI-embedded project >

Still unsure?

We will soon publish another article demonstrating a practical implementation of the mentioned technologies on a small microcontroller.