TinyML Made Easy

Motion Classification
Creating a TinyML Anomaly Detection & Motion Classification project with the
Seeed XIAO ESP32S3 and IMU MPU6050

MJRoBot (Marcelo Rovai)

Published May 16, 2023 © MIT

https://www.hackster.io/mjrobot/exploring-machine-learning-with-the-new-xiao-esp32s3-6463e5
Introduction
In my tutorial, TinyML Made Easy: Image Classification, we explored Image
Classification on the new tiny device of the Seeed XIAO family, the ESP32S3
Sense. The Sense has a camera and a mic incorporated, but what happens if
you want another type of sensor as an IMU? No problem! One great
advantage of the XIAO ESP32S3 is its several pins available as an I2C bus
(SDA/SCL pins).

Seeed Studio

Installing the XIAO ESP32S3 Sense on Arduino IDE

Following my last tutorial, you should have the device installed on the Arduino
IDE. If not, let's do a quick review:

On Arduino IDE, navigate to File > Preferences, and fill in the URL:
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pag
es/package_esp32_dev_index.json

on the field ==> Additional Boards Manager URLs

Next, open boards manager. Go to Tools > Board > Boards Manager… and
enter with esp32. Select and install the most updated package:
On Tools, select the Board (XIAO ESP32S3):

Last, but not least, select the Port where the ESP32S3 is connected.

That is it! The device should be OK. To be sure, run the Blink sketch.

Installing the IMU

You could select your IMU from several devices found in the market, such as
ADXL362 (3-axis), MAX21100 (6-axis), MPU6050 (6-axis), LIS3DHTR
(3-axis), or the LCM20600 (6-axis), which is part of the Seeed Grove - IMU
9DOF (lcm20600+AK09918).

For this project, we will use an IMU, the MPU6050 (or 6500), and a low-cost
(less than 2.00 USD) 6-Axis Accelerometer/Gyroscope unit.

In conclusion, I will also comment about using the LCM20600

The MPU-6500 is a 6-axis Motion Tracking device that combines a 3-axis

gyroscope, 3-axis accelerometer, and a Digital Motion ProcessorTM (DMP) in
a small 3x3x0.9mm package. It also features a 4096-byte FIFO that can lower
the traffic on the serial bus interface and reduce power consumption by
allowing the system processor to burst read sensor data and then go into a
low-power mode.

With its dedicated I2C sensor bus, the MPU-6500 directly accepts inputs from
external I2C devices. MPU-6500, with its 6-axis integration, on-chip DMP, and
run-time calibration firmware, enables manufacturers to eliminate the costly
and complex selection, qualification, and system-level integration of discrete
devices, guaranteeing optimal motion performance for consumers. MPU-6500
is also designed to interface with multiple non-inertial digital sensors, such as
pressure sensors, on its auxiliary I2C port.

I2C - Accelerometer & Gyroscope - I2C/SPI

Usually, the libraries available are for MPU6050, but they work for both devices.

Connecting the HW

Connect the IMU to the XIAO according to the below diagram:

● MPU6050 SCL --> XIAO D5

● MPU6050 SDA --> XIAO D4
● MPU6050 VCC --> XIAO 3.3V
● MPU6050 GND --> XIAO GND
 Image from author

Install the Library

Go to Arduino Library Manager and type MPU6050. Install the latest version.
Download the sketch MPU6050_Acc_Data_Acquisition.in:

/*
* Based on I2C device class (I2Cdev) Arduino sketch for MPU6050 class by Jeff
Rowberg <[email protected]>
* and Edge Impulse Data Forwarder Exampe (Arduino) -
https://docs.edgeimpulse.com/docs/cli-data-forwarder
*
* Developed by M.Rovai @11May23
*/

#include "I2Cdev.h"
#include "MPU6050.h"
#include "Wire.h"

#define FREQUENCY_HZ 50
#define INTERVAL_MS (1000 / (FREQUENCY_HZ + 1))
#define ACC_RANGE 1 // 0: -/+2G; 1: +/-4G

// convert factor g to m/s2 ==> [-32768, +32767] ==> [-2g, +2g]

#define CONVERT_G_TO_MS2 (9.81/(16384.0/(1.+ACC_RANGE)))

static unsigned long last_interval_ms = 0;

MPU6050 imu;
int16_t ax, ay, az;

void setup() {

Serial.begin(115200);

// initialize device
Serial.println("Initializing I2C devices...");
Wire.begin();
imu.initialize();
delay(10);

// // verify connection
// if (imu.testConnection()) {
// Serial.println("IMU connected");
// }
// else {
// Serial.println("IMU Error");
// }
delay(300);

//Set MCU 6050 OffSet Calibration

imu.setXAccelOffset(-4732);
imu.setYAccelOffset(4703);
imu.setZAccelOffset(8867);
imu.setXGyroOffset(61);
imu.setYGyroOffset(-73);
imu.setZGyroOffset(35);

/* Set full-scale accelerometer range.

* 0 = +/- 2g
* 1 = +/- 4g
* 2 = +/- 8g
* 3 = +/- 16g
*/
imu.setFullScaleAccelRange(ACC_RANGE);
}

void loop() {

if (millis() > last_interval_ms + INTERVAL_MS) {

last_interval_ms = millis();

// read raw accel/gyro measurements from device

imu.getAcceleration(&ax, &ay, &az);
 // converting to m/s2
float ax_m_s2 = ax * CONVERT_G_TO_MS2;
float ay_m_s2 = ay * CONVERT_G_TO_MS2;
float az_m_s2 = az * CONVERT_G_TO_MS2;

Serial.print(ax_m_s2);
Serial.print("\t");
Serial.print(ay_m_s2);
Serial.print("\t");
Serial.println(az_m_s2);
}
}

Some comments about the code:

Note that the values generated by the accelerometer and gyroscope have a
range: [-32768, +32767], so for example, if the default accelerometer range is
used, the range in Gs should be: [-2g, +2g]. So, "1G" means 16384.

For conversion to m/s2, for example, you can define the following:

#define CONVERT_G_TO_MS2 (9.81/16384.0)

In the code, I left an option (ACC_RANGE) to be set to 0 (+/-2G) or 1 (+/- 4G).

We will use +/-4G; that should be enough for us. In this case.

We will capture the accelerometer data on a frequency of 50Hz, and the

acceleration data will be sent to the Serial Port as meters per squared second
(m/s2).

When you ran the code with the IMU resting over your table, the
accelerometer data shown on the Serial Monitor should be around: 0.00,
0.00, and 9.81. If the values are a lot different, you should calibrate the
IMU.
The MCU6050 can be calibrated using the sketch: mcu6050-calibration.ino.

Run the code. The following will be displayed on the Serial Monitor:

Send any character (in the above example, "x"), and the calibration should
start.

Note that A message MPU6050 connection failed. Ignore this message. For
some reason, imu.testConnection() is not returning a correct result.

In the end, you will receive the offset values to be used on all your sketches:
Take the values and use them on the setup:

//Set MCU 6050 OffSet Calibration

imu.setXAccelOffset(-4732);
imu.setYAccelOffset(4703);
imu.setZAccelOffset(8867);
imu.setXGyroOffset(61);
imu.setYGyroOffset(-73);
imu.setZGyroOffset(35);

Now, run the sketch MPU6050_Acc_Data_Acquisition.in:

Once you run the above sketch, open the Serial Monitor:
Or check the Plotter:
Move your device in the three axes, and you should see the variation on
Plotter:

The TinyML Motion Classification model

For our tutorial, we will simulate mechanical stresses in transport. Our
problem will be to classify four classes of movement:

● Maritime (pallets in boats)

● Terrestrial (palettes in a Truck or Train)
● Lift (Palettes being handled by Fork-Lift)
● Idle (Palettes in Storage houses)

So, to start, we should collect data. Then, accelerometers will provide the data
on the palette (or container).
From the above images, we can see that primarily horizontal movements
should be associated with the "Terrestrial class, " Vertical movements with the
"Lift Class, " no activity with the "Idle class, " and movent on all three axes to
Maritime class.

Connecting the device to Edge Impulse

For data collection, we should first connect our device to the Edge Impulse
Studio, which will also be used for data pre-processing, model training,
testing, and deployment.

Follow the instructions here to install the Node.js and Edge Impulse CLI on your
computer.

Once the XIAO ESP32S3 is not a fully supported development board by Edge
Impulse, we should, for example, use the CLI Data Forwarder to capture data
from our sensor and send it to the Studio, as shown in this diagram:
 Image from author

You can also capture your data "offline", storing them on an SD card or send data
via Bluetooth or Wifi for your computer. In this video, you can learn alternative
ways to send data to the Edge Impulse Studio.

Connect your device to the serial port and run the previous code to capture
IMU (Accelerometer) data, "printing them" on the serial. This will allow the
Edge Impulse Studio to "capture" them.

Go to the Edge Impulse page and create a project.

The maximum length for an Arduino library name is 63 characters. Note that the
Studio will name the final library using your project name and include
“_inference” to it. In my case, the name that I choose at first will not work when I
will try to deploy the Arduino library because it will result in 64 characters. So, I
need to change it, taking out the “-anomaly-detection” part.

Next, start the CLI Data Forwarder on your terminal, entering (if it is the first
time) the following command:

$ edge-impulse-data-forwarder --clean

Next, enter your EI credentials, and choose your project, variables, and device
names:
Go to your EI Project and verify if the device is connected (the dot should be
green):

Data Collection
As discussed before, we should capture data from all four Transportation
Classes. Imagine that you have a container with a built-in accelerometer:
Now imagine your container is on a boat, facing an angry ocean, on a truck,
etc.:

● Maritime (pallets in boats)

● Terrestrial (palettes in a Truck or Train)
● Lift (Palettes being handled by
● Idle (Palettes in Storage houses)
Below is one sample (raw data) of 10 seconds:

You can capture, for example, 2 minutes (twelve samples of 10 seconds each)
for the four classes. Using the "3 dots" after each one of the samples, select
2, moving them for the Test set (or use the automatic Train/Test Split tool on
the Danger Zone of Dashboard tab). Below are the result datasets:
Data Pre-Processing
The raw data type captured by the accelerometer is a "time series" and should
be converted to "tabular data". We can do this conversion using a sliding
window over the sample data. For example, in the below figure,
we can see 10 seconds of accelerometer data captured with a sample rate
(SR) of 50Hz. A 2 seconds window will capture 300 data points (3 axis x 2
seconds x 50 samples). We will slide this window each 200ms, creating a
larger dataset where each instance has 300 raw features.

You should use the best SR for your case, taking into consideration, Nyquist's
theorem, which states that a periodic signal must be sampled at more than twice
the highest frequency component of the signal.

Data preprocessing is a challenging area for embedded machine learning.

Still, Edge Impulse helps overcome this with its digital signal processing
(DSP) preprocessing step and, more specifically, the Spectral Features.

On the Studio, this dataset will be the input of a Spectral Analysis block, which
is excellent for analyzing repetitive motion, such as data from accelerometers.
This block will perform a DSP (Digital Signal Processing), extracting features
such as "FFT" or "Wavelets". In the most common case, FFT.
The Time Domain Statistical features per axis/channel are:

● RMS
● Skewness
● Kurtosis,

And the Frequency Domain Spectral features per axis/channel are:

● Spectral Power
● Skewness
● Kurtosis

So, for example, for an FFT length of 32 points, the resulting output of the
Spectral Analysis Block will be 21 features per axis (a total of 63 features).

Those 63 features will be the Input Tensor of a Neural Network Classifier and
the Anomaly Detection model (K-Means).

You can learn more by digging into the tutorial TinyML under the hood: Spectral
Analysis.

Model Design
Our classifier will be a Dense Neural Network (DNN) that will have 63 neurons
on its input layer, two hidden layers with 20 and 10 neurons, and an output
layer with four neurons (one per each class), as shown here:
Impulse Design
An impulse takes raw data uses signal processing to extract features and then
uses a learning block to classify new data.

We also take advantage of a second model, the K-means, that can be used
for Anomaly Detection. If we imagine that we could have our known classes
as clusters, any sample that could not fit on that could be an outlier, an
anomaly (for example, a container rolling out of a ship on the ocean).
For that, we can use the same input tensor that goes to the NN Classifier as
the input of a K-means model:
Below is our final Impulse design:
Generating features
At this point in our project, we have defined the pre-processing method and
the model designed. Now it is time to have the job done. First, let's take the
raw data (time-series type) and convert it to tabular data. Go to the Spectral
Features tab, and select Save Parameters:

At the top menu, select Generate Features option and Generate Features
button. Each of our 2 seconds window data will be converted into one data
point of 63 features each.

The Feature Explorer will show those data in 2D using UMAP. Uniform Manifold
Approximation and Projection (UMAP) is a dimension reduction technique that
can be used for visualization similarly to t-SNE but also for general non-linear
dimension reduction.

The visualization makes it possible to verify that the classes present an

excellent separation, which indicates that the classifier should work well.
Optionally you can analyze how important each one of the features is for one
class compared with other classes.

Training
Our model has four layers, as shown below:
As hyperparameters, we will use a Learning Rate of 0.005 and 20% of data
for validation for 30 epochs. After training, we can see that the accuracy is
97%.

And for Anomaly Detection, we should choose the suggested features that are
precisely the most important ones found in the Feature Extraction. The
number of clusters will be 32 as suggested by the Studio:
Testing
Using 20% of the data left behind during the data capture phase, we can
verify how our model will behave with unknown data; if not 100% (what is
expected), the result was not that good (8%), mainly due to the terrestrial
class. Once we have four classes (which output should add 1.0), we can set
up a lower threshold for a class to be considered valid (for example, 0.4):
Now, the Test accuracy will go up to 97%.
You should also use your device (which is still connected to the Studio) and
perform some Live Classification.

Be aware that here you will capture real data with your device and upload it to
the Studio, where an inference will be taken using the trained model (But the
model is NOT in your device).

Deploy
Now it is time for magic˜! The Studio will package all the needed libraries,
preprocessing functions, and trained models, downloading them to your
computer. You should select the option Arduino Library and at the bottom,
select Quantized (Int8) and Build. A Zip file will be created and downloaded to
your computer.

On your Arduino IDE, go to the Sketch tab and select the option Add.ZIP
Library and Choose the.zip file downloaded by the Studio:
Inference
Now it is time for a real test. We will make inferences wholly disconnected
from the Studio. Let's change one of the code examples created when you
deploy the Arduino Library.

In your Arduino IDE, go to File/Examples tab and look for your project, and on
examples, select nano_ble_sense_accelerometer:

Of course, this is not your board, but we can have the code working with only
a few changes.
For example, at the beginning of the code, you have the library related to
Arduino Sense IMU:

/* Includes --------------------------------------------------------------- */
#include <XIAO-ESP32S3-Motion-Classification_inferencing.h>
#include <Arduino_LSM9DS1.h>

Change the "includes" portion with the code related to the IMU:

#include <XIAO-ESP32S3-Motion-Classification_inferencing.h>
#include "I2Cdev.h"
#include "MPU6050.h"
#include "Wire.h"

Change the Constant Defines

/* Constant defines ------------------------------------------------------- */

MPU6050 imu;
int16_t ax, ay, az;

#define ACC_RANGE 1 // 0: -/+2G; 1: +/-4G

#define CONVERT_G_TO_MS2 (9.81/(16384/(1.+ACC_RANGE)))
#define MAX_ACCEPTED_RANGE (2*9.81)+(2*9.81)*ACC_RANGE

On the setup function, initiate the IMU, set the offset values and range:

// initialize device
Serial.println("Initializing I2C devices...");
Wire.begin();
imu.initialize();
delay(10);
//Set MCU 6050 OffSet Calibration
imu.setXAccelOffset(-4732);
imu.setYAccelOffset(4703);
imu.setZAccelOffset(8867);
imu.setXGyroOffset(61);
imu.setYGyroOffset(-73);
imu.setZGyroOffset(35);
imu.setFullScaleAccelRange(ACC_RANGE);
At the loop function, the buffers: buffer[ix], buffer[ix + 1], and buffer[ix + 2] will
receive the 3-axis data captured by the accelerometer. On the original code,
you have the line:

IMU.readAcceleration(buffer[ix], buffer[ix + 1], buffer[ix + 2]);

Change it with this block of code:

imu.getAcceleration(&ax, &ay, &az);

buffer[ix + 0] = ax;
buffer[ix + 1] = ay;
buffer[ix + 2] = az;

You should change the order of the following two blocks of code. First, you
make the conversion to raw data to "Meters per squared second (ms2)",
followed by the test regarding the maximum acceptance range (that here is in
ms2, but on Arduino, was in Gs):

buffer[ix + 0] *= CONVERT_G_TO_MS2;
buffer[ix + 1] *= CONVERT_G_TO_MS2;
buffer[ix + 2] *= CONVERT_G_TO_MS2;

for (int i = 0; i < 3; i++) {

if (fabs(buffer[ix + i]) > MAX_ACCEPTED_RANGE) {
buffer[ix + i] = ei_get_sign(buffer[ix + i]) * MAX_ACCEPTED_RANGE;
}
}

And that is it! You can now upload the code to your device and proceed with
the inferences. The complete code is available on the project's GitHub.

If you get an error trying to upload the code to the XIAO ESP32S3 as below,
you should switch off ESP NN acceleration.
To do that, locate ei_classifier_config.h in exported Arduino library folder:
/scr/edge-impulse-sdk/classifier/:

Locate the line with #define EI_CLASSIFIER_TFLITE_ENABLE_ESP_NN 1, and

change it from 1 to 0:
Now you should try your movements, seeing the result of the inference of
each class on the images:
Conclusion
The Seeed XIAO ESP32S is a giant tiny device! It is powerful, not expensive,
low power, and suitable for use on the most common embedded machine
learning applications. Even though Edge Impulse does not officially support
XIAO BLE Sense, we realized it could be easily connected with the Studio.

Regarding the IMU, this project used the low-cost MPU6050 but could also
use other IMUs, for example, the LCM20600 (6-axis), which is part of the
Seeed Grove - IMU 9DOF (lcm20600+AK09918).

One advantage of the last device is that it has integrated a Grove connector,
which can be helpful in teaching in the case you are using the XIAO with an
extension board, as shown below:

You can follow the instruction here to connect the IMU with the MCU. Only
note that for using the Grove ICM20600 Accelerometer, it is essential to
update the files I2Cdev.cpp and I2Cdev.h that you will download from the
library provided by Seeed Studio. For that, replace both files from this link.
You can find on the GitHub project a sketch for testing the IMU:
accelerometer_test.ino.

On the project's GitHub repository, you will find the last version of all codes and
other docs: XIAO-ESP32S3 - IMU.

Knowing more
If you want to learn more about Embedded Machine Learning (TinyML),
please see these references:

● "TinyML - Machine Learning for Embedding Devices" - UNIFEI

● "Professional Certificate in Tiny Machine Learning (TinyML)" –
edX/Harvard
● "Introduction to Embedded Machine Learning" - Coursera/Edge
Impulse
● "Computer Vision with Embedded Machine Learning" - Coursera/Edge
Impulse
● "Deep Learning with Python" by François Chollet
● “TinyML” by Pete Warden, Daniel Situnayake
● "TinyML Cookbook" by Gian Marco Iodice
● "AI at the Edge" by Daniel Situnayake, Jenny Plunkett

On the TinyML4D website, You can find lots of educational materials on TinyML.
They are all free and open-source for educational uses – we ask that if you use
the material, please cite them! TinyML4D is an initiative to make TinyML
education available to everyone globally.
That's all, folks!

As always, I hope this project can help others find their way into the exciting
world of AI!

link: MJRoBot.org

Greetings from the south of the world!

See you at my next project!

Thank you

Marcelo