AI-based Multi-Modal Sensor Fusion for Handwriting Trajectory Reconstruction

Context

Handwriting remains one of the most important and practical skills of modern society. It allows to efficiently capture information of different modalities like text and sketches. However, keeping and navigating the information on paper may be cumbersome and unreliable. Capturing handwriting and sketches using an electronic pen and saving them in digital form takes the best of two worlds, uniting the ease of use with the persistence of information.

There are different ways to capture the handwriting, however, they normally require some external devices for tracking the position of the pen or a special kind of paper, what in turn limits the freedom of notetaking. An alternative approach to enable surface independent handwriting acquisition is the application of AI-based multi-modal sensor fusion techniques, combining the data from compact inertial and image sensors. Visual-inertial positioning proved to be accurate and robust in the robotic and automotive applications, however, integrating these algorithms into an electronic pen poses a particular challenge, considering the limited physical space and compute resources available in the typical Systems-on-Chip for use with TinyML. The goal of this thesis is to develop and evaluate a prototype of the AI-based visual-inertial trajectory reconstruction model suitable for the resource constrained MCU’s.

Targets

Study the state-of-the-art of visual-inertial odometry and multimodal neural networks for trajectory reconstruction, especially applied to handwriting reconstruction, as well as the neural network optimization techniques for TinyML.
Research the concept of handwriting trajectory reconstruction using multimodal neural networks and the optimization pipeline for efficient deployment of such neural networks onto the resource constraint device.
Develop a pen prototype, collect the training data and evaluate the performance of the designed trajectory reconstruction algorithm.

Requirements

Solid Python skills
Experience with Deep Learning frameworks like Pytorch or Keras/Tensorflow
Experience with Embedded C/C++
(Optional) Experience with NN deployment onto resource-constrained devices like microcontrollers