Safety and Security for machine learning - Layer activation tracing analysis

Safety and Security for machine learning - Layer activation tracing analysis

. .


The classical approach for highest safety requirements (ASIL-D for automotive) aims at finding all systematic system and software faults in the development phase. For certain classes of faults, careful work (processes, tools, etc.) can largely achieve this goal. However, for systems that include AI accelerators, there are always sporadic effects resulting in malfunctions or short-term performance degradation. These can occur, for example, as a result of colliding accesses to shared resources such as memory or I/O, or an unrecognized need for synchronization between processes, as well as effects caused by attacks or input distortions. Since the explanation of the black box behavior of an AI accelerator is a new field of research, occurring effects cannot be explained with conventional or researched methods. One attempt to explain classification results of the accelerator would be to look at the activation patterns within an AI accelerator. Using a tool build by ITIV, it is possible to trace layer activations during inference of input images. Evaluating these traces, finding metrics to describe those and using them to make assertions over the operation health of these accelerators is your task.


  • Literature research into AI explainability and accelerators
  • Evaluation and implementation of strategies to evaluate activation traces and decision algorithms on system health
  • Trade-Off evaluation between resource usage and health prediction accuracy

Following task is needed for a ‘very good’ grade:

  • Proposal of strategy for intrusion detection based on layer traces


  • Interest in embedded systems, AI research and new design methods
  • Very good knowledge in Python (preferably with knowledge in AI learning libraries, such as PyTorch, Tensorflow)
  • Ability to work independently

Before the start of the concrete work, an exposé has to be written and approved by the supervisor.

(Pictures generated with Dalle 2)