Knowledge Distillation for Interpretable Neural Networks

Context

In the context of cyber-physical systems, the deployment of artificial intelligence is restricted to models of limited size. One approach to addressing this problem is offloading AI functionality to the cloud. This, however, introduces a dependency on a typically wireless connection.

As an alternative, the learned representations of a large model (teacher model) are transferred to a smaller model (student model). Within the scope of a thesis, this process, known as Knowledge Distillation (KD), is to be applied to intrinsically explainable AI models.

To this end, the thesis will involve researching and synthesizing the current state of the art on KD in the context of intrinsically explainable AI, with a focus on prototype-based neural networks. Suitable metrics for evaluating the performance of the KD process and the resulting student model are to be identified through literature review and/or designed from scratch. For one selected architecture, a KD concept is to be developed and evaluated against these metrics on an appropriate dataset.

The specific objectives will be defined in collaboration with the applicant and tailored to the type of thesis being pursued.

Targets

• Structured analysis and classification of relevant literature
• Design and implementation of an approach to KD in intrinsically explainable neural networks
• Evaluation and analysis based on predefined metrics

Prerequisites

• Experience in developing machine learning models and PyTorch
• Programming experience in Python
• Background knowledge in probability theory
• Basic knowledge of Explainable AI (XAI) methods
• Familiarity with academic writing and literature review