On Wednesday, February 8th, at 14h00 in Aula Dal Passo at Roma Tor Vergata, RoMaDS (https://www.mat.uniroma2.it/~rds/about.php) will host Francesco Tudisco (GSSI - Gran Sasso Science Institute) with the seminar

"Efficient training of low-rank neural networks"

Abstract:

Neural networks have achieved tremendous success in a variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. At the same time, overparametrization seems to be necessary in order to overcome the highly nonconvex nature of the optimization problem. An optimal trade-off is then to be found in order to reduce networks' dimensions while maintaining high performance. Popular approaches in the literature are based on pruning techniques that look for "winning tickets", smaller subnetworks achieving approximately the initial performance. However, these techniques are not able to reduce the memory footprint of the training phase and can be unstable with respect to the input weights. In this talk, we will present a training algorithm that looks for "low-rank lottery tickets" by interpreting the training phase as a continuous ODE and by integrating it within the manifold of low-rank matrices. The low-rank subnetworks and their ranks are determined and adapted during the training phase, allowing the overall time and memory resources required by both training and inference phases to be reduced significantly. We will illustrate the efficiency of this approach on a variety of fully connected and convolutional networks.

The talk is based on:

S Schotthöfer, E Zangrando, J Kusch, G Ceruti, F Tudisco

Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations

NeurIPS 2022 https://arxiv.org/pdf/2205.13571.pdf

We encourage in-person partecipation. Should you be unable to come, here is the link to the event on Teams:

https://teams.microsoft.com/l/meetup-join/19%3arfsL73KX-fw86y1YnXq2nk5VnZFwPU-iIPEmqet8NCg1%40thread.tacv2/1675333086807?context=%7b%22Tid%22%3a%2224c5be2a-d764-40c5-9975-82d08ae47d0e%22%2c%22Oid%22%3a%22650fc4a8-4cec-4bd2-87bc-90d134074fe6%22%7d

The seminar is part of the Excellence Project Math@TOV.

You can find a schedule with the next events at the following link: https://www.mat.uniroma2.it/~rds/events.php .