On Monday, October 26, at 14h00 in Aula Dal Passo in the Math Department of the University of Rome Tor Vergata, RoMaDS (https://www.mat.uniroma2.it/~rds/events.php) will host
Andrea Agazzi (University of Bern) with the seminar
“Stochastic scaling limits of deep transformer architectures”
Abstract: In this talk, we study the evolution of tokens across the depth of randomly initialized transformer models, modeling them as a system of interacting particles driven by common noise in the infinite-depth and infinite-width limits. We focus on the interplay between self-attention and feed-forward layers and its role in the emergence of clustering. Using stochastic analysis, we identify explicit thresholds for local synchronization and relate them to the sign of Lyapunov exponents governing the separation of nearby tokens.
We encourage in-person partecipation. Should you be unable to come, here is the link to the Teams streaming: