ETH-FDS seminar series

More information about ETH Foundations of Data Science can be found here

Modal title

Modal content

Please subscribe here if you would you like to be notified about these presentations via e-mail. Moreover you can subscribe to the iCal/ics Calender.

Autumn Semester 2020

Date / Time

Speaker

Title

Location

19 November 2020
16:00-17:00

Ivan Dokmanić
Universität Basel

Event Details

ETH-FDS seminar

Title	Injective Neural Networks for Inference and Inverse Problems
Speaker, Affiliation	Ivan Dokmanić, Universität Basel
Date, Time	19 November 2020, 16:00-17:00
Location
Abstract	Injectivity plays an important role in generative models where it enables inference; in inverse problems and compressed sensing with generative priors it is a precursor to well posedness. We establish sharp characterizations of injectivity of fully-connected and convolutional ReLU layers and networks. We begin by a layerwise analysis and show that an expansivity factor of two is necessary and sufficient for injectivity by constructing appropriate weight matrices. We show that global injectivity with iid Gaussian matrices, a commonly used tractable model, requires larger expansivity between 3.4 and 5.7. We also characterize the stability of inverting an injective network via worst-case Lipschitz constants of the inverse. Next, we use arguments from differential topology to study injectivity of deep networks and prove that any Lipschitz map can be approximated by an injective ReLU network; we . Finally, using an argument based on random projections, we show that an end-to-end---rather than layerwise---doubling of the dimension suffices for injectivity. We close with numerical experiments on injective generative models showing that injectivity improves inference.
Assets	Video Ivan Dokmanić - ETH-FDS talk on 19 November 2020file_download

Injective Neural Networks for Inference and Inverse Problemsread_more

27 November 2020
16:00-17:00

Matus Telgarsky
University of Illinois Urbana-Champaign

Event Details

ETH-FDS seminar

Title	The dual of the margin: improved analyses and rates of gradient descent's implicit bias
Speaker, Affiliation	Matus Telgarsky, University of Illinois Urbana-Champaign
Date, Time	27 November 2020, 16:00-17:00
Location
Abstract	The implicit bias of gradient descent has arisen as a promising explanation for the good generalization properties of deep networks (Soudry-Hoffer-Nacson-Gunasekar-Srebro, 2018). The purpose of this talk is to demonstrate the effectiveness of a certain dual problem in the analysis of this implicit bias. Concretely, this talk will develop this dual, as well as a variety of consequences in linear and nonlinear settings. In the linear case, this dual perspective firstly will allow a characterization of the implicit bias, even outside the standard setting of exponentially-tailed losses; in this sense, it is gradient descent, and not a particular loss structure which leads to implicit bias. Secondly, invoking duality in the margin convergence analysis will yield a fast 1/t rate; by contrast, all prior analyses never surpassed 1/sqrt{t}, even in the well-studied boosting setting. In the nonlinear case, duality will enable the proof of a gradient alignment property: asymptotically, the parameters and their gradients become colinear. Although abstract, this property in turn implies various existing and new margin maximization results. Joint work with Ziwei Ji. bio: Matus Telgarsky is an assistant professor at the University of Illinois, Urbana-Champaign, specializing in deep learning theory. He was fortunate to receive a PhD at UCSD under Sanjoy Dasgupta. Other highlights include: co-founding, in 2017, the Midwest ML Symposium (MMLS) with Po-Ling Loh; receiving a 2018 NSF CAREER award; organizing a Simons Insititute summer 2019 program on deep learning with Samy Bengio, Aleskander Madry, and Elchanan Mossel. In 2020, meanwhile, he's thankful to be alive and employed.
Assets	Video Matus Telgarsky - ETH-FDS talk on 27 November 2020file_download

The dual of the margin: improved analyses and rates of gradient descent's implicit biasread_more

Archive: SS 24 AS 23 SS 23 AS 22 SS 22 AS 21 SS 21 AS 20 SS 20 AS 19 SS 19