Representation learning is motivated by the fact that learning systems, both biological and artificial, receive unstructured information from the external world: artificial networks trained for object recognition take collections of pixels as inputs; visual information processing in biological systems starts in photoreceptors, where incoming light is converted into biological signals. To make certain aspects of this information explicit and easily accessible (e.g., the position, dimension and colour of objects in an image), complex processing is required. Central questions are what information should be made explicit in a representation, and how to do so. In this talk, we will approach these problems based on two metaphors. The first one is the cocktail-party problem, where a number of conversations happen in parallel in a room, and the task is to recover (or separate) the voices of the individual speakers from recorded mixtures—also termed blind source separation. The second one is what we call the independent-listeners problem: given two listeners in front of some loudspeakers, the question is whether, when processing what they hear, they will make the same information explicit, identifying similar constitutive elements. Rather than the reconstruction of a ground truth, in the second problem we are interested in comparing the representations extracted by the two listeners. These questions can be studied with the approach of independent component analysis (ICA). This entails establishing whether, under some technical assumptions, representations can be uniquely specified—up to some ambiguities deemed tolerable, and except for a small number of corner cases. In technical terms, this corresponds to characterizing identifiability of the model. We will explore the motivations and objectives of research on identifiability in representation learning, highlighting the challenges and the proposed solutions. Although the talk will be mostly based on literature on (nonlinear) ICA, we will also discuss the implications for unsupervised learning and probabilistic modelling in general.
06/03/2023