Federated Learning: Current landscape and challenges
TL;DR: Presentation of tasks and challenges in FL, state-of-the-art work and open challenges.
Abstract:
DNN training has traditionally been conducted in a centralised setting, with data and computational resources residing in large data centers. This has largely been enabled by technological leaps in accelerator design and cheap storage. However, users and regulators have been pushing for increased privacy in data collection. This, coupled with computational advances embedded hardware and SoC design have rendered training on-device a tractable alternative.
Federated Learning has been proposed as an alternative, privacy-preserving design for training DNNs in a more decentralised manner. Popularise by its adoption in Google keyboard, FL has been gaining more and more traction nowadays. However, it also brings new challenges on the table, including data (non-IIDness) and system heterogeneity. Tackling these in a robust and fair manner requires bespoke solutions that span across the areas of distributed optimisation, personalisation, as well as on-device learning and secure computation.
In this talk, we first present the basics of Federated Learning going through alternative architectures and designs. We explain the main challenges of training DNNs in a federated manner, both at system and learning level, and talk about different optimisation and aggregation techniques used in the literature. Last, we visit tangential topics such as personalisation, multi-task and continual learning as well as relevant privacy preserving techniques, such as Secure Multi-Party Computation and Differential Privacy.