Perturbations, Optimization, and Statistics (Neural Information Processing series)

In nearly all machine learning, decisions must be made given current knowledge. Surprisingly, making what is believed to be the best decision is not always the best strategy, even when learning in a supervised learning setting. An emerging body of work on learning under different rules applies perturbations to decision and learning procedures. These methods provide simple and highly efficient learning rules with improved theoretical guarantees. This book describes perturbation-based methods developed in machine learning to augment novel optimization methods with strong statistical guarantees, offering readers a state-of-the-art overview.

Chapters address recent modeling ideas that have arisen within the perturbations framework, including Perturb & MAP, herding, and the use of neural networks to map generic noise to distribution over highly structured data. They describe new learning procedures for perturbation models, including an improved EM algorithm and a learning algorithm that aims to match moments of model samples to moments of data. They discuss understanding the relation of perturbation models to their traditional counterparts, with one chapter showing that the perturbations viewpoint can lead to new algorithms in the traditional setting. And they consider perturbation-based regularization in neural networks, offering a more complete understanding of dropout and studying perturbations in the context of deep neural networks.

在幾乎所有的機器學習中，根據當前的知識必須做出決策。令人驚訝的是，即使在監督式學習的情境下，做出被認為是最佳決策並不總是最好的策略。在學習規則不同的情況下，出現了一系列基於擾動的學習方法，這些方法對決策和學習程序進行擾動，提供了簡單且高效的學習規則，並改進了理論保證。本書描述了在機器學習中發展的基於擾動的方法，以增強新型優化方法的統計保證，為讀者提供了最新的概述。

各章節介紹了在擾動框架內出現的最新建模思想，包括擾動與MAP、牧羊和使用神經網絡將通用噪聲映射到高度結構化數據的方法。它們描述了擾動模型的新學習程序，包括改進的EM算法和一種旨在將模型樣本的矩匹配到數據矩的學習算法。它們討論了擾動模型與傳統模型之間的關係，其中一章顯示了擾動觀點可以在傳統情境下引導出新的算法。此外，它們還考慮了在神經網絡中基於擾動的正則化，提供了對dropout的更全面理解，並研究了在深度神經網絡的背景下的擾動。