Information Theory, Inference & Learning Algorithms (Hardcover)

Information theory and inference, taught together in this exciting textbook, lie at the heart of many important areas of modern technology - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics and cryptography. The book introduces theory in tandem with applications. Information theory is taught alongside practical communication systems such as arithmetic coding for data compression and sparse-graph codes for error-correction. Inference techniques, including message-passing algorithms, Monte Carlo methods and variational approximations, are developed alongside applications to clustering, convolutional codes, independent component analysis, and neural networks. Uniquely, the book covers state-of-the-art error-correcting codes, including low-density-parity-check codes, turbo codes, and digital fountain codes - the twenty-first-century standards for satellite communications, disk drives, and data broadcast. Richly illustrated, filled with worked examples and over 400 exercises, some with detailed solutions, the book is ideal for self-learning, and for undergraduate or graduate courses. It also provides an unparalleled entry point for professionals in areas as diverse as computational biology, financial engineering and machine learning.

Contents

1. Introduction to information theory; 2. Probability, entropy, and inference; 3. More about inference; Part I. Data Compression: 4. The source coding theorem; 5. Symbol codes; 6. Stream codes; 7. Codes for integers; Part II. Noisy-Channel Coding: 8. Correlated random variables; 9. Communication over a noisy channel; 10. The noisy-channel coding theorem; 11. Error-correcting codes and real channels; Part III. Further Topics in Information Theory: 12. Hash codes: codes for efficient information retrieval; 13. Binary codes; 14. Very good linear codes exist; 15. Further exercises on information theory; 16. Message passing; 17. Communication over constrained noiseless channels; 18. Crosswords and codebreaking; 19. Why have sex? Information acquisition and evolution; Part IV. Probabilities and Inference: 20. An example inference task: clustering; 21. Exact inference by complete enumeration; 22. Maximum likelihood and clustering; 23. Useful probability distributions; 24. Exact marginalization; 25. Exact marginalization in trellises; 26. Exact marginalization in graphs; 27. Laplace's method; 28. Model comparison and Occam's razor; 29. Monte Carlo methods; 30. Efficient Monte Carlo methods; 31. Ising models; 32. Exact Monte Carlo sampling; 33. Variational methods; 34. Independent component analysis and latent variable modelling; 35. Random inference topics; 36. Decision theory; 37. Bayesian inference and sampling theory; Part V. Neural Networks: 38. Introduction to neural networks; 39. The single neuron as a classifier; 40. Capacity of a single neuron; 41. Learning as inference; 42. Hopfield networks; 43. Boltzmann machines; 44. Supervised learning in multilayer networks; 45. Gaussian processes; 46. Deconvolution; Part VI. Sparse Graph Codes; 47. Low-density parity-check codes; 48. Convolutional codes and turbo codes; 49. Repeat-accumulate codes; 50. Digital fountain codes; Part VII. Appendices: A. Notation; B. Some physics; C. Some mathematics; Bibliography; Index.

資訊理論和推論是現代技術中許多重要領域的核心，包括通訊、信號處理、數據挖掘、機器學習、模式識別、計算神經科學、生物信息學和密碼學。這本令人興奮的教科書將理論與應用並重地介紹。資訊理論與實際通訊系統並行教授，例如用於數據壓縮的算術編碼和用於錯誤修正的稀疏圖碼。推論技術，包括消息傳遞算法、蒙特卡羅方法和變分近似，與聚類、卷積碼、獨立成分分析和神經網絡等應用並行發展。獨特的是，本書還介紹了最先進的錯誤修正碼，包括低密度奇偶校驗碼、渦輪碼和數字噴泉碼，這些是21世紀衛星通信、磁盤驅動器和數據廣播的標準。本書豐富的插圖、大量的實例和400多個練習題，其中一些有詳細的解答，非常適合自學和本科或研究生課程使用。它還為計算生物學、金融工程和機器學習等不同領域的專業人士提供了無與倫比的入門點。

目錄:
1. 資訊理論簡介；2. 概率、熵和推論；3. 更多關於推論；第一部分. 數據壓縮: 4. 源編碼定理；5. 符號編碼；6. 流編碼；7. 整數編碼；第二部分. 有噪聲通道編碼: 8. 相關隨機變量；9. 在有噪聲通道上的通訊；10. 有噪聲通道編碼定理；11. 錯誤修正碼和真實通道；第三部分. 資訊理論的進一步主題: 12. 哈希編碼：高效信息檢索的編碼；13. 二進制編碼；14. 非常好的線性編碼存在；15. 關於資訊理論的進一步練習；16. 消息傳遞；17. 在受限無噪聲通道上的通訊；18. 填字遊戲和密碼破解；19. 為什麼要有性？信息獲取和進化；第四部分. 概率和推論: 20. 一個推論任務的例子：聚類；21. 通過完全列舉進行精確推論；22. 最大似然和聚類；23. 有用的概率分佈；24. 精確邊緣化；25. 在格子中的精確邊緣化；26. 在圖中的精確邊緣化；27. 拉普拉斯方法；28. 模型比較和奧卡姆剃刀；29. 蒙特卡羅方法；30. 高效的蒙特卡羅方法；31. 伊辛模型；32. 精確的蒙特卡羅抽樣；33. 變分方法；34. 獨立成分分析和潛變量建模；35. 隨機推論主題；36. 決策理論；37. 貝葉斯推論和抽樣理論；第五部分. 神經網絡: 38. 神經網絡簡介；39. 單個神經元作為分類器；40. 單個神經元的容量；41. 學習作為推論；42. 霍普菲爾德網絡；43. 波茨曼機；44. 多層網絡中的監督學習；45. 高斯過程；46. 解卷積；第六部分. 稀疏圖碼: 47. 低密度奇偶校驗碼；48. 卷積碼和渦輪碼；49. 重複累加碼；50. 數字噴泉碼；第七部分. 附錄: A. 符號；B. 一些物理；C. 一些數學；參考文獻；索引。