強化學習
賈如春 總主編；王琦，陶曉玲，蒲源 主編；楊振傑，楊毅遠，馬國政，牛雅哲 副主編
- 出版商: 清華大學
- 出版日期: 2026-07-01
- 售價: $359
- 語言: 簡體中文
- 頁數: 215
- ISBN: 7302720746
- ISBN-13: 9787302720744
-
相關分類:
Reinforcement
下單後立即進貨 (約4週~6週)
商品描述
本書系統介紹強化學習領域的理論基礎、關鍵算法及其在實際場景中的廣泛應用，以深入淺出的方式全面覆蓋強化學習領域的核心知識體系與技術方法，包括強化學習的基本概念、經典算法、深度強化學習、獎勵空間探索、模仿學習、基於模型的強化學習及該領域的**研究進展。此外，本書還將對視覺強化學習、多任務強化學習、蒙特卡洛樹搜索、通用智能體等重要方向進行深入分析，並結合豐富的實際案例與代碼實現引導讀者從算法原理過渡到實際應用，幫助讀者快速掌握強化學習技術並解決復雜的實際問題。 本書理論與實踐並重，適合作為高等學校人工智能、數據科學、機器人控制等相關專業本科生與研究生的教材，也可供對強化學習感興趣的工程師、研究人員及開發者閱讀。
作者簡介
賈如春，男，副教授，CISP信息安全高級講師，信息安全國賽技能大賽裁判，《全國高等教育信息安全人才培養規劃叢書》編委會主任，先後發表國家級核心期刊論文80余篇，獲得國家發明軟著專利20余項，主持、參與省市級科研項目8項，獨著、主編《網絡安全實用教程》、《數據安全與災備管理》、《信息安全基礎》、《計算機病毒與防禦》等20余本國家高等教育信息安全人才培養專業規劃系列教材。
目錄大綱
目錄
第 1章緒論 ........................................................................................................... 1
1.1搜索最優解的不同方式 ................................................................................. 2
1.2用強化學習尋找最優解 ................................................................................. 4
1.3用深度學習建模各類非線性單元.................................................................... 5
1.4標準化研究問題的形式 ................................................................................. 6
1.5 Gym實踐 ................................................................................................... 7 參考文獻............................................................................................................11
第 2章 Q-learning和 SARSA.............................................................................12
2.1問題背景 ....................................................................................................12
2.2 Q-learning..................................................................................................12
2.2.1介紹 ................................................................................................12
2.2.2更新規則 .........................................................................................12
2.2.3偽代碼.............................................................................................13 SARSA ......................................................................................................13
2.3
2.3.1介紹 ................................................................................................13
2.3.2更新規則 .........................................................................................14
2.3.3偽代碼.............................................................................................14
2.4同策略與異策略的理解 ................................................................................14
2.5 Q-learning與 SARSA的對比 ......................................................................15
2.6實現：CliffWalking環境 .............................................................................15
2.6.1 Q-learning實現 ...............................................................................16
2.6.2 SARSA實現....................................................................................18
2.6.3測試與對比 ......................................................................................21 參考文獻............................................................................................................21
第 3章深度學習基礎 .............................................................................................22
3.1深度學習概述 .............................................................................................22
3.2神經網絡的基本構成 ...................................................................................23
X|強化學習
3.2.1 神經元與激活函數 ............................................................................23
3.2.2 層結構與前向傳播 ............................................................................24
3.2.3 損失函數 .........................................................................................25
3.3反向傳播與優化算法 ...................................................................................25
3.3.1 梯度下降算法...................................................................................26
3.3.2 其他優化算法...................................................................................27
3.4卷積神經網絡 .............................................................................................28
3.4.1 卷積層.............................................................................................28
3.4.2 池化層.............................................................................................28
3.4.3 全連接層 .........................................................................................29
3.5循環神經網絡 .............................................................................................29
3.5.1 RNN的結構 ....................................................................................29
3.5.2 LSTM和 GRU ................................................................................30
3.5.3 CNN、RNN、LSTM和 GRU的示例代碼與分析...............................31
3.6 深度學習框架 .............................................................................................33 TensorFlow......................................................................................33
3.6.1
3.6.2 PyTorch ..........................................................................................34 Keras ..............................................................................................35
3.6.3
3.7深度強化學習 .............................................................................................35
3.7.1 深度 Q網絡 ....................................................................................35
3.7.2 異步優勢 Actor-Critic ......................................................................36
3.7.3 近端策略優化...................................................................................36
3.8總結...........................................................................................................37 參考文獻............................................................................................................37
第 4章深度 Q網絡 ..............................................................................................39
4.1問題背景 ....................................................................................................39
4.2深度 Q網絡 ...............................................................................................39
4.3 DQN改進版本 ...........................................................................................40
4.3.1 雙重 DQN .......................................................................................40
4.3.2 雙流 DQN .......................................................................................41
4.3.3 優先經驗回放...................................................................................41
4.3.4 彩虹 DQN .......................................................................................42
4.3.5 噪聲 DQN .......................................................................................42
4.3.6 分布式 DQN....................................................................................43
4.4實現: Cartpole ...........................................................................................44 參考文獻............................................................................................................46
目錄 | XI
第 5章策略梯度....................................................................................................47
5.1問題定義 ....................................................................................................47
5.1.1目標策略的定義 ...............................................................................47
5.1.2優化目標的定義 ...............................................................................47
5.1.3從值函數到策略梯度優化 ..................................................................48
5.2核心思想 ....................................................................................................48
5.2.1策略梯度推導...................................................................................48
5.2.2引入累計獎勵...................................................................................50
5.2.3引入動作值函數 ...............................................................................50
5.2.4梯度上升與策略更新.........................................................................50
5.2.5總結 ................................................................................................51
5.3 REINFORCE算法 .....................................................................................51
5.3.1算法背景 .........................................................................................51
5.3.2算法推導與更新規則.........................................................................52
5.3.3算法流程 .........................................................................................52
5.3.4優缺點分析 ......................................................................................53
5.4 Actor-Critic方法 ........................................................................................53
5.4.1算法背景 .........................................................................................53
5.4.2 Actor的更新規則.............................................................................53
5.4.3 Critic的更新規則 ............................................................................54
5.4.4算法流程 .........................................................................................55
5.4.5優缺點分析 ......................................................................................55
5.5 Advantage Actor-Critic方法 .......................................................................55
5.5.1算法背景 .........................................................................................55
5.5.2優勢函數與 TD-誤差 ........................................................................56
5.5.3更新規則 .........................................................................................56
5.5.4算法流程 .........................................................................................56
5.5.5優缺點分析 ......................................................................................57
5.6實現：Cartpole環境...................................................................................57
5.6.1 REINFORCE實現...........................................................................58
5.6.2 Actor-Critic/A2C實現 .....................................................................59 參考文獻............................................................................................................62
第 6章策略梯度進階：從 TRPO到 PPO............................................................63
6.1 TRPO:信賴域策略優化 ..............................................................................63
6.1.1背景與問題 ......................................................................................63
6.1.2策略提升 .........................................................................................64
6.1.3替代函數與信賴域 ............................................................................64
XII |強化學習
6.1.4自然梯度與參數化優化 .....................................................................65
6.1.5 TRPO的局限性 ..............................................................................66
6.2 PPO:近端策略優化 ....................................................................................66
6.2.1優化目標 .........................................................................................67
6.2.2 PPO的偽代碼 .................................................................................67
6.3 PPO的核心實現細節..................................................................................68
6.3.1向量化架構 ......................................................................................68
6.3.2權重初始化 ......................................................................................69
6.3.3 Adam優化器與學習率退火...............................................................69
6.3.4廣義優勢估計...................................................................................70
6.3.5目標函數裁剪...................................................................................70
6.3.6小批量更新與優勢歸一化 ..................................................................71
6.3.7熵正則化與全局梯度裁剪 ..................................................................71
6.3.8全局梯度裁剪...................................................................................71
6.4應用場景 ....................................................................................................72
6.5總結與展望.................................................................................................73 參考文獻............................................................................................................73
第 7章蒙特卡洛樹搜索..........................................................................................74
7.1概覽...........................................................................................................75
7.1.1 Rollout算法 ....................................................................................75
7.1.2蒙特卡洛樹搜索 ...............................................................................77
7.1.3蒙特卡洛樹搜索的特性與挑戰 ...........................................................78
7.2 MCTS的起源與發展 ..................................................................................78
7.2.1多臂老虎機問題 ...............................................................................79
7.2.2 UCB及其擴展算法 ..........................................................................81
7.2.3與其他樹搜索算法的聯系與區別 ........................................................83
7.3經典算法中的 MCTS ..................................................................................84
7.3.1 AlphaZero中的 MCTS.....................................................................84
7.3.2 MuZero中的 MCTS.........................................................................86
7.3.3 MCTS與策略優化的聯系 .................................................................87
7.4 MCTS算法前沿進展 ..................................................................................89
7.4.1擴展到連續動作空間：Sampled MuZero ............................................89
7.4.2提高樣本效率：EfficientZero與 Gumbel MuZero ...............................90
7.4.3擴展到隨機環境：Stochastic MuZero.................................................91
7.4.4擴展到離線設定：MuZero Unplugged................................................92
7.4.5 MCTS並行化..................................................................................93
7.4.6總結 ................................................................................................94
目錄 | XIII
7.5實踐：井字棋 .............................................................................................94
7.5.1環境構建 .........................................................................................94
7.5.2蒙特卡洛樹搜索節點的定義...............................................................96
7.5.3 MCTS的核心步驟實現 ....................................................................96
7.5.4 MCTS算法的整體實現 ....................................................................98
7.5.5人類與機器人玩家的交互 ..................................................................99
7.5.6代碼運行示例.................................................................................100
7.5.7總結 ..............................................................................................101
7.6總結與展望...............................................................................................101 參考文獻..........................................................................................................101
第 8章基於模型的強化學習 .................................................................................103
8.1世界模型的起源 ........................................................................................104
8.2世界模型的發展: Dreamer系列 .................................................................105
8.2.1算法介紹 .......................................................................................105
8.2.2對稱對數預測.................................................................................106
8.2.3世界模型學習.................................................................................107
8.2.4 Actor-Critic學習 ...........................................................................109
8.2.5實驗效果 .......................................................................................110
8.3世界模型的發展: TD-MPC系列 ................................................................111
8.3.1 TD-MPC2:可擴展且魯棒的世界模型 ..............................................111
8.3.2多樣化任務中的表現.......................................................................111
8.3.3算法介紹 .......................................................................................112
8.3.4模型學習 .......................................................................................113
8.3.5策略學習 .......................................................................................113
8.3.6網絡架構 .......................................................................................114
8.3.7帶有策略先驗的 MPC ....................................................................114
8.3.8訓練通用 TD-MPC2智能體............................................................115
8.3.9實驗效果 .......................................................................................115
8.4世界模型的發展: Transformer系列 ............................................................118
8.4.1 STORM：隨機 Transformer世界模型.............................................118
8.4.2 IRIS：基於世界模型的高效數據學習智能體 .....................................119
8.4.3 Unizero：用於高效規劃的可擴展隱空間世界模型 .............................119
8.5世界模型的前沿 ........................................................................................120
8.5.1模型主幹分析.................................................................................120
8.5.2應用 ..............................................................................................120
8.6總結與展望...............................................................................................121 參考文獻..........................................................................................................121
XIV |強化學習
第 9章獎勵空間探索 ...........................................................................................123
9.1獎勵空間概述 ...........................................................................................123
9.1.1獎勵的稀疏性.................................................................................123
9.1.2獎勵數值的多尺度變化 ...................................................................124
9.1.3獎勵塑形與逆強化學習 ...................................................................125
9.2稀疏獎勵 ..................................................................................................125
9.2.1基於好奇心的樸素方法 ...................................................................125
9.2.2基於好奇心的進階方法 ...................................................................126
9.3多尺度獎勵...............................................................................................130
9.3.1獎勵裁剪 .......................................................................................132
9.3.2 PopArt..........................................................................................133
9.3.3值函數重縮放.................................................................................134
9.4實踐：內在獎勵模型與數值歸一化技術.......................................................137
9.4.1內在獎勵模型：ICM與 RND .........................................................137
9.4.2數值歸一化 ....................................................................................140
9.4.3總結 ..............................................................................................140 參考文獻..........................................................................................................141
第 10章多任務強化學習......................................................................................142
10.1多任務學習基礎 ......................................................................................142
10.2架構設計 ................................................................................................143
10.2.1模塊化架構 .................................................................................143
10.2.2參數組合架構 ..............................................................................146
10.2.3其他研究.....................................................................................148
10.3優化技巧 ................................................................................................149
10.3.1多任務梯度矯正...........................................................................149
10.3.2其他 ...........................................................................................152
10.4任務自適應 .............................................................................................153
10.5預訓練與微調..........................................................................................155
10.5.1背景 ...........................................................................................156
10.5.2學會調制.....................................................................................157
10.6總結與展望 .............................................................................................158 參考文獻..........................................................................................................159
第 11章視覺強化學習和數據增強 ........................................................................162
11.1背景知識 ................................................................................................163
11.1.1視覺強化學習任務的建模 .............................................................163
11.1.2數據增強.....................................................................................166
目錄 | XV
11.2視覺強化學習中增強數據的生成方法 ........................................................168
11.2.1觀察狀態增強 ..............................................................................169
11.2.2轉移元組增強 ..............................................................................170
11.2.3軌跡增強.....................................................................................171
11.2.4總結 ...........................................................................................171
11.3視覺強化學習中增強數據的利用範式 ........................................................172
11.3.1基於增強數據的隱式策略正則.......................................................173
11.3.2結合輔助任務的顯式策略正則.......................................................176
11.3.3結合策略解耦的任務專屬表征.......................................................179
11.3.4基於無監督學習的跨任務表征.......................................................181
11.3.5總結 ...........................................................................................181 參考文獻..........................................................................................................182
第 12章模仿學習................................................................................................187
12.1模仿學習的基礎理論................................................................................187
12.1.1學習目標與問題定義 ....................................................................187
12.1.2模仿學習的基本範式 ....................................................................187
12.1.3分布偏移與應對策略 ....................................................................188
12.1.4理論保證.....................................................................................188
12.2模仿學習的經典方法和模型......................................................................188
12.2.1行為克隆.....................................................................................189
12.2.2基於反饋的算法...........................................................................189
12.2.3逆強化學習 .................................................................................190
12.2.4生成對抗模仿學習 .......................................................................190
12.2.5其他模型.....................................................................................190
12.2.6方法比較與適用場景 ....................................................................191
12.2.7模仿學習的案例分析 ....................................................................191
12.3模仿學習的實際應用................................................................................193
12.3.1機器人控制 .................................................................................193
12.3.2自動駕駛.....................................................................................193
12.3.3遊戲 AI.......................................................................................194
12.3.4醫療健康.....................................................................................194
12.3.5其他領域.....................................................................................194
12.3.6總結與展望 .................................................................................195 參考文獻..........................................................................................................195
第 13章通用智能體 ............................................................................................196
Gato ......................................................................................................197
13.1
XVI |強化學習
13.1.1簡介 ...........................................................................................197
13.1.2方法 ...........................................................................................198
13.1.3主要結果.....................................................................................201
13.1.4局限性 ........................................................................................202 JAT .......................................................................................................202
13.2
13.2.1簡介 ...........................................................................................202
13.2.2模型架構.....................................................................................203
13.2.3數據集與訓練 ..............................................................................204
13.2.4實驗 ...........................................................................................205 Genie .....................................................................................................206
13.3
13.3.1簡介 ...........................................................................................206
13.3.2方法 ...........................................................................................207
13.3.3實驗 ...........................................................................................209 Humanoid ..............................................................................................210
13.4
13.4.1簡介 ...........................................................................................210
13.4.2方法 ...........................................................................................210
13.4.3實驗 ...........................................................................................212
13.5強化學習與語言模型的融合......................................................................213
13.5.1強化學習與語言模型的融合研究 ...................................................213
13.5.2多模態交互世界模型 ....................................................................214
13.6總結與展望 .............................................................................................215 參考文獻..........................................................................................................215
