數據中心智能調度關鍵技術與應用

第1章 數據中心概述 ······································································.1
1.1 數據中心簡介 ····················································································.2
1.1.1 什麼是數據中心 ·········································································.2
1.1.2 數據中心的需求和挑戰 ································································.4
1.2 雲計算數據中心資源調度需求分析 ·························································.4
1.2.1 技術需求 ··················································································.4
1.2.2 技術目標 ··················································································.5
1.3 雲計算數據中心資源調度研究進展 ·························································.5
1.4 雲計算數據中心資源調度方案分析 ·························································.6
1.4.1 Google解決方案 ········································································.6
1.4.2 Amazon解決方案 ·······································································.7
1.4.3 IBM解決方案 ············································································.8
1.4.4 HP解決方案··············································································10
1.4.5 VMware解決方案 ·······································································10
1.4.6 阿裏雲解決方案 ·········································································12
1.4.7 華為雲解決方案 ·········································································14
1.4.8 其他廠家解決方案 ······································································15
1.5 雲計算數據中心資源調度標準進展 ·························································17
1.6 雲資源管理調度關鍵技術及研究熱點 ······················································18
本章小結 ·································································································20
思考題·····································································································21
參考文獻 ·································································································21
第2章 雲計算概述 ········································································.25
2.1 雲計算的發展背景 ··············································································26
2.2 雲計算是集大成者 ··············································································28
2.2.1 並行計算 ················································································.28
2.2.2 網格計算 ················································································.29
2.2.3 效用計算 ················································································.29
2.2.4 普適計算 ················································································.30
2.2.5 軟件即服務 ·············································································.30
2.2.6 虛擬化技術 ·············································································.31
2.3 雲計算的驅動因素 ············································································.31
2.3.1 雲計算的發展現狀和趨勢 ···························································.33
2.3.2 雲計算應用初步分類 ·································································.35
2.4 雲計算產業鏈中的不同角色 ································································.36
2.5 雲計算的主要特征和技術挑戰 ·····························································.37
2.5.1 雲計算的主要特征 ····································································.37
2.5.2 挑戰性問題 ·············································································.38
本章小結 ································································································.46
思考題 ···································································································.46
參考文獻 ································································································.46
第3章 大數據處理 ········································································.51
3.1 大數據的發展背景及定義 ···································································.52
3.2 大數據問題 ······················································································.55
3.2.1 速度方面的問題 ·······································································.55
3.2.2 種類及架構問題 ·······································································.56
3.2.3 體量及靈活性問題 ····································································.56
3.2.4 成本問題 ················································································.57
3.2.5 價值挖掘問題 ··········································································.57
3.2.6 存儲及安全問題 ·······································································.58
3.2.7 互聯互通與數據共享問題 ···························································.59
3.3 大數據與雲計算的辯證關系 ································································.60
3.4 大數據技術 ······················································································.61
3.4.1 基礎架構支持 ··········································································.62
3.4.2 數據采集 ················································································.64
3.4.3 數據存儲 ················································································.65
3.4.4 數據計算 ················································································.68
3.4.5 數據展現與交互 ·······································································.73
本章小結 ································································································.75
思考題 ···································································································.76
參考文獻 ································································································.76
第4章 人工智能平臺的資源調度概述 ················································.78
4.1 引言 ·································································································79
4.2 深度學習的分布式並行訓練系統架構 ······················································79
4.3 深度學習的分布式並行策略 ··································································81
4.3.1 深度學習的基礎概念 ···································································82
4.3.2 分布式並行訓練算法 ···································································82
4.3.3 研究現狀分析 ············································································85
4.4 分布式並行訓練的時效分析 ··································································91
本章小結 ·································································································96
思考題·····································································································96
參考文獻 ·································································································97
第5章 基於深度學習的雲服務負載預測方法 ·······································.99
5.1 引言 ······························································································.100
5.2 相關工作 ························································································.101
5.2.1 基於回歸方法的雲服務負載預測方法···········································.101
5.2.2 基於學習的雲服務負載預測方法 ·················································.102
5.2.3 討論分析 ···············································································.103
5.3 系統模型 ························································································.104
5.4 esDNN：基於高效監督學習的深度神經網絡 ··········································.106
5.4.1 多元時間序列預測的滑動窗口 ····················································.106
5.4.2 esDNN算法 ···········································································.109
5.5 性能測試 ························································································.112
5.5.1 數據集和環境配置 ···································································.112
5.5.2 與基於無監督學習方法的比較 ····················································.113
5.5.3 與基於神經網絡方法的比較 ·······················································.114
5.5.4 與其他方面的比較 ···································································.118
本章小結 ······························································································.119
思考題··································································································.120
參考文獻 ······························································································.120
第6章 雲應用程序和可再生能源的自適應管理方法 ····························.123
6.1 引言 ······························································································.124
6.2 相關工作 ························································································.125
6.2.1 DVFS和虛擬機整合·································································.125
6.2.2 Brownout ···············································································.126
6.2.3 數據中心冷卻系統的整體管理 ····················································.126
6.2.4 可再生能源 ············································································.126
6.3 系統模型 ························································································.127
6.4 問題建模 ························································································.129
6.4.1 功率消耗 ···············································································.129
6.4.2 工作負載模型 ·········································································.130
6.4.3 優化目標 ···············································································.131
6.5 根據可再生資源進行調度決策 ····························································.132
6.5.1 Green-Aware 調度算法 ·····························································.132
6.5.2 交互式工作負載的Brownout算法 ···············································.133
6.5.3 批處理工作負載的延遲算法 ·······················································.134
6.5.4 主機調度 ···············································································.135
6.5.5 可再生能源預測 ······································································.136
6.6 原型系統的實現 ··············································································.137
6.7 性能評估 ························································································.139
6.7.1 環境設置 ···············································································.140
6.7.2 工作負載 ···············································································.140
6.7.3 應用 ·····················································································.141
6.7.4 結果 ·····················································································.141
本章小結 ·······························································································.145
思考題 ··································································································.145
參考文獻 ·······························································································.146
第7章 雲計算環境下基於虛擬機整合的自適應節能調度 ······················.149
7.1 緒論 ······························································································.150
7.2 虛擬機整合技術 ··············································································.150
7.3 相關研究工作 ··················································································.152
7.4 問題定義 ························································································.153
7.5 數據中心能耗模型 ···········································································.154
7.5.1 服務器功耗模型 ······································································.154
7.5.2 服務器能耗模型 ······································································.155
7.5.3 雲數據中心總能耗模型 ·····························································.156
7.5.4 數據中心節能調度下限 ·····························································.156
7.6 SAVE算法描述 ···············································································.158
7.6.1 概述 ·····················································································.158
7.6.2 分配算法 ···············································································.159
7.6.3 遷移算法 ···············································································.161
7.7 實驗驗證與分析 ··············································································.165
7.7.1 實驗準備 ···············································································.165
7.7.2 數據準備 ···············································································.167
7.7.3 基線方法 ···············································································.167
7.7.4 結果分析 ···············································································.169
本章小結 ······························································································.175
思考題··································································································.175
參考文獻 ······························································································.175
第8章 MapReduce模型中數據傾斜問題的算法 ·································.177
8.1 緒論 ······························································································.178
8.1.1 背景及意義 ············································································.178
8.1.2 研究現狀 ···············································································.179
8.1.3 研究內容 ···············································································.180
8.2 數據傾斜相關理論研究 ·····································································.181
8.2.1 數據傾斜 ···············································································.181
8.2.2 算法介紹 ···············································································.183
8.2.3 算法綜合對比 ·········································································.190
8.3 多任務數據傾斜調度算法設計 ····························································.193
8.3.1 問題描述與建模 ······································································.193
8.3.2 Revised Johnson1954算法（RJA） ··············································.195
8.3.3 離線多任務調度算法 ································································.199
8.3.4 在線多任務調度算法 ································································.201
8.4 單任務數據傾斜算法設計 ··································································.203
8.4.1 YarnTune概述 ·········································································.203
8.4.2 檢測數據傾斜 ·········································································.206
8.4.3 YarnTune核心功能 ··································································.208
8.5 系統測試和分析 ··············································································.212
8.5.1 軟硬件測試環境 ······································································.212
8.5.2 多任務數據傾斜測試 ································································.212
8.5.3 單任務數據傾斜測試 ································································.217
本章小結 ······························································································.220
思考題··································································································.221
參考文獻 ······························································································.221
第9章 Spark中的數據均衡分配算法研究 ·········································.223
9.1 Spark設計思想 ················································································.224
9.1.1 Spark概述 ·············································································.224
9.1.2 Spark計算模型 ·······································································.225
9.1.3 Spark整體架構 ·······································································.226
9.2 Spark數據存儲體系 ··········································································.227
9.2.1 存儲整體架構 ·········································································.227
9.2.2 數據寫入過程 ·········································································.228
9.2.3 數據讀取過程 ·········································································.229
9.3 Spark Shuffle分析 ············································································.230
9.3.1 Shuffle概述 ············································································.230
9.3.2 Shuffle寫操作 ·········································································.231
9.3.3 Shuffle讀操作 ·········································································.232
9.4 Spark分區方法 ················································································.233
9.4.1 HashPartition分區 ····································································.233
9.4.2 RangePartition分區 ··································································.234
9.5 問題描述與建模 ··············································································.235
9.5.1 相關定義 ···············································································.235
9.5.2 問題建模 ···············································································.236
9.6 數據均衡分配算法整體設計 ·······························································.237
9.6.1 抽樣算法 ···············································································.238
9.6.2 數據均衡分區算法 ···································································.240
9.6.3 權重調整算法 ·········································································.242
9.6.4 任務分配算法 ·········································································.245
9.7 算法復雜度分析 ··············································································.246
9.8 MRFair概述 ···················································································.246
9.8.1 MRFair的目標與特征 ·······························································.246
9.8.2 MRFair系統架構 ·····································································.247
9.8.3 MRFair數據均衡分配示例 ·························································.248
9.9 MRFair傾斜檢測時機及算法 ······························································.249
9.9.1 MRFair傾斜檢測時機 ·······························································.249
9.9.2 MRFair傾斜檢測算法 ·······························································.250
9.10 MRFair數據重新分配時機及算法 ······················································.250
9.10.1 MRFair數據重新分配時機 ·······················································.250
9.10.2 MRFair數據重新分配算法 ·······················································.251
9.11 MRFair核心模塊 ············································································.253
9.12 系統測試環境 ················································································.255
9.12.1 軟硬件測試環境 ·····································································.255
9.12.2 測試數據 ··············································································.256
9.12.3 對比算法或系統 ·····································································.257
9.12.4 評價指標 ··············································································.257
9.13 Reduce Partition數據均衡分配算法測試 ··············································.257
9.13.1 WordCount基準測試 ·······························································.257
9.13.2 Sort基準測試 ········································································.260
9.14 MRFair數據均衡分配算法測試 ·························································.263
9.14.1 WordCount基準測試 ·······························································.263
9.14.2 Sort基準測試 ········································································.265
本章小結 ······························································································.266
思考題··································································································.267
參考文獻 ······························································································.267
第10章 深度學習框架TensorFlow的高效分布式並行算法研究 ·············.269
10.1 分布式並行算法的背景及意義 ···························································.270
10.1.1 問題背景 ··············································································.270
10.1.2 研究意義 ··············································································.271
10.2 研究現狀及內容 ·············································································.272
10.2.1 研究現狀 ··············································································.272
10.2.2 研究內容 ··············································································.272
10.3 深度學習理論研究 ··········································································.273
10.3.1 大數據與雲計算 ····································································.273
10.3.2 機器學習 ··············································································.274
10.3.3 深度學習 ··············································································.275
10.4 TensorFlow深度學習框架研究 ··························································.277
10.4.1 TensorFlow系統架構 ······························································.277
10.4.2 TensorFlow數據流圖 ······························································.280
10.4.3 TensorFlow會話管理 ······························································.281
10.4.4 TensorFlow分布式執行 ···························································.282
10.4.5 TensorFlow數據輸入 ······························································.283
10.5 TensorFlow分布式架構分析 ·····························································.285
10.5.1 TensorFlow遠程調用 ······························································.285
10.5.2 現有TensorFlow分布式模型 ····················································.286
10.6 優化算法設計與實現 ·······································································.289
10.6.1 數據並行上的優化 ·································································.289
10.6.2 模型並行上的優化 ·································································.297
10.7 實驗環境配置 ················································································.304
10.7.1 硬件環境 ··············································································.304
10.7.2 軟件環境 ··············································································.304
10.7.3 實驗對象 ··············································································.305
10.7.4 實驗數據 ··············································································.305
10.8 實驗結果展示與分析 ·······································································.307
10.8.1 數據並行算法測試··································································.307
10.8.2 模型並行算法測試··································································.312
本章小結 ·······························································································.314
思考題 ··································································································.314
參考文獻 ·······························································································.314
第11章 基於深度強化學習和模仿學習的任務完工時間優化調度 ···········.317
11.1 任務調度 ······················································································.318
11.2 相關研究 ······················································································.321
11.3 雲資源調度問題定義 ·······································································.322
11.4 調度方案 ······················································································.327
11.4.1 DeepRM_Online介紹 ······························································.327
11.4.2 強化學習模型 ········································································.328
11.4.3 深度強化學習訓練算法····························································.330
11.5 算法分析 ······················································································.333
11.6 實驗分析與驗證 ·············································································.337
11.6.1 實驗準備 ··············································································.337
11.6.2 數據準備 ··············································································.337
11.6.3 基線算法 ··············································································.338
11.6.4 結果分析 ··············································································.339
本章小結 ·······························································································.341
思考題 ··································································································.341
參考文獻 ·······························································································.342
第12章 總結與展望 ······································································.345
12.1 綠色節能數據中心的綜合解決方案·····················································.346
12.2 多數據中心（多調度域）的調度策略和算法動態可選擇 ·························.348
12.3 支持深度學習模型的分布式並行調度 ·················································.349
12.4 從基礎資源調度拓展到應用任務調度 ·················································.350
參考文獻 ·······························································································.351