機器學習

第 1 章 機器學習技術簡介 ···············································································1
1.1 機器學習簡介 ·······················································································1
1.1.1 機器學習的概念············································································1
1.1.2 機器學習的算法模型······································································1
1.1.3 機器學習應用程序開發步驟·····························································2
1.2 機器學習的實現工具 ··············································································3
1.3 Python 平臺搭建 ····················································································3
1.3.1 集成開發環境 Anaconda ··································································4
1.3.2 集成開發環境 PyCharm···································································7
1.3.3 搭建虛擬環境············································································.10
1.3.4 配置虛擬環境············································································.13
1.4 Spark 平臺搭建···················································································.17
1.4.1 Spark 的部署方式·······································································.17
1.4.2 安裝 JDK··················································································.18
1.4.3 安裝 Scala·················································································.21
1.4.4 安裝開發工具 IDEA ····································································.22
1.4.5 安裝 Spark ················································································.24
1.4.6 安裝 Maven···············································································.25
1.5 基於 Python 創建項目 ··········································································.27
1.6 基於 Spark 創建項目············································································.29
習題 1 ·····································································································.32
第 2 章 數據預處理 ·····················································································.34
2.1 數據預處理的概念 ··············································································.34
2.1.1 數據清洗··················································································.34
2.1.2 數據轉換··················································································.35
2.2 基於 Python 的數據預處理 ····································································.37
2.3 基於 Spark 的數據預處理······································································.43
習題 2·······························································································.46
第 3 章 分類模型 ························································································.48
3.1 分類模型的概念 ·················································································.48
3.2 分類模型的算法原理 ···········································································.51
3.2.1 決策樹算法···············································································.51
3.2.2 最近鄰算法···············································································.56
3.2.3 樸素貝葉斯算法·········································································.58
3.2.4 邏輯回歸算法············································································.59
3.2.5 支持向量機算法·········································································.59
3.3 基於 Python 的分類建模實例 ·································································.60
3.4 基於 Spark 的分類建模實例···································································.63
習題 3 ·····································································································.67
第 4 章 聚類模型 ························································································.70
4.1 聚類模型的概念 ·················································································.70
4.1.1 聚類模型概述············································································.70
4.1.2 聚類模型中的相似度計算方法·······················································.71
4.1.3 聚類算法的評價·········································································.73
4.2 聚類模型的算法原理 ···········································································.76
4.2.1 K-means 算法 ············································································.76
4.2.2 AGNES 算法 ·············································································.77
4.2.3 DBSCAN 算法···········································································.78
4.2.4 GMM 算法················································································.79
4.2.5 二分 K-means 算法 ·····································································.79
4.2.6 隱式狄利克雷分配算法································································.80
4.3 基於 Python 的聚類建模實例 ·································································.81
4.4 基於 Spark 的聚類建模實例···································································.86
習題 4 ·····································································································.93
第 5 章 回歸模型 ························································································.95
5.1 回歸模型的概念 ·················································································.95
5.2 回歸模型的算法原理 ···········································································.95
5.2.1 線性回歸算法············································································.95
5.2.2 廣義線性回歸算法······································································102
5.3 基於 Python 的回歸建模實例 ·································································103
5.4 基於 Spark 的回歸建模實例···································································110
習題 5 ·····································································································112
第 6 章 關聯模型 ························································································114
6.1 關聯模型的概念 ·················································································114
6.2 關聯模型的算法原理 ···········································································114
6.2.1 關聯規則算法············································································114
6.2.2 協同過濾算法············································································116
6.3 基於 Python 的關聯建模實例 ·································································120
6.4 基於 Spark 的關聯建模實例···································································122
習題 6 ·····································································································131
第 7 章 數據降維 ························································································133
7.1 數據降維的概念 ·················································································133
7.2 數據降維算法 ····················································································134
7.2.1 主成分分析···············································································134
7.2.2 奇異值分解···············································································136
7.2.3 線性判別分析············································································140
7.3 基於 Python 的數據降維實例 ·································································141
7.4 基於 Spark 的數據降維實例···································································146
習題 7 ·····································································································149
第 8 章 神經網絡 ························································································151
8.1 神經網絡的概念 ·················································································151
8.2 神經網絡的算法原理 ···········································································153
8.2.1 多層感知機···············································································153
8.2.2 捲積神經網絡············································································155
8.3 基於 Python 的神經網絡實例 ·································································159
8.4 基於 Spark 的神經網絡實例···································································166
習題 8 ·····································································································168
第 9 章 項目實戰 1：食品安全信息處理與識別··················································170
9.1 項目背景···························································································170
9.2 數據獲取···························································································170
9.2.1 用 SecureCRT 連接 MongoDB 查看數據···········································170
9.2.2 用 Python 連接 MongoDB 讀取數據 ················································172
9.3 數據預處理 ·······················································································173
9.3.1 數據轉換··················································································173
9.3.2 數據清洗··················································································173
9.4 機器學習建模與分析 ···········································································174
9.4.1 將信息集合劃分為訓練集和測試集·················································174
9.4.2 將 NAME_AND_CONTENT 字段數值化··········································175
9.4.3 針對訓練集建立分類模型進行訓練·················································179
9.4.4 用測試集檢驗分類模型的性能·······················································180
9.4.5 結果可視化···············································································180
9.5 項目總結···························································································181
習題 9 ·····································································································182
第 10 章 項目實戰 2：基於 Hive 數據倉庫的商品推薦·········································183
10.1 項目背景···························································································183
10.2 數據獲取·························································································183
10.2.1 用 Navicat 連接數據庫查看數據 ···················································183
10.2.2 用 Spark 獲取數據到 Hive 的 ODS 數據倉庫····································185
10.3 數據預處理······················································································189
10.3.1 對線下購物數據進行預處理，並存入 Hive 數據倉庫的 DW 層 ············189
10.3.2 對線上購物數據進行預處理，並存入 Hive 數據倉庫的 DW 層 ············190
10.4 機器學習建模與分析··········································································192
10.4.1 對線下購物數據進行分析，並將商品推薦結果寫入 MySQL ···············192
10.4.2 對線上購物數據進行分析，並將商品推薦結果寫入 MySQL ···············195
10.5 項目總結·························································································199
習題 10····································································································199
參考文獻·····································································································200