Learning PySpark

Tomasz Drabas, Denny Lee




Build data-intensive applications locally and deploy at scale using the combined powers of Python and Spark 2.0

About This Book

  • Learn why and how you can efficiently use Python to process data and build machine learning models in Apache Spark 2.0
  • Develop and deploy efficient, scalable real-time Spark solutions
  • Take your understanding of using Spark with Python to the next level with this jump start guide

Who This Book Is For

If you are a Python developer who wants to learn about the Apache Spark 2.0 ecosystem, this book is for you. A firm understanding of Python is expected to get the best out of the book. Familiarity with Spark would be useful, but is not mandatory.

What You Will Learn

  • Learn about Apache Spark and the Spark 2.0 architecture
  • Build and interact with Spark DataFrames using Spark SQL
  • Learn how to solve graph and deep learning problems using GraphFrames and TensorFrames respectively
  • Read, transform, and understand data and use it to train machine learning models
  • Build machine learning models with MLlib and ML
  • Learn how to submit your applications programmatically using spark-submit
  • Deploy locally built applications to a cluster

In Detail

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark.

You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command.

By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.

Style and approach

This book takes a very comprehensive, step-by-step approach so you understand how the Spark ecosystem can be used with Python to develop efficient, scalable solutions. Every chapter is standalone and written in a very easy-to-understand manner, with a focus on both the hows and the whys of each concept.


使用Python和Spark 2.0的結合力量,在本地建立資料密集型應用程式並進行大規模部署。

- 學習如何有效地使用Python在Apache Spark 2.0中處理資料並建立機器學習模型。
- 開發並部署高效且可擴展的即時Spark解決方案。
- 透過這本入門指南,將你對使用Python與Spark的理解提升到更高的水平。

本書適合對Apache Spark 2.0生態系統有興趣的Python開發人員。預期讀者對Python有扎實的理解,熟悉Spark將有所幫助,但不是必需的。

- 了解Apache Spark和Spark 2.0架構。
- 使用Spark SQL建立和操作Spark DataFrames。
- 學習使用GraphFrames和TensorFrames解決圖形和深度學習問題。
- 讀取、轉換和理解資料,並用它來訓練機器學習模型。
- 使用MLlib和ML建立機器學習模型。
- 學習如何使用spark-submit以程式化方式提交應用程式。
- 將本地建立的應用程式部署到叢集。

Apache Spark是一個高效的叢集計算開源框架,具有強大的資料並行處理和容錯能力。本書將向你展示如何利用Python的力量並在Spark生態系統中使用它。你將首先深入了解Spark 2.0架構,並學習如何為Spark設置Python環境。


通過閱讀本書,你將建立對Spark Python API的深厚理解,並學會如何使用它來建立資料密集型應用程式。