Azure Data Factory by Example: Practical Implementation for Data Engineers

Data engineers who need to hit the ground running will use this book to build skills in Azure Data Factory v2 (ADF). The tutorial-first approach to ADF taken in this book gets you working from the first chapter, explaining key ideas naturally as you encounter them. From creating your first data factory to building complex, metadata-driven nested pipelines, the book guides you through essential concepts in Microsoft's cloud-based ETL/ELT platform. It introduces components indispensable for the movement and transformation of data in the cloud. Then it demonstrates the tools necessary to orchestrate, monitor, and manage those components.
This edition, updated for 2024, includes the latest developments to the Azure Data Factory service:

• Enhancements to existing pipeline activities such as Execute Pipeline, along with the introduction of new activities such as Script, and activities designed specifically to interact with Azure Synapse Analytics.
• Improvements to flow control provided by activity deactivation and the Fail activity.
• The introduction of reusable data flow components such as user-defined functions and flowlets.
• Extensions to integration runtime capabilities including Managed VNet support.
• The ability to trigger pipelines in response to custom events.
• Tools for implementing boilerplate processes such as change data capture and metadata-driven data copying.

What You Will Learn

• Create pipelines, activities, datasets, and linked services
• Build reusable components using variables, parameters, and expressions
• Move data into and around Azure services automatically
• Transform data natively using ADF data flows and Power Query data wrangling
• Master flow-of-control and triggers for tightly orchestrated pipeline execution
• Publish and monitor pipelines easily and with confidence

Who This Book Is For
Data engineers and ETL developers taking their first steps in Azure Data Factory, SQL Server Integration Services users making the transition toward doing ETL in Microsoft's Azure cloud, and SQL Server database administrators involved in data warehousing and ETL operations

需要立即上手的資料工程師可以使用這本書來建立 Azure Data Factory v2 (ADF) 的技能。本書以教學為主的方式，從第一章開始讓你動手操作，並在遇到關鍵概念時自然地解釋。從建立第一個資料工廠到建立複雜的、以元數據驅動的巢狀管道，本書將引導你了解 Microsoft 的基於雲端的 ETL/ELT 平台中的重要概念。它介紹了在雲端中移動和轉換數據所不可或缺的組件，並演示了用於協調、監控和管理這些組件的工具。

這本更新至 2024 年的版本包括 Azure Data Factory 服務的最新發展：

• 對現有管道活動（如執行管道）進行了增強，並引入了新的活動（如腳本活動），以及專門與 Azure Synapse Analytics 互動的活動。


• 通過活動停用和失敗活動提供的流程控制改進。


• 引入了可重複使用的數據流組件，如用戶定義函數和流程單元。


• 擴展了集成運行時的功能，包括受管虛擬網路支援。


• 能夠根據自定義事件觸發管道。


• 用於實現樣板流程（如變更數據捕獲和元數據驅動的數據複製）的工具。

你將學到什麼

• 創建管道、活動、數據集和連接服務


• 使用變數、參數和表達式建立可重複使用的組件


• 自動將數據移入和移出 Azure 服務


• 使用 ADF 數據流和 Power Query 數據整理原生地進行數據轉換


• 掌握流程控制和觸發器，實現緊密協調的管道執行


• 輕鬆且有信心地發布和監控管道

適合閱讀對象
對 Azure Data Factory 感興趣的資料工程師和 ETL 開發人員、正在將 ETL 轉移到 Microsoft 的 Azure 雲端的 SQL Server Integration Services 使用者，以及參與數據倉儲和 ETL 操作的 SQL Server 資料庫管理員。

​Richard Swinbank is a data engineer and Microsoft Data Platform MVP. He specializes in building and automating analytics platforms using Microsoft technologies from the SQL Server stack to the Azure cloud. He is a fervent advocate of DataOps, with a technical focus on bringing automation to both analytics development and operations. An active member of the data community and keen knowledge-sharer, Richard is a volunteer, organizer, speaker, blogger, open source contributor, and author. He holds a PhD in computer science from the University of Birmingham (UK).

Richard Swinbank 是一位資料工程師和微軟數據平台 MVP。他專注於使用從 SQL Server 堆疊到 Azure 雲端的微軟技術建立和自動化分析平台。他是 DataOps 的熱情倡導者，技術上致力於將自動化引入分析開發和運營。作為數據社區的活躍成員和熱衷的知識分享者，Richard 是一位志願者、組織者、演講者、部落客、開源貢獻者和作者。他擁有英國伯明翰大學的計算機科學博士學位。