Engineering Online Experimentation and ML Evaluations: Architecture, Statistics and Machine Learning for Production-Scale Systems
暫譯: 工程線上實驗與機器學習評估：生產規模系統的架構、統計與機器學習

Lei, Ming

Engineering Online Experimentation and ML Evaluations: Architecture, Statistics and Machine Learning for Production-Scale Systems
  • 出版商: Apress
  • 出版日期: 2026-07-30
  • 售價: $2,170
  • 貴賓價: 9.5 折 $2,061
  • 語言: 英文
  • 頁數: 518
  • 裝訂: Quality Paper - also called trade paper
  • ISBN: 9798868827204
  • ISBN-13: 9798868827204
  • 相關分類: Data-mining、Machine Learning

    • 海外代購書籍(需單獨結帳)

相關主題

商品描述

Online experimentation is now essential for modern software and machine learning teams. This book provides an engineer-first, end-to-end guide to building and operating production-ready experimentation platforms.

The book begins with Part I establishing the core foundations of credible experimentation, including hypothesis testing, power analysis, sample sizing, metric design, and common pitfalls such as peeking, multiple testing, and novelty or learning effects. Part II focuses on platform engineering--traffic and identity management, mutual exclusion, event and logging design, ETL/ELT pipelines, building a stats engine with SciPy and statsmodels, SRM detection, integrating deployments with feature flags and canaries, and setting up guardrail and health monitoring. Part III presents advanced designs that improve speed and sensitivity: sequential testing with alpha spending, bootstrap intervals for ratios and quantiles, A/B/n testing with ANOVA, interleaving for ranking systems, switchback and geo experiments, and multi-armed bandits. Part IV connects experimentation to ML workflows, covering offline, shadow, canary, and A/B evaluation pipelines; Bayesian optimization for adaptive experimentation; counterfactual and IPS methods for learning from logs; and safe retraining supported by strong governance.

What you will learn:

    Design trustworthy experiments with proper metrics, guardrails, α/power/MDE settings, and safeguards against peeking and multiple-testing errors Build a production-ready experimentation stack with assignment, identity/diversion, logging, ETL/ELT, a stats engine, and SRM checks Run advanced designs at scale, including sequential tests, bootstrap CIs, interleaving, switchback/geo experiments, and multi-armed bandits Evaluate ML systems from offline to online, leverage experiment logs for learning, and enable safe retraining with governance

Who this book is for:

The primary audience for this book includes Data Engineers, ML Engineers, and Platform or Software Architects. It is also well suited for Product and Data Scientists who want a deeper understanding of experimentation systems and the engineering principles behind them.

商品描述(中文翻譯)

線上實驗對於現代軟體和機器學習團隊來說已經變得至關重要。本書提供了一個以工程師為中心的端到端指南，幫助讀者建立和運營生產就緒的實驗平台。

本書的第一部分建立了可信實驗的核心基礎，包括假設檢驗、效能分析、樣本大小、指標設計以及常見的陷阱，如偷看、多重測試和新穎性或學習效應。第二部分專注於平台工程——流量和身份管理、互斥、事件和日誌設計、ETL/ELT 管道、使用 SciPy 和 statsmodels 建立統計引擎、SRM 檢測、將部署與功能標誌和金絲雀集成，以及設置護欄和健康監控。第三部分介紹了提高速度和靈敏度的高級設計：使用 alpha 消耗的序列測試、比率和分位數的自助區間、使用 ANOVA 的 A/B/n 測試、排名系統的交錯、切換和地理實驗，以及多臂強盜。第四部分將實驗與機器學習工作流程連接，涵蓋離線、影子、金絲雀和 A/B 評估管道；用於自適應實驗的貝葉斯優化；從日誌中學習的反事實和 IPS 方法；以及由強治理支持的安全再訓練。

您將學到的內容：

- 設計可信的實驗，具備適當的指標、護欄、α/效能/MDE 設定，以及防止偷看和多重測試錯誤的保障措施
- 建立生產就緒的實驗堆疊，包括分配、身份/分流、日誌、ETL/ELT、統計引擎和 SRM 檢查
- 大規模運行高級設計，包括序列測試、自助 CIs、交錯、切換/地理實驗和多臂強盜
- 從離線到線上評估機器學習系統，利用實驗日誌進行學習，並在治理下啟用安全再訓練

本書的讀者對象：

本書的主要讀者包括數據工程師、機器學習工程師以及平台或軟體架構師。它也非常適合希望深入了解實驗系統及其背後工程原則的產品和數據科學家。

作者簡介

Ming Lei is a data and ML engineering leader with 20 years of experience building end-to-end ML systems for Internet Ads and Search, and experimentation platforms for E-commerce. He has designed large-scale systems that operationalize rigorous statistical methods -- such as sequential testing, bootstrapping, multi-armed bandits, and Bayesian optimization -- and support ML evaluation from offline analysis to online deployment. His leadership spans roles at eBay, Meta (Facebook), Google, and Appen. He holds multiple US patents and advanced degrees in computer science (UC Riverside) and economics (Clark University), along with a B.S. in physics (Wuhan University). He is based in the Northwest of US.

作者簡介(中文翻譯)

Ming Lei 是一位擁有 20 年經驗的數據與機器學習工程領導者，專注於為網路廣告和搜尋建立端到端的機器學習系統，以及為電子商務設計實驗平台。他設計了大型系統，將嚴謹的統計方法（如序列測試、重抽樣、多人臂賭博問題和貝葉斯優化）實際運用，並支持從離線分析到線上部署的機器學習評估。他的領導經歷涵蓋了 eBay、Meta（Facebook）、Google 和 Appen 等公司。他擁有多項美國專利，並在計算機科學（加州大學河濱分校）和經濟學（克拉克大學）獲得高級學位，此外還擁有物理學學士學位（武漢大學）。他目前居住在美國西北部。

類似商品

最後瀏覽商品 (4)