Web Corpus Construction (Paperback)

The World Wide Web constitutes the largest existing source of texts written in a great variety of languages. A feasible and sound way of exploiting this data for linguistic research is to compile a static corpus for a given language. There are several adavantages of this approach: (i) Working with such corpora obviates the problems encountered when using Internet search engines in quantitative linguistic research (such as non-transparent ranking algorithms). (ii) Creating a corpus from web data is virtually free. (iii) The size of corpora compiled from the WWW may exceed by several orders of magnitudes the size of language resources offered elsewhere. (iv) The data is locally available to the user, and it can be linguistically post-processed and queried with the tools preferred by her/him. This book addresses the main practical tasks in the creation of web corpora up to giga-token size. Among these tasks are the sampling process (i.e., web crawling) and the usual cleanups including boilerplate removal and removal of duplicated content. Linguistic processing and problems with linguistic processing coming from the different kinds of noise in web corpora are also covered. Finally, the authors show how web corpora can be evaluated and compared to other corpora (such as traditionally compiled corpora).

For additional material please visit the companion website: sites.morganclaypool.com/wcc

Table of Contents: Preface / Acknowledgments / Web Corpora / Data Collection / Post-Processing / Linguistic Processing / Corpus Evaluation and Comparison / Bibliography / Authors' Biographies

網際網路是目前最大的多語言文本資源來源。利用這些資料進行語言研究的一種可行且有效的方式是為特定語言編制一個靜態語料庫。這種方法有幾個優點：(i) 使用這樣的語料庫可以避免在量化語言研究中使用互聯網搜索引擎時遇到的問題（例如非透明的排名算法）。(ii) 從網絡數據創建語料庫幾乎是免費的。(iii) 從網際網路編制的語料庫的大小可能比其他地方提供的語言資源大幾個數量級。(iv) 用戶可以在本地使用這些數據，並且可以使用他們喜歡的工具對其進行語言後處理和查詢。本書介紹了創建網絡語料庫的主要實際任務，規模可達到十億標記。其中包括採樣過程（即網絡爬蟲）和常規的清理工作，包括去除樣板和重複內容。還涵蓋了從網絡語料庫中不同類型的噪音引起的語言處理和相關問題。最後，作者展示了如何評估和比較網絡語料庫與其他語料庫（如傳統編制的語料庫）。如需更多資料，請訪問附帶網站：sites.morganclaypool.com/wcc。目錄：前言/致謝/網絡語料庫/數據收集/後處理/語言處理/語料庫評估和比較/參考文獻/作者簡介。