Crawling the Web

Gautam Pant; Padmini Srinivasan; Filippo Menczer

doi:10.1007/978-3-662-10874-1_7

Crawling the Web

Gautam Pant, Padmini Srinivasan, Filippo Menczer

Research output: Chapter in Book/Report/Conference proceeding › Chapter

Abstract

The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate “focus” within their crawlers to harvest application- or topic-specific collections. In this chapter we discuss the basic issues related to developing an infrastructure for crawlers. This is followed by a review of several topical crawling algorithms, and evaluation metrics that may be used to judge their performance. Given that many innovative applications of Web crawling are still being invented, we briefly discuss some that have already been developed.

Original language	English (US)
Title of host publication	Web Dynamics
Editors	Mark Levene, Alexandra Poulovassilis
Publisher	Springer
Chapter	7
Pages	153-177
ISBN (Electronic)	978366210841
ISBN (Print)	9783642073779
DOIs	https://doi.org/10.1007/978-3-662-10874-1_7
State	Published - 2004
Externally published	Yes

Keywords

Priority Queue
Context Graph
Anchor Text
Relevance Score
Cosine Similarity

Online availability

10.1007/978-3-662-10874-1_7

Library availability

Discover UIUC Full Text

Cite this

@inbook{10ec9bca716f43dabca51571127476fe,

title = "Crawling the Web",

abstract = "The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate “focus” within their crawlers to harvest application- or topic-specific collections. In this chapter we discuss the basic issues related to developing an infrastructure for crawlers. This is followed by a review of several topical crawling algorithms, and evaluation metrics that may be used to judge their performance. Given that many innovative applications of Web crawling are still being invented, we briefly discuss some that have already been developed.",

keywords = "Priority Queue, Context Graph, Anchor Text, Relevance Score, Cosine Similarity",

author = "Gautam Pant and Padmini Srinivasan and Filippo Menczer",

year = "2004",

doi = "10.1007/978-3-662-10874-1_7",

language = "English (US)",

isbn = "9783642073779",

pages = "153--177",

editor = "Mark Levene and Alexandra Poulovassilis",

booktitle = "Web Dynamics",

publisher = "Springer",

address = "Germany",

}

TY - CHAP

T1 - Crawling the Web

AU - Pant, Gautam

AU - Srinivasan, Padmini

AU - Menczer, Filippo

PY - 2004

Y1 - 2004

N2 - The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate “focus” within their crawlers to harvest application- or topic-specific collections. In this chapter we discuss the basic issues related to developing an infrastructure for crawlers. This is followed by a review of several topical crawling algorithms, and evaluation metrics that may be used to judge their performance. Given that many innovative applications of Web crawling are still being invented, we briefly discuss some that have already been developed.

AB - The large size and the dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. While some systems rely on crawlers that exhaustively crawl the Web, others incorporate “focus” within their crawlers to harvest application- or topic-specific collections. In this chapter we discuss the basic issues related to developing an infrastructure for crawlers. This is followed by a review of several topical crawling algorithms, and evaluation metrics that may be used to judge their performance. Given that many innovative applications of Web crawling are still being invented, we briefly discuss some that have already been developed.

KW - Priority Queue

KW - Context Graph

KW - Anchor Text

KW - Relevance Score

KW - Cosine Similarity

U2 - 10.1007/978-3-662-10874-1_7

DO - 10.1007/978-3-662-10874-1_7

M3 - Chapter

SN - 9783642073779

SP - 153

EP - 177

BT - Web Dynamics

A2 - Levene, Mark

A2 - Poulovassilis, Alexandra

PB - Springer

ER -

Crawling the Web

Abstract

Keywords

Online availability

Library availability

Fingerprint

Cite this