Building Web collections for vertical markets

Gaining actionable business intelligence from the Web is an active and promising area of research. An important facet of this line of work is the ability to build collections of Web pages (or Web collections) that are relevant to a vertical market of interest. Such collections provide raw data that can be further processed with text and data mining tools to gain business intelligence. Web crawlers are programs that can be used to build Web collections ranging in size from thousands to millions of pages. Through a systematic study we show that crawlers face a difficult task when building a Web collection for vertical markets due to the competitive nature of the business communities. However, the difficulty of the task can be greatly reduced through limited use of a search engine to identify candidate "information broker" Web pages that may serve as hubs leading to many relevant pages and competing Web sites.