Crawlets: Agents for high performance web search engines

Prasannaa Thati, Po Hao Chang, Gul Agha

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Some of the reasons for unsatisfactory performance of today’s search engines are their centralized approach to web crawling and lack of explicit support from web servers. We propose a modification to conventional crawling in which a search engine uploads simple agents, called crawlets, to web sites. A crawlet crawls pages at a site locally and sends a compact summary back to the search engine. This not only reduces bandwidth requirements and network latencies, but also parallelizes crawling. Crawlets also provide an effective means for achieving the performance gains of personalized web servers, and can make up for the lack of cooperation from conventional web servers. The specialized nature of crawlets allows simple solutions to security and resource control problems, and reduces software requirements at participating web sites. In fact, we propose an implementation that requires no changes to web servers, but only the installation of a few (active) web pages at host sites.

Original languageEnglish (US)
Title of host publicationMobile Agents - 5th International Conference, MA 2001, Proceedings
EditorsGian Pietro Picco
Number of pages16
ISBN (Print)3540429522, 9783540429524
StatePublished - 2001
Event5th International Conference on Mobile Agents, MA 2001 - Atlanta, United States
Duration: Dec 2 2001Dec 4 2001

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other5th International Conference on Mobile Agents, MA 2001
Country/TerritoryUnited States

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Crawlets: Agents for high performance web search engines'. Together they form a unique fingerprint.

Cite this