An Autonomous high performance web crawler using multi-agent ( AWCMA )

Abstract

Web navigation and search are vital research fields in which efficient ways for indexing and downloading pages are required. Due to the tremendous growth of the web documents and high rate of changes ( updates) , traditional crawling methods are becoming less efficient as specially as the network traffic it causes. In this project a multi agent system is used to realize mobile crawlers using the agent messages technology so that the crawling process is done at the web server side, pages parsed and processed locally then the result messages returned back to the crawler manager side to be integrated into the database. A monitoring agent will be used in these web servers to detect changes in their documents and tells the search engine for immediate updates.