A Web crawler is a simple automated computer program that browses the World Wide Web in a logical, automated manner or in an orderly fashion. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. These programs are generally made to be used only once, but they can be programmed for long-term usage as well.
Different names for a web crawler include web spider, web robot, bot, crawler, and automatic indexer or software agent. In common, it starts with a list of URLs to visit, the number of possible crawl able URLs being generated by server-side software has also made it hard for web crawlers to keep away from retrieving duplicate content.
There are various uses for web crawlers, but a web crawler may be used by anyone looking for to collect information on the Internet. Search engines use web crawlers to gather information what is accessible on web pages. A web crawler carry out a textual analysis; that is may search the Internet to determine what words are usually used at present. Market researchers may use a web crawler to determine and charge trends in a given market.
Once a web crawler visits a web page, it "reads" the visible text, the hyperlinks, and the content of the site, such as keyword rich Metatags. Using the information collected from the crawler, a search engine will determine what the site is about and index the information. Then the website is built-in the search engine's database and page ranking method.
0 comments:
Post a Comment