|
Search engines have Spiders, or scripts, that "crawl" around the web to index it. They don't all go everywhere, most of them have to have a link submitted first. Basically it checks for updates and new pages within sites. Most go around the bigger sites every day or two, others it checks every month or so.
Slurp is just the name of one of Yahoo!'s spiders.
|