User Tools

Site Tools


glossary:s:spider
  • Bookmark "Spider" at del.icio.us
  • Bookmark "Spider" at Digg
  • Bookmark "Spider" at Furl
  • Bookmark "Spider" at Reddit
  • Bookmark "Spider" at Ask
  • Bookmark "Spider" at Google
  • Bookmark "Spider" at Netscape
  • Bookmark "Spider" at StumbleUpon
  • Bookmark "Spider" at Technorati
  • Bookmark "Spider" at Live Bookmarks
  • Bookmark "Spider" at Yahoo! Myweb
  • Bookmark "Spider" at Facebook
  • Bookmark "Spider" at Newsvine
  • Bookmark "Spider" at Yahoo! Bookmarks
  • Bookmark "Spider" at Twitter
  • Bookmark "Spider" at myAOL
  • Bookmark "Spider" at Slashdot
  • Bookmark "Spider" at Mister Wong

Spider

SEO Glossary

A spider (also called crawler or robot) is a automatic program that visits your website by following links from other web pages or by a directly submitted request to a search engine. By following oder “crawling” links throughout the Internet, the spider is grabbing content from the visited sites and adding it to the index of a search engine.

When visiting a website the robot (as well as any browser) sends a specific string to the server. For this the HTTP protocol provides a field for a so-called „user-agent“. So you can recognize the visit of a spider in the log file. (e.g. the user agent code of the Google spider “Googlebot” is: Googlebot/2.1)

More Information

  • Bookmark "Spider" at del.icio.us
  • Bookmark "Spider" at Digg
  • Bookmark "Spider" at Furl
  • Bookmark "Spider" at Reddit
  • Bookmark "Spider" at Ask
  • Bookmark "Spider" at Google
  • Bookmark "Spider" at Netscape
  • Bookmark "Spider" at StumbleUpon
  • Bookmark "Spider" at Technorati
  • Bookmark "Spider" at Live Bookmarks
  • Bookmark "Spider" at Yahoo! Myweb
  • Bookmark "Spider" at Facebook
  • Bookmark "Spider" at Newsvine
  • Bookmark "Spider" at Yahoo! Bookmarks
  • Bookmark "Spider" at Twitter
  • Bookmark "Spider" at myAOL
  • Bookmark "Spider" at Slashdot
  • Bookmark "Spider" at Mister Wong
glossary/s/spider.txt · Last modified: 2012/05/05 19:20 (external edit)