Speed Bumps and Walls for SE Crawlers

Certain types of site navigation and site structure may hold back or entirely prevent search engines from getting your website's content. As search engine spiders crawl the internet, they rely on the structural design of hyperlinks to find new documents and revisit those that may have changed.

Complex linking strategies and deep site structures, with little distinctive content and too much graphics may serve as "bumps", while the information that cannot be accessed by spiderable links may qualify as "walls." So don't use fancy editing tricks to tell a story. Use them only to enhance the story.

Possible "Speed Bumps" for SE Crawlers:

  • URLs with more than two dynamic parameters
  • Pages with over 100 unique links to other pages (crawlers may not follow each of them)
  • Pages more than 3 clicks away from the the home page
  • Pages using a "Session ID" or Cookie during navigation (spiders may not be able to use these elements)
  • Pages with "frames" can hold back the site crawling and confuse spiders about which pages to rank in the results

Possible "Walls" for SE Crawlers:

  • Pages requiring a drop down menu (HTML attribute) to access them
  • Image map navigation structure
  • Flash content and navigation
  • Pages accessible only via a select form and submit button
  • Documents accessible only via a search box
  • Documents blocked purposefully (via a robots meta tag or robots.txt)
  • Pages requiring a login
  • Pages that re-direct before showing content

The best way ensuring that a site's contents are completely accessible by spiders is to have direct, static HTML link to to every page you want the search engine to index and rank in the results. Always consider that if a page cannot be directly accessed from the website’s home page, that is the page where most spiders will start their crawling, it is less likely to be indexed by the search engines. A sitemap can be of tremendous help for this purpose. If your site don't have a sitemap, you can create one using our SiteMap Generator tool.

Article Series

This article is part 2 of a 2 part series. Other articles in this series are shown below:
  1. Why do i need SEO?
  2. Site Navigation and SEO
No popular articles found.