Impact Of The Robots.Txt, Meta-Robots & Rel=Nofollow On Seo And Their Aspects To Avoid Seo Mistakes
Most of the people don’t know the exact use of robots.txt, the META-robots and the rel=nofollow anchor attributes and because of this they make misuse of them.
This in turn affects the crawling, page rank distribution and indexing of a site. As a result the search engine optimization of their site is affected.
This article will tell you the impact of robots.txt, the META-robots and the rel=nofollow on SEO.
Robots.txt is a text file which is placed in the top level directory of a website (for example www.myexample.com/robots.txt). It is used to hide or remove any information which you do not want to appear in search results. Webmasters use it to instruct crawlers / web spiders which pages to spider. The robots.txt file lists which web pages should not be crawled by robots, what should be the crawl rate, and the location of XML sitemap.
Following are some of aspects of robots.txt to help you avoiding the major SEO blunders:
- If you block a website, page or any folder by using robots.txt file then it will not be crawled by those search engines which follow the Robot Exclusion Standard. However they will be crawled by spam bots despite your robots'txt file.
- If you block any site, webpage or folder by using robots.txt then it is not crawled by major search engines. But it might appear in search engine results if they find sufficient information about a site on DMOZ or if it has links on other pages. The search engine might show the page in search results, except that it won't show a description.
- Calculation of page rank does not depend upon whether the page is blocked or not by robots.txt. Therefore those inbound links which are pointing these pages do surpass link juice. Since blocked pages cannot be crawled by search engines there will be no outbound links and therefore these pages are considered as dangling pages. This means these pages fade out the page rank of other pages of a site and lead to the loss of page rank of a site.
Meta-robots is a Meta tag that specifies whether the particular page should be indexed or not and whether the outbound links should be followed or not.
Default tag in Meta-robots is “index,follow” which means this particular page should be indexed and the outbound links should be followed if there is no use of rel=nofollow attribute.
The valid values of these Meta tags are index, follow, noindex, nofollow and none.
Following are some of the aspects of meta-robots that help you in avoiding the major SEO blunders:
- If you block a page by using noindex Meta-robots value then it doesn’t mean it will not crawl by search engines. It will crawl by all major search engines where all links of a page will be extracted.
- If the blocked page is crawled by search engines then it doesn’t mean that it will appear in search results. To reason this out, crawling and indexing have to be understood as different terms. As the webpage is crawled, the search engine identifies the meta-robots noindex value. The page will then never show up in the results of search engine.
The page will not be indexed, but the search engine will follow all links on the page:
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
The page will be indexed, but the spider will NOT follow any link on the page:
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
The page won't be indexed, and the links won't be followed.
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
If you want your pages indexed and all links followed, you do not need the robots meta tag at all.
The rel=”nofollow” is an attribute that is used in anchor links which we wish to block so that they do not end up passing any link juice to any of the target pages.
Following are some of the aspects of rel=”nofollow” that help you in avoiding the major SEO blunders:
- If you block a link by using rel=”nofollow” then it doesn’t mean the search engine will not index the page. It will only not count your link for page rank. Therefore, rel=nofollow is not recommended for preventing the indexing of a page.
- In principle, the search engines don’t crawl the “nofollow” links. However, practically, the converse is true. Having a link no-followed does not mean the spider will not find the page. But it does mean that the links don't pass any link juice to the linked page.
Google webmaster central can give you a brief idea on this.
About the author: Matthew Anton is an on-page optimization expert. He has done extensive research on the impact of Robots.Txt, Meta-Robots and Rel=Nofollow on seo.
Category: Internet Business Blog |