Robots txt, Link Authority, and Internal Linking Know-how
April 6th, 2007 by KevIt’s commonly thought that a dozen of pages is better than fewer pages. Google forces to redefine this belief. Google has been constantly improving on their duplicate content filters and began using PageRank limitations.
Such implementations on Google’s part forced webmasters into a dilema of avoiding those filters. The solutions was found. The renowned Shoemoney increased his Google search traffic 1400% by not allowing spiders index some of his pages.
At times the search engine’s preferences do not fit into the humans preferences. Online shoppers like to sort through products by price or look at different sizes and colors. Such pages create near duplicate content which major SE’s do not like. What do you do then?
What Shoemoney and other SEO’s do is they make duplicate pages non indexable using robots.txt file. This is how you protect your page rank and avoid penalties for duplicate content, and, of course, increase traffic.
If you’re not careful about link equity getting low value pages indexed then your valuable pages will not rank as well as they could because too many empty, duplicate content pages were indexed.
When planning the information architecture of your website, you have to keep low value pages off the pages that drive you traffic. Check Google for more information on how to use the robots.txt.











