How Robots.txt May Allow Link Sellers A Loophole

[Adam]

I’ve got to hand it to Andy Beard. Here’s a creative way of getting around Google’s paid link policy. The search engine has effectively told webmasters that they will be penalized for selling links. Many webmasters have gone to instituting nofollow attributes on their paid links so that they aren’t penalized. That penalizes the webmasters buying the links as they don’t get any PR transference from those nofollow links. So will Beard’s Robots.txt solution solve his problem?

It’s possible, but not likely. Robots.txt will allow webmasters the ability to tell Google not to crawl their pages. If you have a page on which you link to other sites that have paid for you to do so then you can prevent Google from crawling that web page altogether. The page will still be indexed, but it won’t be crawled. And the natural follow up question is, “Well, if it isn’t crawled then how will be indexed?”

If there is even one inbound link to that page from anywhere else on the web then it will be indexed. Google may not crawl it, and therefore will not crawl the links on that page so it will potentially never discover that those links are paid links. But what Andy Beard is banking on is that his content being syndicated either by aggregators or scrapers - the first being a legitimate and ethical way of syndication and the second one being unethical. Either way, he wins, as does his link buyers.

Any content that is syndicated will provide inbound links from other sources. Google is unlikely to count syndicated paid links as paid links. The search engine won’t know that they are paid links on the syndicated pages. How would they know? They can figure it out on the original content page, but since it isn’t crawling that page it becomes a non issue. This is what Beard is banking on.

He says he hasn’t implemented the Robots.txt strategy yet. My bet is, if he does, then other webmasters will follow and will get away with selling links for some time. They will even figure out how to syndicate each other’s content effortlessly for a period of time. Eventually, Google will institute another policy. But short term, it looks like it might work.

Nick