Using a robots.txt file to prevent search engines from spidering dynamic variables and pages
Posted by zz-James Moir on 21 June 2016 04:08 PM


By default, when a search engine spider visits your web site it follows all variable links, queries, search boxes and dynamic pages on your web site This can end up in an endless loop causing excessive load on your web site and generating high traffic.


The problem can be solved by uploading a robots.txt file to your web site which will prevent the search engines from following any URLs that begin with /? (these are the common dynamic page references).

Insert this text into your robots.txt file:

User-agent: *
Allow: /*?$
Disallow: /*?