Or you are able to pick a Completely ready-produced robots.txt template that contains a list of the most typical normal and CMS directives. You might also insert a sitemap to the file. This standard was proposed in 1994 by Martijn Koster after a Website crawler penned by Charles Stross performed https://rankingseotoolz.com/robots-txt-generator