User-agent: [Expected, one or more per team] The rule specifies the identify of the automated customer called search engine crawler the rule relates to. Allow for use of all but a single crawler Unnecessarybot may not crawl the internet site, all other bots might. Using the Robots.txt Generator tool is https://seotoolstube.com/