sitemap.xml

Google has extended the robots.txt to have allows, but I don’t see where they would even bother looking for the sitemap entry. It is not in the robots.txt specification which normally only has excludes in it.

The extensions that google supports are not supported by anyone else and if you have signed up with google for the sitemap info, even google won’t use it.

“For instance, Googlebot supports an extended definition of the standard. It understands Allow: lines, as well as * and $ pattern matching. So while the tool shows lines that include these extensions as understood, remember that this applies only to Googlebot and not necessarily to other bots that may crawl your site.”