Robots.txt only contains instructions that tell the search engine where it can, and can’t, look for content to index.
Yours is telling them not to index your log and xxx directories, and by default they can search anywhere else.
Robots.txt only contains instructions that tell the search engine where it can, and can’t, look for content to index.
Yours is telling them not to index your log and xxx directories, and by default they can search anywhere else.