Errors on Carterlake/Saratoga/Webster type website

[Ralph1]

My website seems to work really well, but I notice that certain people when they login generate errors which appear in my website host's log.  These errors are quite regular and the two most common ones are:
[Wed Aug 25 16:46:26 2010] [error] [client {client URL}] File does not exist: /home/myusername/public_html/404.shtml
[Wed Aug 25 16:46:26 2010] [error] [client {client URL}] File does not exist: /home/myusername/public_html/robots.txt

I've deleted the URL which caused the error in the above so as not to incriminate anyone and substituted {client URL} in each line.  I've also changed the name of the directory which causes it to "myusername". I'm not sure if it compromises security if I left the correct directory in the above or not so I have replaced it as a precaution. If I login to my website, I don't create the error. Does anyone know the reason for this and what does it signify? Generally speaking, these two errors appear together, in fact I think they always do.

My website URL is http://sidlawweather.co.uk.
If anyone can suggest the reason for this, I'll be very grateful.

[BigOkie]

My website seems to work really well, but I notice that certain people when they login generate errors which appear in my website host's log.  These errors are quite regular and the two most common ones are:
[Wed Aug 25 16:46:26 2010] [error] [client {client URL}] File does not exist: /home/myusername/public_html/404.shtml
[Wed Aug 25 16:46:26 2010] [error] [client {client URL}] File does not exist: /home/myusername/public_html/robots.txt

I've deleted the URL which caused the error in the above so as not to incriminate anyone and substituted {client URL} in each line.  I've also changed the name of the directory which causes it to "myusername". I'm not sure if it compromises security if I left the correct directory in the above or not so I have replaced it as a precaution. If I login to my website, I don't create the error. Does anyone know the reason for this and what does it signify? Generally speaking, these two errors appear together, in fact I think they always do.

My website URL is http://sidlawweather.co.uk.
If anyone can suggest the reason for this, I'll be very grateful.

Your first clue is the lack of a robots.txt file.  The Carterlake template doesn't provide for one.  Robots.txt is a standards file that spiders (search engine bots) use to get a map of your website.  Spiders first request the robots.txt file...if it can't find one, it starts traversing the website willy-nilly usually.

The 404.shtml file is likely another, or, your htaccess file has a rule pointing to 404.shtml as your default 404 (not found) page and it doesn't exist.  Nothing really to be alarmed with.  IMO.

[Ralph1]

Your first clue is the lack of a robots.txt file.  The Carterlake template doesn't provide for one.  Robots.txt is a standards file that spiders (search engine bots) use to get a map of your website.  Spiders first request the robots.txt file...if it can't find one, it starts traversing the website willy-nilly usually.

The 404.shtml file is likely another, or, your htaccess file has a rule pointing to 404.shtml as your default 404 (not found) page and it doesn't exist.  Nothing really to be alarmed with.  IMO.

OK thanks BigOkie.  Much is explained by your reply. I don't know how to create robots.txt in order to prevent this error, but, as you say it is perhaps not too important.  The 404.shtml error only seems to occur in conjunction with another error - it's as if it is looking for that file which might contain a list of errors.

John

[saratogaWX]

A robots.txt file can be really simple like:

Code: [Select]

User-agent: *

Disallow: /cgi-bin/
Disallow: /wxwuhistory.php

Just create the file in a text editor and upload it.

The missing 404.shtml file just means you haven't created a custom error handler file for missing files.. no worries.. it's not required at all.

Best regards,
Ken

[Ralph1]

OK, thanks Ken. I was away on vacation which explains why I haven't got back to you. I've uploaded the robots.txt file as suggested, so we'll see what happens.

Cheers,

John