Missing data from Average/extremes and climatedataout plus data discrepancy

Hello
The first day of every month I manually Tweet: warmest, coldest, wettest, windiest and wind run for the previous month

Missing data
Today when I looked at the climatedataout page for January I saw that data was missing from 1-16 inc.

I’ve not made any changes to WD and checking my Twitter (3 hourly Tweet), WeatherUnderground and WOW sites shows data for these days. On the 16th I suffered a power cut and had to restart the server that WD was running on so I don’t know if that reset something which is why there is data from the 17th.

I know there are several posts about using Action\Convert log files to data files but before I went down that road… What files do I need to check?

My dailydatalog.txt file contains data for all the missing days as does: 12016extralog.csv, 12016indoorlog.txt, 12016lg.txt, 12016vantagelog.txt but I’m not sure if these are sufficient to try and recreate the data.

Data discrepancy
Furthermore, when I thought I could use the data from dailydatalog.txt for my tweet I noticed a discrepancy in the wind data between dailydatalog.txt and climatedataout page.

One example was 29th Jan: dailydatalog.txt shows max wind as 51 yet climatedataout.html, climatedataout12016.html and January2016.htm pages show this to be 58.6. I haven’t made a detailed check for other differences but a quick look at the data from 17th onwards does seem to show wind discrepancies on a number of days (ignoring possible decimal point rounding), e.g. 21st 12.7 (climatedataout) ; 11 (dailydatalog)

Can anyone advise?

I have attached screen snippets to highlight the missing data and discrepancy issues and a snippet of part of the logfiles folder but I can send any other file that is considered relevant

Regards
Peter
WD: 10.37R b300
WD was running on W2K3 server up to 27th Jan 2016, now on W7 Pro (fully patched)
Davis VP2 serial interface up to 27th Jan 2016 now with WeatherLink IP interface (WeatherLink software not used). These are 2 different consoles.
WD connects to VVP for the data

This move which was made on 27th Jan 2016 is still in ‘observation phase’ but nothing has been seen so far so everything seems to be working okay.
Just to clarify, although I have 2 different VP2 consoles there is only one instance of WD running. WD, WXSIM and VVP on the server have been shutdown.

Added at 20:19 1 Feb 2016
I’ve now just realised that none of the daily graphs (yyymmdd.gif) contain data (created but empty) except for 20160100.gif, 20160130.gif and 20160131.gif and the January2016.htm only contains the summary and rain totals (from the 17th) rather than the daily data and links to the daily graphs. I assume this is all connected.

AvExJan2016.jpg769×487 114 KB

CDOJan2016.jpg905×704 256 KB

first, I would upgrade to the latest version
then if your logfile, 12016lg.txt look OK
then go action, convert WD logfiles to data files, select that file, then click on convert
then you should be able to recreate the noaa report/climete reports
see the youtube video on how to recreate those reports
http://www.weather-display.com/youtubevideos.php

Thanks Brian I’ll have a go at that.

I’ve just downloaded the .zip for b314 but when I go to install it, it tells me that it isn’t compatible with the version of Windows I’m running (W7 Pro 64-bit) but b310 looks like it will install and I got from the same place.

Any advice?

Regards
Peter

not sure why the .zip update would give that error
the latest full install is build 314 , so not sure where you are getting build 310 from?

Sorry Brian, it might have been the way I wrote it.

I downloaded the b314 zip last night after getting your reply. When I went to install it I got a ‘not compatible with my Windows’ error

As I had previously downloaded but not installed b310 I tried that and that seemed to be okay.

This morning I installed b310 which is okay but I’m still getting the ‘not compatible with my Windows’ error for b314
Regards
Peter

WDb314 error1.JPG735×231 37.7 KB

WDb310-No error1.JPG755×411 56.3 KB

Hi Peter,
FYI dailydatalog.txt is a file that is solely used for the UK interactive climate change project IICCP - see my data at https://www.iiccp.co.uk/stations/martynhicks_co_uk/data.html
That project does not use the wind data from the file, but the value is in knots so your reading of 51 converted to mph is 58.6 matching the climatedataout and January2016 files.
Martyn

Thanks Martyn that is useful to know and solves that mystery

Hi Brian
It appears that the Windows incompatibility error is due to a mistake I made when I downloaded and saved the file. I inadvertently changed the file extension from zip to exe, although quite why it caused that particular error I don’t know. Anyway I have now installed b314 okay.

Many apologies :oops:

Regards
Peter

Added 15 Feb 2016
Hi Brian
I tried to re-create the missing data today. I started by looking through 12016lg.txt (as best I could) but didn’t see any problems with it. However, when I followed the instructions in the video to recreate the averages/extremes for January 2016 I could still only see data from 17 Jan.

A closer look at the log file showed a number of problems:

1. The very first line had the year 2015 instead of 2016
2. Couldn’t see any problems after that until I got to down to date 17 time 5 51 when the data was pushed out of alignment by multiple spaces. Then I noticed data from day 17 time 4 42 to day 17 time 5 47 inclusive was missing and the data for 5 48 5 49 and 5 50 was incorrect as it wasn’t in the same value ranges as the times either side.

Rightly or wrongly I corrected the year date using Notepad, removed the multiple spaces and saved the file but this had no effect. I’d had hoped it would pull in the data from the 1st up to at least the point on the 17th where it was corrupted but this didn’t happened. Even using the Log/Data File Viewer/Correct Data feature it still only started at day 17 time 5 2 which oddly enough doesn’t seem to exist in 12016lg.txt but I might be mis-reading the screen for the time values.

I even tried to fudge it by copying a few lines either side to fill in the missing blanks and incorrect data but that didn’t work either. I copied the log files folder on the 1st Feb so I have the original January data. It’s almost as if another file is telling it where to start.

Anything else I can try?
Regards
Peter

Upload the suspect bad log to someplace on your web server. Then visit this site, http://www.stella-maris.org.uk/checklogs.php
and point it to the location of the log file on your server.

It should read your file and you will see a bunch of errors…scroll down the page until you see the spreadsheet looking data in squares. Each error must be repaired manually in your file (except the very last error). Fix the errors on your local file, then re-upload to your server and run it thru the checker again. You should have 1 error when finished, saying something about the time 23:59 is zero’s or something like that.

Once the log file passes with no errors, copy it back to your log file directory and try doing the convert process again. If you have good data it should show up.

Take your time…messing with files can be confusing. Always backup the original file and save it someplace safe.

Thanks Dan :D. I’ll have a go at that.

I’ve had to do this more times than I’d like…not sure what causes it, but it gives WD fits. Using the method I posted has worked for me 100% of the time. The main thing is to make sure you have the first and last 10 minutes of the day in the log file. Those entries are key to WD picking up the change in day. And it’s ok to have gaps in data during the day. By far the error I see the most is similar to “time does not increment” and is usually caused when re-starting WD and it re-writes previous data…as an example, your log file has data from 11:45 to 11:55, but then after restarting WD it appends the same minutes into the log file, so you will see 11:45 to 11:55, then it jumps back to 11:45 and enters a second set of the same minutes before continuing on with the newest data. The log file checker is good at spotting these.

Thanks Dan
I have submitted the log file to the WD log checker but there is a problem. I’ve been in contact with Stuart who says he thinks there is a bug in the programme and he is going to have a look at it in the next day or two.

Ok, just tried the log checker and it worked for me. See attached. You have to scroll down the page after submitting your file…look for the spreadsheet looking boxes with your errors. Like I say, ignore the last error about a blank line, but fix all the others.

Hopefully the picture will help “explain” what I mean about the page errors…just scroll down to find the “real” errors.

logcheck.png975×463 97.7 KB

I got a similiar result (attached) but ignoring the ‘page’ errors and the blank record error seems to show no errors in the log file.

I emailed Stuart with the same attachment:
[i]I’ve just used your log file checker and received lots of errors but to be honest I’m not sure if the errors I’m getting are related to my log
because I got the same errors when trying a file I wasn

I’ve been looking at the log file 12016lg.txt manually and have corrected a couple of very obvious errors. Taking into account what Dan said about the first and last 10 minutes I made sure that the first and last hours of the log file data were present. This still has no effect on recreating the av/extreme report.

I then pulled the log file into the viewer/correction utility (Correct/check the data) but still couldn’t see anything before the 17th. What has flummoxed me is the data being displayed in the log viewer (log file correct.jpg). I have assumed (I know, dangerous) that the log viewer uses the myyyylg.txt file, in this case 12016lg.txt. I have corrected this file so can’t understand where it is getting the data it is displaying in the viewer (log file correct.jpg) as that doesn’t exist in 12016lg.txt (12016lgtxt.jpg). Or am I barking up the wrong tree? I’ve even looked into every 12016*.* file in the logfiles folder for the barometric value 750.7 shown in the viewer but couldn’t find it anywhere. That got removed when I corrected a couple of obvious errors.

12016lgtxt.JPG597×604 207 KB

log file correct.jpg936×734 206 KB

I thought I would provide an update as there have been one or two discoveries but nothing that resolves my problem.

This thread started initially as missing data from the NOAA style and averages/extremes reports. The data for 1-16 Jan 2016 inc was missing although it appear to be present in the logfile 12016lg.txt. When the log was submitted to the log file checker on: http://www.stella-maris.org.uk/checklogs.php it passed but a number of other errors were displayed due to a bug in the programme. Stuart has now fixed this. :cheers:

I corrected some obvious errors which were:

1. Incorrect year (2015) for the very first line of data for the 1st Jan (the time increment was unaffected)
2. On the 17th between 0441 and 0552 the data was either missing or just plain wrong when compared with the times either side. Probably caused by the power cut we had around that time.
3. The 0548 line was preceded with 74 nulls

Except for the first year value the data appeared to be okay for the first and last 10 minutes of the month which I understand to be important and missing data in between isn’t so much of an issue when it comes to functionality.

I corrected the year value and replaced the ‘wrong’ values around the 0441 and 0552 lines for the 17th. I did this using Notepad and by copying the values from either side of those times then tried to recreate the NOAA style and averages/extremes reports but got the same result (no data before the 17th).

Using the WD\View\Averages/Extremes Noaa Style Reports\Averages Extremes tab\Correct/check the data utility I found that only the 17th to the 31st days would display data and the very first line (day 17) contained values which didn’t exist in the log file anymore. (I had previously used this utility to look at the log file before making the corrections. Does it cache the data somewhere? I have shutdown and restarted WD).

However, if I used the WD\View\Data Log File utility I found it can view the days 1-16 Jan as well as the rest of the month even in the corrupt uncorrected file although it does ignore the line that is proceded with the 74 nulls which isn’t a surprise. So one log file utility can read days 1-16 in the corrupt file, one can’t. The one that can’t, cannot read the corrected file days 1-16 either and displays values that now don’t exist in the corrected file.

I also had a look at the first 3 months of the preceding years (back to Jan 2008) of the log file mmyyyylg.txt and found the same year error in Jan 2011 (2010).

I hope all that makes sense.

if you could .zip and email me the logfile in question and I will test/check here

What is the permissible range for barometer readings? I have his log file and he has some very low barometer readings, i.e… 970 to 990. I will email you the file he sent me.

That’s quite ok for his location. The UK can get some very low pressure systems especially up in Scotland. My all time record low baro is 958mb.