Davis VP History Extraction only getting some records

The partial data extraction happened again this morning. As soon as I realised it had happened, I shut WD down before it could record “fresh” data from today, and complicate my chances of retrieving the remainder of last night’s data - but now I can’t view the debug file without re-starting WD, and am scared that will zap the debug file and I’ll lose whatever clues it may contain. I thought I’d be able to identify the debug info by time and date in the wdisplay folder, but have been unable to.

1: what should I do (where should I look) to find the debug info for the partial datalogger extraction from this morning?
2: If I hadn’t noticed the partial datalogger extraction immediately (and WD had started recording “fresh” data from today), what steps would I have needed to take to manually extract the missing data again ? For example:
a: Do I need to delete “fresh” entries from 82016lg.txt? 82016lgten.txt? 82016vantagelog.txt? 82016vantagelogten.txt?
b: How do I re-run the datalogger extraction? (I thought the info was in this thread, but can’t find it now)

thanks

well, I restarted wdisplay anyway (just to get it going again). I had a few goes at setting the time & date for data logger extraction and restarting wdisplay. Each time it got more data - but not all.

1st startup: got about 72 minutes worth from yesterday evening
2nd startup: got another 25 minutes
3rd startup: got another 21 minutes
4th startup: got from 23:35 to 1:55 this morning
5th & 6th: refuses to get any more after 1:55 (“time period too short”??)

Looking at the debug info each time, I see “old records” being extracted (some from over a year ago, some from earlier today) as per the following extracts (note that the extracts are just prior to the extraction aborting, but in some cases, it has been reading and digesting old records for some time prior to the abort - see attached complete file):


Time :23:30
Date :23/08/2016
sending OK after 5 blocks of data*
***formating data
Time :23:31
Date :23/08/2016
**formating data
Time :03:36
Date :02/08/2015 -----------------------------------note the date is over a year ago
year less 2015 2016
finished data block

***formating data
Time :23:33
Date :23/08/2016
***formating data
Time :03:36
Date :02/08/2015 --------------------------------------same old record again
year less 2015 2016
***formating data
Time :23:34
Date :23/08/2016
***formating data
Time :03:36
Date :02/08/2015 ------------------------------------------and again
year less 2015 2016
**formating data
Time :23:35
Date :23/08/2016
finished data block

Aborted
All data downloaded or did not sync correct!
formating the VP data
vp data read in 17
VP data Date:= 23/8/2016
VP data Time:= 23:15
Rain in davis VP download :0.0
Rain in davis VP download added :0.0
Adding wunderground data to the missed data file 23:15
missed wunderground hour normal 11
missed wunderground hour from logger 23
missed wunderground hour difference -8
missed wunderground new hour /day/normalday 15 23 24
missed wunderground new time 08/23/2016 23:15:00
VP data Date:= 23/8/2016
VP data Time:= 23:16


Time :01:50
Date :24/08/2016
sending OK after 5 blocks of data*
***formating data
Time :01:51
Date :24/08/2016
***formating data
Time :08:38
Date :02/08/2016 --------------------------------------------record from a few weeks ago
**formating data
Time :01:52
Date :24/08/2016
finished data block

***formating data
Time :08:38
Date :02/08/2016 ---------------------------------------------same old record again
***formating data
Time :01:54
Date :24/08/2016
**formating data
Time :01:55
Date :24/08/2016
sending OK after 5 blocks of data

Aborted
All data downloaded or did not sync correct!
formating the VP data
vp data read in 134
VP data Date:= 23/8/2016
VP data Time:= 23:36
Rain in davis VP download :0.0
Rain in davis VP download added :0.0
VP data Date:= 23/8/2016
VP data Time:= 23:37

Still having similar problems my end with data being mixed up. Have resorted to using 81 seems to be collecting data ok, then manually deleting/renumbering extraneous entries.

Cheers
Steve

re the old dates (which WD does recognise as being too old in the first stage) , I can add more checking too
but I would delete all the data on the data logger, to clear out those old data, so that it does not send them

  1. If I tick the box to clear the datalogger, I’m concerned then that if something went wrong with WD getting and storing the data, that would blow my chances of having a second attempt at extracting it.

  2. Is there some way of using the history extraction program by itself to extract datalogger data into a text file, that I could then import into MySQL?

I do not mean to have that ticked

I mean to clear the data logger of all the data, eg using weather link
as sometimes there can be ghost data left

More of the same results today:

  • partial data returned at each new attempt at data logger extraction (and WD restart)
  • still returning old data (I mustn’t have cleared-out the data logger)
  • VERY unreliable at getting the data into MySQL (my main repository) - some yesterday, and none from 3 attempts today

I have cleared the datalogger using Weatherlink this morning.
I have the debug files from all 3 attempts today, but they are pretty similar to yesterdays.

I have set up my IP data logger to 10 minute interval and will do some testing here

I have mine on 1 minute intervals (it is tshattuk that is using 10 minute intervals).

Note, that when the datalogger extraction is run on startup first thing every morning and works correctly, the data goes into MySQL fine. It is just when I go to the setting in Control panel / Datalogger and set a start date and time (and restart WD) to try and retrieve missing data, is the MySQL data capture not so good (ie, I can see records go into the datalogfile, but not into MySQL).

not trying to hijack anything. My problem is much the same. I have made three attempts today to extract data. Mine is set to 1 minute intervals. 81 works fine…
cheers
windi

I have done an overnight test at 10 minute logging interval with an IP data logger
worked great, no problem

Note, that when the datalogger extraction is run on startup first thing every morning and works correctly, the data goes into MySQL fine. It is just when I go to the setting in Control panel / Datalogger and set a start date and time (and restart WD) to try and retrieve missing data, is the MySQL data capture not so good (ie, I can see records go into the datalogfile, but not into MySQL).
I do not understand why you would be wanting to to the second part when you say the first part worked fine?

I seem to be missing something

if you could explain in alot of detail what exactly is going wrong?

sorry, not written as well as it could have been…

If all goes well, I don’t need to do anything further. It is only when things DO NOT go well, that I need to use the data logger tool to try and retrieve the data that was not extracted as it should - and it isn’t as successful in retrieving data (and sticking it into MySQL) as I dreamed it might have been.

  • I get the impression that history data is extracted from the Davis datalogger in blocks of 5 records - is that so?

If so, it seems very strange the latest history data records are sometimes interspersed with old records - and sometimes the same old data records appearing multiple times (as if the Davis has done something wong).

More of the same last night (partial data extraction, followed by a few attempts at manual data logger extraction which resulted in much of the missing data being added to the logfiles (but out of sequence of course), and NONE of the missing data being added to MySQL).

  • Despite Weatherlink saying I only have data in the datalogger for the last 4 or 5 days, the debug info from datalogger extraction still reported a record from last year - where is this coming from ???

I tried reverting to build R81, but had some com port error, so will just keep plugging away with the latest (although I’m getting reluctant to turn the computer off at night now because my MySQL data is developing huge holes, and missing rainfall)

I have done lots of testing here, at 1 minute interval
but I am not seeing this problem
but it might show up after the data logger is full

what I suggest you do, is when WD is caught up
then power down the console
remove the data logger for 5 minutes (and from the IP plug)
then re insert, power up console, re connect IP plug
then set the correct time and logging interval (with weather link)
so as to completly reset the data logger

OK, done that - we’ll see what happens.

In order to fill the gaps in MySQL, I will use an export from the datalogger from Weatherlink.

  1. Do you happen to know what units are used for wind speed in exports from Weatherlink ?
  2. It appears that the rain in the Weatherlink exports is just incremental, so I will have to total from one record to the next - is that right (I’m thinking this is probably what you get direct from the Davis as well ??)

well, the next 2 days worked fine, but today was another failure.

  1. It only extracted data until 00:30 this morning as per the 1st debug file attached
  2. I closed WD and started-up Weatherlink, and as per the attached snapshot, it was able to extract all the records around the times where WD had trouble
  3. I then re-started WD, and it recovered the 12 minutes of data OK, but (as per the 2nd debug attachment) still retrieved old records in the process (but this might be as expected)
  4. I set the datalogger to retrieve data from 00:31 and re-started WD. The debug records (3rd attached extract) show no attempt was made the extract data as “the time period was too short” !!

Do you know the answers to my previous questions about wind units of measure and incremental rainfall ?


Weatherlink records 160830.JPG

wdisplay debug 160830_1.txt (4.11 KB)

wdisplay debug 160830_2.txt (1.78 KB)

wdisplay debug 160830_3.txt (191 Bytes)

Ok, now that my data logger is full, I am running into this problem
trying to figure out what is going on (its bascily getting out of sync)

Ok, I it sorted in build 21 .zip update, ready now

OK, updated, and (after setting up the datalogger in the Control panel and restarting) it even retrieved the missing data from overnight and got it into MySQL (the datalogfile and graphs in WD are a bit of a mess of course with duplicate and out-of-sequence data, but that doesn’t bother me too much).

thanks Brian, fingers crossed for tomorrow morning

Hi Brian and Steerage 250,
Have run 21 and the data collection is working now. I did put in earlier (one this thread) a debug report and in it it did have something about out of sync…
Anyways looking forward to tomoz to see if all is well, after a second try.
cheers
windi
:smiley: :smiley: :smiley: :smiley: :smiley: